2024-04-26T21:05:27.501 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T21:05:27.501 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} --owner scheduled_rishabh@teuthology --name rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204 --verbose -- /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204/orig.config.yaml 2024-04-26T21:05:27.565 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T21:05:27.648 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204 branch: wip-rishabh-testing-20240426.111959 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7675204' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240426.111959 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240426.111959 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 816 sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 sleep_before_teardown: 0 subset: 166/512 suite: fs suite_branch: wip-rishabh-testing-20240426.111959 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFf2aXPvj/0ccN2dtFI0yyjVcw+DmIuTnxhXs2ZD1Tf2z0KzCQYseJR3tRObtZllJ0PcB+eblcoJCDqohzYzK/I= smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPaky2vOsP4VUCUWYzQGzwp0GzpvIqL1JGL08QbC+PVx3E8lrj4EDdrqyEW8fN7OTtLBboZJowJLVhXzsT1nIl8= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLXrztibfFKNHYapVSY5yz823xjnIE3KD4MFl4HgKwRzAZh7cjMScP0gqZkBBcKL1GtTp73rhjcigBMjh8FdHew= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_19:30:57 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T21:05:27.648 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa; will attempt to use it 2024-04-26T21:05:27.649 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks 2024-04-26T21:05:27.649 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T21:05:27.651 INFO:teuthology.task.internal:Checking packages... 2024-04-26T21:05:27.674 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bc612ffb9c1d5bc27536bc917a705262a36d4387' 2024-04-26T21:05:27.674 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T21:05:27.674 INFO:teuthology.packaging:ref: None 2024-04-26T21:05:27.674 INFO:teuthology.packaging:tag: None 2024-04-26T21:05:27.674 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240426.111959 2024-04-26T21:05:27.674 INFO:teuthology.packaging:sha1: bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:05:27.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240426.111959 2024-04-26T21:05:27.888 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3306.gbc612ffb 2024-04-26T21:05:27.889 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T21:05:27.896 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T21:05:27.896 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T21:05:27.940 INFO:teuthology.task.internal:Saving configuration 2024-04-26T21:05:27.962 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T21:05:27.987 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T21:05:28.007 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:00:10.173396', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFf2aXPvj/0ccN2dtFI0yyjVcw+DmIuTnxhXs2ZD1Tf2z0KzCQYseJR3tRObtZllJ0PcB+eblcoJCDqohzYzK/I='} 2024-04-26T21:05:28.031 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:00:10.175128', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPaky2vOsP4VUCUWYzQGzwp0GzpvIqL1JGL08QbC+PVx3E8lrj4EDdrqyEW8fN7OTtLBboZJowJLVhXzsT1nIl8='} 2024-04-26T21:05:28.054 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:00:10.171386', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLXrztibfFKNHYapVSY5yz823xjnIE3KD4MFl4HgKwRzAZh7cjMScP0gqZkBBcKL1GtTp73rhjcigBMjh8FdHew='} 2024-04-26T21:05:28.054 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T21:05:28.060 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-26T21:05:28.061 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-26T21:05:28.061 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-26T21:05:28.061 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T21:05:28.176 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5faf5b6160>, signals=[15]) 2024-04-26T21:05:28.177 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T21:05:28.183 INFO:teuthology.task.internal:Opening connections... 2024-04-26T21:05:28.183 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:05:28.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:05:28.264 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-04-26T21:05:28.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:05:28.343 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T21:05:28.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:05:28.421 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T21:05:28.428 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-04-26T21:05:28.446 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-04-26T21:05:28.447 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-04-26T21:05:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-04-26T21:05:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-04-26T21:05:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-04-26T21:05:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-04-26T21:05:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-04-26T21:05:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-04-26T21:05:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T21:05:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-04-26T21:05:28.505 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-04-26T21:05:28.506 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T21:05:28.506 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-04-26T21:05:28.506 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T21:05:28.506 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T21:05:28.506 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T21:05:28.507 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-04-26T21:05:28.531 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-04-26T21:05:28.550 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-04-26T21:05:28.551 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-04-26T21:05:28.609 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-04-26T21:05:28.609 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-04-26T21:05:28.609 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T21:05:28.610 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T21:05:28.610 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-04-26T21:05:28.631 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-04-26T21:05:28.649 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-04-26T21:05:28.649 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-26T21:05:28.707 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-04-26T21:05:28.707 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-04-26T21:05:28.707 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-26T21:05:28.708 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T21:05:28.708 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-26T21:05:28.731 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T21:05:28.742 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T21:05:28.754 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T21:05:28.754 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-04-26T21:05:28.756 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-04-26T21:05:28.759 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-04-26T21:05:28.775 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T21:05:28.781 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T21:05:28.781 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-04-26T21:05:28.814 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-04-26T21:05:28.816 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-04-26T21:05:28.906 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T21:05:28.986 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T21:05:29.001 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-26T21:05:29.001 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-26T21:05:29.001 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-26T21:05:29.001 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T21:05:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:05:29.019 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-26T21:05:29.261 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T21:05:29.261 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-26T21:05:29.262 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-04-26T21:05:29.280 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-437.el9.x86_64 2024-04-26T21:05:29.280 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T21:05:29.280 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-26T21:05:29.281 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum upgrade -y linux-firmware 2024-04-26T21:05:32.700 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-04-26T21:05:34.800 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-26T21:05:39.353 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 6.2 MB 00:00 2024-04-26T21:05:40.918 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-04-26T21:05:42.589 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-26T21:05:47.762 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-26T21:05:49.934 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-04-26T21:05:49.936 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-04-26T21:05:49.936 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-04-26T21:05:50.050 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T21:05:50.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-26T21:05:50.198 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-26T21:05:50.198 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-26T21:05:50.201 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi001.front.sepia.ceph.com to complete... 2024-04-26T21:06:02.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26 21:06:02 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-26T21:06:02.535 INFO:teuthology.orchestra.run.smithi001.stderr:FINISHED --2024-04-26 21:06:02-- 2024-04-26T21:06:02.535 INFO:teuthology.orchestra.run.smithi001.stderr:Total wall clock time: 12s 2024-04-26T21:06:02.535 INFO:teuthology.orchestra.run.smithi001.stderr:Downloaded: 1 files, 341M in 12s (28.5 MB/s) 2024-04-26T21:06:02.537 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-26T21:06:02.537 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-26T21:06:04.726 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying... ######################################## 2024-04-26T21:06:04.847 INFO:teuthology.orchestra.run.smithi001.stdout:Preparing... ######################################## 2024-04-26T21:06:04.955 INFO:teuthology.orchestra.run.smithi001.stdout:Updating / installing... 2024-04-26T21:06:13.265 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-26T21:06:53.250 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-26T21:06:53.251 DEBUG:teuthology.orchestra.run.smithi001:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-26T21:06:53.341 INFO:teuthology.orchestra.run.smithi001.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-26T21:06:53.342 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-26T21:06:53.343 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-26T21:06:53.344 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-26T21:06:53.345 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-26T21:06:53.346 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-26T21:06:53.347 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-26T21:06:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-26T21:06:53.349 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-26T21:06:53.350 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-26T21:06:53.351 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-26T21:06:53.352 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-26T21:06:53.353 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-26T21:06:53.354 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-26T21:06:53.355 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-26T21:06:53.356 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-26T21:06:53.357 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-26T21:06:53.358 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-26T21:06:53.359 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-26T21:06:53.360 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-26T21:06:53.361 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-26T21:06:53.362 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-26T21:06:53.363 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-26T21:06:53.364 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-26T21:06:53.365 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-26T21:06:53.366 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-26T21:06:53.367 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-26T21:06:53.368 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-26T21:06:53.369 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-26T21:06:53.370 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-26T21:06:53.371 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-26T21:06:53.372 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-26T21:06:53.373 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-26T21:06:53.374 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-26T21:06:53.375 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-26T21:06:53.376 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-26T21:06:53.377 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-26T21:06:53.378 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-26T21:06:53.379 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-26T21:06:53.380 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-26T21:06:53.381 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-26T21:06:53.382 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-26T21:06:53.383 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-26T21:06:53.384 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-26T21:06:53.385 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-26T21:06:53.386 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-26T21:06:53.387 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-26T21:06:53.388 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-26T21:06:53.388 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-26T21:06:53.388 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-26T21:06:53.388 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-26T21:06:53.388 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-26T21:06:53.388 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-26T21:06:53.388 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-26T21:06:53.388 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-26T21:06:53.389 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-26T21:06:53.389 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-26T21:06:53.389 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-26T21:06:53.389 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-26T21:06:53.389 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-26T21:06:53.389 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-26T21:06:53.389 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-26T21:06:53.389 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-26T21:06:53.389 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-26T21:06:53.390 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-26T21:06:53.391 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-26T21:06:53.392 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-26T21:06:53.393 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-26T21:06:53.394 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-26T21:06:53.394 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-26T21:06:53.394 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-26T21:06:53.394 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-26T21:06:53.394 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-26T21:06:53.394 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-26T21:06:53.394 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-26T21:06:53.394 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-26T21:06:53.394 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-26T21:06:53.394 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-26T21:06:53.395 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-26T21:06:53.395 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-26T21:06:53.395 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-26T21:06:53.395 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-26T21:06:53.395 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-26T21:06:53.395 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-26T21:06:53.395 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-26T21:06:53.395 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-26T21:06:53.395 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-26T21:06:53.395 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-26T21:06:53.396 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-26T21:06:53.396 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-26T21:06:53.396 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-26T21:06:53.396 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-26T21:06:53.396 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-26T21:06:53.396 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-26T21:06:53.396 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-26T21:06:53.397 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-26T21:06:53.397 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-26T21:06:53.397 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-26T21:06:53.397 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-26T21:06:53.397 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-26T21:06:53.397 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-26T21:06:53.397 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-26T21:06:53.397 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-26T21:06:53.397 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-26T21:06:53.397 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-26T21:06:53.398 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-26T21:06:53.398 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-26T21:06:53.398 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-26T21:06:53.398 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-26T21:06:53.399 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-26T21:06:53.399 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-26T21:06:53.400 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-26T21:06:53.401 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-26T21:06:53.402 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-26T21:06:53.403 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-26T21:06:53.403 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-26T21:06:53.403 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-26T21:06:53.403 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-26T21:06:53.403 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-26T21:06:53.403 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-26T21:06:53.403 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-26T21:06:53.403 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-26T21:06:53.403 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-26T21:06:53.403 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-26T21:06:53.404 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-26T21:06:53.404 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-26T21:06:53.404 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-26T21:06:53.404 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-26T21:06:53.404 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-26T21:06:53.404 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-26T21:06:53.405 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-26T21:06:53.405 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-26T21:06:53.405 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-26T21:06:53.405 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-26T21:06:53.405 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-26T21:06:53.405 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-26T21:06:53.405 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-26T21:06:53.405 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-26T21:06:53.405 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-26T21:06:53.405 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-26T21:06:53.406 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-26T21:06:53.407 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-26T21:06:53.408 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-26T21:06:53.409 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-26T21:06:53.410 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-26T21:06:53.411 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-26T21:06:53.412 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-26T21:06:53.413 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-26T21:06:53.414 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-26T21:06:53.415 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-26T21:06:53.416 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-26T21:06:53.417 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-26T21:06:53.418 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-26T21:06:53.419 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-26T21:06:53.420 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-26T21:06:53.421 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-26T21:06:53.422 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-26T21:06:53.423 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-26T21:06:53.424 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-26T21:06:53.425 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-26T21:06:53.426 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-26T21:06:53.427 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-26T21:06:53.428 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-26T21:06:53.429 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-26T21:06:53.430 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-26T21:06:53.430 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-26T21:06:53.430 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-26T21:06:53.430 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-26T21:06:53.430 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-26T21:06:53.430 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-26T21:06:53.430 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-26T21:06:53.430 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-26T21:06:53.430 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-26T21:06:53.430 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-26T21:06:53.431 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-26T21:06:53.432 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-26T21:06:53.433 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-26T21:06:53.434 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-26T21:06:53.435 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-26T21:06:53.436 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-26T21:06:53.437 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-26T21:06:53.437 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-26T21:06:53.437 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-26T21:06:53.437 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-26T21:06:53.437 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-26T21:06:53.437 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-26T21:06:53.437 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-26T21:06:53.437 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-26T21:06:53.437 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-26T21:06:53.437 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-26T21:06:53.438 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-26T21:06:53.439 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-26T21:06:53.440 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-26T21:06:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-26T21:06:53.442 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-26T21:06:53.443 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-26T21:06:53.444 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-26T21:06:53.445 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-26T21:06:53.446 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-26T21:06:53.447 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-26T21:06:53.448 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-26T21:06:53.449 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-26T21:06:53.450 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-26T21:06:53.451 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-26T21:06:53.452 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-26T21:06:53.453 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-26T21:06:53.454 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-26T21:06:53.455 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-26T21:06:53.456 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-26T21:06:53.457 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-26T21:06:53.458 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-26T21:06:53.459 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-26T21:06:53.460 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-26T21:06:53.461 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-26T21:06:53.462 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-26T21:06:53.463 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-26T21:06:53.464 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-26T21:06:53.465 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-26T21:06:53.466 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-26T21:06:53.467 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-26T21:06:53.468 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-26T21:06:53.469 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-26T21:06:53.470 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-26T21:06:53.471 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-26T21:06:53.472 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-26T21:06:53.473 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-26T21:06:53.474 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-26T21:06:53.475 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-26T21:06:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-26T21:06:53.477 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-26T21:06:53.478 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-26T21:06:53.479 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-26T21:06:53.480 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-26T21:06:53.481 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-26T21:06:53.482 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-26T21:06:53.483 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-26T21:06:53.484 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-26T21:06:53.485 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-26T21:06:53.486 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-26T21:06:53.487 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-26T21:06:53.488 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-26T21:06:53.489 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-26T21:06:53.490 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-26T21:06:53.491 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-26T21:06:53.492 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-26T21:06:53.493 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-26T21:06:53.494 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-26T21:06:53.495 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-26T21:06:53.495 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-26T21:06:53.495 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-26T21:06:53.495 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-26T21:06:53.495 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-26T21:06:53.495 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-26T21:06:53.495 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-26T21:06:53.495 INFO:teuthology.orchestra.run.smithi001.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T21:06:53.499 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-26T21:06:53.500 DEBUG:teuthology.orchestra.run.smithi001:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-26T21:06:53.523 INFO:teuthology.orchestra.run.smithi001.stdout:postinstall scriptlet (using /bin/sh): 2024-04-26T21:06:53.523 INFO:teuthology.orchestra.run.smithi001.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T21:06:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-26T21:06:53.523 INFO:teuthology.orchestra.run.smithi001.stdout:fi 2024-04-26T21:06:53.523 INFO:teuthology.orchestra.run.smithi001.stdout:for file in vmlinuz System.map config; do 2024-04-26T21:06:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-26T21:06:53.523 INFO:teuthology.orchestra.run.smithi001.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: fi 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout:done 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout:fi 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout:fi 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-26T21:06:53.524 INFO:teuthology.orchestra.run.smithi001.stdout:fi 2024-04-26T21:06:53.525 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-04-26T21:06:53.598 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 68.el9 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8264254 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-26T21:06:53.599 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:06:53.600 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T21:06:53.601 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T21:06:53.601 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 6.9.0-rc3-g806ec36b2112 2024-04-26T21:06:53.601 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T21:06:54.207 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-04-26T21:06:55.618 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-26T21:06:55.660 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-04-26T21:06:55.663 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-04-26T21:06:55.706 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.s9MRTiOvJT 2024-04-26T21:06:55.706 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.s9MRTiOvJT 2024-04-26T21:06:55.772 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.s9MRTiOvJT 2024-04-26T21:06:56.033 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.s9MRTiOvJT is 6KB 2024-04-26T21:06:56.082 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.s9MRTiOvJT 2024-04-26T21:06:56.097 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T21:06:56.162 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-26T21:06:56.162 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-26T21:06:56.162 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-26T21:06:56.162 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-26T21:06:56.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-26T21:06:56.324 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-04-26T21:07:26.327 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-26T21:07:26.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-04-26T21:07:26.329 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:07:44.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-04-26T21:07:53.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-04-26T21:07:53.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:08:53.803 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T21:09:05.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-04-26T21:09:05.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:09:06.232 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-04-26T21:09:06.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-04-26T21:09:06.899 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-26T21:09:06.899 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-04-26T21:09:06.960 INFO:teuthology.orchestra.run.smithi001.stdout:6.9.0-rc3-g806ec36b2112 2024-04-26T21:09:06.960 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-26T21:09:06.960 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T21:09:06.960 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T21:09:07.961 DEBUG:teuthology.parallel:result is None 2024-04-26T21:09:07.961 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T21:09:07.969 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T21:09:07.969 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T21:09:07.976 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T21:09:07.979 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T21:09:08.028 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T21:09:08.041 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T21:09:08.073 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T21:09:08.073 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T21:09:08.078 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T21:09:08.081 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T21:09:08.142 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T21:09:08.149 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T21:09:08.149 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T21:09:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:09:08.167 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T21:09:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:09:08.184 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T21:09:08.191 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T21:09:08.209 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T21:09:08.243 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:09:08.257 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:09:08.260 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:09:08.290 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:09:08.308 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:09:08.364 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T21:09:08.368 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T21:09:08.376 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T21:09:08.376 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T21:09:08.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T21:09:08.386 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T21:09:08.468 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T21:09:08.477 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T21:09:08.477 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T21:09:08.484 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T21:09:08.486 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T21:09:08.538 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:09:08.621 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:09:08.719 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:09:08.805 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:09:08.898 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:09:08.898 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T21:09:08.986 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:09:09.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:09:09.084 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:09:09.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:09:09.212 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:09:09.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T21:09:09.279 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:09:09.323 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T21:09:09.383 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:09:09.426 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T21:09:09.469 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:09:09.469 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T21:09:09.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-04-26T21:09:09.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-04-26T21:09:09.546 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-04-26T21:09:09.586 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T21:09:09.610 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T21:09:09.664 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T21:09:10.151 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T21:09:10.159 INFO:teuthology.task.internal:Starting timer... 2024-04-26T21:09:10.159 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T21:09:10.201 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T21:09:10.234 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-26T21:09:10.234 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-04-26T21:09:10.352 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-04-26T21:09:10.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-04-26T21:09:10.409 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-04-26T21:09:10.412 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-04-26T21:09:10.462 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-04-26T21:09:10.464 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T21:09:10.464 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-04-26T21:09:10.527 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-04-26T21:09:10.527 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-04-26T21:09:10.551 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-04-26T21:09:10.552 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-04-26T21:09:10.573 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-04-26T21:09:10.573 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-04-26T21:09:10.574 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T21:09:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:09:10.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T21:09:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:09:10.673 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T21:09:10.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:09:10.698 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T21:09:10.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-04-26T21:09:10.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-04-26T21:09:10.800 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-04-26T21:09:10.824 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T21:09:10.834 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-26T21:09:10.840 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-26T21:09:10.961 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T21:09:10.978 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T21:09:10.980 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi038.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T21:14:22.472 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-04-26T21:14:22.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-04-26T21:14:22.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:14:22.590 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-04-26T21:14:22.748 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-04-26T21:14:22.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-26T21:14:22.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:14:22.831 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-04-26T21:14:22.920 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-04-26T21:14:22.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T21:14:22.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T21:14:23.003 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-26T21:14:23.083 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T21:14:23.083 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T21:14:23.093 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T21:14:23.094 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T21:14:23.094 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T21:14:23.101 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T21:14:23.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T21:14:23.104 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T21:14:23.104 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T21:14:23.138 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T21:14:23.150 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T21:14:23.153 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T21:14:23.167 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T21:14:23.176 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-04-26T21:14:23.187 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-04-26T21:14:23.191 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-04-26T21:14:23.195 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T21:14:23.202 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T21:14:23.202 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-04-26T21:14:23.216 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T21:14:23.216 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T21:14:23.229 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T21:14:23.240 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T21:14:23.269 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-26T21:14:23.273 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T21:14:23.273 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-26T21:14:23.273 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.273 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.273 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.274 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.287 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-26T21:14:23.290 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T21:14:23.290 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-26T21:14:23.290 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.290 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.290 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.290 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.318 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-04-26T21:14:23.349 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-04-26T21:14:23.393 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T21:14:23.442 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T21:14:23.589 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-04-26T21:14:23.598 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T21:14:23.599 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-04-26T21:14:23.599 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.599 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.599 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.599 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T21:14:23.600 INFO:teuthology.run_tasks:Running task install... 2024-04-26T21:14:23.609 DEBUG:teuthology.task.install:project ceph 2024-04-26T21:14:23.609 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'}} 2024-04-26T21:14:23.609 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387'} 2024-04-26T21:14:23.609 INFO:teuthology.task.install:Using flavor: default 2024-04-26T21:14:23.615 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T21:14:23.615 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-26T21:14:23.615 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T21:14:23.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:14:23.618 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T21:14:23.618 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:14:23.619 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'tag': None, 'wait_for_package': False} 2024-04-26T21:14:23.619 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:14:23.763 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T21:14:23.763 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T21:14:23.832 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T21:14:23.832 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T21:14:23.847 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/ 2024-04-26T21:14:23.847 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3306.gbc612ffb 2024-04-26T21:14:23.880 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T21:14:23.880 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:14:23.880 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T21:14:23.911 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T21:14:23.911 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T21:14:23.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T21:14:24.003 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T21:14:24.003 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:14:24.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T21:14:24.014 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T21:14:24.014 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:14:24.014 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T21:14:24.043 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T21:14:24.043 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T21:14:24.060 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T21:14:24.098 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-26T21:14:24.099 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc612ffb9c1d5bc27536bc917a705262a36d4387/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T21:14:24.111 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T21:14:24.126 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-04-26T21:14:24.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-04-26T21:14:24.188 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T21:14:24.234 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T21:14:24.252 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-04-26T21:14:24.254 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-04-26T21:14:24.355 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T21:14:24.366 INFO:teuthology.orchestra.run.smithi038.stdout:48 files removed 2024-04-26T21:14:24.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T21:14:24.439 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-04-26T21:14:24.444 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-04-26T21:14:24.480 INFO:teuthology.orchestra.run.smithi186.stdout:48 files removed 2024-04-26T21:14:24.508 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T21:14:24.778 INFO:teuthology.orchestra.run.smithi001.stdout:48 files removed 2024-04-26T21:14:24.818 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-26T21:14:25.005 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 249 kB/s | 84 kB 00:00 2024-04-26T21:14:25.168 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 211 kB/s | 84 kB 00:00 2024-04-26T21:14:25.456 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-04-26T21:14:25.560 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 33 kB/s | 12 kB 00:00 2024-04-26T21:14:25.726 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 179 kB/s | 84 kB 00:00 2024-04-26T21:14:25.907 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-04-26T21:14:25.919 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-26T21:14:26.294 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 25 kB/s | 12 kB 00:00 2024-04-26T21:14:26.578 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-26T21:14:26.794 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-04-26T21:14:27.632 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-04-26T21:14:28.537 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-26T21:14:28.834 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-04-26T21:14:30.186 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-26T21:14:32.048 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 19 MB 00:01 2024-04-26T21:14:32.696 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-26T21:14:34.155 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-04-26T21:14:34.359 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-26T21:14:35.382 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-26T21:14:36.035 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-26T21:14:37.110 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 6.0 MB/s | 6.2 MB 00:01 2024-04-26T21:14:37.464 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-26T21:14:39.048 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-04-26T21:14:40.407 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-26T21:14:42.516 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:14:42.517 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:14:42.527 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T21:14:42.528 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T21:14:42.529 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T21:14:42.530 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T21:14:42.531 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T21:14:42.533 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T21:14:42.534 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T21:14:42.596 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T21:14:42.624 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T21:14:42.625 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T21:14:42.626 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T21:14:42.627 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T21:14:42.628 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T21:14:42.629 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T21:14:42.630 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T21:14:42.631 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T21:14:42.632 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T21:14:42.633 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T21:14:42.634 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout:Install 148 Packages 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 4 Packages 2024-04-26T21:14:42.635 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:14:42.636 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 222 M 2024-04-26T21:14:42.636 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-26T21:14:42.689 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-26T21:14:44.028 INFO:teuthology.orchestra.run.smithi186.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-04-26T21:14:44.342 INFO:teuthology.orchestra.run.smithi186.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 2.8 MB/s | 889 kB 00:00 2024-04-26T21:14:44.425 INFO:teuthology.orchestra.run.smithi186.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 1.7 MB/s | 145 kB 00:00 2024-04-26T21:14:44.609 INFO:teuthology.orchestra.run.smithi186.stdout:(4/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 5.0 MB/s | 5.4 MB 00:01 2024-04-26T21:14:44.784 INFO:teuthology.orchestra.run.smithi186.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 8.7 MB/s | 1.5 MB 00:00 2024-04-26T21:14:45.050 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:14:45.050 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:14:45.061 INFO:teuthology.orchestra.run.smithi038.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T21:14:45.062 INFO:teuthology.orchestra.run.smithi038.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T21:14:45.063 INFO:teuthology.orchestra.run.smithi038.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T21:14:45.064 INFO:teuthology.orchestra.run.smithi038.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T21:14:45.065 INFO:teuthology.orchestra.run.smithi038.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T21:14:45.068 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T21:14:45.069 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T21:14:45.084 INFO:teuthology.orchestra.run.smithi186.stdout:(6/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 3.6 MB/s | 2.4 MB 00:00 2024-04-26T21:14:45.133 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-04-26T21:14:45.150 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-26T21:14:45.150 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-04-26T21:14:45.150 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T21:14:45.151 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T21:14:45.152 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T21:14:45.153 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T21:14:45.154 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T21:14:45.155 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T21:14:45.156 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T21:14:45.157 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T21:14:45.157 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T21:14:45.157 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T21:14:45.157 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T21:14:45.157 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T21:14:45.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T21:14:45.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T21:14:45.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T21:14:45.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T21:14:45.157 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T21:14:45.158 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T21:14:45.159 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T21:14:45.160 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T21:14:45.161 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T21:14:45.162 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================= 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout:Install 148 Packages 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 4 Packages 2024-04-26T21:14:45.163 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:14:45.164 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 222 M 2024-04-26T21:14:45.164 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-04-26T21:14:45.309 INFO:teuthology.orchestra.run.smithi186.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 9.0 MB/s | 4.7 MB 00:00 2024-04-26T21:14:45.593 INFO:teuthology.orchestra.run.smithi186.stdout:(8/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 10 MB/s | 21 MB 00:02 2024-04-26T21:14:45.651 INFO:teuthology.orchestra.run.smithi186.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 434 kB/s | 25 kB 00:00 2024-04-26T21:14:46.293 INFO:teuthology.orchestra.run.smithi186.stdout:(10/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 11 MB/s | 11 MB 00:00 2024-04-26T21:14:46.385 INFO:teuthology.orchestra.run.smithi186.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 2.4 MB/s | 225 kB 00:00 2024-04-26T21:14:46.452 INFO:teuthology.orchestra.run.smithi186.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 407 kB/s | 27 kB 00:00 2024-04-26T21:14:46.457 INFO:teuthology.orchestra.run.smithi038.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-04-26T21:14:46.527 INFO:teuthology.orchestra.run.smithi186.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 423 kB/s | 32 kB 00:00 2024-04-26T21:14:46.686 INFO:teuthology.orchestra.run.smithi186.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 4.6 MB/s | 747 kB 00:00 2024-04-26T21:14:46.770 INFO:teuthology.orchestra.run.smithi186.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 1.9 MB/s | 163 kB 00:00 2024-04-26T21:14:46.853 INFO:teuthology.orchestra.run.smithi186.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.5 MB/s | 127 kB 00:00 2024-04-26T21:14:46.953 INFO:teuthology.orchestra.run.smithi186.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 4.9 MB/s | 501 kB 00:00 2024-04-26T21:14:47.325 INFO:teuthology.orchestra.run.smithi038.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.0 MB/s | 889 kB 00:00 2024-04-26T21:14:47.491 INFO:teuthology.orchestra.run.smithi038.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 873 kB/s | 145 kB 00:00 2024-04-26T21:14:47.595 INFO:teuthology.orchestra.run.smithi186.stdout:(18/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 8.4 MB/s | 5.4 MB 00:00 2024-04-26T21:14:47.663 INFO:teuthology.orchestra.run.smithi186.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 673 kB/s | 45 kB 00:00 2024-04-26T21:14:47.746 INFO:teuthology.orchestra.run.smithi186.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 1.6 MB/s | 135 kB 00:00 2024-04-26T21:14:47.830 INFO:teuthology.orchestra.run.smithi186.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.9 MB/s | 161 kB 00:00 2024-04-26T21:14:47.914 INFO:teuthology.orchestra.run.smithi186.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 3.8 MB/s | 321 kB 00:00 2024-04-26T21:14:47.997 INFO:teuthology.orchestra.run.smithi186.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.5 MB/s | 298 kB 00:00 2024-04-26T21:14:48.081 INFO:teuthology.orchestra.run.smithi186.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.2 MB/s | 99 kB 00:00 2024-04-26T21:14:48.164 INFO:teuthology.orchestra.run.smithi186.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.0 MB/s | 85 kB 00:00 2024-04-26T21:14:48.408 INFO:teuthology.orchestra.run.smithi038.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 2.6 MB/s | 2.4 MB 00:00 2024-04-26T21:14:48.648 INFO:teuthology.orchestra.run.smithi186.stdout:(26/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 6.4 MB/s | 3.1 MB 00:00 2024-04-26T21:14:48.731 INFO:teuthology.orchestra.run.smithi186.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-26T21:14:48.800 INFO:teuthology.orchestra.run.smithi038.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 3.9 MB/s | 1.5 MB 00:00 2024-04-26T21:14:48.899 INFO:teuthology.orchestra.run.smithi186.stdout:(28/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 4.4 MB/s | 17 MB 00:03 2024-04-26T21:14:48.923 INFO:teuthology.orchestra.run.smithi186.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3306.g 178 kB/s | 34 kB 00:00 2024-04-26T21:14:48.976 INFO:teuthology.orchestra.run.smithi186.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.8 MB/s | 142 kB 00:00 2024-04-26T21:14:49.600 INFO:teuthology.orchestra.run.smithi038.stdout:(6/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 5.9 MB/s | 4.7 MB 00:00 2024-04-26T21:14:49.701 INFO:teuthology.orchestra.run.smithi038.stdout:(7/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-04-26T21:14:49.785 INFO:teuthology.orchestra.run.smithi186.stdout:(31/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 12 MB/s | 49 MB 00:04 2024-04-26T21:14:49.843 INFO:teuthology.orchestra.run.smithi186.stdout:(32/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 3.9 MB/s | 3.5 MB 00:00 2024-04-26T21:14:49.868 INFO:teuthology.orchestra.run.smithi186.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.9 MB/s | 249 kB 00:00 2024-04-26T21:14:49.919 INFO:teuthology.orchestra.run.smithi186.stdout:(34/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 658 kB/s | 49 kB 00:00 2024-04-26T21:14:49.944 INFO:teuthology.orchestra.run.smithi186.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3306.gb 219 kB/s | 16 kB 00:00 2024-04-26T21:14:50.078 INFO:teuthology.orchestra.run.smithi186.stdout:(36/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 1.7 MB/s | 273 kB 00:00 2024-04-26T21:14:50.111 INFO:teuthology.orchestra.run.smithi186.stdout:(37/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 4.5 MB/s | 762 kB 00:00 2024-04-26T21:14:50.161 INFO:teuthology.orchestra.run.smithi186.stdout:(38/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 218 kB/s | 18 kB 00:00 2024-04-26T21:14:50.245 INFO:teuthology.orchestra.run.smithi186.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 80 kB 00:00 2024-04-26T21:14:50.312 INFO:teuthology.orchestra.run.smithi186.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-26T21:14:50.362 INFO:teuthology.orchestra.run.smithi186.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-26T21:14:50.387 INFO:teuthology.orchestra.run.smithi186.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-04-26T21:14:50.438 INFO:teuthology.orchestra.run.smithi186.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-26T21:14:50.488 INFO:teuthology.orchestra.run.smithi186.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-26T21:14:50.538 INFO:teuthology.orchestra.run.smithi186.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.5 MB/s | 125 kB 00:00 2024-04-26T21:14:50.572 INFO:teuthology.orchestra.run.smithi186.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 5.9 MB/s | 1.2 MB 00:00 2024-04-26T21:14:50.597 INFO:teuthology.orchestra.run.smithi186.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-26T21:14:50.892 INFO:teuthology.orchestra.run.smithi038.stdout:(8/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x86 13 MB/s | 17 MB 00:01 2024-04-26T21:14:50.981 INFO:teuthology.orchestra.run.smithi186.stdout:(48/152): ceph-mgr-diskprediction-local-19.0.0- 3.7 MB/s | 7.4 MB 00:02 2024-04-26T21:14:51.126 INFO:teuthology.orchestra.run.smithi038.stdout:(9/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 4.3 MB/s | 21 MB 00:04 2024-04-26T21:14:51.151 INFO:teuthology.orchestra.run.smithi038.stdout:(10/152): ceph-selinux-19.0.0-3306.gbc612ffb.el 97 kB/s | 25 kB 00:00 2024-04-26T21:14:51.202 INFO:teuthology.orchestra.run.smithi038.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 4.4 MB/s | 225 kB 00:00 2024-04-26T21:14:51.260 INFO:teuthology.orchestra.run.smithi038.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 464 kB/s | 27 kB 00:00 2024-04-26T21:14:51.319 INFO:teuthology.orchestra.run.smithi038.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 544 kB/s | 32 kB 00:00 2024-04-26T21:14:51.411 INFO:teuthology.orchestra.run.smithi038.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 8.0 MB/s | 747 kB 00:00 2024-04-26T21:14:51.478 INFO:teuthology.orchestra.run.smithi038.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 2.4 MB/s | 163 kB 00:00 2024-04-26T21:14:51.536 INFO:teuthology.orchestra.run.smithi038.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 2.1 MB/s | 127 kB 00:00 2024-04-26T21:14:51.612 INFO:teuthology.orchestra.run.smithi038.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 6.5 MB/s | 501 kB 00:00 2024-04-26T21:14:51.656 INFO:teuthology.orchestra.run.smithi186.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 158 kB/s | 107 kB 00:00 2024-04-26T21:14:51.914 INFO:teuthology.orchestra.run.smithi186.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 513 kB/s | 689 kB 00:01 2024-04-26T21:14:51.979 INFO:teuthology.orchestra.run.smithi038.stdout:(18/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 15 MB/s | 5.4 MB 00:00 2024-04-26T21:14:52.037 INFO:teuthology.orchestra.run.smithi038.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 776 kB/s | 45 kB 00:00 2024-04-26T21:14:52.096 INFO:teuthology.orchestra.run.smithi038.stdout:(20/152): python3-ceph-common-19.0.0-3306.gbc61 2.3 MB/s | 135 kB 00:00 2024-04-26T21:14:52.155 INFO:teuthology.orchestra.run.smithi038.stdout:(21/152): python3-cephfs-19.0.0-3306.gbc612ffb. 2.7 MB/s | 161 kB 00:00 2024-04-26T21:14:52.164 INFO:teuthology.orchestra.run.smithi186.stdout:(51/152): bison-3.7.4-5.el9.x86_64.rpm 602 kB/s | 943 kB 00:01 2024-04-26T21:14:52.190 INFO:teuthology.orchestra.run.smithi186.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 40 kB/s | 21 kB 00:00 2024-04-26T21:14:52.222 INFO:teuthology.orchestra.run.smithi038.stdout:(22/152): python3-rados-19.0.0-3306.gbc612ffb.e 4.7 MB/s | 321 kB 00:00 2024-04-26T21:14:52.280 INFO:teuthology.orchestra.run.smithi038.stdout:(23/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 5.0 MB/s | 298 kB 00:00 2024-04-26T21:14:52.331 INFO:teuthology.orchestra.run.smithi038.stdout:(24/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.9 MB/s | 99 kB 00:00 2024-04-26T21:14:52.389 INFO:teuthology.orchestra.run.smithi038.stdout:(25/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-26T21:14:52.632 INFO:teuthology.orchestra.run.smithi038.stdout:(26/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 13 MB/s | 3.1 MB 00:00 2024-04-26T21:14:52.748 INFO:teuthology.orchestra.run.smithi186.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 61 kB/s | 33 kB 00:00 2024-04-26T21:14:52.748 INFO:teuthology.orchestra.run.smithi038.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 3.4 MB/s | 171 kB 00:00 2024-04-26T21:14:52.748 INFO:teuthology.orchestra.run.smithi038.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 683 kB/s | 34 kB 00:00 2024-04-26T21:14:52.790 INFO:teuthology.orchestra.run.smithi186.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 358 kB/s | 313 kB 00:00 2024-04-26T21:14:52.799 INFO:teuthology.orchestra.run.smithi038.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 2.1 MB/s | 142 kB 00:00 2024-04-26T21:14:52.999 INFO:teuthology.orchestra.run.smithi038.stdout:(30/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 3.2 MB/s | 11 MB 00:03 2024-04-26T21:14:53.001 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 1.5 MB/s | 21 MB 00:13 2024-04-26T21:14:53.074 INFO:teuthology.orchestra.run.smithi038.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 13 MB/s | 3.5 MB 00:00 2024-04-26T21:14:53.125 INFO:teuthology.orchestra.run.smithi038.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 4.9 MB/s | 249 kB 00:00 2024-04-26T21:14:53.166 INFO:teuthology.orchestra.run.smithi186.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 39 kB/s | 18 kB 00:00 2024-04-26T21:14:53.183 INFO:teuthology.orchestra.run.smithi038.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 845 kB/s | 49 kB 00:00 2024-04-26T21:14:53.242 INFO:teuthology.orchestra.run.smithi038.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 282 kB/s | 16 kB 00:00 2024-04-26T21:14:53.266 INFO:teuthology.orchestra.run.smithi186.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 23 kB/s | 11 kB 00:00 2024-04-26T21:14:53.309 INFO:teuthology.orchestra.run.smithi038.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 4.0 MB/s | 273 kB 00:00 2024-04-26T21:14:53.401 INFO:teuthology.orchestra.run.smithi038.stdout:(36/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 8.1 MB/s | 762 kB 00:00 2024-04-26T21:14:53.494 INFO:teuthology.orchestra.run.smithi038.stdout:(37/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 196 kB/s | 18 kB 00:00 2024-04-26T21:14:53.735 INFO:teuthology.orchestra.run.smithi038.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 334 kB/s | 80 kB 00:00 2024-04-26T21:14:53.824 INFO:teuthology.orchestra.run.smithi186.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 81 kB/s | 45 kB 00:00 2024-04-26T21:14:53.852 INFO:teuthology.orchestra.run.smithi038.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 6.7 MB/s | 803 kB 00:00 2024-04-26T21:14:53.858 INFO:teuthology.orchestra.run.smithi186.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 232 kB/s | 160 kB 00:00 2024-04-26T21:14:53.919 INFO:teuthology.orchestra.run.smithi038.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-26T21:14:53.994 INFO:teuthology.orchestra.run.smithi038.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-26T21:14:54.128 INFO:teuthology.orchestra.run.smithi038.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.2 MB 00:00 2024-04-26T21:14:54.195 INFO:teuthology.orchestra.run.smithi038.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-26T21:14:54.262 INFO:teuthology.orchestra.run.smithi038.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-26T21:14:54.328 INFO:teuthology.orchestra.run.smithi038.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-26T21:14:54.412 INFO:teuthology.orchestra.run.smithi038.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-04-26T21:14:54.671 INFO:teuthology.orchestra.run.smithi038.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 689 kB 00:00 2024-04-26T21:14:54.813 INFO:teuthology.orchestra.run.smithi038.stdout:(48/152): ceph-mgr-diskprediction-local-19.0.0- 4.1 MB/s | 7.4 MB 00:01 2024-04-26T21:14:54.847 INFO:teuthology.orchestra.run.smithi038.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 5.3 MB/s | 943 kB 00:00 2024-04-26T21:14:55.008 INFO:teuthology.orchestra.run.smithi186.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 506 kB/s | 582 kB 00:01 2024-04-26T21:14:55.133 INFO:teuthology.orchestra.run.smithi186.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 506 kB/s | 662 kB 00:01 2024-04-26T21:14:55.289 INFO:teuthology.orchestra.run.smithi038.stdout:(50/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 12 MB/s | 49 MB 00:04 2024-04-26T21:14:55.314 INFO:teuthology.orchestra.run.smithi038.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 213 kB/s | 107 kB 00:00 2024-04-26T21:14:55.390 INFO:teuthology.orchestra.run.smithi038.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 3.1 MB/s | 313 kB 00:00 2024-04-26T21:14:55.440 INFO:teuthology.orchestra.run.smithi038.stdout:(53/152): elfutils-libelf-devel-0.190-2.el9.x86 36 kB/s | 21 kB 00:00 2024-04-26T21:14:55.515 INFO:teuthology.orchestra.run.smithi038.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 190 kB/s | 33 kB 00:00 2024-04-26T21:14:55.526 INFO:teuthology.orchestra.run.smithi186.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 49 kB/s | 25 kB 00:00 2024-04-26T21:14:55.540 INFO:teuthology.orchestra.run.smithi038.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 183 kB/s | 18 kB 00:00 2024-04-26T21:14:55.582 INFO:teuthology.orchestra.run.smithi038.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 166 kB/s | 11 kB 00:00 2024-04-26T21:14:55.641 INFO:teuthology.orchestra.run.smithi038.stdout:(57/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-04-26T21:14:55.683 INFO:teuthology.orchestra.run.smithi038.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-04-26T21:14:55.700 INFO:teuthology.orchestra.run.smithi186.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 84 kB/s | 47 kB 00:00 2024-04-26T21:14:55.708 INFO:teuthology.orchestra.run.smithi038.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 360 kB/s | 45 kB 00:00 2024-04-26T21:14:55.775 INFO:teuthology.orchestra.run.smithi038.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 582 kB 00:00 2024-04-26T21:14:55.800 INFO:teuthology.orchestra.run.smithi038.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 277 kB/s | 25 kB 00:00 2024-04-26T21:14:55.851 INFO:teuthology.orchestra.run.smithi038.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-04-26T21:14:55.893 INFO:teuthology.orchestra.run.smithi038.stdout:(63/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.6 MB/s | 662 kB 00:00 2024-04-26T21:14:55.918 INFO:teuthology.orchestra.run.smithi038.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-04-26T21:14:55.943 INFO:teuthology.orchestra.run.smithi038.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-04-26T21:14:55.968 INFO:teuthology.orchestra.run.smithi038.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 468 kB/s | 35 kB 00:00 2024-04-26T21:14:55.984 INFO:teuthology.orchestra.run.smithi186.stdout:(63/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 807 kB/s | 3.0 MB 00:03 2024-04-26T21:14:56.094 INFO:teuthology.orchestra.run.smithi038.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-26T21:14:56.302 INFO:teuthology.orchestra.run.smithi038.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-04-26T21:14:56.344 INFO:teuthology.orchestra.run.smithi038.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 3.6 MB/s | 911 kB 00:00 2024-04-26T21:14:56.394 INFO:teuthology.orchestra.run.smithi038.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 133 kB 00:00 2024-04-26T21:14:56.426 INFO:teuthology.orchestra.run.smithi186.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 325 kB/s | 292 kB 00:00 2024-04-26T21:14:56.493 INFO:teuthology.orchestra.run.smithi186.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 238 kB/s | 188 kB 00:00 2024-04-26T21:14:56.520 INFO:teuthology.orchestra.run.smithi038.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-04-26T21:14:56.543 INFO:teuthology.orchestra.run.smithi186.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 63 kB/s | 35 kB 00:00 2024-04-26T21:14:56.637 INFO:teuthology.orchestra.run.smithi038.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 20 MB/s | 5.8 MB 00:00 2024-04-26T21:14:56.662 INFO:teuthology.orchestra.run.smithi038.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 617 kB/s | 87 kB 00:00 2024-04-26T21:14:56.721 INFO:teuthology.orchestra.run.smithi038.stdout:(74/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 5.7 MB/s | 4.4 MB 00:00 2024-04-26T21:14:56.805 INFO:teuthology.orchestra.run.smithi038.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.7 MB/s | 244 kB 00:00 2024-04-26T21:14:56.855 INFO:teuthology.orchestra.run.smithi038.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-04-26T21:14:56.922 INFO:teuthology.orchestra.run.smithi038.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 219 kB/s | 26 kB 00:00 2024-04-26T21:14:56.980 INFO:teuthology.orchestra.run.smithi038.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 153 kB/s | 19 kB 00:00 2024-04-26T21:14:57.006 INFO:teuthology.orchestra.run.smithi038.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-26T21:14:57.073 INFO:teuthology.orchestra.run.smithi038.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 378 kB/s | 35 kB 00:00 2024-04-26T21:14:57.140 INFO:teuthology.orchestra.run.smithi038.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-04-26T21:14:57.190 INFO:teuthology.orchestra.run.smithi038.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-26T21:14:57.215 INFO:teuthology.orchestra.run.smithi038.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-04-26T21:14:57.307 INFO:teuthology.orchestra.run.smithi038.stdout:(84/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-04-26T21:14:57.324 INFO:teuthology.orchestra.run.smithi038.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 388 kB/s | 42 kB 00:00 2024-04-26T21:14:57.349 INFO:teuthology.orchestra.run.smithi038.stdout:(86/152): python3-packaging-20.9-5.el9.noarch.r 485 kB/s | 77 kB 00:00 2024-04-26T21:14:57.400 INFO:teuthology.orchestra.run.smithi038.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-26T21:14:57.425 INFO:teuthology.orchestra.run.smithi038.stdout:(88/152): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 269 kB 00:00 2024-04-26T21:14:57.450 INFO:teuthology.orchestra.run.smithi038.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 279 kB 00:00 2024-04-26T21:14:57.475 INFO:teuthology.orchestra.run.smithi038.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 675 kB/s | 51 kB 00:00 2024-04-26T21:14:57.501 INFO:teuthology.orchestra.run.smithi038.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 712 kB/s | 54 kB 00:00 2024-04-26T21:14:57.551 INFO:teuthology.orchestra.run.smithi038.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 554 kB/s | 42 kB 00:00 2024-04-26T21:14:57.576 INFO:teuthology.orchestra.run.smithi038.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 587 kB/s | 44 kB 00:00 2024-04-26T21:14:57.635 INFO:teuthology.orchestra.run.smithi038.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-04-26T21:14:57.660 INFO:teuthology.orchestra.run.smithi038.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 429 kB/s | 36 kB 00:00 2024-04-26T21:14:57.711 INFO:teuthology.orchestra.run.smithi038.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2024-04-26T21:14:57.736 INFO:teuthology.orchestra.run.smithi038.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 305 kB 00:00 2024-04-26T21:14:57.787 INFO:teuthology.orchestra.run.smithi038.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 768 kB/s | 58 kB 00:00 2024-04-26T21:14:57.820 INFO:teuthology.orchestra.run.smithi038.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-26T21:14:57.879 INFO:teuthology.orchestra.run.smithi038.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 693 kB/s | 64 kB 00:00 2024-04-26T21:14:57.904 INFO:teuthology.orchestra.run.smithi038.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 537 kB/s | 45 kB 00:00 2024-04-26T21:14:58.013 INFO:teuthology.orchestra.run.smithi038.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-04-26T21:14:58.205 INFO:teuthology.orchestra.run.smithi038.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.8 MB/s | 863 kB 00:00 2024-04-26T21:14:58.447 INFO:teuthology.orchestra.run.smithi038.stdout:(104/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-04-26T21:14:58.656 INFO:teuthology.orchestra.run.smithi038.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 853 kB/s | 548 kB 00:00 2024-04-26T21:14:58.697 INFO:teuthology.orchestra.run.smithi038.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-04-26T21:14:58.756 INFO:teuthology.orchestra.run.smithi038.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 561 kB/s | 309 kB 00:00 2024-04-26T21:14:58.798 INFO:teuthology.orchestra.run.smithi038.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-04-26T21:14:58.831 INFO:teuthology.orchestra.run.smithi038.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-04-26T21:14:58.927 INFO:teuthology.orchestra.run.smithi186.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 682 kB/s | 1.6 MB 00:02 2024-04-26T21:14:58.965 INFO:teuthology.orchestra.run.smithi038.stdout:(110/152): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-26T21:14:58.990 INFO:teuthology.orchestra.run.smithi038.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 350 kB/s | 67 kB 00:00 2024-04-26T21:14:59.016 INFO:teuthology.orchestra.run.smithi038.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 821 kB/s | 151 kB 00:00 2024-04-26T21:14:59.050 INFO:teuthology.orchestra.run.smithi038.stdout:(113/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-04-26T21:14:59.099 INFO:teuthology.orchestra.run.smithi038.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 520 kB/s | 43 kB 00:00 2024-04-26T21:14:59.117 INFO:teuthology.orchestra.run.smithi038.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-04-26T21:14:59.150 INFO:teuthology.orchestra.run.smithi038.stdout:(116/152): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-04-26T21:14:59.153 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-04-26T21:14:59.167 INFO:teuthology.orchestra.run.smithi038.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 223 kB/s | 15 kB 00:00 2024-04-26T21:14:59.193 INFO:teuthology.orchestra.run.smithi038.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-26T21:14:59.243 INFO:teuthology.orchestra.run.smithi038.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-04-26T21:14:59.268 INFO:teuthology.orchestra.run.smithi038.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 469 kB/s | 35 kB 00:00 2024-04-26T21:14:59.327 INFO:teuthology.orchestra.run.smithi038.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-04-26T21:14:59.352 INFO:teuthology.orchestra.run.smithi038.stdout:(122/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.6 MB/s | 303 kB 00:00 2024-04-26T21:14:59.411 INFO:teuthology.orchestra.run.smithi038.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-26T21:14:59.436 INFO:teuthology.orchestra.run.smithi038.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-04-26T21:14:59.461 INFO:teuthology.orchestra.run.smithi038.stdout:(125/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-26T21:14:59.486 INFO:teuthology.orchestra.run.smithi038.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-04-26T21:14:59.512 INFO:teuthology.orchestra.run.smithi038.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 306 kB/s | 23 kB 00:00 2024-04-26T21:14:59.537 INFO:teuthology.orchestra.run.smithi038.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-26T21:14:59.562 INFO:teuthology.orchestra.run.smithi038.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-04-26T21:14:59.613 INFO:teuthology.orchestra.run.smithi038.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 614 kB/s | 46 kB 00:00 2024-04-26T21:14:59.638 INFO:teuthology.orchestra.run.smithi038.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-04-26T21:14:59.697 INFO:teuthology.orchestra.run.smithi038.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-04-26T21:14:59.722 INFO:teuthology.orchestra.run.smithi038.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 531 kB/s | 58 kB 00:00 2024-04-26T21:14:59.747 INFO:teuthology.orchestra.run.smithi038.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-26T21:14:59.798 INFO:teuthology.orchestra.run.smithi038.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 219 kB/s | 16 kB 00:00 2024-04-26T21:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-04-26T21:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 716 kB/s | 90 kB 00:00 2024-04-26T21:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-04-26T21:14:59.899 INFO:teuthology.orchestra.run.smithi038.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-26T21:14:59.924 INFO:teuthology.orchestra.run.smithi038.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2024-04-26T21:14:59.949 INFO:teuthology.orchestra.run.smithi038.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 473 kB/s | 36 kB 00:00 2024-04-26T21:14:59.974 INFO:teuthology.orchestra.run.smithi038.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-04-26T21:15:00.008 INFO:teuthology.orchestra.run.smithi038.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-26T21:15:00.033 INFO:teuthology.orchestra.run.smithi038.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-04-26T21:15:00.059 INFO:teuthology.orchestra.run.smithi038.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-04-26T21:15:00.084 INFO:teuthology.orchestra.run.smithi038.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-26T21:15:00.166 INFO:teuthology.orchestra.run.smithi038.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-04-26T21:15:00.218 INFO:teuthology.orchestra.run.smithi038.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-26T21:15:00.352 INFO:teuthology.orchestra.run.smithi038.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 10 MB/s | 1.3 MB 00:00 2024-04-26T21:15:00.502 INFO:teuthology.orchestra.run.smithi038.stdout:(150/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 9.4 MB/s | 3.1 MB 00:00 2024-04-26T21:15:00.552 INFO:teuthology.orchestra.run.smithi186.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 561 kB/s | 911 kB 00:01 2024-04-26T21:15:00.602 INFO:teuthology.orchestra.run.smithi038.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.6 MB/s | 2.4 MB 00:00 2024-04-26T21:15:00.961 INFO:teuthology.orchestra.run.smithi038.stdout:(152/152): librados2-19.0.0-3306.gbc612ffb.el9. 3.9 MB/s | 3.4 MB 00:00 2024-04-26T21:15:00.970 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-04-26T21:15:00.970 INFO:teuthology.orchestra.run.smithi038.stdout:Total 14 MB/s | 222 MB 00:15 2024-04-26T21:15:01.302 INFO:teuthology.orchestra.run.smithi186.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 178 kB/s | 133 kB 00:00 2024-04-26T21:15:01.809 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:15:01.811 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-26T21:15:01.835 INFO:teuthology.orchestra.run.smithi001.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-26T21:15:01.836 INFO:teuthology.orchestra.run.smithi001.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-26T21:15:01.838 INFO:teuthology.orchestra.run.smithi001.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-26T21:15:01.839 INFO:teuthology.orchestra.run.smithi001.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-26T21:15:01.842 INFO:teuthology.orchestra.run.smithi001.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-26T21:15:01.845 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-26T21:15:01.847 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-26T21:15:01.923 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-04-26T21:15:01.943 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-04-26T21:15:01.943 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-04-26T21:15:01.943 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 6.6 k 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 889 k 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 145 k 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 1.5 M 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 142 k 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 3.5 M 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 7.4 M 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 49 k 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 11 M 2024-04-26T21:15:01.944 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 49 M 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 273 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 762 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 225 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-shell x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 27 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 18 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 32 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 747 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 127 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-26T21:15:01.945 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 161 k 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 321 k 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 298 k 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 99 k 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 85 k 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 171 k 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-04-26T21:15:01.946 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.4 M 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 3.1 M 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 21 M 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 34 k 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 2.4 M 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 249 k 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 4.7 M 2024-04-26T21:15:01.947 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 17 M 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3306.gbc612ffb.el9 ceph-noarch 16 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 25 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 163 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-26T21:15:01.948 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 501 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 5.4 M 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-26T21:15:01.949 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 45 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:19.0.0-3306.gbc612ffb.el9 ceph 135 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-26T21:15:01.950 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-26T21:15:01.951 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-26T21:15:01.952 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-26T21:15:01.953 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-26T21:15:01.954 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:Install 148 Packages 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 4 Packages 2024-04-26T21:15:01.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:15:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 222 M 2024-04-26T21:15:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-04-26T21:15:02.119 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-04-26T21:15:02.278 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-04-26T21:15:02.278 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-04-26T21:15:02.369 INFO:teuthology.orchestra.run.smithi186.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 816 kB/s | 4.7 MB 00:05 2024-04-26T21:15:02.652 INFO:teuthology.orchestra.run.smithi186.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 735 kB/s | 4.4 MB 00:06 2024-04-26T21:15:03.238 INFO:teuthology.orchestra.run.smithi001.stdout:(1/152): ceph-19.0.0-3306.gbc612ffb.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-04-26T21:15:03.302 INFO:teuthology.orchestra.run.smithi186.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 135 kB/s | 87 kB 00:00 2024-04-26T21:15:03.472 INFO:teuthology.orchestra.run.smithi001.stdout:(2/152): ceph-fuse-19.0.0-3306.gbc612ffb.el9.x8 3.7 MB/s | 889 kB 00:00 2024-04-26T21:15:03.530 INFO:teuthology.orchestra.run.smithi001.stdout:(3/152): ceph-immutable-object-cache-19.0.0-330 2.5 MB/s | 145 kB 00:00 2024-04-26T21:15:03.822 INFO:teuthology.orchestra.run.smithi001.stdout:(4/152): ceph-mds-19.0.0-3306.gbc612ffb.el9.x86 8.1 MB/s | 2.4 MB 00:00 2024-04-26T21:15:04.023 INFO:teuthology.orchestra.run.smithi001.stdout:(5/152): ceph-mgr-19.0.0-3306.gbc612ffb.el9.x86 7.6 MB/s | 1.5 MB 00:00 2024-04-26T21:15:04.053 INFO:teuthology.orchestra.run.smithi186.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 614 kB/s | 1.0 MB 00:01 2024-04-26T21:15:04.164 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-04-26T21:15:04.165 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-04-26T21:15:04.498 INFO:teuthology.orchestra.run.smithi001.stdout:(6/152): ceph-base-19.0.0-3306.gbc612ffb.el9.x8 3.5 MB/s | 5.4 MB 00:01 2024-04-26T21:15:04.640 INFO:teuthology.orchestra.run.smithi001.stdout:(7/152): ceph-mon-19.0.0-3306.gbc612ffb.el9.x86 7.6 MB/s | 4.7 MB 00:00 2024-04-26T21:15:04.836 INFO:teuthology.orchestra.run.smithi186.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 312 kB/s | 244 kB 00:00 2024-04-26T21:15:05.165 INFO:teuthology.orchestra.run.smithi001.stdout:(8/152): ceph-common-19.0.0-3306.gbc612ffb.el9. 9.6 MB/s | 21 MB 00:02 2024-04-26T21:15:05.218 INFO:teuthology.orchestra.run.smithi001.stdout:(9/152): ceph-selinux-19.0.0-3306.gbc612ffb.el9 516 kB/s | 25 kB 00:00 2024-04-26T21:15:05.611 INFO:teuthology.orchestra.run.smithi186.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 320 kB/s | 248 kB 00:00 2024-04-26T21:15:05.724 INFO:teuthology.orchestra.run.smithi001.stdout:(10/152): ceph-radosgw-19.0.0-3306.gbc612ffb.el 9.8 MB/s | 11 MB 00:01 2024-04-26T21:15:05.791 INFO:teuthology.orchestra.run.smithi001.stdout:(11/152): cephfs-mirror-19.0.0-3306.gbc612ffb.e 3.3 MB/s | 225 kB 00:00 2024-04-26T21:15:05.874 INFO:teuthology.orchestra.run.smithi001.stdout:(12/152): cephfs-shell-19.0.0-3306.gbc612ffb.el 328 kB/s | 27 kB 00:00 2024-04-26T21:15:05.958 INFO:teuthology.orchestra.run.smithi001.stdout:(13/152): libcephfs-devel-19.0.0-3306.gbc612ffb 383 kB/s | 32 kB 00:00 2024-04-26T21:15:06.119 INFO:teuthology.orchestra.run.smithi186.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 50 kB/s | 26 kB 00:00 2024-04-26T21:15:06.141 INFO:teuthology.orchestra.run.smithi001.stdout:(14/152): libcephfs2-19.0.0-3306.gbc612ffb.el9. 4.0 MB/s | 747 kB 00:00 2024-04-26T21:15:06.217 INFO:teuthology.orchestra.run.smithi001.stdout:(15/152): libcephsqlite-19.0.0-3306.gbc612ffb.e 2.1 MB/s | 163 kB 00:00 2024-04-26T21:15:06.292 INFO:teuthology.orchestra.run.smithi001.stdout:(16/152): librados-devel-19.0.0-3306.gbc612ffb. 1.7 MB/s | 127 kB 00:00 2024-04-26T21:15:06.408 INFO:teuthology.orchestra.run.smithi001.stdout:(17/152): libradosstriper1-19.0.0-3306.gbc612ff 4.3 MB/s | 501 kB 00:00 2024-04-26T21:15:06.595 INFO:teuthology.orchestra.run.smithi186.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 40 kB/s | 19 kB 00:00 2024-04-26T21:15:07.043 INFO:teuthology.orchestra.run.smithi001.stdout:(18/152): ceph-osd-19.0.0-3306.gbc612ffb.el9.x8 6.6 MB/s | 17 MB 00:02 2024-04-26T21:15:07.110 INFO:teuthology.orchestra.run.smithi001.stdout:(19/152): python3-ceph-argparse-19.0.0-3306.gbc 680 kB/s | 45 kB 00:00 2024-04-26T21:15:07.227 INFO:teuthology.orchestra.run.smithi001.stdout:(20/152): librgw2-19.0.0-3306.gbc612ffb.el9.x86 6.6 MB/s | 5.4 MB 00:00 2024-04-26T21:15:07.260 INFO:teuthology.orchestra.run.smithi001.stdout:(21/152): python3-ceph-common-19.0.0-3306.gbc61 906 kB/s | 135 kB 00:00 2024-04-26T21:15:07.311 INFO:teuthology.orchestra.run.smithi001.stdout:(22/152): python3-cephfs-19.0.0-3306.gbc612ffb. 1.9 MB/s | 161 kB 00:00 2024-04-26T21:15:07.328 INFO:teuthology.orchestra.run.smithi186.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 235 kB/s | 172 kB 00:00 2024-04-26T21:15:07.352 INFO:teuthology.orchestra.run.smithi001.stdout:(23/152): python3-rados-19.0.0-3306.gbc612ffb.e 3.5 MB/s | 321 kB 00:00 2024-04-26T21:15:07.386 INFO:teuthology.orchestra.run.smithi001.stdout:(24/152): python3-rbd-19.0.0-3306.gbc612ffb.el9 3.9 MB/s | 298 kB 00:00 2024-04-26T21:15:07.419 INFO:teuthology.orchestra.run.smithi001.stdout:(25/152): python3-rgw-19.0.0-3306.gbc612ffb.el9 1.5 MB/s | 99 kB 00:00 2024-04-26T21:15:07.445 INFO:teuthology.orchestra.run.smithi001.stdout:(26/152): rbd-fuse-19.0.0-3306.gbc612ffb.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-26T21:15:07.503 INFO:teuthology.orchestra.run.smithi001.stdout:(27/152): rbd-nbd-19.0.0-3306.gbc612ffb.el9.x86 2.9 MB/s | 171 kB 00:00 2024-04-26T21:15:07.562 INFO:teuthology.orchestra.run.smithi001.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3306.g 596 kB/s | 34 kB 00:00 2024-04-26T21:15:07.637 INFO:teuthology.orchestra.run.smithi001.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3306.gbc612ff 1.9 MB/s | 142 kB 00:00 2024-04-26T21:15:07.887 INFO:teuthology.orchestra.run.smithi186.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 62 kB/s | 35 kB 00:00 2024-04-26T21:15:07.954 INFO:teuthology.orchestra.run.smithi001.stdout:(30/152): rbd-mirror-19.0.0-3306.gbc612ffb.el9. 5.8 MB/s | 3.1 MB 00:00 2024-04-26T21:15:08.063 INFO:teuthology.orchestra.run.smithi001.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3306.gbc612 8.3 MB/s | 3.5 MB 00:00 2024-04-26T21:15:08.163 INFO:teuthology.orchestra.run.smithi001.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3306.gbc 2.5 MB/s | 249 kB 00:00 2024-04-26T21:15:08.247 INFO:teuthology.orchestra.run.smithi001.stdout:(33/152): ceph-mgr-rook-19.0.0-3306.gbc612ffb.e 599 kB/s | 49 kB 00:00 2024-04-26T21:15:08.322 INFO:teuthology.orchestra.run.smithi001.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3306.gb 222 kB/s | 16 kB 00:00 2024-04-26T21:15:08.414 INFO:teuthology.orchestra.run.smithi001.stdout:(35/152): ceph-volume-19.0.0-3306.gbc612ffb.el9 2.9 MB/s | 273 kB 00:00 2024-04-26T21:15:08.438 INFO:teuthology.orchestra.run.smithi186.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 829 kB/s | 5.8 MB 00:07 2024-04-26T21:15:08.931 INFO:teuthology.orchestra.run.smithi001.stdout:(36/152): ceph-test-19.0.0-3306.gbc612ffb.el9.x 13 MB/s | 49 MB 00:03 2024-04-26T21:15:08.998 INFO:teuthology.orchestra.run.smithi001.stdout:(37/152): cephadm-19.0.0-3306.gbc612ffb.el9.noa 1.3 MB/s | 762 kB 00:00 2024-04-26T21:15:09.023 INFO:teuthology.orchestra.run.smithi001.stdout:(38/152): cephfs-top-19.0.0-3306.gbc612ffb.el9. 201 kB/s | 18 kB 00:00 2024-04-26T21:15:09.190 INFO:teuthology.orchestra.run.smithi001.stdout:(39/152): ceph-mgr-diskprediction-local-19.0.0- 6.0 MB/s | 7.4 MB 00:01 2024-04-26T21:15:09.254 INFO:teuthology.orchestra.run.smithi186.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 307 kB/s | 250 kB 00:00 2024-04-26T21:15:09.924 INFO:teuthology.orchestra.run.smithi001.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 89 kB/s | 80 kB 00:00 2024-04-26T21:15:10.016 INFO:teuthology.orchestra.run.smithi001.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 237 kB/s | 195 kB 00:00 2024-04-26T21:15:10.029 INFO:teuthology.orchestra.run.smithi186.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 287 kB/s | 222 kB 00:00 2024-04-26T21:15:10.338 INFO:teuthology.orchestra.run.smithi186.stdout:(83/152): python3-babel-2.9.1-2.el9.noarch.rpm 868 kB/s | 6.0 MB 00:07 2024-04-26T21:15:10.423 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-04-26T21:15:10.507 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T21:15:10.545 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T21:15:10.599 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T21:15:10.654 INFO:teuthology.orchestra.run.smithi186.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 123 kB/s | 77 kB 00:00 2024-04-26T21:15:10.666 INFO:teuthology.orchestra.run.smithi001.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 489 kB/s | 803 kB 00:01 2024-04-26T21:15:10.774 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T21:15:10.800 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T21:15:10.844 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T21:15:10.858 INFO:teuthology.orchestra.run.smithi001.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 272 kB/s | 253 kB 00:00 2024-04-26T21:15:10.930 INFO:teuthology.orchestra.run.smithi186.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 71 kB/s | 42 kB 00:00 2024-04-26T21:15:11.206 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T21:15:11.309 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:15:11.375 INFO:teuthology.orchestra.run.smithi001.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 150 kB/s | 106 kB 00:00 2024-04-26T21:15:11.425 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:15:11.452 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:15:11.488 INFO:teuthology.orchestra.run.smithi186.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 323 kB/s | 269 kB 00:00 2024-04-26T21:15:11.528 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:15:11.565 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:15:11.630 INFO:teuthology.orchestra.run.smithi186.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 227 kB/s | 159 kB 00:00 2024-04-26T21:15:11.675 INFO:teuthology.orchestra.run.smithi001.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 166 kB/s | 135 kB 00:00 2024-04-26T21:15:11.720 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T21:15:11.821 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T21:15:11.848 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:15:11.954 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:15:11.986 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:15:12.046 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:15:12.067 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:15:12.093 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:15:12.159 INFO:teuthology.orchestra.run.smithi001.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 586 kB/s | 1.2 MB 00:02 2024-04-26T21:15:12.197 INFO:teuthology.orchestra.run.smithi186.stdout:(88/152): python3-pytz-2021.1-5.el9.noarch.rpm 90 kB/s | 51 kB 00:00 2024-04-26T21:15:12.200 INFO:teuthology.orchestra.run.smithi001.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 151 kB/s | 125 kB 00:00 2024-04-26T21:15:12.252 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:15:12.284 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:15:12.322 INFO:teuthology.orchestra.run.smithi186.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 335 kB/s | 279 kB 00:00 2024-04-26T21:15:12.396 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T21:15:12.442 INFO:teuthology.orchestra.run.smithi001.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 689 kB 00:00 2024-04-26T21:15:12.499 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T21:15:12.501 INFO:teuthology.orchestra.run.smithi001.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-26T21:15:12.533 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T21:15:12.543 INFO:teuthology.orchestra.run.smithi001.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 2.7 MB/s | 943 kB 00:00 2024-04-26T21:15:12.569 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T21:15:12.576 INFO:teuthology.orchestra.run.smithi001.stdout:(51/152): python3-urllib3-1.26.5-5.el9.noarch.r 238 kB/s | 215 kB 00:00 2024-04-26T21:15:12.602 INFO:teuthology.orchestra.run.smithi001.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 216 kB/s | 21 kB 00:00 2024-04-26T21:15:12.645 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T21:15:12.685 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T21:15:12.710 INFO:teuthology.orchestra.run.smithi001.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 1.8 MB/s | 313 kB 00:00 2024-04-26T21:15:12.727 INFO:teuthology.orchestra.run.smithi001.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 263 kB/s | 33 kB 00:00 2024-04-26T21:15:12.729 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T21:15:12.761 INFO:teuthology.orchestra.run.smithi001.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 369 kB/s | 18 kB 00:00 2024-04-26T21:15:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T21:15:12.777 INFO:teuthology.orchestra.run.smithi001.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 224 kB/s | 11 kB 00:00 2024-04-26T21:15:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T21:15:12.806 INFO:teuthology.orchestra.run.smithi186.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 88 kB/s | 54 kB 00:00 2024-04-26T21:15:12.820 INFO:teuthology.orchestra.run.smithi001.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-26T21:15:12.836 INFO:teuthology.orchestra.run.smithi001.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 781 kB/s | 45 kB 00:00 2024-04-26T21:15:12.870 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T21:15:12.920 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T21:15:12.959 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T21:15:12.970 INFO:teuthology.orchestra.run.smithi001.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-04-26T21:15:13.003 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T21:15:13.004 INFO:teuthology.orchestra.run.smithi001.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 3.4 MB/s | 582 kB 00:00 2024-04-26T21:15:13.029 INFO:teuthology.orchestra.run.smithi001.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 441 kB/s | 25 kB 00:00 2024-04-26T21:15:13.062 INFO:teuthology.orchestra.run.smithi001.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 824 kB/s | 47 kB 00:00 2024-04-26T21:15:13.081 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T21:15:13.113 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T21:15:13.129 INFO:teuthology.orchestra.run.smithi001.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 292 kB 00:00 2024-04-26T21:15:13.157 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T21:15:13.163 INFO:teuthology.orchestra.run.smithi001.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.9 MB/s | 188 kB 00:00 2024-04-26T21:15:13.185 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T21:15:13.188 INFO:teuthology.orchestra.run.smithi001.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 611 kB/s | 35 kB 00:00 2024-04-26T21:15:13.257 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T21:15:13.300 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T21:15:13.356 INFO:teuthology.orchestra.run.smithi186.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 76 kB/s | 42 kB 00:00 2024-04-26T21:15:13.377 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T21:15:13.439 INFO:teuthology.orchestra.run.smithi001.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.6 MB/s | 3.0 MB 00:00 2024-04-26T21:15:13.522 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T21:15:13.578 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T21:15:13.747 INFO:teuthology.orchestra.run.smithi001.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 5.2 MB/s | 1.6 MB 00:00 2024-04-26T21:15:13.906 INFO:teuthology.orchestra.run.smithi186.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 80 kB/s | 44 kB 00:00 2024-04-26T21:15:13.931 INFO:teuthology.orchestra.run.smithi001.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 4.9 MB/s | 911 kB 00:00 2024-04-26T21:15:14.023 INFO:teuthology.orchestra.run.smithi001.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2024-04-26T21:15:14.115 INFO:teuthology.orchestra.run.smithi186.stdout:(93/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 838 kB/s | 5.1 MB 00:06 2024-04-26T21:15:14.123 INFO:teuthology.orchestra.run.smithi001.stdout:(70/152): openblas-openmp-0.3.21-2.el9.x86_64.r 4.9 MB/s | 4.7 MB 00:00 2024-04-26T21:15:14.133 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T21:15:14.148 INFO:teuthology.orchestra.run.smithi001.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 616 kB/s | 133 kB 00:00 2024-04-26T21:15:14.207 INFO:teuthology.orchestra.run.smithi001.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-26T21:15:14.225 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T21:15:14.261 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T21:15:14.304 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T21:15:14.340 INFO:teuthology.orchestra.run.smithi001.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.7 MB/s | 1.0 MB 00:00 2024-04-26T21:15:14.350 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T21:15:14.392 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T21:15:14.433 INFO:teuthology.orchestra.run.smithi001.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-04-26T21:15:14.533 INFO:teuthology.orchestra.run.smithi001.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-26T21:15:14.539 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T21:15:14.595 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T21:15:14.600 INFO:teuthology.orchestra.run.smithi001.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 390 kB/s | 26 kB 00:00 2024-04-26T21:15:14.651 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T21:15:14.666 INFO:teuthology.orchestra.run.smithi001.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 291 kB/s | 19 kB 00:00 2024-04-26T21:15:14.690 INFO:teuthology.orchestra.run.smithi186.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 66 kB/s | 36 kB 00:00 2024-04-26T21:15:14.733 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T21:15:14.749 INFO:teuthology.orchestra.run.smithi186.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 262 kB/s | 221 kB 00:00 2024-04-26T21:15:14.767 INFO:teuthology.orchestra.run.smithi001.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-26T21:15:14.771 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T21:15:14.808 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T21:15:14.842 INFO:teuthology.orchestra.run.smithi001.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 469 kB/s | 35 kB 00:00 2024-04-26T21:15:14.855 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T21:15:14.890 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T21:15:14.934 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T21:15:14.976 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T21:15:15.101 INFO:teuthology.orchestra.run.smithi001.stdout:(80/152): postgresql-server-13.14-1.el9.x86_64. 5.4 MB/s | 5.8 MB 00:01 2024-04-26T21:15:15.118 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T21:15:15.165 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T21:15:15.167 INFO:teuthology.orchestra.run.smithi001.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-26T21:15:15.186 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T21:15:15.228 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T21:15:15.243 INFO:teuthology.orchestra.run.smithi001.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-04-26T21:15:15.302 INFO:teuthology.orchestra.run.smithi001.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-26T21:15:15.326 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T21:15:15.341 INFO:teuthology.orchestra.run.smithi186.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 73 kB/s | 47 kB 00:00 2024-04-26T21:15:15.362 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T21:15:15.385 INFO:teuthology.orchestra.run.smithi001.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 514 kB/s | 42 kB 00:00 2024-04-26T21:15:15.402 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T21:15:15.435 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T21:15:15.460 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T21:15:15.519 INFO:teuthology.orchestra.run.smithi001.stdout:(85/152): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-04-26T21:15:15.552 INFO:teuthology.orchestra.run.smithi001.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 1.6 MB/s | 269 kB 00:00 2024-04-26T21:15:15.555 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T21:15:15.586 INFO:teuthology.orchestra.run.smithi001.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.4 MB/s | 159 kB 00:00 2024-04-26T21:15:15.620 INFO:teuthology.orchestra.run.smithi001.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-26T21:15:15.641 INFO:teuthology.orchestra.run.smithi186.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 342 kB/s | 305 kB 00:00 2024-04-26T21:15:15.645 INFO:teuthology.orchestra.run.smithi001.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 884 kB/s | 51 kB 00:00 2024-04-26T21:15:15.684 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T21:15:15.719 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T21:15:15.738 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T21:15:15.762 INFO:teuthology.orchestra.run.smithi001.stdout:(90/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.6 MB/s | 5.1 MB 00:00 2024-04-26T21:15:15.772 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T21:15:15.787 INFO:teuthology.orchestra.run.smithi001.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 322 kB/s | 54 kB 00:00 2024-04-26T21:15:15.812 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T21:15:15.812 INFO:teuthology.orchestra.run.smithi001.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 844 kB/s | 42 kB 00:00 2024-04-26T21:15:15.838 INFO:teuthology.orchestra.run.smithi001.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 892 kB/s | 44 kB 00:00 2024-04-26T21:15:15.843 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T21:15:15.871 INFO:teuthology.orchestra.run.smithi001.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-04-26T21:15:15.881 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T21:15:15.888 INFO:teuthology.orchestra.run.smithi001.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 726 kB/s | 36 kB 00:00 2024-04-26T21:15:15.916 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T21:15:15.922 INFO:teuthology.orchestra.run.smithi001.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 963 kB/s | 47 kB 00:00 2024-04-26T21:15:15.932 INFO:teuthology.orchestra.run.smithi186.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 98 kB/s | 58 kB 00:00 2024-04-26T21:15:15.949 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T21:15:15.955 INFO:teuthology.orchestra.run.smithi001.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-26T21:15:15.981 INFO:teuthology.orchestra.run.smithi001.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-04-26T21:15:16.008 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T21:15:16.022 INFO:teuthology.orchestra.run.smithi001.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-04-26T21:15:16.040 INFO:teuthology.orchestra.run.smithi001.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-26T21:15:16.073 INFO:teuthology.orchestra.run.smithi001.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 908 kB/s | 45 kB 00:00 2024-04-26T21:15:16.132 INFO:teuthology.orchestra.run.smithi001.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 245 kB/s | 22 kB 00:00 2024-04-26T21:15:16.222 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T21:15:16.291 INFO:teuthology.orchestra.run.smithi001.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 3.9 MB/s | 863 kB 00:00 2024-04-26T21:15:16.291 INFO:teuthology.orchestra.run.smithi186.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 144 kB/s | 94 kB 00:00 2024-04-26T21:15:16.441 INFO:teuthology.orchestra.run.smithi001.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-04-26T21:15:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T21:15:16.499 INFO:teuthology.orchestra.run.smithi001.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 338 kB/s | 19 kB 00:00 2024-04-26T21:15:16.533 INFO:teuthology.orchestra.run.smithi186.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 106 kB/s | 64 kB 00:00 2024-04-26T21:15:16.558 INFO:teuthology.orchestra.run.smithi001.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-26T21:15:16.558 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T21:15:16.584 INFO:teuthology.orchestra.run.smithi186.stdout:(101/152): lua-devel-5.4.4-4.el9.x86_64.rpm 449 kB/s | 22 kB 00:00 2024-04-26T21:15:16.596 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T21:15:16.617 INFO:teuthology.orchestra.run.smithi001.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 451 kB/s | 26 kB 00:00 2024-04-26T21:15:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T21:15:16.675 INFO:teuthology.orchestra.run.smithi001.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 859 kB/s | 49 kB 00:00 2024-04-26T21:15:16.700 INFO:teuthology.orchestra.run.smithi186.stdout:(102/152): protobuf-compiler-3.14.0-13.el9.x86_ 7.3 MB/s | 863 kB 00:00 2024-04-26T21:15:16.734 INFO:teuthology.orchestra.run.smithi001.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-26T21:15:16.756 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T21:15:16.807 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T21:15:16.825 INFO:teuthology.orchestra.run.smithi001.stdout:(110/152): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-26T21:15:16.859 INFO:teuthology.orchestra.run.smithi186.stdout:(103/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 79 kB/s | 45 kB 00:00 2024-04-26T21:15:16.861 INFO:teuthology.orchestra.run.smithi001.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-26T21:15:16.919 INFO:teuthology.orchestra.run.smithi001.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-04-26T21:15:16.969 INFO:teuthology.orchestra.run.smithi001.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 5.0 MB/s | 548 kB 00:00 2024-04-26T21:15:16.994 INFO:teuthology.orchestra.run.smithi001.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 586 kB/s | 43 kB 00:00 2024-04-26T21:15:17.044 INFO:teuthology.orchestra.run.smithi001.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 433 kB/s | 32 kB 00:00 2024-04-26T21:15:17.061 INFO:teuthology.orchestra.run.smithi001.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 226 kB/s | 15 kB 00:00 2024-04-26T21:15:17.103 INFO:teuthology.orchestra.run.smithi001.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-26T21:15:17.162 INFO:teuthology.orchestra.run.smithi001.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-26T21:15:17.195 INFO:teuthology.orchestra.run.smithi001.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-26T21:15:17.220 INFO:teuthology.orchestra.run.smithi001.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 611 kB/s | 35 kB 00:00 2024-04-26T21:15:17.262 INFO:teuthology.orchestra.run.smithi001.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-04-26T21:15:17.321 INFO:teuthology.orchestra.run.smithi001.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 2.5 MB/s | 144 kB 00:00 2024-04-26T21:15:17.388 INFO:teuthology.orchestra.run.smithi001.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-04-26T21:15:17.413 INFO:teuthology.orchestra.run.smithi001.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 117 kB/s | 11 kB 00:00 2024-04-26T21:15:17.447 INFO:teuthology.orchestra.run.smithi001.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 307 kB/s | 18 kB 00:00 2024-04-26T21:15:17.459 INFO:teuthology.orchestra.run.smithi186.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 722 kB/s | 548 kB 00:00 2024-04-26T21:15:17.473 INFO:teuthology.orchestra.run.smithi001.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 398 kB/s | 23 kB 00:00 2024-04-26T21:15:17.506 INFO:teuthology.orchestra.run.smithi001.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 335 kB/s | 19 kB 00:00 2024-04-26T21:15:17.526 INFO:teuthology.orchestra.run.smithi186.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 463 kB/s | 309 kB 00:00 2024-04-26T21:15:17.531 INFO:teuthology.orchestra.run.smithi001.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 340 kB/s | 20 kB 00:00 2024-04-26T21:15:17.568 INFO:teuthology.orchestra.run.smithi186.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-04-26T21:15:17.569 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T21:15:17.598 INFO:teuthology.orchestra.run.smithi001.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-26T21:15:17.611 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T21:15:17.624 INFO:teuthology.orchestra.run.smithi001.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 508 kB/s | 46 kB 00:00 2024-04-26T21:15:17.646 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T21:15:17.657 INFO:teuthology.orchestra.run.smithi001.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-26T21:15:17.668 INFO:teuthology.orchestra.run.smithi186.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-26T21:15:17.691 INFO:teuthology.orchestra.run.smithi001.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 871 kB/s | 58 kB 00:00 2024-04-26T21:15:17.724 INFO:teuthology.orchestra.run.smithi001.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-26T21:15:17.748 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T21:15:17.750 INFO:teuthology.orchestra.run.smithi001.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 284 kB/s | 16 kB 00:00 2024-04-26T21:15:17.777 INFO:teuthology.orchestra.run.smithi186.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-04-26T21:15:17.783 INFO:teuthology.orchestra.run.smithi001.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-26T21:15:17.804 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T21:15:17.809 INFO:teuthology.orchestra.run.smithi001.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 487 kB/s | 28 kB 00:00 2024-04-26T21:15:17.842 INFO:teuthology.orchestra.run.smithi001.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 533 kB/s | 31 kB 00:00 2024-04-26T21:15:17.876 INFO:teuthology.orchestra.run.smithi001.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-04-26T21:15:17.894 INFO:teuthology.orchestra.run.smithi186.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 579 kB/s | 67 kB 00:00 2024-04-26T21:15:17.894 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T21:15:17.901 INFO:teuthology.orchestra.run.smithi001.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-26T21:15:17.931 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T21:15:17.935 INFO:teuthology.orchestra.run.smithi001.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 617 kB/s | 36 kB 00:00 2024-04-26T21:15:17.960 INFO:teuthology.orchestra.run.smithi001.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 882 kB/s | 51 kB 00:00 2024-04-26T21:15:18.002 INFO:teuthology.orchestra.run.smithi001.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-04-26T21:15:18.027 INFO:teuthology.orchestra.run.smithi001.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-04-26T21:15:18.028 INFO:teuthology.orchestra.run.smithi186.stdout:(110/152): libarrow-9.0.0-11.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-04-26T21:15:18.053 INFO:teuthology.orchestra.run.smithi186.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 953 kB/s | 151 kB 00:00 2024-04-26T21:15:18.077 INFO:teuthology.orchestra.run.smithi001.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-04-26T21:15:18.103 INFO:teuthology.orchestra.run.smithi001.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 268 kB/s | 20 kB 00:00 2024-04-26T21:15:18.117 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T21:15:18.137 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T21:15:18.145 INFO:teuthology.orchestra.run.smithi186.stdout:(112/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-04-26T21:15:18.217 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T21:15:18.220 INFO:teuthology.orchestra.run.smithi186.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-04-26T21:15:18.253 INFO:teuthology.orchestra.run.smithi186.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 400 kB/s | 43 kB 00:00 2024-04-26T21:15:18.328 INFO:teuthology.orchestra.run.smithi001.stdout:(146/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.1 MB/s | 16 MB 00:02 2024-04-26T21:15:18.329 INFO:teuthology.orchestra.run.smithi186.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-26T21:15:18.353 INFO:teuthology.orchestra.run.smithi001.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 698 kB/s | 192 kB 00:00 2024-04-26T21:15:18.362 INFO:teuthology.orchestra.run.smithi186.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-04-26T21:15:18.428 INFO:teuthology.orchestra.run.smithi001.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 4.9 MB/s | 1.6 MB 00:00 2024-04-26T21:15:18.454 INFO:teuthology.orchestra.run.smithi186.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-04-26T21:15:18.480 INFO:teuthology.orchestra.run.smithi186.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-26T21:15:18.588 INFO:teuthology.orchestra.run.smithi186.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-04-26T21:15:18.614 INFO:teuthology.orchestra.run.smithi186.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-04-26T21:15:18.629 INFO:teuthology.orchestra.run.smithi001.stdout:(149/152): librados2-19.0.0-3306.gbc612ffb.el9. 11 MB/s | 3.4 MB 00:00 2024-04-26T21:15:18.714 INFO:teuthology.orchestra.run.smithi186.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.6 MB/s | 206 kB 00:00 2024-04-26T21:15:18.773 INFO:teuthology.orchestra.run.smithi186.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-04-26T21:15:18.831 INFO:teuthology.orchestra.run.smithi186.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-04-26T21:15:18.873 INFO:teuthology.orchestra.run.smithi186.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-04-26T21:15:18.894 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T21:15:18.940 INFO:teuthology.orchestra.run.smithi186.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-04-26T21:15:18.982 INFO:teuthology.orchestra.run.smithi186.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-04-26T21:15:18.987 INFO:teuthology.orchestra.run.smithi001.stdout:(150/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 5.0 MB/s | 3.1 MB 00:00 2024-04-26T21:15:19.013 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T21:15:19.049 INFO:teuthology.orchestra.run.smithi186.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-04-26T21:15:19.091 INFO:teuthology.orchestra.run.smithi186.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-04-26T21:15:19.200 INFO:teuthology.orchestra.run.smithi186.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 427 kB/s | 46 kB 00:00 2024-04-26T21:15:19.258 INFO:teuthology.orchestra.run.smithi186.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-04-26T21:15:19.308 INFO:teuthology.orchestra.run.smithi186.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 727 kB/s | 79 kB 00:00 2024-04-26T21:15:19.367 INFO:teuthology.orchestra.run.smithi186.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 532 kB/s | 58 kB 00:00 2024-04-26T21:15:19.417 INFO:teuthology.orchestra.run.smithi186.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-04-26T21:15:19.476 INFO:teuthology.orchestra.run.smithi186.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-04-26T21:15:19.526 INFO:teuthology.orchestra.run.smithi186.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 828 kB/s | 90 kB 00:00 2024-04-26T21:15:19.585 INFO:teuthology.orchestra.run.smithi186.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-04-26T21:15:19.635 INFO:teuthology.orchestra.run.smithi186.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 284 kB/s | 31 kB 00:00 2024-04-26T21:15:19.694 INFO:teuthology.orchestra.run.smithi186.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-04-26T21:15:19.744 INFO:teuthology.orchestra.run.smithi186.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-04-26T21:15:19.802 INFO:teuthology.orchestra.run.smithi186.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-04-26T21:15:19.853 INFO:teuthology.orchestra.run.smithi186.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 469 kB/s | 51 kB 00:00 2024-04-26T21:15:19.911 INFO:teuthology.orchestra.run.smithi186.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-26T21:15:19.962 INFO:teuthology.orchestra.run.smithi186.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 826 kB/s | 90 kB 00:00 2024-04-26T21:15:20.037 INFO:teuthology.orchestra.run.smithi186.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-26T21:15:20.062 INFO:teuthology.orchestra.run.smithi186.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-04-26T21:15:20.154 INFO:teuthology.orchestra.run.smithi186.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-26T21:15:20.204 INFO:teuthology.orchestra.run.smithi186.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-26T21:15:20.307 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T21:15:20.393 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T21:15:20.421 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T21:15:20.454 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T21:15:20.521 INFO:teuthology.orchestra.run.smithi186.stdout:(148/152): librbd1-19.0.0-3306.gbc612ffb.el9.x8 9.9 MB/s | 3.1 MB 00:00 2024-04-26T21:15:20.596 INFO:teuthology.orchestra.run.smithi186.stdout:(149/152): librados2-19.0.0-3306.gbc612ffb.el9. 7.7 MB/s | 3.4 MB 00:00 2024-04-26T21:15:20.637 INFO:teuthology.orchestra.run.smithi001.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 626 kB/s | 1.3 MB 00:02 2024-04-26T21:15:20.697 INFO:teuthology.orchestra.run.smithi186.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 7.7 MB/s | 1.3 MB 00:00 2024-04-26T21:15:20.783 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T21:15:20.806 INFO:teuthology.orchestra.run.smithi186.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-04-26T21:15:20.825 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:15:20.916 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:15:20.947 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T21:15:20.983 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T21:15:21.489 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T21:15:21.520 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:15:21.574 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:15:21.605 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T21:15:21.838 INFO:teuthology.orchestra.run.smithi001.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 764 kB/s | 2.4 MB 00:03 2024-04-26T21:15:21.847 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-04-26T21:15:21.847 INFO:teuthology.orchestra.run.smithi001.stdout:Total 11 MB/s | 222 MB 00:19 2024-04-26T21:15:23.997 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:15:24.026 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:15:24.598 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:15:24.639 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:15:24.647 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-04-26T21:15:24.755 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:15:24.755 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T21:15:24.755 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T21:15:24.755 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:24.790 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:15:24.903 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-04-26T21:15:24.903 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-04-26T21:15:25.639 INFO:teuthology.orchestra.run.smithi186.stdout:(152/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.2 MB/s | 16 MB 00:13 2024-04-26T21:15:25.646 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-26T21:15:25.646 INFO:teuthology.orchestra.run.smithi186.stdout:Total 5.2 MB/s | 222 MB 00:43 2024-04-26T21:15:26.710 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-26T21:15:26.855 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-26T21:15:26.855 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-26T21:15:27.833 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-04-26T21:15:27.833 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-04-26T21:15:28.635 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-26T21:15:28.635 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-26T21:15:34.310 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-04-26T21:15:34.404 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T21:15:34.610 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T21:15:34.612 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-26T21:15:34.701 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-26T21:15:34.730 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T21:15:34.738 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-26T21:15:34.792 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-26T21:15:34.802 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T21:15:34.826 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-26T21:15:34.839 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T21:15:34.851 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-26T21:15:34.894 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T21:15:34.983 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-26T21:15:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T21:15:35.336 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:15:35.413 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-26T21:15:35.458 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:15:35.495 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:15:35.519 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:15:35.595 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:15:35.624 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:15:35.701 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:15:35.729 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:15:35.776 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T21:15:35.829 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T21:15:35.856 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:15:35.857 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:15:35.889 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:15:35.961 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:15:35.983 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:15:36.046 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:15:36.068 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:15:36.091 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:15:36.178 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-26T21:15:36.244 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-26T21:15:36.268 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:15:36.284 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:15:36.317 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:15:36.421 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:15:36.424 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T21:15:36.452 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:15:36.530 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T21:15:36.548 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 14/156 2024-04-26T21:15:36.557 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T21:15:36.578 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:15:36.592 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T21:15:36.642 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-26T21:15:36.659 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T21:15:36.699 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T21:15:36.743 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T21:15:36.776 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T21:15:36.804 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T21:15:36.847 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:15:36.848 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-04-26T21:15:36.848 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-04-26T21:15:36.848 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-04-26T21:15:36.848 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-04-26T21:15:36.848 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-04-26T21:15:36.848 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-04-26T21:15:36.848 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-04-26T21:15:36.848 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:36.873 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T21:15:36.926 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T21:15:36.945 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:15:36.964 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T21:15:37.000 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T21:15:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-26T21:15:37.023 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:15:37.039 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:15:37.040 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:15:37.040 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T21:15:37.040 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:37.076 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T21:15:37.110 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T21:15:37.154 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T21:15:37.183 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T21:15:37.209 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-26T21:15:37.254 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T21:15:37.305 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T21:15:37.333 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-26T21:15:37.379 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-26T21:15:37.380 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T21:15:37.428 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-26T21:15:37.526 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T21:15:37.535 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-26T21:15:37.584 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T21:15:37.592 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-26T21:15:37.656 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-26T21:15:37.705 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-26T21:15:37.744 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-26T21:15:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-26T21:15:37.945 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-26T21:15:37.979 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-26T21:15:38.037 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-26T21:15:38.120 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T21:15:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-26T21:15:38.200 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 31/156 2024-04-26T21:15:38.203 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T21:15:38.232 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T21:15:38.256 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 32/156 2024-04-26T21:15:38.275 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T21:15:38.290 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 33/156 2024-04-26T21:15:38.313 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T21:15:38.355 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T21:15:38.418 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-26T21:15:38.478 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-26T21:15:38.500 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T21:15:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T21:15:38.589 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-26T21:15:38.593 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:15:38.612 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T21:15:38.620 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:15:38.620 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:15:38.620 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T21:15:38.620 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:15:38.620 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:15:38.621 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:38.686 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T21:15:38.716 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T21:15:38.753 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T21:15:38.799 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T21:15:38.801 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-26T21:15:38.835 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T21:15:38.852 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:15:38.871 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-26T21:15:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:15:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:15:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T21:15:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:15:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:15:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:38.878 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T21:15:38.921 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T21:15:39.057 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T21:15:39.101 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T21:15:39.131 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T21:15:39.172 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T21:15:39.267 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T21:15:39.314 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T21:15:39.344 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:15:39.354 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T21:15:39.368 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:15:39.368 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:15:39.368 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T21:15:39.369 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:15:39.369 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:15:39.369 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:39.379 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T21:15:39.404 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T21:15:39.475 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T21:15:39.480 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T21:15:39.533 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T21:15:39.597 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T21:15:39.602 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-26T21:15:39.615 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T21:15:39.653 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T21:15:39.681 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T21:15:39.704 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-26T21:15:39.715 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T21:15:39.737 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-26T21:15:39.754 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T21:15:39.782 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-26T21:15:39.785 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T21:15:39.794 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T21:15:39.815 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T21:15:39.822 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 43/156 2024-04-26T21:15:39.846 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T21:15:39.850 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T21:15:39.883 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T21:15:39.885 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-26T21:15:39.925 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T21:15:39.928 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T21:15:39.959 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T21:15:40.025 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:15:40.047 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T21:15:40.106 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T21:15:40.137 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T21:15:40.139 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-26T21:15:40.166 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:15:40.185 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T21:15:40.213 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T21:15:40.231 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-26T21:15:40.244 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T21:15:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T21:15:40.284 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T21:15:40.314 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-26T21:15:40.339 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T21:15:40.416 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-26T21:15:40.455 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-26T21:15:40.493 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-26T21:15:40.588 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-26T21:15:40.630 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-26T21:15:40.696 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-26T21:15:40.735 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:15:40.758 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-26T21:15:40.763 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:15:40.796 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T21:15:40.860 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T21:15:40.886 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:15:40.960 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-26T21:15:41.017 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-26T21:15:41.058 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-26T21:15:41.081 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T21:15:41.112 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-26T21:15:41.127 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T21:15:41.161 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T21:15:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-26T21:15:41.262 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T21:15:41.304 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-26T21:15:41.311 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T21:15:41.349 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-26T21:15:41.386 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-26T21:15:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T21:15:41.435 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-26T21:15:41.438 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T21:15:41.616 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T21:15:41.636 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-26T21:15:41.643 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T21:15:41.721 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T21:15:41.822 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-26T21:15:41.867 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-26T21:15:41.907 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-26T21:15:41.954 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-26T21:15:41.987 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:15:42.022 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:15:42.023 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-26T21:15:42.089 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-26T21:15:42.139 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-26T21:15:42.192 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-26T21:15:42.242 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-26T21:15:42.321 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-26T21:15:42.374 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T21:15:42.375 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-26T21:15:42.484 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T21:15:42.525 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-26T21:15:42.587 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-26T21:15:42.690 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-26T21:15:42.927 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-26T21:15:43.039 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:15:43.064 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:15:43.106 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-26T21:15:43.171 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:15:43.171 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-26T21:15:43.341 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T21:15:43.374 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:15:43.399 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:15:43.399 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:15:43.399 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T21:15:43.399 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:15:43.399 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:15:43.399 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:43.450 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:15:43.462 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:15:43.563 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T21:15:43.616 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T21:15:43.734 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T21:15:43.824 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T21:15:43.850 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T21:15:43.883 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T21:15:44.200 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-26T21:15:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T21:15:44.259 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-26T21:15:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:15:44.293 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-26T21:15:44.353 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:15:44.384 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T21:15:44.420 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T21:15:44.421 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-26T21:15:44.489 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-26T21:15:44.592 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T21:15:44.598 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-26T21:15:44.619 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:15:44.635 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-26T21:15:44.644 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:15:44.644 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:15:44.644 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T21:15:44.644 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:15:44.644 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:15:44.644 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:44.686 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:15:44.710 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:15:44.710 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:15:44.710 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T21:15:44.710 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:44.762 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:15:44.786 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:15:44.786 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:15:44.786 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T21:15:44.786 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:15:44.786 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:15:44.786 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:44.891 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-26T21:15:44.897 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T21:15:44.924 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-26T21:15:44.931 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:15:45.002 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:15:45.025 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T21:15:45.032 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-26T21:15:45.114 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:15:45.137 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:15:45.138 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:15:45.940 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-26T21:15:46.101 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-26T21:15:47.275 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:15:47.312 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:15:47.761 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-26T21:15:47.852 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:15:47.863 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-26T21:15:47.899 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:15:47.915 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-26T21:15:47.968 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-26T21:15:47.990 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:15:47.990 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T21:15:47.990 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T21:15:47.990 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:15:48.017 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:15:48.377 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-26T21:15:48.426 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:15:48.549 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 99/156 2024-04-26T21:15:48.616 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 100/156 2024-04-26T21:15:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-26T21:15:49.269 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-26T21:15:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:15:49.396 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 103/156 2024-04-26T21:15:49.422 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 104/156 2024-04-26T21:15:50.001 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T21:15:50.043 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T21:15:50.084 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T21:15:50.116 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T21:15:50.143 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T21:15:50.345 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T21:15:50.398 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T21:15:50.428 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T21:15:51.030 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:15:51.067 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:15:51.088 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:15:51.993 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T21:15:52.092 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T21:15:52.126 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T21:15:52.218 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T21:15:52.263 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T21:15:52.263 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:15:52.372 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:15:52.386 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:15:52.405 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:15:52.405 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:15:52.426 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:15:52.498 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:15:52.565 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:15:53.110 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 105/156 2024-04-26T21:15:53.171 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:15:53.348 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 106/156 2024-04-26T21:15:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-26T21:15:53.348 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-26T21:15:53.348 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:15:53.397 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:15:58.595 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:15:58.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T21:15:58.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T21:15:58.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T21:15:58.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T21:15:58.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T21:15:58.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T21:15:58.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T21:15:58.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T21:15:58.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T21:15:58.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T21:15:58.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T21:15:58.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T21:15:58.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T21:15:58.602 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:15:58.603 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T21:15:58.604 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T21:15:58.605 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T21:15:58.606 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T21:15:58.606 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T21:15:58.606 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T21:15:58.606 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T21:15:58.606 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:15:58.606 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T21:15:59.328 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:15:59.328 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-04-26T21:15:59.328 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-04-26T21:15:59.328 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-04-26T21:15:59.328 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-04-26T21:15:59.328 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-04-26T21:15:59.329 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-04-26T21:15:59.329 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-04-26T21:15:59.329 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:15:59.503 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:15:59.520 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:15:59.520 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:15:59.520 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T21:15:59.520 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T21:16:00.572 INFO:teuthology.orchestra.run.smithi038.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:00.573 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:00.574 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T21:16:00.575 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T21:16:00.576 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T21:16:00.577 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T21:16:00.578 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T21:16:00.579 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T21:16:00.580 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T21:16:00.581 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T21:16:00.582 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T21:16:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T21:16:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T21:16:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T21:16:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T21:16:00.583 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-04-26T21:16:00.583 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-04-26T21:16:00.953 DEBUG:teuthology.parallel:result is None 2024-04-26T21:16:01.006 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:16:01.031 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:16:01.031 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:01.031 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T21:16:01.031 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:16:01.031 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:16:01.031 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:01.257 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:16:01.279 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:16:01.279 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:01.279 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T21:16:01.279 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:16:01.279 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:16:01.279 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:01.724 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:16:01.746 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:16:01.747 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:01.747 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T21:16:01.747 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:16:01.747 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:16:01.747 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:01.854 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T21:16:01.911 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T21:16:01.967 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T21:16:02.154 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T21:16:02.199 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T21:16:02.278 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T21:16:02.375 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:16:02.516 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:16:02.557 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T21:16:02.589 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T21:16:02.617 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T21:16:03.077 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:16:03.107 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:16:03.139 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T21:16:03.201 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T21:16:03.230 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:16:04.273 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:16:04.315 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:16:05.271 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:16:05.298 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:16:05.401 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:16:05.569 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T21:16:05.600 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:16:05.624 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:16:05.624 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:05.624 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T21:16:05.624 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:16:05.624 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:16:05.624 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:05.668 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:16:05.678 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:16:05.778 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T21:16:05.824 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T21:16:06.744 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T21:16:06.770 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:16:06.792 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:16:06.793 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:06.793 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T21:16:06.793 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:16:06.793 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:16:06.793 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:06.846 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:16:06.868 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:16:06.868 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:06.868 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T21:16:06.868 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:06.913 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:16:06.936 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:16:06.936 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:06.936 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T21:16:06.936 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:16:06.936 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:16:06.936 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 107/156 2024-04-26T21:16:06.937 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-04-26T21:16:06.937 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-04-26T21:16:06.937 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-04-26T21:16:06.937 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-04-26T21:16:06.937 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-04-26T21:16:06.937 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-04-26T21:16:06.937 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-04-26T21:16:06.937 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:07.160 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:16:07.211 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 108/156 2024-04-26T21:16:07.212 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:07.212 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-26T21:16:07.212 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:07.239 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:16:07.261 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:16:07.261 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:07.261 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T21:16:07.261 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:16:07.261 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:16:07.262 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:08.893 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:16:08.976 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 109/156 2024-04-26T21:16:08.976 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:08.977 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-26T21:16:08.977 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:16:08.977 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-26T21:16:08.977 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:09.244 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:16:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 110/156 2024-04-26T21:16:09.322 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:09.322 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-26T21:16:09.322 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:16:09.322 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-26T21:16:09.322 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:09.836 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:16:09.910 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 111/156 2024-04-26T21:16:09.910 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:09.910 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-26T21:16:09.910 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:16:09.910 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-26T21:16:09.911 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:10.046 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-26T21:16:10.109 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-26T21:16:10.185 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-26T21:16:10.430 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-26T21:16:10.516 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-26T21:16:10.608 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-26T21:16:10.724 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:16:10.894 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:16:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-26T21:16:11.007 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-26T21:16:11.039 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-26T21:16:11.712 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:16:11.751 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 123/156 2024-04-26T21:16:11.812 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 124/156 2024-04-26T21:16:11.868 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T21:16:11.909 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T21:16:11.930 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 125/156 2024-04-26T21:16:11.951 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T21:16:11.969 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:16:11.982 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T21:16:12.018 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T21:16:12.209 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T21:16:12.255 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T21:16:12.285 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T21:16:12.829 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:16:12.866 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:16:12.885 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:16:13.335 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 126/156 2024-04-26T21:16:13.379 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:16:13.747 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T21:16:13.830 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T21:16:13.861 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T21:16:13.953 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T21:16:13.991 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T21:16:13.991 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:16:14.067 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:16:14.100 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:16:14.100 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:16:14.174 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:16:14.242 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:16:14.547 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 127/156 2024-04-26T21:16:14.579 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:16:14.732 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 128/156 2024-04-26T21:16:14.917 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 129/156 2024-04-26T21:16:14.964 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:16:15.044 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 130/156 2024-04-26T21:16:15.044 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:15.044 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-26T21:16:15.044 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:16:15.045 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-26T21:16:15.045 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:15.107 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:16:15.148 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 131/156 2024-04-26T21:16:15.297 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 132/156 2024-04-26T21:16:15.381 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 133/156 2024-04-26T21:16:16.469 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-26T21:16:16.618 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:16:16.695 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 135/156 2024-04-26T21:16:16.695 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:16.695 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-26T21:16:16.695 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:16:16.695 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-26T21:16:16.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:16.768 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:16:16.848 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3306.gbc612 136/156 2024-04-26T21:16:16.848 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:16.848 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-26T21:16:16.848 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:17.036 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:16:17.120 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 137/156 2024-04-26T21:16:17.121 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:17.121 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-26T21:16:17.121 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:16:17.121 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-26T21:16:17.121 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:17.536 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:16:17.611 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 138/156 2024-04-26T21:16:17.611 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-26T21:16:17.611 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-26T21:16:17.611 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:16:17.611 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-26T21:16:17.611 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:20.256 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:16:20.257 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T21:16:20.258 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T21:16:20.259 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T21:16:20.259 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T21:16:20.259 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T21:16:20.272 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T21:16:20.272 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T21:16:20.273 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T21:16:20.274 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T21:16:20.275 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T21:16:20.276 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T21:16:20.277 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T21:16:20.278 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T21:16:20.279 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T21:16:20.280 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T21:16:20.281 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T21:16:20.282 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:16:20.282 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T21:16:22.345 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:22.346 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T21:16:22.347 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T21:16:22.348 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T21:16:22.349 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T21:16:22.350 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T21:16:22.351 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T21:16:22.352 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T21:16:22.353 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T21:16:22.354 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T21:16:22.355 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T21:16:22.356 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T21:16:22.356 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T21:16:22.356 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:16:22.356 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-26T21:16:22.735 DEBUG:teuthology.parallel:result is None 2024-04-26T21:16:22.917 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 139/156 2024-04-26T21:16:22.961 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 140/156 2024-04-26T21:16:23.083 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 141/156 2024-04-26T21:16:23.132 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-26T21:16:23.180 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-26T21:16:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 144/156 2024-04-26T21:16:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-26T21:16:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 146/156 2024-04-26T21:16:24.562 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:16:24.606 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:16:24.662 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-26T21:16:26.120 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 148/156 2024-04-26T21:16:26.232 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-26T21:16:26.310 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-26T21:16:26.487 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-26T21:16:26.539 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-26T21:16:26.539 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:16:26.681 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-26T21:16:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:16:26.729 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:16:26.853 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-26T21:16:26.987 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:16:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:16:27.260 INFO:teuthology.orchestra.run.smithi001.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-26T21:16:27.260 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 1/156 2024-04-26T21:16:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2/156 2024-04-26T21:16:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 3/156 2024-04-26T21:16:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 4/156 2024-04-26T21:16:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3306.gbc612 5/156 2024-04-26T21:16:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 6/156 2024-04-26T21:16:33.177 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 7/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 8/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 9/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 10/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 11/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 12/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 13/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 14/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_ 15/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 16/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 17/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_6 18/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86 19/156 2024-04-26T21:16:33.178 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 20/156 2024-04-26T21:16:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el 21/156 2024-04-26T21:16:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9. 22/156 2024-04-26T21:16:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_6 23/156 2024-04-26T21:16:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 24/156 2024-04-26T21:16:33.179 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 25/156 2024-04-26T21:16:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 26/156 2024-04-26T21:16:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 27/156 2024-04-26T21:16:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 28/156 2024-04-26T21:16:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 29/156 2024-04-26T21:16:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb. 30/156 2024-04-26T21:16:33.192 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noa 31/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.n 32/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc6 33/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el 34/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 35/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.e 36/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 37/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 38/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 39/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-26T21:16:33.193 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-26T21:16:33.194 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-26T21:16:33.195 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-26T21:16:33.196 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-26T21:16:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-26T21:16:33.198 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-26T21:16:33.199 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-26T21:16:33.200 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 149/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 151/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-26T21:16:33.201 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-26T21:16:35.609 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-26T21:16:35.609 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:35.609 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-04-26T21:16:35.609 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: automake-1.16.2-8.el9.noarch 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: bison-3.7.4-5.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.610 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.611 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-shell-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top-2:19.0.0-3306.gbc612ffb.el9.noarch 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: flex-2.6.4-9.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-26T21:16:35.612 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-26T21:16:35.613 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: lua-5.4.4-4.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-26T21:16:35.614 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-13.14-1.el9.x86_64 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-26T21:16:35.615 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-26T21:16:35.616 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-26T21:16:35.617 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-26T21:16:35.618 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-26T21:16:35.619 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:19.0.0-3306.gbc612ffb.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: re2-1:20211101-3.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-26T21:16:35.620 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-26T21:16:35.621 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-26T21:16:35.621 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:16:35.621 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-04-26T21:16:36.284 DEBUG:teuthology.parallel:result is None 2024-04-26T21:16:36.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:16:36.532 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T21:16:36.595 INFO:teuthology.orchestra.run.smithi001.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T21:16:36.595 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T21:16:36.596 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T21:16:36.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:16:36.871 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T21:16:36.898 INFO:teuthology.orchestra.run.smithi038.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T21:16:36.899 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T21:16:36.899 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T21:16:36.901 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:16:37.045 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T21:16:37.071 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-3306.gbc612ffb.el9 2024-04-26T21:16:37.071 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3306.gbc612ffb.el9 2024-04-26T21:16:37.072 INFO:teuthology.task.install:The correct ceph version 19.0.0-3306.gbc612ffb is installed. 2024-04-26T21:16:37.073 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T21:16:37.073 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:16:37.073 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T21:16:37.165 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:16:37.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T21:16:37.198 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:16:37.198 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T21:16:37.229 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T21:16:37.229 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:16:37.229 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T21:16:37.297 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T21:16:37.404 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:16:37.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T21:16:37.434 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T21:16:37.504 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:16:37.504 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T21:16:37.535 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T21:16:37.603 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T21:16:37.603 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:16:37.603 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T21:16:37.674 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T21:16:37.785 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:16:37.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T21:16:37.811 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T21:16:37.876 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:16:37.876 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T21:16:37.900 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T21:16:37.964 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T21:16:37.964 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:16:37.964 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T21:16:38.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T21:16:38.148 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:16:38.148 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T21:16:38.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T21:16:38.237 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:16:38.238 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T21:16:38.261 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T21:16:38.325 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T21:16:38.421 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-26T21:16:38.421 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:16:38.422 INFO:tasks.cephadm:Cluster fsid is 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:16:38.422 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T21:16:38.423 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.1', 'mon.b': '172.21.15.38', 'mon.c': '172.21.15.186'} 2024-04-26T21:16:38.423 INFO:tasks.cephadm:First mon is mon.a on smithi001 2024-04-26T21:16:38.423 INFO:tasks.cephadm:First mgr is x 2024-04-26T21:16:38.423 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T21:16:38.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo hostname $(hostname -s) 2024-04-26T21:16:38.492 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-04-26T21:16:38.516 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-04-26T21:16:38.540 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T21:16:38.540 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:16:38.688 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bc612ffb9c1d5bc27536bc917a705262a36d4387', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78918/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3306-gbc612ffb', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3306.gbc612ffb'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-26 13:02:42.256491', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240426.111959', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-26T21:16:38.932 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240426.111959, sha1 bc612ffb9c1d5bc27536bc917a705262a36d4387 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:16:38.933 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm 2024-04-26T21:16:38.933 DEBUG:teuthology.orchestra.run.smithi001:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T21:16:39.419 INFO:teuthology.orchestra.run.smithi001.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 21:16 /home/ubuntu/cephtest/cephadm 2024-04-26T21:16:39.419 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T21:16:39.854 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 21:16 /home/ubuntu/cephtest/cephadm 2024-04-26T21:16:39.855 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240426.111959/bc612ffb9c1d5bc27536bc917a705262a36d4387/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T21:16:40.306 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 781514 Apr 26 21:16 /home/ubuntu/cephtest/cephadm 2024-04-26T21:16:40.306 DEBUG:teuthology.orchestra.run.smithi001:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T21:16:40.351 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T21:16:40.371 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T21:16:40.406 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 on all hosts... 2024-04-26T21:16:40.406 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T21:16:40.411 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T21:16:40.413 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 pull 2024-04-26T21:16:41.944 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T21:16:42.012 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T21:16:42.836 INFO:teuthology.orchestra.run.smithi001.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T21:17:19.538 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T21:17:19.538 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T21:17:19.538 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T21:17:19.538 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-04-26T21:17:19.538 INFO:teuthology.orchestra.run.smithi186.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T21:17:19.538 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-26T21:17:19.538 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T21:17:23.059 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:17:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T21:17:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T21:17:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: "repo_digests": [ 2024-04-26T21:17:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T21:17:23.059 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:17:23.059 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:17:33.252 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-26T21:17:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T21:17:33.252 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07", 2024-04-26T21:17:33.253 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-04-26T21:17:33.253 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2" 2024-04-26T21:17:33.253 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-04-26T21:17:33.253 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-26T21:17:33.278 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph 2024-04-26T21:17:33.376 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-04-26T21:17:33.404 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-04-26T21:17:33.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /etc/ceph 2024-04-26T21:17:33.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-04-26T21:17:33.555 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-04-26T21:17:33.583 INFO:tasks.cephadm:Writing seed config... 2024-04-26T21:17:33.585 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-26T21:17:33.585 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-26T21:17:33.585 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T21:17:33.585 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-26T21:17:33.586 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-26T21:17:33.586 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-26T21:17:33.586 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-26T21:17:33.586 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-26T21:17:33.586 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-26T21:17:33.586 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-26T21:17:33.586 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-26T21:17:33.587 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-26T21:17:33.587 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-26T21:17:33.587 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-26T21:17:33.587 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-26T21:17:33.587 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-26T21:17:33.587 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-26T21:17:33.587 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-04-26T21:17:33.587 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-26T21:17:33.588 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-26T21:17:33.588 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-26T21:17:33.588 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-26T21:17:33.588 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-26T21:17:33.588 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-26T21:17:33.588 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T21:17:33.588 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T21:17:33.589 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T21:17:33.589 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T21:17:33.589 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T21:17:33.589 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-26T21:17:33.589 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-26T21:17:33.589 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-26T21:17:33.589 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-26T21:17:33.590 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-26T21:17:33.590 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-26T21:17:33.590 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T21:17:33.590 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T21:17:33.590 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-26T21:17:33.590 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-26T21:17:33.590 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-26T21:17:33.591 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-26T21:17:33.591 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-26T21:17:33.591 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-26T21:17:33.591 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-26T21:17:33.591 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T21:17:33.591 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-26T21:17:33.593 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:17:33.593 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T21:17:33.654 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 44ef3d6e-0412-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-26T21:17:33.655 DEBUG:teuthology.orchestra.run.smithi001:mon.a> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T21:17:33.702 DEBUG:teuthology.orchestra.run.smithi001:mgr.x> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T21:17:33.749 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T21:17:33.749 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 -v bootstrap --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.1 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:17:34.033 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-04-26T21:17:34.033 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387', '-v', 'bootstrap', '--fsid', '44ef3d6e-0412-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.1', '--skip-admin-label'] 2024-04-26T21:17:34.033 INFO:teuthology.orchestra.run.smithi001.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T21:17:34.034 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying podman|docker is present... 2024-04-26T21:17:34.094 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T21:17:34.095 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying lvm2 is present... 2024-04-26T21:17:34.095 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying time synchronization is in place... 2024-04-26T21:17:34.111 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T21:17:34.111 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T21:17:34.128 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T21:17:34.128 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T21:17:34.149 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout enabled 2024-04-26T21:17:34.168 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout active 2024-04-26T21:17:34.169 INFO:teuthology.orchestra.run.smithi001.stdout:Unit chronyd.service is enabled and running 2024-04-26T21:17:34.169 INFO:teuthology.orchestra.run.smithi001.stdout:Repeating the final host check... 2024-04-26T21:17:34.239 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-26T21:17:34.239 INFO:teuthology.orchestra.run.smithi001.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-26T21:17:34.239 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl is present 2024-04-26T21:17:34.239 INFO:teuthology.orchestra.run.smithi001.stdout:lvcreate is present 2024-04-26T21:17:34.254 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T21:17:34.254 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T21:17:34.271 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T21:17:34.271 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T21:17:34.293 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout enabled 2024-04-26T21:17:34.312 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout active 2024-04-26T21:17:34.312 INFO:teuthology.orchestra.run.smithi001.stdout:Unit chronyd.service is enabled and running 2024-04-26T21:17:34.312 INFO:teuthology.orchestra.run.smithi001.stdout:Host looks OK 2024-04-26T21:17:34.312 INFO:teuthology.orchestra.run.smithi001.stdout:Cluster fsid: 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:17:34.312 INFO:teuthology.orchestra.run.smithi001.stdout:Acquiring lock 140012893822112 on /run/cephadm/44ef3d6e-0412-11ef-bc93-c7b262605968.lock 2024-04-26T21:17:34.312 INFO:teuthology.orchestra.run.smithi001.stdout:Lock 140012893822112 acquired on /run/cephadm/44ef3d6e-0412-11ef-bc93-c7b262605968.lock 2024-04-26T21:17:34.313 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying IP 172.21.15.1 port 3300 ... 2024-04-26T21:17:34.313 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying IP 172.21.15.1 port 6789 ... 2024-04-26T21:17:34.314 INFO:teuthology.orchestra.run.smithi001.stdout:Base mon IP(s) is [172.21.15.1:3300, 172.21.15.1:6789], mon addrv is [v2:172.21.15.1:3300,v1:172.21.15.1:6789] 2024-04-26T21:17:34.324 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.1 metric 100 2024-04-26T21:17:34.324 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.1 metric 100 2024-04-26T21:17:34.332 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T21:17:34.333 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-26T21:17:34.333 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-26T21:17:34.341 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T21:17:34.341 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T21:17:34.341 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T21:17:34.341 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-26T21:17:34.341 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-04-26T21:17:34.341 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T21:17:34.342 INFO:teuthology.orchestra.run.smithi001.stdout:Mon IP `172.21.15.1` is in CIDR network `172.21.0.0/20` 2024-04-26T21:17:34.343 INFO:teuthology.orchestra.run.smithi001.stdout:Mon IP `172.21.15.1` is in CIDR network `172.21.0.0/20` 2024-04-26T21:17:34.343 INFO:teuthology.orchestra.run.smithi001.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T21:17:34.343 INFO:teuthology.orchestra.run.smithi001.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T21:17:34.345 INFO:teuthology.orchestra.run.smithi001.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T21:17:35.031 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T21:17:35.031 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387... 2024-04-26T21:17:35.031 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Getting image source signatures 2024-04-26T21:17:35.031 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying blob sha256:f340d36a00bce3feed0ad509aeafc9092c1c625bf553b8a613c380a79149b87e 2024-04-26T21:17:35.031 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-26T21:17:35.031 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying config sha256:55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 2024-04-26T21:17:35.031 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-26T21:17:36.925 INFO:teuthology.orchestra.run.smithi001.stdout:ceph: stdout ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T21:17:36.925 INFO:teuthology.orchestra.run.smithi001.stdout:Ceph version: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev) 2024-04-26T21:17:36.925 INFO:teuthology.orchestra.run.smithi001.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T21:17:38.409 INFO:teuthology.orchestra.run.smithi001.stdout:stat: stdout 167 167 2024-04-26T21:17:38.410 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial keys... 2024-04-26T21:17:39.544 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQDyGSxmvlbzMhAAiwA1eVy/uaWHWl4fQs28aw== 2024-04-26T21:17:40.641 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQD0GSxmoBUvABAAVHq4ntybyb6eftVInZ1yEA== 2024-04-26T21:17:41.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQD1GSxmMJDKCBAAc39iGOnsRNZiOq8gksLBNA== 2024-04-26T21:17:41.804 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial monmap... 2024-04-26T21:17:42.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T21:17:42.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T21:17:42.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:17:42.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T21:17:42.868 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool for a [v2:172.21.15.1:3300,v1:172.21.15.1:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T21:17:42.868 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-04-26T21:17:42.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: set fsid to 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:17:42.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T21:17:42.869 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:17:42.869 INFO:teuthology.orchestra.run.smithi001.stdout:Creating mon... 2024-04-26T21:17:44.011 INFO:teuthology.orchestra.run.smithi001.stdout:create mon.a on 2024-04-26T21:17:44.423 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-26T21:17:44.671 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-26T21:17:44.969 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-44ef3d6e-0412-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-44ef3d6e-0412-11ef-bc93-c7b262605968.target. 2024-04-26T21:17:44.969 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-44ef3d6e-0412-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-44ef3d6e-0412-11ef-bc93-c7b262605968.target. 2024-04-26T21:17:45.382 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.a 2024-04-26T21:17:45.382 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to reset failed state of unit ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T21:17:45.695 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-44ef3d6e-0412-11ef-bc93-c7b262605968.target.wants/ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@.service. 2024-04-26T21:17:46.476 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:46 smithi001 podman[27668]: 2024-04-26 21:17:46.303712288 +0000 UTC m=+0.337351581 container create 4a152655b8269cd838536639525d1e087bc6577e305f3047f7d0a00441033d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-04-26T21:17:46.720 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T21:17:46.720 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-04-26T21:17:46.735 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T21:17:46.735 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T21:17:46.735 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-04-26T21:17:46.735 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T21:17:46.735 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mon to start... 2024-04-26T21:17:46.735 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mon... 2024-04-26T21:17:46.884 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:46 smithi001 podman[27668]: 2024-04-26 21:17:46.605999864 +0000 UTC m=+0.639639119 container init 4a152655b8269cd838536639525d1e087bc6577e305f3047f7d0a00441033d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T21:17:46.884 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:46 smithi001 podman[27668]: 2024-04-26 21:17:46.617708772 +0000 UTC m=+0.651348071 container start 4a152655b8269cd838536639525d1e087bc6577e305f3047f7d0a00441033d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-26T21:17:46.884 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:46 smithi001 bash[27668]: 4a152655b8269cd838536639525d1e087bc6577e305f3047f7d0a00441033d69 2024-04-26T21:17:46.884 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:46 smithi001 systemd[1]: Started Ceph mon.a for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:17:46.884 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:46 smithi001 ceph-mon[27686]: mkfs 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:17:46.885 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:46 smithi001 ceph-mon[27686]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T21:17:47.964 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T21:17:47.964 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: monmap epoch 1 2024-04-26T21:17:47.965 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:17:47.965 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: last_changed 2024-04-26T21:17:42.249569+0000 2024-04-26T21:17:47.965 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: created 2024-04-26T21:17:42.249569+0000 2024-04-26T21:17:47.965 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: min_mon_release 19 (squid) 2024-04-26T21:17:47.965 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: election_strategy: 1 2024-04-26T21:17:47.965 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T21:17:47.965 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: fsmap 2024-04-26T21:17:47.965 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T21:17:47.965 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: mgrmap e1: no daemons active 2024-04-26T21:17:47.965 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:47 smithi001 ceph-mon[27686]: from='client.? 172.21.15.1:0/1004637484' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T21:17:48.239 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T21:17:48.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout id: 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:17:48.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T21:17:48.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:17:48.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout services: 2024-04-26T21:17:48.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.732475s) 2024-04-26T21:17:48.240 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T21:17:48.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T21:17:48.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:17:48.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout data: 2024-04-26T21:17:48.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T21:17:48.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T21:17:48.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T21:17:48.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T21:17:48.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:17:48.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.335+0000 7f65f3a00700 1 Processor -- start 2024-04-26T21:17:48.244 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.335+0000 7f65f3a00700 1 -- start start 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.335+0000 7f65f3a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec112060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.335+0000 7f65f3a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec112630 con 0x7f65ec10fc70 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.336+0000 7f65f1000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec112060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.337+0000 7f65f1000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec112060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46996/0 (socket says 172.21.15.1:46996) 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.337+0000 7f65f1000700 1 -- 172.21.15.1:0/1436772055 learned_addr learned my addr 172.21.15.1:0/1436772055 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.337+0000 7f65f1000700 1 -- 172.21.15.1:0/1436772055 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ec1127b0 con 0x7f65ec10fc70 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.338+0000 7f65f1000700 1 --2- 172.21.15.1:0/1436772055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec112060 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f65e00098a0 tx=0x7f65e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965b6f1c404c199 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.338+0000 7f65eb400700 1 -- 172.21.15.1:0/1436772055 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65e0004160 con 0x7f65ec10fc70 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.339+0000 7f65eb400700 1 -- 172.21.15.1:0/1436772055 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65e0004300 con 0x7f65ec10fc70 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.339+0000 7f65eb400700 1 -- 172.21.15.1:0/1436772055 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65e00036a0 con 0x7f65ec10fc70 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.339+0000 7f65f3a00700 1 -- 172.21.15.1:0/1436772055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 msgr2=0x7f65ec112060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.339+0000 7f65f3a00700 1 --2- 172.21.15.1:0/1436772055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec112060 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f65e00098a0 tx=0x7f65e0009bb0 comp rx=0 tx=0).stop 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.340+0000 7f65f3a00700 1 -- 172.21.15.1:0/1436772055 shutdown_connections 2024-04-26T21:17:48.245 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.340+0000 7f65f3a00700 1 --2- 172.21.15.1:0/1436772055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec112060 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:17:48.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.340+0000 7f65f3a00700 1 -- 172.21.15.1:0/1436772055 >> 172.21.15.1:0/1436772055 conn(0x7f65ec07e460 msgr2=0x7f65ec080880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:17:48.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.340+0000 7f65f3a00700 1 -- 172.21.15.1:0/1436772055 shutdown_connections 2024-04-26T21:17:48.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.340+0000 7f65f3a00700 1 -- 172.21.15.1:0/1436772055 wait complete. 2024-04-26T21:17:48.246 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.341+0000 7f65f3a00700 1 Processor -- start 2024-04-26T21:17:48.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.342+0000 7f65f3a00700 1 -- start start 2024-04-26T21:17:48.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.342+0000 7f65f3a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec1a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:17:48.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.342+0000 7f65f3a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec112c80 con 0x7f65ec10fc70 2024-04-26T21:17:48.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.343+0000 7f65f1000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec1a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:17:48.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.343+0000 7f65f1000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec1a2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47010/0 (socket says 172.21.15.1:47010) 2024-04-26T21:17:48.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.343+0000 7f65f1000700 1 -- 172.21.15.1:0/1004637484 learned_addr learned my addr 172.21.15.1:0/1004637484 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:17:48.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.343+0000 7f65f1000700 1 -- 172.21.15.1:0/1004637484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ec1a2f90 con 0x7f65ec10fc70 2024-04-26T21:17:48.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.343+0000 7f65f1000700 1 --2- 172.21.15.1:0/1004637484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec1a2a50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009fb0 tx=0x7f65e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:17:48.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.344+0000 7f65e9600700 1 -- 172.21.15.1:0/1004637484 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65e0003de0 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.344+0000 7f65e9600700 1 -- 172.21.15.1:0/1004637484 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65e0011630 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.344+0000 7f65e9600700 1 -- 172.21.15.1:0/1004637484 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f65e0011930 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.344+0000 7f65f3a00700 1 -- 172.21.15.1:0/1004637484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ec1a3280 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.344+0000 7f65f3a00700 1 -- 172.21.15.1:0/1004637484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ec1a5f10 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.344+0000 7f65e9600700 1 -- 172.21.15.1:0/1004637484 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f65e0011ad0 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.345+0000 7f65e9600700 1 -- 172.21.15.1:0/1004637484 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f65e0004260 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.345+0000 7f65f3a00700 1 -- 172.21.15.1:0/1004637484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ec064a80 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.349+0000 7f65e9600700 1 -- 172.21.15.1:0/1004637484 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f65e0033020 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.430+0000 7f65f3a00700 1 -- 172.21.15.1:0/1004637484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f65ec02e730 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.431+0000 7f65e9600700 1 -- 172.21.15.1:0/1004637484 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f65e0011dc0 con 0x7f65ec10fc70 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.432+0000 7f65f3a00700 1 -- 172.21.15.1:0/1004637484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 msgr2=0x7f65ec1a2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.432+0000 7f65f3a00700 1 --2- 172.21.15.1:0/1004637484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec1a2a50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009fb0 tx=0x7f65e0004750 comp rx=0 tx=0).stop 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.432+0000 7f65f3a00700 1 -- 172.21.15.1:0/1004637484 shutdown_connections 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.432+0000 7f65f3a00700 1 --2- 172.21.15.1:0/1004637484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65ec10fc70 0x7f65ec1a2a50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:17:48.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.432+0000 7f65f3a00700 1 -- 172.21.15.1:0/1004637484 >> 172.21.15.1:0/1004637484 conn(0x7f65ec07e460 msgr2=0x7f65ec07f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:17:48.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.432+0000 7f65f3a00700 1 -- 172.21.15.1:0/1004637484 shutdown_connections 2024-04-26T21:17:48.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:47.432+0000 7f65f3a00700 1 -- 172.21.15.1:0/1004637484 wait complete. 2024-04-26T21:17:48.251 INFO:teuthology.orchestra.run.smithi001.stdout:mon is available 2024-04-26T21:17:48.251 INFO:teuthology.orchestra.run.smithi001.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T21:17:49.305 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:49 smithi001 ceph-mon[27686]: from='client.? 172.21.15.1:0/3809640038' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T21:17:49.305 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:49 smithi001 ceph-mon[27686]: from='client.? 172.21.15.1:0/3809640038' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout fsid = 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.1:3300,v1:172.21.15.1:6789] 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T21:17:49.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.869+0000 7f57e5c00700 1 Processor -- start 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.870+0000 7f57e5c00700 1 -- start start 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.870+0000 7f57e5c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e0112060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.870+0000 7f57e5c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f57e0112630 con 0x7f57e010fc70 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.870+0000 7f57dea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e0112060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.871+0000 7f57dea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e0112060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49630/0 (socket says 172.21.15.1:49630) 2024-04-26T21:17:49.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.871+0000 7f57dea00700 1 -- 172.21.15.1:0/3773876969 learned_addr learned my addr 172.21.15.1:0/3773876969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.871+0000 7f57dea00700 1 -- 172.21.15.1:0/3773876969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e01127b0 con 0x7f57e010fc70 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.871+0000 7f57dea00700 1 --2- 172.21.15.1:0/3773876969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e0112060 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009e80 tx=0x7f57d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352d12426f600891 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.872+0000 7f57de000700 1 -- 172.21.15.1:0/3773876969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57d0017070 con 0x7f57e010fc70 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.872+0000 7f57de000700 1 -- 172.21.15.1:0/3773876969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57d0003a20 con 0x7f57e010fc70 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.872+0000 7f57de000700 1 -- 172.21.15.1:0/3773876969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57d0003d20 con 0x7f57e010fc70 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.873+0000 7f57e5c00700 1 -- 172.21.15.1:0/3773876969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 msgr2=0x7f57e0112060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.873+0000 7f57e5c00700 1 --2- 172.21.15.1:0/3773876969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e0112060 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009e80 tx=0x7f57d000b330 comp rx=0 tx=0).stop 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.873+0000 7f57e5c00700 1 -- 172.21.15.1:0/3773876969 shutdown_connections 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.873+0000 7f57e5c00700 1 --2- 172.21.15.1:0/3773876969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e0112060 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.873+0000 7f57e5c00700 1 -- 172.21.15.1:0/3773876969 >> 172.21.15.1:0/3773876969 conn(0x7f57e007e460 msgr2=0x7f57e0080880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.873+0000 7f57e5c00700 1 -- 172.21.15.1:0/3773876969 shutdown_connections 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.873+0000 7f57e5c00700 1 -- 172.21.15.1:0/3773876969 wait complete. 2024-04-26T21:17:49.583 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.874+0000 7f57e5c00700 1 Processor -- start 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.874+0000 7f57e5c00700 1 -- start start 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.875+0000 7f57e5c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e01a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.875+0000 7f57e5c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f57e0112c80 con 0x7f57e010fc70 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.875+0000 7f57dea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e01a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.875+0000 7f57dea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e01a2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49634/0 (socket says 172.21.15.1:49634) 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.875+0000 7f57dea00700 1 -- 172.21.15.1:0/3809640038 learned_addr learned my addr 172.21.15.1:0/3809640038 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.875+0000 7f57dea00700 1 -- 172.21.15.1:0/3809640038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e01a3010 con 0x7f57e010fc70 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.876+0000 7f57dea00700 1 --2- 172.21.15.1:0/3809640038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e01a2ad0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f57d0000f80 tx=0x7f57d0004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.876+0000 7f57d7e00700 1 -- 172.21.15.1:0/3809640038 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57d0022020 con 0x7f57e010fc70 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.876+0000 7f57d7e00700 1 -- 172.21.15.1:0/3809640038 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57d0018040 con 0x7f57e010fc70 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.876+0000 7f57d7e00700 1 -- 172.21.15.1:0/3809640038 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f57d0017070 con 0x7f57e010fc70 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.876+0000 7f57e5c00700 1 -- 172.21.15.1:0/3809640038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e01a32a0 con 0x7f57e010fc70 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.876+0000 7f57e5c00700 1 -- 172.21.15.1:0/3809640038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e01a5f90 con 0x7f57e010fc70 2024-04-26T21:17:49.584 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.877+0000 7f57d7e00700 1 -- 172.21.15.1:0/3809640038 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f57d0025020 con 0x7f57e010fc70 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.877+0000 7f57d7e00700 1 -- 172.21.15.1:0/3809640038 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f57d000f050 con 0x7f57e010fc70 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.878+0000 7f57d4c00700 1 -- 172.21.15.1:0/3809640038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b8005330 con 0x7f57e010fc70 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.881+0000 7f57d7e00700 1 -- 172.21.15.1:0/3809640038 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f57d002c030 con 0x7f57e010fc70 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.959+0000 7f57d4c00700 1 -- 172.21.15.1:0/3809640038 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f57b8002400 con 0x7f57e010fc70 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.968+0000 7f57d7e00700 1 -- 172.21.15.1:0/3809640038 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+463 (secure 0 0 0) 0x7f57d0037480 con 0x7f57e010fc70 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.970+0000 7f57d7e00700 1 -- 172.21.15.1:0/3809640038 <== mon.0 v2:172.21.15.1:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f57d0035070 con 0x7f57e010fc70 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.971+0000 7f57d4c00700 1 -- 172.21.15.1:0/3809640038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 msgr2=0x7f57e01a2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.971+0000 7f57d4c00700 1 --2- 172.21.15.1:0/3809640038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e01a2ad0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f57d0000f80 tx=0x7f57d0004ad0 comp rx=0 tx=0).stop 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.972+0000 7f57d4c00700 1 -- 172.21.15.1:0/3809640038 shutdown_connections 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.972+0000 7f57d4c00700 1 --2- 172.21.15.1:0/3809640038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f57e010fc70 0x7f57e01a2ad0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.972+0000 7f57d4c00700 1 -- 172.21.15.1:0/3809640038 >> 172.21.15.1:0/3809640038 conn(0x7f57e007e460 msgr2=0x7f57e0080720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:17:49.585 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.972+0000 7f57d4c00700 1 -- 172.21.15.1:0/3809640038 shutdown_connections 2024-04-26T21:17:49.586 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:48.972+0000 7f57d4c00700 1 -- 172.21.15.1:0/3809640038 wait complete. 2024-04-26T21:17:49.586 INFO:teuthology.orchestra.run.smithi001.stdout:Generating new minimal ceph.conf... 2024-04-26T21:17:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:50 smithi001 ceph-mon[27686]: from='client.? 172.21.15.1:0/3259116064' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:17:50.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.179+0000 7effac400700 1 Processor -- start 2024-04-26T21:17:50.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.179+0000 7effac400700 1 -- start start 2024-04-26T21:17:50.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.179+0000 7effac400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa4112060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:17:50.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.179+0000 7effac400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7effa4112630 con 0x7effa410fc70 2024-04-26T21:17:50.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.180+0000 7effa9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa4112060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:17:50.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.180+0000 7effa9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa4112060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49638/0 (socket says 172.21.15.1:49638) 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.180+0000 7effa9a00700 1 -- 172.21.15.1:0/3356225244 learned_addr learned my addr 172.21.15.1:0/3356225244 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.181+0000 7effa9a00700 1 -- 172.21.15.1:0/3356225244 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa41127b0 con 0x7effa410fc70 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.181+0000 7effa9a00700 1 --2- 172.21.15.1:0/3356225244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa4112060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7eff98009e80 tx=0x7eff9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e406521325fce0e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.182+0000 7effa3e00700 1 -- 172.21.15.1:0/3356225244 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff98017070 con 0x7effa410fc70 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.182+0000 7effa3e00700 1 -- 172.21.15.1:0/3356225244 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7eff9800b8c0 con 0x7effa410fc70 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.182+0000 7effa3e00700 1 -- 172.21.15.1:0/3356225244 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff98004490 con 0x7effa410fc70 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.182+0000 7effac400700 1 -- 172.21.15.1:0/3356225244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 msgr2=0x7effa4112060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.182+0000 7effac400700 1 --2- 172.21.15.1:0/3356225244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa4112060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7eff98009e80 tx=0x7eff9800b330 comp rx=0 tx=0).stop 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.183+0000 7effac400700 1 -- 172.21.15.1:0/3356225244 shutdown_connections 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.183+0000 7effac400700 1 --2- 172.21.15.1:0/3356225244 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa4112060 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.183+0000 7effac400700 1 -- 172.21.15.1:0/3356225244 >> 172.21.15.1:0/3356225244 conn(0x7effa407e460 msgr2=0x7effa4080880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.183+0000 7effac400700 1 -- 172.21.15.1:0/3356225244 shutdown_connections 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.183+0000 7effac400700 1 -- 172.21.15.1:0/3356225244 wait complete. 2024-04-26T21:17:50.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.184+0000 7effac400700 1 Processor -- start 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.184+0000 7effac400700 1 -- start start 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.184+0000 7effac400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa41a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.184+0000 7effac400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7effa4113760 con 0x7effa410fc70 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.184+0000 7effa9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa41a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.184+0000 7effa9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa41a3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49642/0 (socket says 172.21.15.1:49642) 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.184+0000 7effa9a00700 1 -- 172.21.15.1:0/3259116064 learned_addr learned my addr 172.21.15.1:0/3259116064 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.185+0000 7effa9a00700 1 -- 172.21.15.1:0/3259116064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa41a3790 con 0x7effa410fc70 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.185+0000 7effa9a00700 1 --2- 172.21.15.1:0/3259116064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa41a3250 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7eff9800b7b0 tx=0x7eff9800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.185+0000 7effa2000700 1 -- 172.21.15.1:0/3259116064 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff98020020 con 0x7effa410fc70 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.185+0000 7effa2000700 1 -- 172.21.15.1:0/3259116064 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7eff98003d10 con 0x7effa410fc70 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.185+0000 7effa2000700 1 -- 172.21.15.1:0/3259116064 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff98017040 con 0x7effa410fc70 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.185+0000 7effac400700 1 -- 172.21.15.1:0/3259116064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effa41a3a20 con 0x7effa410fc70 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.186+0000 7effac400700 1 -- 172.21.15.1:0/3259116064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effa41a6710 con 0x7effa410fc70 2024-04-26T21:17:50.869 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.186+0000 7effa2000700 1 -- 172.21.15.1:0/3259116064 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7eff98023020 con 0x7effa410fc70 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.186+0000 7effa2000700 1 -- 172.21.15.1:0/3259116064 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7eff9800f050 con 0x7effa410fc70 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.187+0000 7effac400700 1 -- 172.21.15.1:0/3259116064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effa4064a80 con 0x7effa410fc70 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.190+0000 7effa2000700 1 -- 172.21.15.1:0/3259116064 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7eff9802a020 con 0x7effa410fc70 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.268+0000 7effac400700 1 -- 172.21.15.1:0/3259116064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7effa41a6bb0 con 0x7effa410fc70 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.269+0000 7effa2000700 1 -- 172.21.15.1:0/3259116064 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+173 (secure 0 0 0) 0x7eff980128c0 con 0x7effa410fc70 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.271+0000 7effac400700 1 -- 172.21.15.1:0/3259116064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 msgr2=0x7effa41a3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.271+0000 7effac400700 1 --2- 172.21.15.1:0/3259116064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa41a3250 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7eff9800b7b0 tx=0x7eff9800be90 comp rx=0 tx=0).stop 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.272+0000 7effac400700 1 -- 172.21.15.1:0/3259116064 shutdown_connections 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.272+0000 7effac400700 1 --2- 172.21.15.1:0/3259116064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effa410fc70 0x7effa41a3250 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.272+0000 7effac400700 1 -- 172.21.15.1:0/3259116064 >> 172.21.15.1:0/3259116064 conn(0x7effa407e460 msgr2=0x7effa407ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.272+0000 7effac400700 1 -- 172.21.15.1:0/3259116064 shutdown_connections 2024-04-26T21:17:50.870 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:50.272+0000 7effac400700 1 -- 172.21.15.1:0/3259116064 wait complete. 2024-04-26T21:17:50.871 INFO:teuthology.orchestra.run.smithi001.stdout:Restarting the monitor... 2024-04-26T21:17:51.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:50 smithi001 systemd[1]: Stopping Ceph mon.a for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:17:51.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a[27682]: 2024-04-26T21:17:51.369+0000 7f6d10600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:17:51.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a[27682]: 2024-04-26T21:17:51.369+0000 7f6d10600700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T21:17:51.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:51 smithi001 podman[27928]: 2024-04-26 21:17:51.5244253 +0000 UTC m=+0.291740554 container died 4a152655b8269cd838536639525d1e087bc6577e305f3047f7d0a00441033d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-26T21:17:52.225 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:51 smithi001 podman[27928]: 2024-04-26 21:17:51.987559214 +0000 UTC m=+0.754874471 container cleanup 4a152655b8269cd838536639525d1e087bc6577e305f3047f7d0a00441033d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:17:52.225 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:51 smithi001 bash[27928]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a 2024-04-26T21:17:52.500 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:52 smithi001 podman[27942]: 2024-04-26 21:17:52.224172314 +0000 UTC m=+0.680202136 container remove 4a152655b8269cd838536639525d1e087bc6577e305f3047f7d0a00441033d69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T21:17:53.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:52 smithi001 systemd[1]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T21:17:53.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:52 smithi001 systemd[1]: Stopped Ceph mon.a for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:17:53.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:52 smithi001 systemd[1]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.a.service: Consumed 2.066s CPU time. 2024-04-26T21:17:53.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:52 smithi001 systemd[1]: Starting Ceph mon.a for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:17:53.461 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 podman[28035]: 2024-04-26 21:17:53.088883232 +0000 UTC m=+0.037523246 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:17:53.461 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 podman[28035]: 2024-04-26 21:17:53.183214899 +0000 UTC m=+0.131854933 container create acb0745845a29cd3647612d262c5a4fcb6636ebbd587c642c0996a67e66de0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-26T21:17:53.461 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 podman[28035]: 2024-04-26 21:17:53.403145213 +0000 UTC m=+0.351785270 container init acb0745845a29cd3647612d262c5a4fcb6636ebbd587c642c0996a67e66de0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:17:53.461 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 podman[28035]: 2024-04-26 21:17:53.414141541 +0000 UTC m=+0.362781595 container start acb0745845a29cd3647612d262c5a4fcb6636ebbd587c642c0996a67e66de0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-04-26T21:17:53.499 INFO:teuthology.orchestra.run.smithi001.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: set uid:gid to 167:167 (ceph:ceph) 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev), process ceph-mon, pid 2 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: pidfile_write: ignore empty --pid-file 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: load: jerasure load: lrc 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: RocksDB version: 7.9.2 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Git sha 0 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Compile date 2024-04-26 11:39:23 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: DB SUMMARY 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: DB Session ID: PNXVYQP1DL2A3CMLQTGS 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: CURRENT file: CURRENT 2024-04-26T21:17:53.723 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: IDENTITY file: IDENTITY 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104504 ; 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.error_if_exists: 0 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.create_if_missing: 0 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.paranoid_checks: 1 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.env: 0x55b607741cc0 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.fs: PosixFileSystem 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.info_log: 0x55b61a565dc0 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_file_opening_threads: 16 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.statistics: (nil) 2024-04-26T21:17:53.724 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.use_fsync: 0 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_log_file_size: 0 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.keep_log_file_num: 1000 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.recycle_log_file_num: 0 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.allow_fallocate: 1 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.allow_mmap_reads: 0 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.allow_mmap_writes: 0 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.use_direct_reads: 0 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.create_missing_column_families: 0 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.db_log_dir: 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.wal_dir: 2024-04-26T21:17:53.725 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T21:17:53.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T21:17:53.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T21:17:53.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T21:17:53.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T21:17:53.729 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T21:17:53.729 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.advise_random_on_open: 1 2024-04-26T21:17:53.729 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.db_write_buffer_size: 0 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.write_buffer_manager: 0x55b61a5370e0 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.rate_limiter: (nil) 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.wal_recovery_mode: 2 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.enable_thread_tracking: 0 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.enable_pipelined_write: 0 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.unordered_write: 0 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T21:17:53.730 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.row_cache: None 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.wal_filter: None 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.allow_ingest_behind: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.two_write_queues: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.manual_wal_flush: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.wal_compression: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.atomic_flush: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.log_readahead_size: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.best_efforts_recovery: 0 2024-04-26T21:17:53.731 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.allow_data_in_errors: 0 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.db_host_id: __hostname__ 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_background_jobs: 2 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_background_compactions: -1 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_subcompactions: 1 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_total_wal_size: 0 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T21:17:53.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_open_files: -1 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bytes_per_sync: 0 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_readahead_size: 0 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_background_flushes: -1 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Compression algorithms supported: 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: kZSTD supported: 0 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: kXpressCompression supported: 0 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: kLZ4HCCompression supported: 1 2024-04-26T21:17:53.733 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: kZlibCompression supported: 1 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: kSnappyCompression supported: 1 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: kLZ4Compression supported: 1 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: kBZip2Compression supported: 0 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.merge_operator: 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_filter: None 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_filter_factory: None 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.sst_partitioner_factory: None 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T21:17:53.734 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b61a565f80) 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: cache_index_and_filter_blocks: 1 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: pin_top_level_index_and_filter: 1 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: index_type: 0 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: data_block_index_type: 0 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: index_shortening: 1 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: checksum: 4 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: no_block_cache: 0 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache: 0x55b61a5691f0 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache_name: BinnedLRUCache 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache_options: 2024-04-26T21:17:53.735 INFO:journalctl@ceph.mon.a.smithi001.stdout: capacity : 536870912 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: num_shard_bits : 4 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: strict_capacity_limit : 0 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: high_pri_pool_ratio: 0.000 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_cache_compressed: (nil) 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: persistent_cache: (nil) 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_size: 4096 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_size_deviation: 10 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_restart_interval: 16 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: index_block_restart_interval: 1 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: metadata_block_size: 4096 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: partition_filters: 0 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: use_delta_encoding: 1 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: filter_policy: bloomfilter 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: whole_key_filtering: 1 2024-04-26T21:17:53.736 INFO:journalctl@ceph.mon.a.smithi001.stdout: verify_compression: 0 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout: read_amp_bytes_per_bit: 0 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout: format_version: 5 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout: enable_index_compression: 1 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout: block_align: 0 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout: max_auto_readahead_size: 262144 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout: prepopulate_block_cache: 0 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout: initial_auto_readahead_size: 8192 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout: num_file_reads_for_auto_readahead: 2 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.write_buffer_size: 33554432 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_write_buffer_number: 2 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compression: NoCompression 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bottommost_compression: Disabled 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.prefix_extractor: nullptr 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T21:17:53.737 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.num_levels: 7 2024-04-26T21:17:53.738 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T21:17:53.738 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T21:17:53.738 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T21:17:53.738 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T21:17:53.738 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T21:17:53.738 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T21:17:53.738 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T21:17:53.738 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T21:17:53.739 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T21:17:53.739 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T21:17:53.739 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T21:17:53.739 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T21:17:53.739 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T21:17:53.739 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compression_opts.level: 32767 2024-04-26T21:17:53.739 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compression_opts.strategy: 0 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compression_opts.enabled: false 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.target_file_size_base: 67108864 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T21:17:53.740 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.arena_block_size: 1048576 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.disable_auto_compactions: 0 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T21:17:53.741 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.inplace_update_support: 0 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T21:17:53.742 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.bloom_locality: 0 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.max_successive_merges: 0 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.paranoid_file_checks: 0 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.force_consistency_checks: 1 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.report_bg_io_stats: 0 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.ttl: 2592000 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.enable_blob_files: false 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.min_blob_size: 0 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.blob_file_size: 268435456 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T21:17:53.743 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.blob_file_starting_level: 0 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 267c8afb-0ad3-49b1-beba-9b904833a657 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714166273471686, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714166273473880, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99725, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 97607, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14030, "raw_average_key_size": 49, "raw_value_size": 89969, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714166273, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "267c8afb-0ad3-49b1-beba-9b904833a657", "db_session_id": "PNXVYQP1DL2A3CMLQTGS", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714166273474072, "job": 1, "event": "recovery_finished"} 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b61a686000 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: DB pointer 0x55b61a672000 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T21:17:53.744 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** DB Stats ** 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** Compaction Stats [default] ** 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: L0 2/0 99.25 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: Sum 2/0 99.25 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T21:17:53.745 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** Compaction Stats [default] ** 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 65.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: Cumulative compaction: 0.00 GB write, 13.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T21:17:53.746 INFO:journalctl@ceph.mon.a.smithi001.stdout: Interval compaction: 0.00 GB write, 13.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: Block cache BinnedLRUCache@0x55b61a5691f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: starting mon.a rank 0 at public addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] at bind addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: mon.a@-1(???) e1 preinit fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: mon.a@-1(???).mds e1 new map 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: mon.a@-1(???).mds e1 print_map 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: e1 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: legacy client fscid: -1 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout: No filesystems configured 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T21:17:53.747 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 bash[28035]: acb0745845a29cd3647612d262c5a4fcb6636ebbd587c642c0996a67e66de0e9 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 systemd[1]: Started Ceph mon.a for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: monmap epoch 1 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: last_changed 2024-04-26T21:17:42.249569+0000 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: created 2024-04-26T21:17:42.249569+0000 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: min_mon_release 19 (squid) 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: election_strategy: 1 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: fsmap 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: osdmap e1: 0 total, 0 up, 0 in 2024-04-26T21:17:53.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:53 smithi001 ceph-mon[28054]: mgrmap e1: no daemons active 2024-04-26T21:17:54.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.106+0000 7fb5f1200700 1 Processor -- start 2024-04-26T21:17:54.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.106+0000 7fb5f1200700 1 -- start start 2024-04-26T21:17:54.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.107+0000 7fb5f1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec1092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:17:54.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.107+0000 7fb5f1200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec1097f0 con 0x7fb5ec108e90 2024-04-26T21:17:54.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.107+0000 7fb5e9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec1092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:17:54.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.108+0000 7fb5e9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec1092b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49646/0 (socket says 172.21.15.1:49646) 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.108+0000 7fb5e9e00700 1 -- 172.21.15.1:0/1650609234 learned_addr learned my addr 172.21.15.1:0/1650609234 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.108+0000 7fb5e9e00700 1 -- 172.21.15.1:0/1650609234 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ec109ff0 con 0x7fb5ec108e90 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.108+0000 7fb5e9e00700 1 --2- 172.21.15.1:0/1650609234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec1092b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0098a0 tx=0x7fb5dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5345e38355a16a87 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.109+0000 7fb5e8a00700 1 -- 172.21.15.1:0/1650609234 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5dc004160 con 0x7fb5ec108e90 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.109+0000 7fb5e8a00700 1 -- 172.21.15.1:0/1650609234 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb5dc004300 con 0x7fb5ec108e90 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.109+0000 7fb5e8a00700 1 -- 172.21.15.1:0/1650609234 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5dc011af0 con 0x7fb5ec108e90 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.110+0000 7fb5f1200700 1 -- 172.21.15.1:0/1650609234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 msgr2=0x7fb5ec1092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.110+0000 7fb5f1200700 1 --2- 172.21.15.1:0/1650609234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec1092b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0098a0 tx=0x7fb5dc009bb0 comp rx=0 tx=0).stop 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.110+0000 7fb5f1200700 1 -- 172.21.15.1:0/1650609234 shutdown_connections 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.110+0000 7fb5f1200700 1 --2- 172.21.15.1:0/1650609234 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec1092b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.110+0000 7fb5f1200700 1 -- 172.21.15.1:0/1650609234 >> 172.21.15.1:0/1650609234 conn(0x7fb5ec104450 msgr2=0x7fb5ec106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.110+0000 7fb5f1200700 1 -- 172.21.15.1:0/1650609234 shutdown_connections 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.110+0000 7fb5f1200700 1 -- 172.21.15.1:0/1650609234 wait complete. 2024-04-26T21:17:54.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.111+0000 7fb5f1200700 1 Processor -- start 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.111+0000 7fb5f1200700 1 -- start start 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.112+0000 7fb5f1200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.112+0000 7fb5f1200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec10d430 con 0x7fb5ec108e90 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.112+0000 7fb5e9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.112+0000 7fb5e9e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec19f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49662/0 (socket says 172.21.15.1:49662) 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.112+0000 7fb5e9e00700 1 -- 172.21.15.1:0/3737882478 learned_addr learned my addr 172.21.15.1:0/3737882478 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.112+0000 7fb5e9e00700 1 -- 172.21.15.1:0/3737882478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ec19f560 con 0x7fb5ec108e90 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.113+0000 7fb5e9e00700 1 --2- 172.21.15.1:0/3737882478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec19f020 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc011ac0 tx=0x7fb5dc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.113+0000 7fb5e2a00700 1 -- 172.21.15.1:0/3737882478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5dc003970 con 0x7fb5ec108e90 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.113+0000 7fb5e2a00700 1 -- 172.21.15.1:0/3737882478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fb5dc003b10 con 0x7fb5ec108e90 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.113+0000 7fb5f1200700 1 -- 172.21.15.1:0/3737882478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ec19f7f0 con 0x7fb5ec108e90 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.113+0000 7fb5e2a00700 1 -- 172.21.15.1:0/3737882478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5dc024610 con 0x7fb5ec108e90 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.113+0000 7fb5f1200700 1 -- 172.21.15.1:0/3737882478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ec1a2490 con 0x7fb5ec108e90 2024-04-26T21:17:54.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.114+0000 7fb5e2a00700 1 -- 172.21.15.1:0/3737882478 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb5dc024bd0 con 0x7fb5ec108e90 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.114+0000 7fb5e2a00700 1 -- 172.21.15.1:0/3737882478 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb5dc019960 con 0x7fb5ec108e90 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.115+0000 7fb5f1200700 1 -- 172.21.15.1:0/3737882478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5ec064a80 con 0x7fb5ec108e90 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.118+0000 7fb5e2a00700 1 -- 172.21.15.1:0/3737882478 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fb5dc01a9e0 con 0x7fb5ec108e90 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.199+0000 7fb5f1200700 1 -- 172.21.15.1:0/3737882478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fb5ec1a2660 con 0x7fb5ec108e90 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.203+0000 7fb5e2a00700 1 -- 172.21.15.1:0/3737882478 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fb5dc019b60 con 0x7fb5ec108e90 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.204+0000 7fb5f1200700 1 -- 172.21.15.1:0/3737882478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 msgr2=0x7fb5ec19f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.204+0000 7fb5f1200700 1 --2- 172.21.15.1:0/3737882478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec19f020 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc011ac0 tx=0x7fb5dc011ba0 comp rx=0 tx=0).stop 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.204+0000 7fb5f1200700 1 -- 172.21.15.1:0/3737882478 shutdown_connections 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.204+0000 7fb5f1200700 1 --2- 172.21.15.1:0/3737882478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5ec108e90 0x7fb5ec19f020 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.204+0000 7fb5f1200700 1 -- 172.21.15.1:0/3737882478 >> 172.21.15.1:0/3737882478 conn(0x7fb5ec104450 msgr2=0x7fb5ec104eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.204+0000 7fb5f1200700 1 -- 172.21.15.1:0/3737882478 shutdown_connections 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:54.205+0000 7fb5f1200700 1 -- 172.21.15.1:0/3737882478 wait complete. 2024-04-26T21:17:54.806 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T21:17:54.824 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:17:54.824 INFO:teuthology.orchestra.run.smithi001.stdout:Creating mgr... 2024-04-26T21:17:54.826 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T21:17:54.826 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T21:17:55.249 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.x 2024-04-26T21:17:55.249 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to reset failed state of unit ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-26T21:17:55.514 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-44ef3d6e-0412-11ef-bc93-c7b262605968.target.wants/ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@.service. 2024-04-26T21:17:55.541 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:55 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3737882478' entity='client.admin' 2024-04-26T21:17:55.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:55 smithi001 systemd[1]: Starting Ceph mgr.x for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:17:56.174 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:55 smithi001 podman[28276]: 2024-04-26 21:17:55.82639034 +0000 UTC m=+0.038228613 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:17:56.174 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:55 smithi001 podman[28276]: 2024-04-26 21:17:55.935180967 +0000 UTC m=+0.147019267 container create 7e9c0a8e9d3d3b56a801776c3e57b4686c3d283dc3d59d8eb07ff308578d49f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:17:56.175 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:56 smithi001 podman[28276]: 2024-04-26 21:17:56.172659374 +0000 UTC m=+0.384497724 container init 7e9c0a8e9d3d3b56a801776c3e57b4686c3d283dc3d59d8eb07ff308578d49f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:17:56.300 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T21:17:56.300 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-04-26T21:17:56.321 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T21:17:56.321 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T21:17:56.321 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-04-26T21:17:56.321 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T21:17:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T21:17:56.341 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-04-26T21:17:56.358 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T21:17:56.358 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T21:17:56.358 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-04-26T21:17:56.358 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T21:17:56.358 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr to start... 2024-04-26T21:17:56.358 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr... 2024-04-26T21:17:56.449 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:56 smithi001 podman[28276]: 2024-04-26 21:17:56.183797096 +0000 UTC m=+0.395635366 container start 7e9c0a8e9d3d3b56a801776c3e57b4686c3d283dc3d59d8eb07ff308578d49f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:17:56.449 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:56 smithi001 bash[28276]: 7e9c0a8e9d3d3b56a801776c3e57b4686c3d283dc3d59d8eb07ff308578d49f1 2024-04-26T21:17:56.450 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:56 smithi001 systemd[1]: Started Ceph mgr.x for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:17:56.706 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:56 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:17:56.447+0000 7f7a90be0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:17:56.706 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:56 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:17:56.705+0000 7f7a90be0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:17:57.035 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:56 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:17:56.843+0000 7f7a90be0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:17:57.317 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:57 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:17:57.266+0000 7f7a90be0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:17:57.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:17:57 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2873531950' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T21:17:57.702 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:57 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:17:57.381+0000 7f7a90be0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:17:57.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:17:57.712 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:17:57.712 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:17:57.712 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T21:17:57.712 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T21:17:57.712 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T21:17:57.712 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T21:17:57.712 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:17:57.712 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:17:57.713 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T21:17:57.714 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:17:57.714 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T21:17:57.714 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T21:17:57.714 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T21:17:57.722 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T21:17:57.723 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T21:17:46.701371+0000", 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.942+0000 7fd9d9200700 1 Processor -- start 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.942+0000 7fd9d9200700 1 -- start start 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.942+0000 7fd9d9200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d41081a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.942+0000 7fd9d9200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d4108770 con 0x7fd9d4107d80 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.943+0000 7fd9d1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d41081a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:17:57.724 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.943+0000 7fd9d1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d41081a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49686/0 (socket says 172.21.15.1:49686) 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.943+0000 7fd9d1e00700 1 -- 172.21.15.1:0/3891720916 learned_addr learned my addr 172.21.15.1:0/3891720916 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.944+0000 7fd9d1e00700 1 -- 172.21.15.1:0/3891720916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d4108f80 con 0x7fd9d4107d80 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.944+0000 7fd9d1e00700 1 --2- 172.21.15.1:0/3891720916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d41081a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40098a0 tx=0x7fd9c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2dc5c0c96d0597b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.944+0000 7fd9d0a00700 1 -- 172.21.15.1:0/3891720916 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9c4004160 con 0x7fd9d4107d80 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.944+0000 7fd9d0a00700 1 -- 172.21.15.1:0/3891720916 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd9c4004300 con 0x7fd9d4107d80 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.944+0000 7fd9d0a00700 1 -- 172.21.15.1:0/3891720916 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9c4011af0 con 0x7fd9d4107d80 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.945+0000 7fd9d9200700 1 -- 172.21.15.1:0/3891720916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 msgr2=0x7fd9d41081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.945+0000 7fd9d9200700 1 --2- 172.21.15.1:0/3891720916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d41081a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40098a0 tx=0x7fd9c4009bb0 comp rx=0 tx=0).stop 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.945+0000 7fd9d9200700 1 -- 172.21.15.1:0/3891720916 shutdown_connections 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.945+0000 7fd9d9200700 1 --2- 172.21.15.1:0/3891720916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d41081a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.945+0000 7fd9d9200700 1 -- 172.21.15.1:0/3891720916 >> 172.21.15.1:0/3891720916 conn(0x7fd9d4103340 msgr2=0x7fd9d4105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.945+0000 7fd9d9200700 1 -- 172.21.15.1:0/3891720916 shutdown_connections 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.945+0000 7fd9d9200700 1 -- 172.21.15.1:0/3891720916 wait complete. 2024-04-26T21:17:57.725 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.947+0000 7fd9d9200700 1 Processor -- start 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.947+0000 7fd9d9200700 1 -- start start 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.947+0000 7fd9d9200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.947+0000 7fd9d9200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d410c330 con 0x7fd9d4107d80 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.947+0000 7fd9d1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.948+0000 7fd9d1e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49692/0 (socket says 172.21.15.1:49692) 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.948+0000 7fd9d1e00700 1 -- 172.21.15.1:0/2873531950 learned_addr learned my addr 172.21.15.1:0/2873531950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.948+0000 7fd9d1e00700 1 -- 172.21.15.1:0/2873531950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d419f530 con 0x7fd9d4107d80 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.948+0000 7fd9d1e00700 1 --2- 172.21.15.1:0/2873531950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d419eff0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4000c00 tx=0x7fd9c4011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.948+0000 7fd9caa00700 1 -- 172.21.15.1:0/2873531950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9c4003b20 con 0x7fd9d4107d80 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.949+0000 7fd9caa00700 1 -- 172.21.15.1:0/2873531950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd9c401a8c0 con 0x7fd9d4107d80 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.949+0000 7fd9d9200700 1 -- 172.21.15.1:0/2873531950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d419f7c0 con 0x7fd9d4107d80 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.949+0000 7fd9caa00700 1 -- 172.21.15.1:0/2873531950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9c4024940 con 0x7fd9d4107d80 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.949+0000 7fd9d9200700 1 -- 172.21.15.1:0/2873531950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d41a24b0 con 0x7fd9d4107d80 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.949+0000 7fd9caa00700 1 -- 172.21.15.1:0/2873531950 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd9c4022020 con 0x7fd9d4107d80 2024-04-26T21:17:57.726 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.949+0000 7fd9caa00700 1 -- 172.21.15.1:0/2873531950 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd9c402ddd0 con 0x7fd9d4107d80 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.950+0000 7fd9d9200700 1 -- 172.21.15.1:0/2873531950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d4064a80 con 0x7fd9d4107d80 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:56.953+0000 7fd9caa00700 1 -- 172.21.15.1:0/2873531950 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fd9c4042ec0 con 0x7fd9d4107d80 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:57.033+0000 7fd9d9200700 1 -- 172.21.15.1:0/2873531950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd9d402e5a0 con 0x7fd9d4107d80 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:57.034+0000 7fd9caa00700 1 -- 172.21.15.1:0/2873531950 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd9c4031020 con 0x7fd9d4107d80 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:57.036+0000 7fd9d9200700 1 -- 172.21.15.1:0/2873531950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 msgr2=0x7fd9d419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:57.036+0000 7fd9d9200700 1 --2- 172.21.15.1:0/2873531950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d419eff0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4000c00 tx=0x7fd9c4011b70 comp rx=0 tx=0).stop 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:57.036+0000 7fd9d9200700 1 -- 172.21.15.1:0/2873531950 shutdown_connections 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:57.036+0000 7fd9d9200700 1 --2- 172.21.15.1:0/2873531950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9d4107d80 0x7fd9d419eff0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:57.036+0000 7fd9d9200700 1 -- 172.21.15.1:0/2873531950 >> 172.21.15.1:0/2873531950 conn(0x7fd9d4103340 msgr2=0x7fd9d4104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:57.036+0000 7fd9d9200700 1 -- 172.21.15.1:0/2873531950 shutdown_connections 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:17:57.036+0000 7fd9d9200700 1 -- 172.21.15.1:0/2873531950 wait complete. 2024-04-26T21:17:57.727 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (1/15)... 2024-04-26T21:17:59.317 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:59 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:17:59.023+0000 7f7a90be0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:17:59.318 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:59 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:17:59.256+0000 7f7a90be0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:17:59.817 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:59 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:17:59.474+0000 7f7a90be0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:18:00.172 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:17:59 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:17:59.870+0000 7f7a90be0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:18:00.173 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:17:59.999+0000 7f7a90be0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:18:00.422 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:00.260+0000 7f7a90be0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:18:00.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:00 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1556949170' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T21:18:00.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:00.542+0000 7f7a90be0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:18:01.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:18:01.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:18:01.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:18:01.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T21:18:01.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T21:18:01.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T21:18:01.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T21:18:01.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:01.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T21:18:01.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T21:18:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T21:18:01.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T21:18:01.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T21:17:46.701371+0000", 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.328+0000 7f9ccd400700 1 Processor -- start 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.328+0000 7f9ccd400700 1 -- start start 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.328+0000 7f9ccd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc81092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.329+0000 7f9ccd400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc81097f0 con 0x7f9cc8108e90 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.329+0000 7f9cc6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc81092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:01.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.329+0000 7f9cc6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc81092b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49954/0 (socket says 172.21.15.1:49954) 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.329+0000 7f9cc6800700 1 -- 172.21.15.1:0/1864842282 learned_addr learned my addr 172.21.15.1:0/1864842282 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.329+0000 7f9cc6800700 1 -- 172.21.15.1:0/1864842282 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc8109ff0 con 0x7f9cc8108e90 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.330+0000 7f9cc6800700 1 --2- 172.21.15.1:0/1864842282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc81092b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009e80 tx=0x7f9cb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63501ce333f230bb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.330+0000 7f9cc5400700 1 -- 172.21.15.1:0/1864842282 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9cb8017070 con 0x7f9cc8108e90 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.330+0000 7f9cc5400700 1 -- 172.21.15.1:0/1864842282 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9cb800b8c0 con 0x7f9cc8108e90 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.330+0000 7f9cc5400700 1 -- 172.21.15.1:0/1864842282 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9cb8004490 con 0x7f9cc8108e90 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.331+0000 7f9ccd400700 1 -- 172.21.15.1:0/1864842282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 msgr2=0x7f9cc81092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.331+0000 7f9ccd400700 1 --2- 172.21.15.1:0/1864842282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc81092b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009e80 tx=0x7f9cb800b330 comp rx=0 tx=0).stop 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.331+0000 7f9ccd400700 1 -- 172.21.15.1:0/1864842282 shutdown_connections 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.331+0000 7f9ccd400700 1 --2- 172.21.15.1:0/1864842282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc81092b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.331+0000 7f9ccd400700 1 -- 172.21.15.1:0/1864842282 >> 172.21.15.1:0/1864842282 conn(0x7f9cc8104450 msgr2=0x7f9cc8106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.331+0000 7f9ccd400700 1 -- 172.21.15.1:0/1864842282 shutdown_connections 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.331+0000 7f9ccd400700 1 -- 172.21.15.1:0/1864842282 wait complete. 2024-04-26T21:18:01.159 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.332+0000 7f9ccd400700 1 Processor -- start 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.332+0000 7f9ccd400700 1 -- start start 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.332+0000 7f9ccd400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc807eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.332+0000 7f9ccd400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc810d430 con 0x7f9cc8108e90 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.333+0000 7f9cc6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc807eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.333+0000 7f9cc6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc807eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49966/0 (socket says 172.21.15.1:49966) 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.333+0000 7f9cc6800700 1 -- 172.21.15.1:0/1556949170 learned_addr learned my addr 172.21.15.1:0/1556949170 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.333+0000 7f9cc6800700 1 -- 172.21.15.1:0/1556949170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc807f000 con 0x7f9cc8108e90 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.333+0000 7f9cc6800700 1 --2- 172.21.15.1:0/1556949170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc807eac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800b7b0 tx=0x7f9cb800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.334+0000 7f9cbf400700 1 -- 172.21.15.1:0/1556949170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9cb8020020 con 0x7f9cc8108e90 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.334+0000 7f9cbf400700 1 -- 172.21.15.1:0/1556949170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9cb8003d10 con 0x7f9cc8108e90 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.334+0000 7f9cbf400700 1 -- 172.21.15.1:0/1556949170 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9cb8017040 con 0x7f9cc8108e90 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.334+0000 7f9ccd400700 1 -- 172.21.15.1:0/1556949170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc807d240 con 0x7f9cc8108e90 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.334+0000 7f9ccd400700 1 -- 172.21.15.1:0/1556949170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc807d780 con 0x7f9cc8108e90 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.334+0000 7f9cbf400700 1 -- 172.21.15.1:0/1556949170 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9cb8023020 con 0x7f9cc8108e90 2024-04-26T21:18:01.160 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.334+0000 7f9cbf400700 1 -- 172.21.15.1:0/1556949170 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9cb800f050 con 0x7f9cc8108e90 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.335+0000 7f9ccd400700 1 -- 172.21.15.1:0/1556949170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc8064a80 con 0x7f9cc8108e90 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.337+0000 7f9cbf400700 1 -- 172.21.15.1:0/1556949170 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f9cb802a020 con 0x7f9cc8108e90 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.419+0000 7f9ccd400700 1 -- 172.21.15.1:0/1556949170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9cc807dde0 con 0x7f9cc8108e90 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.419+0000 7f9cbf400700 1 -- 172.21.15.1:0/1556949170 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9cb8003ec0 con 0x7f9cc8108e90 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.421+0000 7f9ccd400700 1 -- 172.21.15.1:0/1556949170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 msgr2=0x7f9cc807eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.421+0000 7f9ccd400700 1 --2- 172.21.15.1:0/1556949170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc807eac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800b7b0 tx=0x7f9cb800be90 comp rx=0 tx=0).stop 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.421+0000 7f9ccd400700 1 -- 172.21.15.1:0/1556949170 shutdown_connections 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.421+0000 7f9ccd400700 1 --2- 172.21.15.1:0/1556949170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cc8108e90 0x7f9cc807eac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.421+0000 7f9ccd400700 1 -- 172.21.15.1:0/1556949170 >> 172.21.15.1:0/1556949170 conn(0x7f9cc8104450 msgr2=0x7f9cc8104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.421+0000 7f9ccd400700 1 -- 172.21.15.1:0/1556949170 shutdown_connections 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:00.421+0000 7f9ccd400700 1 -- 172.21.15.1:0/1556949170 wait complete. 2024-04-26T21:18:01.161 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (2/15)... 2024-04-26T21:18:01.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:01 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:01.370+0000 7f7a90be0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:18:02.318 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:01 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:01.935+0000 7f7a90be0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:18:02.318 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:02 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:02.067+0000 7f7a90be0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:18:02.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:02 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:02.340+0000 7f7a90be0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:18:03.301 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:03 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:03.016+0000 7f7a90be0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:18:03.301 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:03 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:03.142+0000 7f7a90be0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:18:03.567 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:03 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:03.476+0000 7f7a90be0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:18:03.905 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:03 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:03.666+0000 7f7a90be0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:18:03.906 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:03 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:03.801+0000 7f7a90be0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:18:04.288 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:03 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3958505612' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T21:18:04.289 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:03 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:03.969+0000 7f7a90be0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:18:04.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:04 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:04.354+0000 7f7a90be0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:18:04.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:18:04.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:18:04.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:18:04.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T21:18:04.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T21:18:04.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T21:18:04.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T21:18:04.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T21:18:04.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:04.636 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T21:18:04.636 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:04.636 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T21:18:04.636 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T21:18:04.636 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T21:18:04.636 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T21:18:04.636 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T21:18:04.636 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T21:18:04.636 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:04.642 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:04.643 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T21:17:46.701371+0000", 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.813+0000 7f8f37200700 1 Processor -- start 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.813+0000 7f8f37200700 1 -- start start 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.813+0000 7f8f37200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f3810bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.813+0000 7f8f37200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3807cdf0 con 0x7f8f38109740 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.813+0000 7f8f35e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f3810bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.814+0000 7f8f35e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f3810bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49982/0 (socket says 172.21.15.1:49982) 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.814+0000 7f8f35e00700 1 -- 172.21.15.1:0/4078050888 learned_addr learned my addr 172.21.15.1:0/4078050888 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:04.644 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.814+0000 7f8f35e00700 1 -- 172.21.15.1:0/4078050888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3810c070 con 0x7f8f38109740 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.814+0000 7f8f35e00700 1 --2- 172.21.15.1:0/4078050888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f3810bb30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8f280098a0 tx=0x7f8f28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc3401ffa41aae0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.815+0000 7f8f34a00700 1 -- 172.21.15.1:0/4078050888 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f28004160 con 0x7f8f38109740 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.815+0000 7f8f34a00700 1 -- 172.21.15.1:0/4078050888 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8f28004300 con 0x7f8f38109740 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.815+0000 7f8f34a00700 1 -- 172.21.15.1:0/4078050888 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f28011af0 con 0x7f8f38109740 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.815+0000 7f8f37200700 1 -- 172.21.15.1:0/4078050888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 msgr2=0x7f8f3810bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.815+0000 7f8f37200700 1 --2- 172.21.15.1:0/4078050888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f3810bb30 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8f280098a0 tx=0x7f8f28009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.815+0000 7f8f37200700 1 -- 172.21.15.1:0/4078050888 shutdown_connections 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.815+0000 7f8f37200700 1 --2- 172.21.15.1:0/4078050888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f3810bb30 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.816+0000 7f8f37200700 1 -- 172.21.15.1:0/4078050888 >> 172.21.15.1:0/4078050888 conn(0x7f8f38103110 msgr2=0x7f8f38105550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.816+0000 7f8f37200700 1 -- 172.21.15.1:0/4078050888 shutdown_connections 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.816+0000 7f8f37200700 1 -- 172.21.15.1:0/4078050888 wait complete. 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.817+0000 7f8f37200700 1 Processor -- start 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.817+0000 7f8f37200700 1 -- start start 2024-04-26T21:18:04.645 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.817+0000 7f8f37200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f38108fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:04.648 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.817+0000 7f8f37200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3810cfe0 con 0x7f8f38109740 2024-04-26T21:18:04.648 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.817+0000 7f8f35e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f38108fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:04.648 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.817+0000 7f8f35e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f38108fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49984/0 (socket says 172.21.15.1:49984) 2024-04-26T21:18:04.648 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.817+0000 7f8f35e00700 1 -- 172.21.15.1:0/3958505612 learned_addr learned my addr 172.21.15.1:0/3958505612 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.818+0000 7f8f35e00700 1 -- 172.21.15.1:0/3958505612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f38109500 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.818+0000 7f8f35e00700 1 --2- 172.21.15.1:0/3958505612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f38108fc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8f28011ac0 tx=0x7f8f28011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.818+0000 7f8f2ea00700 1 -- 172.21.15.1:0/3958505612 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f28003b20 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.818+0000 7f8f2ea00700 1 -- 172.21.15.1:0/3958505612 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8f2801a8c0 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.818+0000 7f8f37200700 1 -- 172.21.15.1:0/3958505612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f381076f0 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.818+0000 7f8f2ea00700 1 -- 172.21.15.1:0/3958505612 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f280248d0 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.818+0000 7f8f37200700 1 -- 172.21.15.1:0/3958505612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f38107c30 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.819+0000 7f8f2ea00700 1 -- 172.21.15.1:0/3958505612 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8f28022020 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.819+0000 7f8f2ea00700 1 -- 172.21.15.1:0/3958505612 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8f280193c0 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.820+0000 7f8f37200700 1 -- 172.21.15.1:0/3958505612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f38064a80 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.823+0000 7f8f2ea00700 1 -- 172.21.15.1:0/3958505612 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f8f2802ddf0 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.900+0000 7f8f37200700 1 -- 172.21.15.1:0/3958505612 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f8f38107eb0 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.901+0000 7f8f2ea00700 1 -- 172.21.15.1:0/3958505612 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f8f2801aa70 con 0x7f8f38109740 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.902+0000 7f8f37200700 1 -- 172.21.15.1:0/3958505612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 msgr2=0x7f8f38108fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:04.649 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.902+0000 7f8f37200700 1 --2- 172.21.15.1:0/3958505612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f38108fc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8f28011ac0 tx=0x7f8f28011ba0 comp rx=0 tx=0).stop 2024-04-26T21:18:04.650 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.902+0000 7f8f37200700 1 -- 172.21.15.1:0/3958505612 shutdown_connections 2024-04-26T21:18:04.650 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.902+0000 7f8f37200700 1 --2- 172.21.15.1:0/3958505612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f38109740 0x7f8f38108fc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:04.650 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.902+0000 7f8f37200700 1 -- 172.21.15.1:0/3958505612 >> 172.21.15.1:0/3958505612 conn(0x7f8f38103110 msgr2=0x7f8f38103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:04.650 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.902+0000 7f8f37200700 1 -- 172.21.15.1:0/3958505612 shutdown_connections 2024-04-26T21:18:04.650 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:03.903+0000 7f8f37200700 1 -- 172.21.15.1:0/3958505612 wait complete. 2024-04-26T21:18:04.650 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (3/15)... 2024-04-26T21:18:05.476 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:05.182+0000 7f7a90be0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:18:05.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:05.475+0000 7f7a90be0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:18:05.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: Activating manager daemon x 2024-04-26T21:18:05.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: mgrmap e2: x(active, starting, since 0.0107529s) 2024-04-26T21:18:05.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: from='mgr.14100 172.21.15.1:0/2377538149' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T21:18:05.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: from='mgr.14100 172.21.15.1:0/2377538149' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T21:18:05.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: from='mgr.14100 172.21.15.1:0/2377538149' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T21:18:05.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: from='mgr.14100 172.21.15.1:0/2377538149' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:18:05.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: from='mgr.14100 172.21.15.1:0/2377538149' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T21:18:05.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: Manager daemon x is now available 2024-04-26T21:18:05.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: from='mgr.14100 172.21.15.1:0/2377538149' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T21:18:05.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: from='mgr.14100 172.21.15.1:0/2377538149' entity='mgr.x' 2024-04-26T21:18:05.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: from='mgr.14100 172.21.15.1:0/2377538149' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T21:18:05.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:05 smithi001 ceph-mon[28054]: from='mgr.14100 172.21.15.1:0/2377538149' entity='mgr.x' 2024-04-26T21:18:06.806 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:06 smithi001 ceph-mon[28054]: from='mgr.14100 172.21.15.1:0/2377538149' entity='mgr.x' 2024-04-26T21:18:06.807 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:06 smithi001 ceph-mon[28054]: mgrmap e3: x(active, since 1.02053s) 2024-04-26T21:18:08.328 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:18:08.328 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:18:08.328 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:18:08.328 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T21:18:08.328 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T21:18:08.329 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T21:18:08.329 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T21:18:08.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:08.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T21:18:08.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T21:18:08.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-04-26T21:18:08.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:18:08.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T21:18:08.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T21:18:08.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T21:18:08.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:08.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T21:18:08.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:08.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T21:18:08.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T21:18:08.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:08.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T21:18:08.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T21:18:08.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T21:17:46.701371+0000", 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-04-26T21:18:08.345 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.252+0000 7fd396c00700 1 Processor -- start 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.253+0000 7fd396c00700 1 -- start start 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.253+0000 7fd396c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39007d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.253+0000 7fd396c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd39007d850 con 0x7fd39007ee20 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.254+0000 7fd38fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39007d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.254+0000 7fd38fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39007d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50044/0 (socket says 172.21.15.1:50044) 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.254+0000 7fd38fe00700 1 -- 172.21.15.1:0/2378638419 learned_addr learned my addr 172.21.15.1:0/2378638419 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.254+0000 7fd38fe00700 1 -- 172.21.15.1:0/2378638419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39007d9d0 con 0x7fd39007ee20 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.254+0000 7fd38fe00700 1 --2- 172.21.15.1:0/2378638419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39007d280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd380009e80 tx=0x7fd38000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29512dd3fbeb68f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.255+0000 7fd38f400700 1 -- 172.21.15.1:0/2378638419 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd380017070 con 0x7fd39007ee20 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.255+0000 7fd38f400700 1 -- 172.21.15.1:0/2378638419 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd38000b8c0 con 0x7fd39007ee20 2024-04-26T21:18:08.346 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.255+0000 7fd38f400700 1 -- 172.21.15.1:0/2378638419 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd380004490 con 0x7fd39007ee20 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.255+0000 7fd396c00700 1 -- 172.21.15.1:0/2378638419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 msgr2=0x7fd39007d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.256+0000 7fd396c00700 1 --2- 172.21.15.1:0/2378638419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39007d280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd380009e80 tx=0x7fd38000b330 comp rx=0 tx=0).stop 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.256+0000 7fd396c00700 1 -- 172.21.15.1:0/2378638419 shutdown_connections 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.256+0000 7fd396c00700 1 --2- 172.21.15.1:0/2378638419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39007d280 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.256+0000 7fd396c00700 1 -- 172.21.15.1:0/2378638419 >> 172.21.15.1:0/2378638419 conn(0x7fd390104450 msgr2=0x7fd390106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.256+0000 7fd396c00700 1 -- 172.21.15.1:0/2378638419 shutdown_connections 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.256+0000 7fd396c00700 1 -- 172.21.15.1:0/2378638419 wait complete. 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.257+0000 7fd396c00700 1 Processor -- start 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.257+0000 7fd396c00700 1 -- start start 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.257+0000 7fd396c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.258+0000 7fd396c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd39010f420 con 0x7fd39007ee20 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.258+0000 7fd38fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.258+0000 7fd38fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39019eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50050/0 (socket says 172.21.15.1:50050) 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.258+0000 7fd38fe00700 1 -- 172.21.15.1:0/2636133614 learned_addr learned my addr 172.21.15.1:0/2636133614 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:08.347 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.258+0000 7fd38fe00700 1 -- 172.21.15.1:0/2636133614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39019f3f0 con 0x7fd39007ee20 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.258+0000 7fd38fe00700 1 --2- 172.21.15.1:0/2636133614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39019eeb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd38000b7b0 tx=0x7fd38000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.259+0000 7fd38d600700 1 -- 172.21.15.1:0/2636133614 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd380020020 con 0x7fd39007ee20 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.259+0000 7fd38d600700 1 -- 172.21.15.1:0/2636133614 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd380003d10 con 0x7fd39007ee20 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.259+0000 7fd38d600700 1 -- 172.21.15.1:0/2636133614 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd380017040 con 0x7fd39007ee20 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.259+0000 7fd396c00700 1 -- 172.21.15.1:0/2636133614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd39019f680 con 0x7fd39007ee20 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.259+0000 7fd396c00700 1 -- 172.21.15.1:0/2636133614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3901a2370 con 0x7fd39007ee20 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.260+0000 7fd38d600700 1 -- 172.21.15.1:0/2636133614 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7fd380023020 con 0x7fd39007ee20 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.260+0000 7fd396c00700 1 -- 172.21.15.1:0/2636133614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd390064a80 con 0x7fd39007ee20 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.261+0000 7fd38d600700 1 --2- 172.21.15.1:0/2636133614 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fd374039490 0x7fd37403b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.261+0000 7fd38d600700 1 -- 172.21.15.1:0/2636133614 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd38000f050 con 0x7fd39007ee20 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.261+0000 7fd387400700 1 --2- 172.21.15.1:0/2636133614 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fd374039490 0x7fd37403b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.262+0000 7fd387400700 1 --2- 172.21.15.1:0/2636133614 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fd374039490 0x7fd37403b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd378006fd0 tx=0x7fd378006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.267+0000 7fd38d600700 1 -- 172.21.15.1:0/2636133614 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd38002a020 con 0x7fd39007ee20 2024-04-26T21:18:08.348 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.540+0000 7fd38d600700 1 -- 172.21.15.1:0/2636133614 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fd380012950 con 0x7fd39007ee20 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.588+0000 7fd396c00700 1 -- 172.21.15.1:0/2636133614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd3901a2540 con 0x7fd39007ee20 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.589+0000 7fd38d600700 1 -- 172.21.15.1:0/2636133614 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd38001a2d0 con 0x7fd39007ee20 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.592+0000 7fd396c00700 1 -- 172.21.15.1:0/2636133614 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fd374039490 msgr2=0x7fd37403b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.592+0000 7fd396c00700 1 --2- 172.21.15.1:0/2636133614 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fd374039490 0x7fd37403b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd378006fd0 tx=0x7fd378006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.592+0000 7fd396c00700 1 -- 172.21.15.1:0/2636133614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 msgr2=0x7fd39019eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.592+0000 7fd396c00700 1 --2- 172.21.15.1:0/2636133614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39019eeb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd38000b7b0 tx=0x7fd38000be90 comp rx=0 tx=0).stop 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.592+0000 7fd396c00700 1 -- 172.21.15.1:0/2636133614 shutdown_connections 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.592+0000 7fd396c00700 1 --2- 172.21.15.1:0/2636133614 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fd374039490 0x7fd37403b950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.592+0000 7fd396c00700 1 --2- 172.21.15.1:0/2636133614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd39007ee20 0x7fd39019eeb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.592+0000 7fd396c00700 1 -- 172.21.15.1:0/2636133614 >> 172.21.15.1:0/2636133614 conn(0x7fd390104450 msgr2=0x7fd390104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.592+0000 7fd396c00700 1 -- 172.21.15.1:0/2636133614 shutdown_connections 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:07.592+0000 7fd396c00700 1 -- 172.21.15.1:0/2636133614 wait complete. 2024-04-26T21:18:08.349 INFO:teuthology.orchestra.run.smithi001.stdout:mgr is available 2024-04-26T21:18:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:08 smithi001 ceph-mon[28054]: mgrmap e4: x(active, since 2s) 2024-04-26T21:18:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:08 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2636133614' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T21:18:09.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:09 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4197779394' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout fsid = 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T21:18:09.896 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.936+0000 7fbc0e000700 1 Processor -- start 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.936+0000 7fbc0e000700 1 -- start start 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.936+0000 7fbc0e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc081092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.936+0000 7fbc0e000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081097f0 con 0x7fbc08108e90 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.937+0000 7fbc07400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc081092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:09.897 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.937+0000 7fbc07400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc081092b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49596/0 (socket says 172.21.15.1:49596) 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.937+0000 7fbc07400700 1 -- 172.21.15.1:0/530937103 learned_addr learned my addr 172.21.15.1:0/530937103 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.937+0000 7fbc07400700 1 -- 172.21.15.1:0/530937103 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc08109ff0 con 0x7fbc08108e90 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.938+0000 7fbc07400700 1 --2- 172.21.15.1:0/530937103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc081092b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009e80 tx=0x7fbbf800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a66df1365e424ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.938+0000 7fbc06a00700 1 -- 172.21.15.1:0/530937103 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbf8017070 con 0x7fbc08108e90 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.938+0000 7fbc06a00700 1 -- 172.21.15.1:0/530937103 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbbf800b8c0 con 0x7fbc08108e90 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.938+0000 7fbc06a00700 1 -- 172.21.15.1:0/530937103 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbf8004490 con 0x7fbc08108e90 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.939+0000 7fbc0e000700 1 -- 172.21.15.1:0/530937103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 msgr2=0x7fbc081092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.939+0000 7fbc0e000700 1 --2- 172.21.15.1:0/530937103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc081092b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009e80 tx=0x7fbbf800b330 comp rx=0 tx=0).stop 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.939+0000 7fbc0e000700 1 -- 172.21.15.1:0/530937103 shutdown_connections 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.939+0000 7fbc0e000700 1 --2- 172.21.15.1:0/530937103 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc081092b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.939+0000 7fbc0e000700 1 -- 172.21.15.1:0/530937103 >> 172.21.15.1:0/530937103 conn(0x7fbc08104450 msgr2=0x7fbc08106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.939+0000 7fbc0e000700 1 -- 172.21.15.1:0/530937103 shutdown_connections 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.939+0000 7fbc0e000700 1 -- 172.21.15.1:0/530937103 wait complete. 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.940+0000 7fbc0e000700 1 Processor -- start 2024-04-26T21:18:09.898 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.940+0000 7fbc0e000700 1 -- start start 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.941+0000 7fbc0e000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc0819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.941+0000 7fbc0e000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbc0810d430 con 0x7fbc08108e90 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.941+0000 7fbc07400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc0819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.941+0000 7fbc07400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc0819cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49602/0 (socket says 172.21.15.1:49602) 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.941+0000 7fbc07400700 1 -- 172.21.15.1:0/4197779394 learned_addr learned my addr 172.21.15.1:0/4197779394 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.941+0000 7fbc07400700 1 -- 172.21.15.1:0/4197779394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0819d2c0 con 0x7fbc08108e90 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.942+0000 7fbc07400700 1 --2- 172.21.15.1:0/4197779394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc0819cd80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800b7b0 tx=0x7fbbf800be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.942+0000 7fbc04c00700 1 -- 172.21.15.1:0/4197779394 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbf8020020 con 0x7fbc08108e90 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.942+0000 7fbc04c00700 1 -- 172.21.15.1:0/4197779394 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbbf8003d10 con 0x7fbc08108e90 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.942+0000 7fbc0e000700 1 -- 172.21.15.1:0/4197779394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0819d550 con 0x7fbc08108e90 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.942+0000 7fbc04c00700 1 -- 172.21.15.1:0/4197779394 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbbf8017040 con 0x7fbc08108e90 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.942+0000 7fbc0e000700 1 -- 172.21.15.1:0/4197779394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc081a01f0 con 0x7fbc08108e90 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.943+0000 7fbc04c00700 1 -- 172.21.15.1:0/4197779394 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fbbf8023020 con 0x7fbc08108e90 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.943+0000 7fbc0e000700 1 -- 172.21.15.1:0/4197779394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc08064a80 con 0x7fbc08108e90 2024-04-26T21:18:09.899 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.944+0000 7fbc04c00700 1 --2- 172.21.15.1:0/4197779394 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fbbec0395b0 0x7fbbec03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.944+0000 7fbc04c00700 1 -- 172.21.15.1:0/4197779394 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbbf800f050 con 0x7fbc08108e90 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.944+0000 7fbbfea00700 1 --2- 172.21.15.1:0/4197779394 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fbbec0395b0 0x7fbbec03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.945+0000 7fbbfea00700 1 --2- 172.21.15.1:0/4197779394 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fbbec0395b0 0x7fbbec03ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0006fd0 tx=0x7fbbf0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:08.950+0000 7fbc04c00700 1 -- 172.21.15.1:0/4197779394 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbbf802a020 con 0x7fbc08108e90 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.177+0000 7fbc0e000700 1 -- 172.21.15.1:0/4197779394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fbc081a03c0 con 0x7fbc08108e90 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.179+0000 7fbc04c00700 1 -- 172.21.15.1:0/4197779394 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7fbbf802c300 con 0x7fbc08108e90 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.184+0000 7fbc0e000700 1 -- 172.21.15.1:0/4197779394 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fbbec0395b0 msgr2=0x7fbbec03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.184+0000 7fbc0e000700 1 --2- 172.21.15.1:0/4197779394 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fbbec0395b0 0x7fbbec03ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0006fd0 tx=0x7fbbf0006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.184+0000 7fbc0e000700 1 -- 172.21.15.1:0/4197779394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 msgr2=0x7fbc0819cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.184+0000 7fbc0e000700 1 --2- 172.21.15.1:0/4197779394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc0819cd80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800b7b0 tx=0x7fbbf800be90 comp rx=0 tx=0).stop 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.184+0000 7fbc0e000700 1 -- 172.21.15.1:0/4197779394 shutdown_connections 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.184+0000 7fbc0e000700 1 --2- 172.21.15.1:0/4197779394 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7fbbec0395b0 0x7fbbec03ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.184+0000 7fbc0e000700 1 --2- 172.21.15.1:0/4197779394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc08108e90 0x7fbc0819cd80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.184+0000 7fbc0e000700 1 -- 172.21.15.1:0/4197779394 >> 172.21.15.1:0/4197779394 conn(0x7fbc08104450 msgr2=0x7fbc08104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:09.900 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.184+0000 7fbc0e000700 1 -- 172.21.15.1:0/4197779394 shutdown_connections 2024-04-26T21:18:09.901 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:09.184+0000 7fbc0e000700 1 -- 172.21.15.1:0/4197779394 wait complete. 2024-04-26T21:18:09.901 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling cephadm module... 2024-04-26T21:18:11.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:10 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: ignoring --setuser ceph since I am not root 2024-04-26T21:18:11.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:10 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: ignoring --setgroup ceph since I am not root 2024-04-26T21:18:11.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:10 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1799497194' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T21:18:11.507 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:11 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:11.137+0000 7fdcbb9e0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:18:11.507 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:11 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:11.382+0000 7fdcbb9e0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:18:11.526 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.493+0000 7f3a9b400700 1 Processor -- start 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.493+0000 7f3a9b400700 1 -- start start 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.493+0000 7f3a9b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9410bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.493+0000 7f3a9b400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9407ce50 con 0x7f3a941098c0 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.494+0000 7f3a98a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9410bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.494+0000 7f3a98a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9410bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49618/0 (socket says 172.21.15.1:49618) 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.494+0000 7f3a98a00700 1 -- 172.21.15.1:0/1663475963 learned_addr learned my addr 172.21.15.1:0/1663475963 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.495+0000 7f3a98a00700 1 -- 172.21.15.1:0/1663475963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9410c1f0 con 0x7f3a941098c0 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.495+0000 7f3a98a00700 1 --2- 172.21.15.1:0/1663475963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9410bcb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3a880098a0 tx=0x7f3a88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947427a159dd7d01 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.496+0000 7f3a93400700 1 -- 172.21.15.1:0/1663475963 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a88004160 con 0x7f3a941098c0 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.496+0000 7f3a93400700 1 -- 172.21.15.1:0/1663475963 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3a88004300 con 0x7f3a941098c0 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.496+0000 7f3a93400700 1 -- 172.21.15.1:0/1663475963 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a88011af0 con 0x7f3a941098c0 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.496+0000 7f3a9b400700 1 -- 172.21.15.1:0/1663475963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 msgr2=0x7f3a9410bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.496+0000 7f3a9b400700 1 --2- 172.21.15.1:0/1663475963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9410bcb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3a880098a0 tx=0x7f3a88009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:11.527 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.497+0000 7f3a9b400700 1 -- 172.21.15.1:0/1663475963 shutdown_connections 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.497+0000 7f3a9b400700 1 --2- 172.21.15.1:0/1663475963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9410bcb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.497+0000 7f3a9b400700 1 -- 172.21.15.1:0/1663475963 >> 172.21.15.1:0/1663475963 conn(0x7f3a94103330 msgr2=0x7f3a94105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.497+0000 7f3a9b400700 1 -- 172.21.15.1:0/1663475963 shutdown_connections 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.497+0000 7f3a9b400700 1 -- 172.21.15.1:0/1663475963 wait complete. 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.498+0000 7f3a9b400700 1 Processor -- start 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.498+0000 7f3a9b400700 1 -- start start 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.498+0000 7f3a9b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.498+0000 7f3a9b400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9410d1c0 con 0x7f3a941098c0 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.499+0000 7f3a98a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.499+0000 7f3a98a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9419cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49626/0 (socket says 172.21.15.1:49626) 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.499+0000 7f3a98a00700 1 -- 172.21.15.1:0/1799497194 learned_addr learned my addr 172.21.15.1:0/1799497194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.499+0000 7f3a98a00700 1 -- 172.21.15.1:0/1799497194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9419d220 con 0x7f3a941098c0 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.499+0000 7f3a98a00700 1 --2- 172.21.15.1:0/1799497194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9419cce0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3a88003f40 tx=0x7f3a88011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:11.528 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.500+0000 7f3a91600700 1 -- 172.21.15.1:0/1799497194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a880038b0 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.500+0000 7f3a91600700 1 -- 172.21.15.1:0/1799497194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3a88003a50 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.500+0000 7f3a9b400700 1 -- 172.21.15.1:0/1799497194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a9419d4b0 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.500+0000 7f3a91600700 1 -- 172.21.15.1:0/1799497194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a8801a5c0 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.500+0000 7f3a9b400700 1 -- 172.21.15.1:0/1799497194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a941a0190 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.501+0000 7f3a91600700 1 -- 172.21.15.1:0/1799497194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f3a88024430 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.501+0000 7f3a9b400700 1 -- 172.21.15.1:0/1799497194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a94064a80 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.501+0000 7f3a91600700 1 --2- 172.21.15.1:0/1799497194 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f3a7c039560 0x7f3a7c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.502+0000 7f3a91600700 1 -- 172.21.15.1:0/1799497194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3a8804da70 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.502+0000 7f3a93e00700 1 --2- 172.21.15.1:0/1799497194 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f3a7c039560 0x7f3a7c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.502+0000 7f3a93e00700 1 --2- 172.21.15.1:0/1799497194 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f3a7c039560 0x7f3a7c03ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a80006fd0 tx=0x7f3a80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.507+0000 7f3a91600700 1 -- 172.21.15.1:0/1799497194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3a8802d910 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.800+0000 7f3a9b400700 1 -- 172.21.15.1:0/1799497194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f3a941a0730 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.854+0000 7f3a91600700 1 -- 172.21.15.1:0/1799497194 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f3a8802d3f0 con 0x7f3a941098c0 2024-04-26T21:18:11.529 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.857+0000 7f3a91600700 1 -- 172.21.15.1:0/1799497194 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f3a88018350 con 0x7f3a941098c0 2024-04-26T21:18:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.859+0000 7f3a9b400700 1 -- 172.21.15.1:0/1799497194 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f3a7c039560 msgr2=0x7f3a7c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.859+0000 7f3a9b400700 1 --2- 172.21.15.1:0/1799497194 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f3a7c039560 0x7f3a7c03ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a80006fd0 tx=0x7f3a80006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.859+0000 7f3a9b400700 1 -- 172.21.15.1:0/1799497194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 msgr2=0x7f3a9419cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.859+0000 7f3a9b400700 1 --2- 172.21.15.1:0/1799497194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9419cce0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3a88003f40 tx=0x7f3a88011890 comp rx=0 tx=0).stop 2024-04-26T21:18:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.859+0000 7f3a9b400700 1 -- 172.21.15.1:0/1799497194 shutdown_connections 2024-04-26T21:18:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.859+0000 7f3a9b400700 1 --2- 172.21.15.1:0/1799497194 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f3a7c039560 0x7f3a7c03ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.859+0000 7f3a9b400700 1 --2- 172.21.15.1:0/1799497194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a941098c0 0x7f3a9419cce0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.859+0000 7f3a9b400700 1 -- 172.21.15.1:0/1799497194 >> 172.21.15.1:0/1799497194 conn(0x7f3a94103330 msgr2=0x7f3a94103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.861+0000 7f3a9b400700 1 -- 172.21.15.1:0/1799497194 shutdown_connections 2024-04-26T21:18:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:10.861+0000 7f3a9b400700 1 -- 172.21.15.1:0/1799497194 wait complete. 2024-04-26T21:18:11.817 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:11 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:11.519+0000 7fdcbb9e0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:18:12.121 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:11 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:11.938+0000 7fdcbb9e0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:18:12.121 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:12.057+0000 7fdcbb9e0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:18:12.121 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:11 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1799497194' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T21:18:12.121 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:11 smithi001 ceph-mon[28054]: mgrmap e5: x(active, since 5s) 2024-04-26T21:18:13.062 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:12 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/564185868' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T21:18:13.081 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:18:13.081 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T21:18:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T21:18:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T21:18:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T21:18:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:18:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.119+0000 7f4ec4400700 1 Processor -- start 2024-04-26T21:18:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.120+0000 7f4ec4400700 1 -- start start 2024-04-26T21:18:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.120+0000 7f4ec4400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc1092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.120+0000 7f4ec4400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc1097f0 con 0x7f4ebc108e90 2024-04-26T21:18:13.082 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.121+0000 7f4ec1a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc1092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.121+0000 7f4ec1a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc1092b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49672/0 (socket says 172.21.15.1:49672) 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.121+0000 7f4ec1a00700 1 -- 172.21.15.1:0/2398792775 learned_addr learned my addr 172.21.15.1:0/2398792775 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.122+0000 7f4ec1a00700 1 -- 172.21.15.1:0/2398792775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ebc109ff0 con 0x7f4ebc108e90 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.122+0000 7f4ec1a00700 1 --2- 172.21.15.1:0/2398792775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc1092b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4eb00098a0 tx=0x7f4eb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3b21ec9f7693c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.122+0000 7f4ebbe00700 1 -- 172.21.15.1:0/2398792775 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4eb0004160 con 0x7f4ebc108e90 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.123+0000 7f4ebbe00700 1 -- 172.21.15.1:0/2398792775 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4eb0004300 con 0x7f4ebc108e90 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.123+0000 7f4ebbe00700 1 -- 172.21.15.1:0/2398792775 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4eb0011af0 con 0x7f4ebc108e90 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.123+0000 7f4ec4400700 1 -- 172.21.15.1:0/2398792775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 msgr2=0x7f4ebc1092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.123+0000 7f4ec4400700 1 --2- 172.21.15.1:0/2398792775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc1092b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4eb00098a0 tx=0x7f4eb0009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.123+0000 7f4ec4400700 1 -- 172.21.15.1:0/2398792775 shutdown_connections 2024-04-26T21:18:13.088 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.123+0000 7f4ec4400700 1 --2- 172.21.15.1:0/2398792775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc1092b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.123+0000 7f4ec4400700 1 -- 172.21.15.1:0/2398792775 >> 172.21.15.1:0/2398792775 conn(0x7f4ebc104450 msgr2=0x7f4ebc106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.124+0000 7f4ec4400700 1 -- 172.21.15.1:0/2398792775 shutdown_connections 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.124+0000 7f4ec4400700 1 -- 172.21.15.1:0/2398792775 wait complete. 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.125+0000 7f4ec4400700 1 Processor -- start 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.125+0000 7f4ec4400700 1 -- start start 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.125+0000 7f4ec4400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.125+0000 7f4ec4400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc10d430 con 0x7f4ebc108e90 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.125+0000 7f4ec1a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.125+0000 7f4ec1a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc19ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49688/0 (socket says 172.21.15.1:49688) 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.125+0000 7f4ec1a00700 1 -- 172.21.15.1:0/564185868 learned_addr learned my addr 172.21.15.1:0/564185868 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.126+0000 7f4ec1a00700 1 -- 172.21.15.1:0/564185868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ebc19d340 con 0x7f4ebc108e90 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.126+0000 7f4ec1a00700 1 --2- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc19ce00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0011ac0 tx=0x7f4eb0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.126+0000 7f4eba000700 1 -- 172.21.15.1:0/564185868 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4eb0003970 con 0x7f4ebc108e90 2024-04-26T21:18:13.089 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.126+0000 7f4eba000700 1 -- 172.21.15.1:0/564185868 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4eb0003b10 con 0x7f4ebc108e90 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.126+0000 7f4ec4400700 1 -- 172.21.15.1:0/564185868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ebc19d5d0 con 0x7f4ebc108e90 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.127+0000 7f4eba000700 1 -- 172.21.15.1:0/564185868 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4eb0024610 con 0x7f4ebc108e90 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.127+0000 7f4ec4400700 1 -- 172.21.15.1:0/564185868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ebc1a02c0 con 0x7f4ebc108e90 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.128+0000 7f4ec4400700 1 -- 172.21.15.1:0/564185868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e9c005360 con 0x7f4ebc108e90 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.128+0000 7f4eba000700 1 -- 172.21.15.1:0/564185868 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f4eb0024890 con 0x7f4ebc108e90 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.128+0000 7f4eba000700 1 --2- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f4ea40395b0 0x7f4ea403ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.129+0000 7f4eba000700 1 -- 172.21.15.1:0/564185868 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4eb004d8c0 con 0x7f4ebc108e90 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.129+0000 7f4ec1000700 1 -- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f4ea40395b0 msgr2=0x7f4ea403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/304032214 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.129+0000 7f4ec1000700 1 --2- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f4ea40395b0 0x7f4ea403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.134+0000 7f4eba000700 1 -- 172.21.15.1:0/564185868 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4eb00172b0 con 0x7f4ebc108e90 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.330+0000 7f4ec1000700 1 -- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f4ea40395b0 msgr2=0x7f4ea403ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/304032214 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.330+0000 7f4ec1000700 1 --2- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f4ea40395b0 0x7f4ea403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.422+0000 7f4ec4400700 1 -- 172.21.15.1:0/564185868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f4e9c005e60 con 0x7f4ebc108e90 2024-04-26T21:18:13.090 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.423+0000 7f4eba000700 1 -- 172.21.15.1:0/564185868 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f4eb004d2e0 con 0x7f4ebc108e90 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.426+0000 7f4ec4400700 1 -- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f4ea40395b0 msgr2=0x7f4ea403ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.426+0000 7f4ec4400700 1 --2- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f4ea40395b0 0x7f4ea403ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.426+0000 7f4ec4400700 1 -- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 msgr2=0x7f4ebc19ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.426+0000 7f4ec4400700 1 --2- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc19ce00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0011ac0 tx=0x7f4eb0011ba0 comp rx=0 tx=0).stop 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.426+0000 7f4ec4400700 1 -- 172.21.15.1:0/564185868 shutdown_connections 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.426+0000 7f4ec4400700 1 --2- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f4ea40395b0 0x7f4ea403ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.426+0000 7f4ec4400700 1 --2- 172.21.15.1:0/564185868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ebc108e90 0x7f4ebc19ce00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.426+0000 7f4ec4400700 1 -- 172.21.15.1:0/564185868 >> 172.21.15.1:0/564185868 conn(0x7f4ebc104450 msgr2=0x7f4ebc104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.427+0000 7f4ec4400700 1 -- 172.21.15.1:0/564185868 shutdown_connections 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:12.427+0000 7f4ec4400700 1 -- 172.21.15.1:0/564185868 wait complete. 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for the mgr to restart... 2024-04-26T21:18:13.091 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr epoch 5... 2024-04-26T21:18:13.908 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:13.673+0000 7fdcbb9e0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:18:14.263 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:13.907+0000 7fdcbb9e0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:18:14.263 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:14.126+0000 7fdcbb9e0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:18:14.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:14.517+0000 7fdcbb9e0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:18:14.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:14.644+0000 7fdcbb9e0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:18:15.168 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:14.903+0000 7fdcbb9e0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:18:15.168 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:15 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:15.166+0000 7fdcbb9e0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:18:16.317 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:15 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:15.983+0000 7fdcbb9e0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:18:16.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:16.553+0000 7fdcbb9e0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:18:16.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:16.685+0000 7fdcbb9e0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:18:17.317 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:16.960+0000 7fdcbb9e0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:18:18.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:17.627+0000 7fdcbb9e0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:18:18.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:17.755+0000 7fdcbb9e0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:18:18.405 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:18.087+0000 7fdcbb9e0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:18:18.405 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:18.271+0000 7fdcbb9e0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:18:18.405 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:18.403+0000 7fdcbb9e0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:18:18.817 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:18.560+0000 7fdcbb9e0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:18:19.317 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:18.935+0000 7fdcbb9e0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:18:20.020 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:19 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:19.722+0000 7fdcbb9e0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:18:20.318 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:20.018+0000 7fdcbb9e0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:18:20.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-mon[28054]: Active manager daemon x restarted 2024-04-26T21:18:20.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-mon[28054]: Activating manager daemon x 2024-04-26T21:18:20.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-mon[28054]: osdmap e2: 0 total, 0 up, 0 in 2024-04-26T21:18:20.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-mon[28054]: mgrmap e6: x(active, starting, since 0.00703705s) 2024-04-26T21:18:20.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:18:20.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T21:18:20.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T21:18:20.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T21:18:20.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T21:18:20.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:20 smithi001 ceph-mon[28054]: Manager daemon x is now available 2024-04-26T21:18:21.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:21 smithi001 ceph-mon[28054]: Found migration_current of "None". Setting to last migration. 2024-04-26T21:18:21.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:21 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:21.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:21 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:21.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:21 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:18:21.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:21 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:18:21.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:21 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T21:18:21.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:21 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T21:18:21.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:21 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:21.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:21 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:21.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:21 smithi001 ceph-mon[28054]: mgrmap e7: x(active, since 1.01662s) 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.650+0000 7f5a21600700 1 Processor -- start 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.650+0000 7f5a21600700 1 -- start start 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.651+0000 7f5a21600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c1092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.651+0000 7f5a21600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1c1097f0 con 0x7f5a1c108e90 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.651+0000 7f5a1aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c1092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.651+0000 7f5a1aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c1092b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49694/0 (socket says 172.21.15.1:49694) 2024-04-26T21:18:21.793 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.651+0000 7f5a1aa00700 1 -- 172.21.15.1:0/2989792574 learned_addr learned my addr 172.21.15.1:0/2989792574 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.652+0000 7f5a1aa00700 1 -- 172.21.15.1:0/2989792574 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1c109ff0 con 0x7f5a1c108e90 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.652+0000 7f5a1aa00700 1 --2- 172.21.15.1:0/2989792574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c1092b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0098a0 tx=0x7f5a0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e908ec8aeda3425a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.652+0000 7f5a19600700 1 -- 172.21.15.1:0/2989792574 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a0c004160 con 0x7f5a1c108e90 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.652+0000 7f5a19600700 1 -- 172.21.15.1:0/2989792574 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5a0c004300 con 0x7f5a1c108e90 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.652+0000 7f5a19600700 1 -- 172.21.15.1:0/2989792574 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a0c011af0 con 0x7f5a1c108e90 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.653+0000 7f5a21600700 1 -- 172.21.15.1:0/2989792574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 msgr2=0x7f5a1c1092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.653+0000 7f5a21600700 1 --2- 172.21.15.1:0/2989792574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c1092b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c0098a0 tx=0x7f5a0c009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.653+0000 7f5a21600700 1 -- 172.21.15.1:0/2989792574 shutdown_connections 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.653+0000 7f5a21600700 1 --2- 172.21.15.1:0/2989792574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c1092b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.653+0000 7f5a21600700 1 -- 172.21.15.1:0/2989792574 >> 172.21.15.1:0/2989792574 conn(0x7f5a1c104450 msgr2=0x7f5a1c106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.654+0000 7f5a21600700 1 -- 172.21.15.1:0/2989792574 shutdown_connections 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.654+0000 7f5a21600700 1 -- 172.21.15.1:0/2989792574 wait complete. 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.655+0000 7f5a21600700 1 Processor -- start 2024-04-26T21:18:21.794 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.655+0000 7f5a21600700 1 -- start start 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.656+0000 7f5a21600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.656+0000 7f5a21600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1c10d430 con 0x7f5a1c108e90 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.656+0000 7f5a1aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.656+0000 7f5a1aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c19ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49708/0 (socket says 172.21.15.1:49708) 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.656+0000 7f5a1aa00700 1 -- 172.21.15.1:0/2327663053 learned_addr learned my addr 172.21.15.1:0/2327663053 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.656+0000 7f5a1aa00700 1 -- 172.21.15.1:0/2327663053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1c19d340 con 0x7f5a1c108e90 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.657+0000 7f5a1aa00700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c19ce00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c011ac0 tx=0x7f5a0c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.657+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a0c003970 con 0x7f5a1c108e90 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.657+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5a0c003b10 con 0x7f5a1c108e90 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.657+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a0c024610 con 0x7f5a1c108e90 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.657+0000 7f5a21600700 1 -- 172.21.15.1:0/2327663053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a1c19d5d0 con 0x7f5a1c108e90 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.657+0000 7f5a21600700 1 -- 172.21.15.1:0/2327663053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a1c1a02c0 con 0x7f5a1c108e90 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.659+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f5a0c01a9e0 con 0x7f5a1c108e90 2024-04-26T21:18:21.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.659+0000 7f5a13400700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 0x7f5a0003bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.659+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 --> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5a0003c230 con 0x7f5a00039660 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.659+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5a0c04d8b0 con 0x7f5a1c108e90 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.659+0000 7f5a1a000700 1 -- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 msgr2=0x7f5a0003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/304032214 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.660+0000 7f5a1a000700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 0x7f5a0003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.861+0000 7f5a1a000700 1 -- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 msgr2=0x7f5a0003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/304032214 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:13.861+0000 7f5a1a000700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 0x7f5a0003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:14.262+0000 7f5a1a000700 1 -- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 msgr2=0x7f5a0003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/304032214 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:14.262+0000 7f5a1a000700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 0x7f5a0003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:15.063+0000 7f5a1a000700 1 -- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 msgr2=0x7f5a0003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/304032214 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:15.063+0000 7f5a1a000700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 0x7f5a0003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:16.665+0000 7f5a1a000700 1 -- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 msgr2=0x7f5a0003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/304032214 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:16.665+0000 7f5a1a000700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 0x7f5a0003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:19.868+0000 7f5a1a000700 1 -- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 msgr2=0x7f5a0003bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/304032214 2024-04-26T21:18:21.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:19.869+0000 7f5a1a000700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 0x7f5a0003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:20.026+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f5a0c02c350 con 0x7f5a1c108e90 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:20.027+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 msgr2=0x7f5a0003bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:20.027+0000 7f5a13400700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 0x7f5a0003bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.031+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f5a0c04f9d0 con 0x7f5a1c108e90 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.031+0000 7f5a13400700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5a0003cf50 0x7f5a0003f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.031+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 --> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5a0c012ea0 con 0x7f5a0003cf50 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.034+0000 7f5a1a000700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5a0003cf50 0x7f5a0003f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.036+0000 7f5a1a000700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5a0003cf50 0x7f5a0003f340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5a08003a10 tx=0x7f5a080092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.038+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 <== mgr.14120 v2:172.21.15.1:6800/3159681858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f5a0c012ea0 con 0x7f5a0003cf50 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.048+0000 7f5a21600700 1 -- 172.21.15.1:0/2327663053 --> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f5a1c19dab0 con 0x7f5a0003cf50 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.049+0000 7f5a13400700 1 -- 172.21.15.1:0/2327663053 <== mgr.14120 v2:172.21.15.1:6800/3159681858 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f5a1c19dab0 con 0x7f5a0003cf50 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.049+0000 7f5a21600700 1 -- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5a0003cf50 msgr2=0x7f5a0003f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.049+0000 7f5a21600700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5a0003cf50 0x7f5a0003f340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5a08003a10 tx=0x7f5a080092b0 comp rx=0 tx=0).stop 2024-04-26T21:18:21.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.049+0000 7f5a21600700 1 -- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 msgr2=0x7f5a1c19ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:21.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.049+0000 7f5a21600700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c19ce00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c011ac0 tx=0x7f5a0c011ba0 comp rx=0 tx=0).stop 2024-04-26T21:18:21.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.050+0000 7f5a21600700 1 -- 172.21.15.1:0/2327663053 shutdown_connections 2024-04-26T21:18:21.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.050+0000 7f5a21600700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5a0003cf50 0x7f5a0003f340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:21.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.050+0000 7f5a21600700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a1c108e90 0x7f5a1c19ce00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:21.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.050+0000 7f5a21600700 1 --2- 172.21.15.1:0/2327663053 >> [v2:172.21.15.1:6800/304032214,v1:172.21.15.1:6801/304032214] conn(0x7f5a00039660 0x7f5a0003bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:21.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.050+0000 7f5a21600700 1 -- 172.21.15.1:0/2327663053 >> 172.21.15.1:0/2327663053 conn(0x7f5a1c104450 msgr2=0x7f5a1c104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:21.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.050+0000 7f5a21600700 1 -- 172.21.15.1:0/2327663053 shutdown_connections 2024-04-26T21:18:21.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:21.050+0000 7f5a21600700 1 -- 172.21.15.1:0/2327663053 wait complete. 2024-04-26T21:18:21.798 INFO:teuthology.orchestra.run.smithi001.stdout:mgr epoch 5 is available 2024-04-26T21:18:21.798 INFO:teuthology.orchestra.run.smithi001.stdout:Setting orchestrator backend to cephadm... 2024-04-26T21:18:22.392 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:22 smithi001 ceph-mon[28054]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T21:18:22.392 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:22 smithi001 ceph-mon[28054]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T21:18:22.392 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:22 smithi001 ceph-mon[28054]: [26/Apr/2024:21:18:21] ENGINE Bus STARTING 2024-04-26T21:18:22.392 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:22 smithi001 ceph-mon[28054]: [26/Apr/2024:21:18:21] ENGINE Serving on https://172.21.15.1:7150 2024-04-26T21:18:22.392 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:22 smithi001 ceph-mon[28054]: [26/Apr/2024:21:18:21] ENGINE Serving on http://172.21.15.1:8765 2024-04-26T21:18:22.392 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:22 smithi001 ceph-mon[28054]: [26/Apr/2024:21:18:21] ENGINE Bus STARTED 2024-04-26T21:18:22.392 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:22 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:18:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:23 smithi001 ceph-mon[28054]: mgrmap e8: x(active, since 2s) 2024-04-26T21:18:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:23 smithi001 ceph-mon[28054]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:18:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:23 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:23 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:18:23.404 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.391+0000 7f2240000700 1 Processor -- start 2024-04-26T21:18:23.405 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.391+0000 7f2240000700 1 -- start start 2024-04-26T21:18:23.405 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.391+0000 7f2240000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223810bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:23.405 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.392+0000 7f2240000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f223807cdd0 con 0x7f2238109970 2024-04-26T21:18:23.405 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.392+0000 7f223d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223810bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:23.405 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.392+0000 7f223d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223810bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53816/0 (socket says 172.21.15.1:53816) 2024-04-26T21:18:23.405 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.392+0000 7f223d600700 1 -- 172.21.15.1:0/161022673 learned_addr learned my addr 172.21.15.1:0/161022673 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:23.405 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.392+0000 7f223d600700 1 -- 172.21.15.1:0/161022673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223810c2a0 con 0x7f2238109970 2024-04-26T21:18:23.405 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.393+0000 7f223d600700 1 --2- 172.21.15.1:0/161022673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223810bd60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f222c0098a0 tx=0x7f222c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab4fddf6e3ae4b50 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:23.406 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.393+0000 7f2237e00700 1 -- 172.21.15.1:0/161022673 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f222c004160 con 0x7f2238109970 2024-04-26T21:18:23.406 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.393+0000 7f2237e00700 1 -- 172.21.15.1:0/161022673 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f222c004300 con 0x7f2238109970 2024-04-26T21:18:23.406 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.393+0000 7f2237e00700 1 -- 172.21.15.1:0/161022673 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f222c011af0 con 0x7f2238109970 2024-04-26T21:18:23.406 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.394+0000 7f2240000700 1 -- 172.21.15.1:0/161022673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 msgr2=0x7f223810bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:23.406 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.394+0000 7f2240000700 1 --2- 172.21.15.1:0/161022673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223810bd60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f222c0098a0 tx=0x7f222c009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:23.406 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.394+0000 7f2240000700 1 -- 172.21.15.1:0/161022673 shutdown_connections 2024-04-26T21:18:23.406 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.394+0000 7f2240000700 1 --2- 172.21.15.1:0/161022673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223810bd60 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:23.406 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.394+0000 7f2240000700 1 -- 172.21.15.1:0/161022673 >> 172.21.15.1:0/161022673 conn(0x7f2238103340 msgr2=0x7f2238105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:23.407 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.394+0000 7f2240000700 1 -- 172.21.15.1:0/161022673 shutdown_connections 2024-04-26T21:18:23.407 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.395+0000 7f2240000700 1 -- 172.21.15.1:0/161022673 wait complete. 2024-04-26T21:18:23.407 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.396+0000 7f2240000700 1 Processor -- start 2024-04-26T21:18:23.407 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.396+0000 7f2240000700 1 -- start start 2024-04-26T21:18:23.407 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.396+0000 7f2240000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:23.407 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.396+0000 7f2240000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f223810d200 con 0x7f2238109970 2024-04-26T21:18:23.407 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.396+0000 7f223d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:23.407 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.396+0000 7f223d600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223819ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53830/0 (socket says 172.21.15.1:53830) 2024-04-26T21:18:23.407 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.396+0000 7f223d600700 1 -- 172.21.15.1:0/4058746868 learned_addr learned my addr 172.21.15.1:0/4058746868 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:23.408 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.397+0000 7f223d600700 1 -- 172.21.15.1:0/4058746868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223819f4b0 con 0x7f2238109970 2024-04-26T21:18:23.408 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.397+0000 7f223d600700 1 --2- 172.21.15.1:0/4058746868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223819ef70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f222c011ac0 tx=0x7f222c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:23.408 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.397+0000 7f2236000700 1 -- 172.21.15.1:0/4058746868 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f222c003b20 con 0x7f2238109970 2024-04-26T21:18:23.408 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.397+0000 7f2236000700 1 -- 172.21.15.1:0/4058746868 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f222c01a8c0 con 0x7f2238109970 2024-04-26T21:18:23.408 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.397+0000 7f2240000700 1 -- 172.21.15.1:0/4058746868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223819f740 con 0x7f2238109970 2024-04-26T21:18:23.408 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.397+0000 7f2236000700 1 -- 172.21.15.1:0/4058746868 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f222c0248d0 con 0x7f2238109970 2024-04-26T21:18:23.408 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.398+0000 7f2240000700 1 -- 172.21.15.1:0/4058746868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22381a2430 con 0x7f2238109970 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.399+0000 7f2236000700 1 -- 172.21.15.1:0/4058746868 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f222c022020 con 0x7f2238109970 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.399+0000 7f2240000700 1 -- 172.21.15.1:0/4058746868 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2238064a80 con 0x7f2238109970 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.399+0000 7f2236000700 1 --2- 172.21.15.1:0/4058746868 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f2220039600 0x7f222003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.399+0000 7f2236000700 1 -- 172.21.15.1:0/4058746868 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f222c056760 con 0x7f2238109970 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.399+0000 7f223cc00700 1 --2- 172.21.15.1:0/4058746868 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f2220039600 0x7f222003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.400+0000 7f223cc00700 1 --2- 172.21.15.1:0/4058746868 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f2220039600 0x7f222003bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2224006fd0 tx=0x7f2224006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.405+0000 7f2236000700 1 -- 172.21.15.1:0/4058746868 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f222c0172b0 con 0x7f2238109970 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.653+0000 7f2240000700 1 -- 172.21.15.1:0/4058746868 --> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f2238104460 con 0x7f2220039600 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.668+0000 7f2236000700 1 -- 172.21.15.1:0/4058746868 <== mgr.14120 v2:172.21.15.1:6800/3159681858 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2238104460 con 0x7f2220039600 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.670+0000 7f2240000700 1 -- 172.21.15.1:0/4058746868 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f2220039600 msgr2=0x7f222003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.670+0000 7f2240000700 1 --2- 172.21.15.1:0/4058746868 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f2220039600 0x7f222003bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2224006fd0 tx=0x7f2224006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.670+0000 7f2240000700 1 -- 172.21.15.1:0/4058746868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 msgr2=0x7f223819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:23.409 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.670+0000 7f2240000700 1 --2- 172.21.15.1:0/4058746868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223819ef70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f222c011ac0 tx=0x7f222c011ba0 comp rx=0 tx=0).stop 2024-04-26T21:18:23.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.670+0000 7f2240000700 1 -- 172.21.15.1:0/4058746868 shutdown_connections 2024-04-26T21:18:23.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.671+0000 7f2240000700 1 --2- 172.21.15.1:0/4058746868 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f2220039600 0x7f222003bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:23.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.671+0000 7f2240000700 1 --2- 172.21.15.1:0/4058746868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2238109970 0x7f223819ef70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:23.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.671+0000 7f2240000700 1 -- 172.21.15.1:0/4058746868 >> 172.21.15.1:0/4058746868 conn(0x7f2238103340 msgr2=0x7f2238103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:23.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.671+0000 7f2240000700 1 -- 172.21.15.1:0/4058746868 shutdown_connections 2024-04-26T21:18:23.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:22.671+0000 7f2240000700 1 -- 172.21.15.1:0/4058746868 wait complete. 2024-04-26T21:18:24.951 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T21:18:24.951 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.039+0000 7fe09a000700 1 Processor -- start 2024-04-26T21:18:24.951 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.039+0000 7fe09a000700 1 -- start start 2024-04-26T21:18:24.951 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.040+0000 7fe09a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09410b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:24.951 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.040+0000 7fe09a000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe09410ba40 con 0x7fe09410b050 2024-04-26T21:18:24.951 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.040+0000 7fe093400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09410b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.040+0000 7fe093400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09410b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53832/0 (socket says 172.21.15.1:53832) 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.040+0000 7fe093400700 1 -- 172.21.15.1:0/3155846905 learned_addr learned my addr 172.21.15.1:0/3155846905 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.041+0000 7fe093400700 1 -- 172.21.15.1:0/3155846905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09410c250 con 0x7fe09410b050 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.041+0000 7fe093400700 1 --2- 172.21.15.1:0/3155846905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09410b470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe0840098a0 tx=0x7fe084009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10096397f0ade40a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.041+0000 7fe092000700 1 -- 172.21.15.1:0/3155846905 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe084004160 con 0x7fe09410b050 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.041+0000 7fe092000700 1 -- 172.21.15.1:0/3155846905 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe084004300 con 0x7fe09410b050 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.041+0000 7fe092000700 1 -- 172.21.15.1:0/3155846905 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe084011af0 con 0x7fe09410b050 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.042+0000 7fe09a000700 1 -- 172.21.15.1:0/3155846905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 msgr2=0x7fe09410b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.042+0000 7fe09a000700 1 --2- 172.21.15.1:0/3155846905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09410b470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe0840098a0 tx=0x7fe084009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.042+0000 7fe09a000700 1 -- 172.21.15.1:0/3155846905 shutdown_connections 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.042+0000 7fe09a000700 1 --2- 172.21.15.1:0/3155846905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09410b470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.042+0000 7fe09a000700 1 -- 172.21.15.1:0/3155846905 >> 172.21.15.1:0/3155846905 conn(0x7fe09407e430 msgr2=0x7fe09407e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:24.952 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.043+0000 7fe09a000700 1 -- 172.21.15.1:0/3155846905 shutdown_connections 2024-04-26T21:18:24.955 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.043+0000 7fe09a000700 1 -- 172.21.15.1:0/3155846905 wait complete. 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.044+0000 7fe09a000700 1 Processor -- start 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.044+0000 7fe09a000700 1 -- start start 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.044+0000 7fe09a000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.044+0000 7fe09a000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe09410f600 con 0x7fe09410b050 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.044+0000 7fe093400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.044+0000 7fe093400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09419ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53834/0 (socket says 172.21.15.1:53834) 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.044+0000 7fe093400700 1 -- 172.21.15.1:0/1166564191 learned_addr learned my addr 172.21.15.1:0/1166564191 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.045+0000 7fe093400700 1 -- 172.21.15.1:0/1166564191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09419f4b0 con 0x7fe09410b050 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.045+0000 7fe093400700 1 --2- 172.21.15.1:0/1166564191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09419ef70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe084000c00 tx=0x7fe084011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.045+0000 7fe08be00700 1 -- 172.21.15.1:0/1166564191 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe084003b20 con 0x7fe09410b050 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.046+0000 7fe08be00700 1 -- 172.21.15.1:0/1166564191 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe08401a8c0 con 0x7fe09410b050 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.046+0000 7fe08be00700 1 -- 172.21.15.1:0/1166564191 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe084024940 con 0x7fe09410b050 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.046+0000 7fe09a000700 1 -- 172.21.15.1:0/1166564191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe09419f740 con 0x7fe09410b050 2024-04-26T21:18:24.956 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.046+0000 7fe09a000700 1 -- 172.21.15.1:0/1166564191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0941a2430 con 0x7fe09410b050 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.047+0000 7fe09a000700 1 -- 172.21.15.1:0/1166564191 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe094064a80 con 0x7fe09410b050 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.047+0000 7fe08be00700 1 -- 172.21.15.1:0/1166564191 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe084022020 con 0x7fe09410b050 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.048+0000 7fe08be00700 1 --2- 172.21.15.1:0/1166564191 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fe078039600 0x7fe07803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.048+0000 7fe08be00700 1 -- 172.21.15.1:0/1166564191 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe08404d7d0 con 0x7fe09410b050 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.048+0000 7fe092a00700 1 --2- 172.21.15.1:0/1166564191 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fe078039600 0x7fe07803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.049+0000 7fe092a00700 1 --2- 172.21.15.1:0/1166564191 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fe078039600 0x7fe07803bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe080006fd0 tx=0x7fe080006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.054+0000 7fe08be00700 1 -- 172.21.15.1:0/1166564191 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe08402c3a0 con 0x7fe09410b050 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.284+0000 7fe09a000700 1 -- 172.21.15.1:0/1166564191 --> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fe094108b00 con 0x7fe078039600 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.286+0000 7fe08be00700 1 -- 172.21.15.1:0/1166564191 <== mgr.14120 v2:172.21.15.1:6800/3159681858 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fe094108b00 con 0x7fe078039600 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.288+0000 7fe09a000700 1 -- 172.21.15.1:0/1166564191 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fe078039600 msgr2=0x7fe07803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.288+0000 7fe09a000700 1 --2- 172.21.15.1:0/1166564191 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fe078039600 0x7fe07803bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe080006fd0 tx=0x7fe080006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.289+0000 7fe09a000700 1 -- 172.21.15.1:0/1166564191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 msgr2=0x7fe09419ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.289+0000 7fe09a000700 1 --2- 172.21.15.1:0/1166564191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09419ef70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe084000c00 tx=0x7fe084011b70 comp rx=0 tx=0).stop 2024-04-26T21:18:24.957 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.289+0000 7fe09a000700 1 -- 172.21.15.1:0/1166564191 shutdown_connections 2024-04-26T21:18:24.958 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.289+0000 7fe09a000700 1 --2- 172.21.15.1:0/1166564191 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fe078039600 0x7fe07803bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:24.958 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.289+0000 7fe09a000700 1 --2- 172.21.15.1:0/1166564191 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe09410b050 0x7fe09419ef70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:24.958 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.289+0000 7fe09a000700 1 -- 172.21.15.1:0/1166564191 >> 172.21.15.1:0/1166564191 conn(0x7fe09407e430 msgr2=0x7fe0941083f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:24.958 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.289+0000 7fe09a000700 1 -- 172.21.15.1:0/1166564191 shutdown_connections 2024-04-26T21:18:24.958 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:24.289+0000 7fe09a000700 1 -- 172.21.15.1:0/1166564191 wait complete. 2024-04-26T21:18:24.958 INFO:teuthology.orchestra.run.smithi001.stdout:Generating ssh key... 2024-04-26T21:18:25.292 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-mon[28054]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:18:26.258 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: Generating public/private rsa key pair. 2024-04-26T21:18:26.258 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: Your identification has been saved in /tmp/tmp8ijcpiik/key. 2024-04-26T21:18:26.258 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: Your public key has been saved in /tmp/tmp8ijcpiik/key.pub. 2024-04-26T21:18:26.258 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: The key fingerprint is: 2024-04-26T21:18:26.258 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: SHA256:B85+Jnw+YBdJ1b/jqoo2sperKa7Fp8MJ/MfBrTJKyBo ceph-44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:18:26.258 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: The key's randomart image is: 2024-04-26T21:18:26.258 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: +---[RSA 3072]----+ 2024-04-26T21:18:26.258 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: | ... | 2024-04-26T21:18:26.258 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: | . . | 2024-04-26T21:18:26.258 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: | .. . . | 2024-04-26T21:18:26.259 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: | o .o .| 2024-04-26T21:18:26.259 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: |. . . S .. .| 2024-04-26T21:18:26.259 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: |oo. o +o.. o | 2024-04-26T21:18:26.259 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: |Eo+oo.o.=o+ . .| 2024-04-26T21:18:26.259 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: |o..Oo=.=.*. . | 2024-04-26T21:18:26.259 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: |.o+o*+*oo.oo... | 2024-04-26T21:18:26.259 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: +----[SHA256]-----+ 2024-04-26T21:18:26.615 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.574+0000 7f1659400700 1 Processor -- start 2024-04-26T21:18:26.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.574+0000 7f1659400700 1 -- start start 2024-04-26T21:18:26.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.575+0000 7f1659400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165407d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:26.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.575+0000 7f1659400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f165407d850 con 0x7f165407ee20 2024-04-26T21:18:26.616 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.575+0000 7f1652800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165407d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.575+0000 7f1652800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165407d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53838/0 (socket says 172.21.15.1:53838) 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.575+0000 7f1652800700 1 -- 172.21.15.1:0/544213369 learned_addr learned my addr 172.21.15.1:0/544213369 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.576+0000 7f1652800700 1 -- 172.21.15.1:0/544213369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165407d9d0 con 0x7f165407ee20 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.576+0000 7f1652800700 1 --2- 172.21.15.1:0/544213369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165407d280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16440098a0 tx=0x7f1644009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c3649e9ef98bc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.576+0000 7f1651400700 1 -- 172.21.15.1:0/544213369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1644004160 con 0x7f165407ee20 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.576+0000 7f1651400700 1 -- 172.21.15.1:0/544213369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1644004300 con 0x7f165407ee20 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.576+0000 7f1651400700 1 -- 172.21.15.1:0/544213369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1644011af0 con 0x7f165407ee20 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.577+0000 7f1659400700 1 -- 172.21.15.1:0/544213369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 msgr2=0x7f165407d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.577+0000 7f1659400700 1 --2- 172.21.15.1:0/544213369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165407d280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f16440098a0 tx=0x7f1644009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.577+0000 7f1659400700 1 -- 172.21.15.1:0/544213369 shutdown_connections 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.577+0000 7f1659400700 1 --2- 172.21.15.1:0/544213369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165407d280 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:26.620 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.577+0000 7f1659400700 1 -- 172.21.15.1:0/544213369 >> 172.21.15.1:0/544213369 conn(0x7f1654104450 msgr2=0x7f1654106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.577+0000 7f1659400700 1 -- 172.21.15.1:0/544213369 shutdown_connections 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.577+0000 7f1659400700 1 -- 172.21.15.1:0/544213369 wait complete. 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.578+0000 7f1659400700 1 Processor -- start 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.579+0000 7f1659400700 1 -- start start 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.579+0000 7f1659400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.579+0000 7f1659400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f165410f460 con 0x7f165407ee20 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.579+0000 7f1652800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.579+0000 7f1652800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165419ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53842/0 (socket says 172.21.15.1:53842) 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.579+0000 7f1652800700 1 -- 172.21.15.1:0/580860366 learned_addr learned my addr 172.21.15.1:0/580860366 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.579+0000 7f1652800700 1 -- 172.21.15.1:0/580860366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165419f4d0 con 0x7f165407ee20 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.580+0000 7f1652800700 1 --2- 172.21.15.1:0/580860366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165419ef90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1644011ac0 tx=0x7f1644011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.580+0000 7f164b400700 1 -- 172.21.15.1:0/580860366 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1644003970 con 0x7f165407ee20 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.580+0000 7f164b400700 1 -- 172.21.15.1:0/580860366 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1644003b10 con 0x7f165407ee20 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.580+0000 7f164b400700 1 -- 172.21.15.1:0/580860366 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1644024610 con 0x7f165407ee20 2024-04-26T21:18:26.621 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.580+0000 7f1659400700 1 -- 172.21.15.1:0/580860366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f165419f760 con 0x7f165407ee20 2024-04-26T21:18:26.622 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.580+0000 7f1659400700 1 -- 172.21.15.1:0/580860366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16541a2450 con 0x7f165407ee20 2024-04-26T21:18:26.622 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.581+0000 7f164b400700 1 -- 172.21.15.1:0/580860366 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f164401a9e0 con 0x7f165407ee20 2024-04-26T21:18:26.622 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.581+0000 7f1659400700 1 -- 172.21.15.1:0/580860366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1654064a80 con 0x7f165407ee20 2024-04-26T21:18:26.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.582+0000 7f164b400700 1 --2- 172.21.15.1:0/580860366 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f1638039600 0x7f163803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:26.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.582+0000 7f164b400700 1 -- 172.21.15.1:0/580860366 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f164404d8c0 con 0x7f165407ee20 2024-04-26T21:18:26.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.582+0000 7f1651e00700 1 --2- 172.21.15.1:0/580860366 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f1638039600 0x7f163803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:26.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.583+0000 7f1651e00700 1 --2- 172.21.15.1:0/580860366 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f1638039600 0x7f163803bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1640006fd0 tx=0x7f1640006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:26.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.587+0000 7f164b400700 1 -- 172.21.15.1:0/580860366 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f164400b3e0 con 0x7f165407ee20 2024-04-26T21:18:26.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.811+0000 7f1659400700 1 -- 172.21.15.1:0/580860366 --> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f1654105650 con 0x7f1638039600 2024-04-26T21:18:26.624 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.903+0000 7f164b400700 1 -- 172.21.15.1:0/580860366 <== mgr.14120 v2:172.21.15.1:6800/3159681858 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1654105650 con 0x7f1638039600 2024-04-26T21:18:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.906+0000 7f1659400700 1 -- 172.21.15.1:0/580860366 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f1638039600 msgr2=0x7f163803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.906+0000 7f1659400700 1 --2- 172.21.15.1:0/580860366 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f1638039600 0x7f163803bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1640006fd0 tx=0x7f1640006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.906+0000 7f1659400700 1 -- 172.21.15.1:0/580860366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 msgr2=0x7f165419ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.906+0000 7f1659400700 1 --2- 172.21.15.1:0/580860366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165419ef90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1644011ac0 tx=0x7f1644011ba0 comp rx=0 tx=0).stop 2024-04-26T21:18:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.907+0000 7f1659400700 1 -- 172.21.15.1:0/580860366 shutdown_connections 2024-04-26T21:18:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.907+0000 7f1659400700 1 --2- 172.21.15.1:0/580860366 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f1638039600 0x7f163803bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.907+0000 7f1659400700 1 --2- 172.21.15.1:0/580860366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f165407ee20 0x7f165419ef90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.907+0000 7f1659400700 1 -- 172.21.15.1:0/580860366 >> 172.21.15.1:0/580860366 conn(0x7f1654104450 msgr2=0x7f1654104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.907+0000 7f1659400700 1 -- 172.21.15.1:0/580860366 shutdown_connections 2024-04-26T21:18:26.625 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:25.907+0000 7f1659400700 1 -- 172.21.15.1:0/580860366 wait complete. 2024-04-26T21:18:27.172 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:26 smithi001 ceph-mon[28054]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:18:27.173 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:26 smithi001 ceph-mon[28054]: Generating ssh key... 2024-04-26T21:18:27.173 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:26 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:27.173 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:26 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:28.103 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:28 smithi001 ceph-mon[28054]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:18:28.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCePt3ZKYGfsJpFkEgl6aRIKOdWMdEvJyJbwOefZ4nno7pSW2D3F1hKbgwvuuYk1lCyJjgseVOa9NWmWdEca54tzKrCVlva9tk7EbhjkTNGHPKID58mJotEku2LIFqoCNfxvzVroj2+mNyKaIKctSnS1dl/U6pGrWDQQyfObCXDiZ+9oYPTLG8tlCE92JBz+q4KXKAqyxqNqHqjhUJDOnw1EXInSUcFNMd15AZpc1Fi/lXycteCcBMXK24zyuUiu1vwvgDvuHhFRKGbJo7O1pnPeUVuGA0+iby0qkPNBVxwuXMeYMNyJOqTzYwm8qBS5UUMWaEjUt62RpCFllD+8I8cw9DdXgfn4O7YqX7WMpQ14GsefERhgvYPv4s+h+9rUgFXOJsolkCzHLnvmTwsRWTOOBMS1TN0Ku6uknmfaYWnxEFvDCWE9aiD8rwxjjKPTHzPZmRn7DDFuU+N3zLXXKv2109sp/QTiKb+gSrTNjD9VVneIEkilh6MsiIh2RsxtrU= ceph-44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:18:28.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.254+0000 7f924f200700 1 Processor -- start 2024-04-26T21:18:28.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.254+0000 7f924f200700 1 -- start start 2024-04-26T21:18:28.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.255+0000 7f924f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f9248112060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:28.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.255+0000 7f924f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9248112630 con 0x7f924810fc70 2024-04-26T21:18:28.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.255+0000 7f9247e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f9248112060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:28.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.255+0000 7f9247e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f9248112060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53844/0 (socket says 172.21.15.1:53844) 2024-04-26T21:18:28.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.255+0000 7f9247e00700 1 -- 172.21.15.1:0/1930863656 learned_addr learned my addr 172.21.15.1:0/1930863656 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:28.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.256+0000 7f9247e00700 1 -- 172.21.15.1:0/1930863656 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92481127b0 con 0x7f924810fc70 2024-04-26T21:18:28.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.256+0000 7f9247e00700 1 --2- 172.21.15.1:0/1930863656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f9248112060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f92380098a0 tx=0x7f9238009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f711c4430294d3ef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.256+0000 7f9247400700 1 -- 172.21.15.1:0/1930863656 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9238004160 con 0x7f924810fc70 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.256+0000 7f9247400700 1 -- 172.21.15.1:0/1930863656 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9238004300 con 0x7f924810fc70 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.256+0000 7f9247400700 1 -- 172.21.15.1:0/1930863656 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9238011af0 con 0x7f924810fc70 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.257+0000 7f924f200700 1 -- 172.21.15.1:0/1930863656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 msgr2=0x7f9248112060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.257+0000 7f924f200700 1 --2- 172.21.15.1:0/1930863656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f9248112060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f92380098a0 tx=0x7f9238009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.257+0000 7f924f200700 1 -- 172.21.15.1:0/1930863656 shutdown_connections 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.257+0000 7f924f200700 1 --2- 172.21.15.1:0/1930863656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f9248112060 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.257+0000 7f924f200700 1 -- 172.21.15.1:0/1930863656 >> 172.21.15.1:0/1930863656 conn(0x7f924807e460 msgr2=0x7f9248080880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.257+0000 7f924f200700 1 -- 172.21.15.1:0/1930863656 shutdown_connections 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.258+0000 7f924f200700 1 -- 172.21.15.1:0/1930863656 wait complete. 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.259+0000 7f924f200700 1 Processor -- start 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.259+0000 7f924f200700 1 -- start start 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.259+0000 7f924f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f92481a3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:28.416 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.259+0000 7f924f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9248113760 con 0x7f924810fc70 2024-04-26T21:18:28.417 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.259+0000 7f9247e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f92481a3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:28.417 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.259+0000 7f9247e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f92481a3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53848/0 (socket says 172.21.15.1:53848) 2024-04-26T21:18:28.417 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.259+0000 7f9247e00700 1 -- 172.21.15.1:0/1325690079 learned_addr learned my addr 172.21.15.1:0/1325690079 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:28.417 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.260+0000 7f9247e00700 1 -- 172.21.15.1:0/1325690079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92481a37c0 con 0x7f924810fc70 2024-04-26T21:18:28.417 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.260+0000 7f9247e00700 1 --2- 172.21.15.1:0/1325690079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f92481a3280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9238011ac0 tx=0x7f9238011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:28.417 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.260+0000 7f9245600700 1 -- 172.21.15.1:0/1325690079 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9238003b20 con 0x7f924810fc70 2024-04-26T21:18:28.417 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.260+0000 7f9245600700 1 -- 172.21.15.1:0/1325690079 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f923801a8c0 con 0x7f924810fc70 2024-04-26T21:18:28.418 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.260+0000 7f9245600700 1 -- 172.21.15.1:0/1325690079 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92380248d0 con 0x7f924810fc70 2024-04-26T21:18:28.418 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.261+0000 7f924f200700 1 -- 172.21.15.1:0/1325690079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92481a3ab0 con 0x7f924810fc70 2024-04-26T21:18:28.418 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.261+0000 7f924f200700 1 -- 172.21.15.1:0/1325690079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92481a6740 con 0x7f924810fc70 2024-04-26T21:18:28.418 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.262+0000 7f924f200700 1 -- 172.21.15.1:0/1325690079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9248064a80 con 0x7f924810fc70 2024-04-26T21:18:28.418 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.262+0000 7f9245600700 1 -- 172.21.15.1:0/1325690079 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f9238022020 con 0x7f924810fc70 2024-04-26T21:18:28.418 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.263+0000 7f9245600700 1 --2- 172.21.15.1:0/1325690079 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f922c039600 0x7f922c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:28.418 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.263+0000 7f9245600700 1 -- 172.21.15.1:0/1325690079 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f923804d8b0 con 0x7f924810fc70 2024-04-26T21:18:28.419 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.263+0000 7f923f400700 1 --2- 172.21.15.1:0/1325690079 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f922c039600 0x7f922c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:28.422 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.264+0000 7f923f400700 1 --2- 172.21.15.1:0/1325690079 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f922c039600 0x7f922c03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9230006fd0 tx=0x7f9230006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:28.422 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.268+0000 7f9245600700 1 -- 172.21.15.1:0/1325690079 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f923801ab20 con 0x7f924810fc70 2024-04-26T21:18:28.423 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.498+0000 7f924f200700 1 -- 172.21.15.1:0/1325690079 --> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f924807f610 con 0x7f922c039600 2024-04-26T21:18:28.423 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.499+0000 7f9245600700 1 -- 172.21.15.1:0/1325690079 <== mgr.14120 v2:172.21.15.1:6800/3159681858 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f924807f610 con 0x7f922c039600 2024-04-26T21:18:28.423 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.502+0000 7f924f200700 1 -- 172.21.15.1:0/1325690079 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f922c039600 msgr2=0x7f922c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:28.423 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.502+0000 7f924f200700 1 --2- 172.21.15.1:0/1325690079 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f922c039600 0x7f922c03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9230006fd0 tx=0x7f9230006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:28.423 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.502+0000 7f924f200700 1 -- 172.21.15.1:0/1325690079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 msgr2=0x7f92481a3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:28.423 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.502+0000 7f924f200700 1 --2- 172.21.15.1:0/1325690079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f92481a3280 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9238011ac0 tx=0x7f9238011ba0 comp rx=0 tx=0).stop 2024-04-26T21:18:28.423 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.502+0000 7f924f200700 1 -- 172.21.15.1:0/1325690079 shutdown_connections 2024-04-26T21:18:28.424 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.502+0000 7f924f200700 1 --2- 172.21.15.1:0/1325690079 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f922c039600 0x7f922c03bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:28.424 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.502+0000 7f924f200700 1 --2- 172.21.15.1:0/1325690079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f924810fc70 0x7f92481a3280 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:28.424 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.502+0000 7f924f200700 1 -- 172.21.15.1:0/1325690079 >> 172.21.15.1:0/1325690079 conn(0x7f924807e460 msgr2=0x7f924807ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:28.424 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.503+0000 7f924f200700 1 -- 172.21.15.1:0/1325690079 shutdown_connections 2024-04-26T21:18:28.424 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:27.503+0000 7f924f200700 1 -- 172.21.15.1:0/1325690079 wait complete. 2024-04-26T21:18:28.424 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T21:18:28.425 INFO:teuthology.orchestra.run.smithi001.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T21:18:28.429 INFO:teuthology.orchestra.run.smithi001.stdout:Adding host smithi001... 2024-04-26T21:18:30.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:30 smithi001 ceph-mon[28054]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi001", "addr": "172.21.15.1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:18:31.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:31 smithi001 ceph-mon[28054]: Deploying cephadm binary to smithi001 2024-04-26T21:18:36.630 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Added host 'smithi001' with addr '172.21.15.1' 2024-04-26T21:18:36.630 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.087+0000 7f3178400700 1 Processor -- start 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.087+0000 7f3178400700 1 -- start start 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.087+0000 7f3178400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317010b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.087+0000 7f3178400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f317010ba40 con 0x7f317010b050 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.088+0000 7f3175a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317010b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.088+0000 7f3175a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317010b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33864/0 (socket says 172.21.15.1:33864) 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.088+0000 7f3175a00700 1 -- 172.21.15.1:0/370661904 learned_addr learned my addr 172.21.15.1:0/370661904 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.088+0000 7f3175a00700 1 -- 172.21.15.1:0/370661904 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317010c250 con 0x7f317010b050 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.089+0000 7f3175a00700 1 --2- 172.21.15.1:0/370661904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317010b470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3164009e80 tx=0x7f316400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ece32da6bdbdf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.089+0000 7f316fe00700 1 -- 172.21.15.1:0/370661904 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3164017070 con 0x7f317010b050 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.089+0000 7f316fe00700 1 -- 172.21.15.1:0/370661904 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f316400b8c0 con 0x7f317010b050 2024-04-26T21:18:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.089+0000 7f316fe00700 1 -- 172.21.15.1:0/370661904 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3164004490 con 0x7f317010b050 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.089+0000 7f3178400700 1 -- 172.21.15.1:0/370661904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 msgr2=0x7f317010b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.089+0000 7f3178400700 1 --2- 172.21.15.1:0/370661904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317010b470 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3164009e80 tx=0x7f316400b330 comp rx=0 tx=0).stop 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.090+0000 7f3178400700 1 -- 172.21.15.1:0/370661904 shutdown_connections 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.090+0000 7f3178400700 1 --2- 172.21.15.1:0/370661904 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317010b470 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.090+0000 7f3178400700 1 -- 172.21.15.1:0/370661904 >> 172.21.15.1:0/370661904 conn(0x7f317007e430 msgr2=0x7f317007e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.090+0000 7f3178400700 1 -- 172.21.15.1:0/370661904 shutdown_connections 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.090+0000 7f3178400700 1 -- 172.21.15.1:0/370661904 wait complete. 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.091+0000 7f3178400700 1 Processor -- start 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.091+0000 7f3178400700 1 -- start start 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.091+0000 7f3178400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.091+0000 7f3178400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f317010f600 con 0x7f317010b050 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.092+0000 7f3175a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:36.632 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.092+0000 7f3175a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317019eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33880/0 (socket says 172.21.15.1:33880) 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.092+0000 7f3175a00700 1 -- 172.21.15.1:0/3525607996 learned_addr learned my addr 172.21.15.1:0/3525607996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.092+0000 7f3175a00700 1 -- 172.21.15.1:0/3525607996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317019f430 con 0x7f317010b050 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.092+0000 7f3175a00700 1 --2- 172.21.15.1:0/3525607996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317019eef0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3164004a60 tx=0x7f316400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.093+0000 7f316e000700 1 -- 172.21.15.1:0/3525607996 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3164020020 con 0x7f317010b050 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.093+0000 7f316e000700 1 -- 172.21.15.1:0/3525607996 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3164003c60 con 0x7f317010b050 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.093+0000 7f3178400700 1 -- 172.21.15.1:0/3525607996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f317019f6c0 con 0x7f317010b050 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.093+0000 7f316e000700 1 -- 172.21.15.1:0/3525607996 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3164017040 con 0x7f317010b050 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.093+0000 7f3178400700 1 -- 172.21.15.1:0/3525607996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31701a23b0 con 0x7f317010b050 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.094+0000 7f316e000700 1 -- 172.21.15.1:0/3525607996 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3164023020 con 0x7f317010b050 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.094+0000 7f3178400700 1 -- 172.21.15.1:0/3525607996 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3170064a80 con 0x7f317010b050 2024-04-26T21:18:36.633 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.095+0000 7f316e000700 1 --2- 172.21.15.1:0/3525607996 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3158039600 0x7f315803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.095+0000 7f316e000700 1 -- 172.21.15.1:0/3525607996 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f316400f050 con 0x7f317010b050 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.095+0000 7f3175000700 1 --2- 172.21.15.1:0/3525607996 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3158039600 0x7f315803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.096+0000 7f3175000700 1 --2- 172.21.15.1:0/3525607996 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3158039600 0x7f315803bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f315c006fd0 tx=0x7f315c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.100+0000 7f316e000700 1 -- 172.21.15.1:0/3525607996 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f316402a020 con 0x7f317010b050 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:29.334+0000 7f3178400700 1 -- 172.21.15.1:0/3525607996 --> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi001", "addr": "172.21.15.1", "target": ["mon-mgr", ""]}) -- 0x7f3170108a50 con 0x7f3158039600 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.916+0000 7f316e000700 1 -- 172.21.15.1:0/3525607996 <== mgr.14120 v2:172.21.15.1:6800/3159681858 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+47 (secure 0 0 0) 0x7f3170108a50 con 0x7f3158039600 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.920+0000 7f3178400700 1 -- 172.21.15.1:0/3525607996 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3158039600 msgr2=0x7f315803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.920+0000 7f3178400700 1 --2- 172.21.15.1:0/3525607996 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3158039600 0x7f315803bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f315c006fd0 tx=0x7f315c006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.920+0000 7f3178400700 1 -- 172.21.15.1:0/3525607996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 msgr2=0x7f317019eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.920+0000 7f3178400700 1 --2- 172.21.15.1:0/3525607996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317019eef0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3164004a60 tx=0x7f316400bd50 comp rx=0 tx=0).stop 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.921+0000 7f3178400700 1 -- 172.21.15.1:0/3525607996 shutdown_connections 2024-04-26T21:18:36.634 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.921+0000 7f3178400700 1 --2- 172.21.15.1:0/3525607996 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3158039600 0x7f315803bac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:36.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.921+0000 7f3178400700 1 --2- 172.21.15.1:0/3525607996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f317010b050 0x7f317019eef0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:36.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.921+0000 7f3178400700 1 -- 172.21.15.1:0/3525607996 >> 172.21.15.1:0/3525607996 conn(0x7f317007e430 msgr2=0x7f3170108340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:36.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.921+0000 7f3178400700 1 -- 172.21.15.1:0/3525607996 shutdown_connections 2024-04-26T21:18:36.635 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:35.921+0000 7f3178400700 1 -- 172.21.15.1:0/3525607996 wait complete. 2024-04-26T21:18:36.635 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying unmanaged mon service... 2024-04-26T21:18:37.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:36 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:37.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:36 smithi001 ceph-mon[28054]: Added host smithi001 2024-04-26T21:18:37.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:36 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:18:38.858 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.379+0000 7fafc4200700 1 Processor -- start 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.379+0000 7fafc4200700 1 -- start start 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.380+0000 7fafc4200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1076e0 0x7fafbc109b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.380+0000 7fafc4200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fafbc07ce50 con 0x7fafbc1076e0 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.380+0000 7fafc1800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1076e0 0x7fafbc109b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.380+0000 7fafc1800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1076e0 0x7fafbc109b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33888/0 (socket says 172.21.15.1:33888) 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.380+0000 7fafc1800700 1 -- 172.21.15.1:0/3761595050 learned_addr learned my addr 172.21.15.1:0/3761595050 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.381+0000 7fafc1800700 1 -- 172.21.15.1:0/3761595050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafbc10a050 con 0x7fafbc1076e0 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.381+0000 7fafc1800700 1 --2- 172.21.15.1:0/3761595050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1076e0 0x7fafbc109b10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fafb00098a0 tx=0x7fafb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a06b3bd93fbca3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.381+0000 7fafbbe00700 1 -- 172.21.15.1:0/3761595050 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafb0004160 con 0x7fafbc1076e0 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.381+0000 7fafbbe00700 1 -- 172.21.15.1:0/3761595050 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fafb0004300 con 0x7fafbc1076e0 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.381+0000 7fafbbe00700 1 -- 172.21.15.1:0/3761595050 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafb0011af0 con 0x7fafbc1076e0 2024-04-26T21:18:38.859 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.382+0000 7fafc4200700 1 -- 172.21.15.1:0/3761595050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1076e0 msgr2=0x7fafbc109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.382+0000 7fafc4200700 1 --2- 172.21.15.1:0/3761595050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1076e0 0x7fafbc109b10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fafb00098a0 tx=0x7fafb0009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.382+0000 7fafc4200700 1 -- 172.21.15.1:0/3761595050 shutdown_connections 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.382+0000 7fafc4200700 1 --2- 172.21.15.1:0/3761595050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1076e0 0x7fafbc109b10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.382+0000 7fafc4200700 1 -- 172.21.15.1:0/3761595050 >> 172.21.15.1:0/3761595050 conn(0x7fafbc103340 msgr2=0x7fafbc105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.383+0000 7fafc4200700 1 -- 172.21.15.1:0/3761595050 shutdown_connections 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.383+0000 7fafc4200700 1 -- 172.21.15.1:0/3761595050 wait complete. 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.384+0000 7fafc4200700 1 Processor -- start 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.384+0000 7fafc4200700 1 -- start start 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.385+0000 7fafc4200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1a34a0 0x7fafbc1a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.385+0000 7fafc4200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fafbc10b080 con 0x7fafbc1a34a0 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.385+0000 7fafc1800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1a34a0 0x7fafbc1a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.385+0000 7fafc1800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1a34a0 0x7fafbc1a38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33890/0 (socket says 172.21.15.1:33890) 2024-04-26T21:18:38.860 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.385+0000 7fafc1800700 1 -- 172.21.15.1:0/453723523 learned_addr learned my addr 172.21.15.1:0/453723523 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.386+0000 7fafc1800700 1 -- 172.21.15.1:0/453723523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafbc1a3e00 con 0x7fafbc1a34a0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.386+0000 7fafc1800700 1 --2- 172.21.15.1:0/453723523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1a34a0 0x7fafbc1a38c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fafb0015000 tx=0x7fafb0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.386+0000 7fafba000700 1 -- 172.21.15.1:0/453723523 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafb00036a0 con 0x7fafbc1a34a0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.386+0000 7fafba000700 1 -- 172.21.15.1:0/453723523 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fafb001a460 con 0x7fafbc1a34a0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.386+0000 7fafc4200700 1 -- 172.21.15.1:0/453723523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafbc1a4090 con 0x7fafbc1a34a0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.386+0000 7fafba000700 1 -- 172.21.15.1:0/453723523 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fafb001b5d0 con 0x7fafbc1a34a0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.387+0000 7fafc4200700 1 -- 172.21.15.1:0/453723523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafbc1a6b70 con 0x7fafbc1a34a0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.388+0000 7fafba000700 1 -- 172.21.15.1:0/453723523 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fafb001a610 con 0x7fafbc1a34a0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.388+0000 7fafc4200700 1 -- 172.21.15.1:0/453723523 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafbc064a80 con 0x7fafbc1a34a0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.388+0000 7fafba000700 1 --2- 172.21.15.1:0/453723523 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fafa4039600 0x7fafa403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.389+0000 7fafba000700 1 -- 172.21.15.1:0/453723523 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fafb004dce0 con 0x7fafbc1a34a0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.389+0000 7fafc0e00700 1 --2- 172.21.15.1:0/453723523 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fafa4039600 0x7fafa403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:38.861 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.390+0000 7fafc0e00700 1 --2- 172.21.15.1:0/453723523 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fafa4039600 0x7fafa403bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fafa8006fd0 tx=0x7fafa8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.395+0000 7fafba000700 1 -- 172.21.15.1:0/453723523 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fafb002c3a0 con 0x7fafbc1a34a0 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.649+0000 7fafc4200700 1 -- 172.21.15.1:0/453723523 --> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fafbc063860 con 0x7fafa4039600 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.661+0000 7fafba000700 1 -- 172.21.15.1:0/453723523 <== mgr.14120 v2:172.21.15.1:6800/3159681858 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fafbc063860 con 0x7fafa4039600 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.664+0000 7fafc4200700 1 -- 172.21.15.1:0/453723523 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fafa4039600 msgr2=0x7fafa403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.664+0000 7fafc4200700 1 --2- 172.21.15.1:0/453723523 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fafa4039600 0x7fafa403bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fafa8006fd0 tx=0x7fafa8006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.664+0000 7fafc4200700 1 -- 172.21.15.1:0/453723523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1a34a0 msgr2=0x7fafbc1a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.664+0000 7fafc4200700 1 --2- 172.21.15.1:0/453723523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1a34a0 0x7fafbc1a38c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fafb0015000 tx=0x7fafb0011940 comp rx=0 tx=0).stop 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.664+0000 7fafc4200700 1 -- 172.21.15.1:0/453723523 shutdown_connections 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.664+0000 7fafc4200700 1 --2- 172.21.15.1:0/453723523 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fafa4039600 0x7fafa403bac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.664+0000 7fafc4200700 1 --2- 172.21.15.1:0/453723523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafbc1a34a0 0x7fafbc1a38c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.664+0000 7fafc4200700 1 -- 172.21.15.1:0/453723523 >> 172.21.15.1:0/453723523 conn(0x7fafbc103340 msgr2=0x7fafbc105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.664+0000 7fafc4200700 1 -- 172.21.15.1:0/453723523 shutdown_connections 2024-04-26T21:18:38.862 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:37.664+0000 7fafc4200700 1 -- 172.21.15.1:0/453723523 wait complete. 2024-04-26T21:18:38.863 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying unmanaged mgr service... 2024-04-26T21:18:39.013 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:38 smithi001 ceph-mon[28054]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:18:39.013 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:38 smithi001 ceph-mon[28054]: Saving service mon spec with placement count:5 2024-04-26T21:18:39.014 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:38 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:40.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T21:18:40.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.596+0000 7f3fa6c00700 1 Processor -- start 2024-04-26T21:18:40.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.596+0000 7f3fa6c00700 1 -- start start 2024-04-26T21:18:40.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.596+0000 7f3fa6c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa007d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:40.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.596+0000 7f3fa6c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa007d8b0 con 0x7f3fa007ee80 2024-04-26T21:18:40.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.597+0000 7f3fa5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa007d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:40.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.597+0000 7f3fa5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa007d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38882/0 (socket says 172.21.15.1:38882) 2024-04-26T21:18:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.597+0000 7f3fa5800700 1 -- 172.21.15.1:0/4227972498 learned_addr learned my addr 172.21.15.1:0/4227972498 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.597+0000 7f3fa5800700 1 -- 172.21.15.1:0/4227972498 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa007da30 con 0x7f3fa007ee80 2024-04-26T21:18:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.598+0000 7f3fa5800700 1 --2- 172.21.15.1:0/4227972498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa007d2e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3f940098a0 tx=0x7f3f94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ecee720c83f63d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.598+0000 7f3f9fe00700 1 -- 172.21.15.1:0/4227972498 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f94004160 con 0x7f3fa007ee80 2024-04-26T21:18:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.598+0000 7f3f9fe00700 1 -- 172.21.15.1:0/4227972498 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3f94004300 con 0x7f3fa007ee80 2024-04-26T21:18:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.598+0000 7f3f9fe00700 1 -- 172.21.15.1:0/4227972498 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f94011af0 con 0x7f3fa007ee80 2024-04-26T21:18:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.599+0000 7f3fa6c00700 1 -- 172.21.15.1:0/4227972498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 msgr2=0x7f3fa007d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.599+0000 7f3fa6c00700 1 --2- 172.21.15.1:0/4227972498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa007d2e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3f940098a0 tx=0x7f3f94009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.599+0000 7f3fa6c00700 1 -- 172.21.15.1:0/4227972498 shutdown_connections 2024-04-26T21:18:40.508 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.599+0000 7f3fa6c00700 1 --2- 172.21.15.1:0/4227972498 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa007d2e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:40.509 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.599+0000 7f3fa6c00700 1 -- 172.21.15.1:0/4227972498 >> 172.21.15.1:0/4227972498 conn(0x7f3fa0104230 msgr2=0x7f3fa0106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:40.509 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.599+0000 7f3fa6c00700 1 -- 172.21.15.1:0/4227972498 shutdown_connections 2024-04-26T21:18:40.509 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.599+0000 7f3fa6c00700 1 -- 172.21.15.1:0/4227972498 wait complete. 2024-04-26T21:18:40.509 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.600+0000 7f3fa6c00700 1 Processor -- start 2024-04-26T21:18:40.509 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.600+0000 7f3fa6c00700 1 -- start start 2024-04-26T21:18:40.509 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.600+0000 7f3fa6c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa019eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:40.509 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.600+0000 7f3fa6c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa010f1f0 con 0x7f3fa007ee80 2024-04-26T21:18:40.509 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.601+0000 7f3fa5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa019eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:40.509 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.601+0000 7f3fa5800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa019eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38892/0 (socket says 172.21.15.1:38892) 2024-04-26T21:18:40.509 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.601+0000 7f3fa5800700 1 -- 172.21.15.1:0/582764646 learned_addr learned my addr 172.21.15.1:0/582764646 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.601+0000 7f3fa5800700 1 -- 172.21.15.1:0/582764646 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa019f2e0 con 0x7f3fa007ee80 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.601+0000 7f3fa5800700 1 --2- 172.21.15.1:0/582764646 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa019eda0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f94000c00 tx=0x7f3f94011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.602+0000 7f3f9e000700 1 -- 172.21.15.1:0/582764646 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f94003b20 con 0x7f3fa007ee80 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.602+0000 7f3f9e000700 1 -- 172.21.15.1:0/582764646 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3f9401a8c0 con 0x7f3fa007ee80 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.602+0000 7f3f9e000700 1 -- 172.21.15.1:0/582764646 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f94024940 con 0x7f3fa007ee80 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.602+0000 7f3fa6c00700 1 -- 172.21.15.1:0/582764646 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa019f570 con 0x7f3fa007ee80 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.602+0000 7f3fa6c00700 1 -- 172.21.15.1:0/582764646 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa01a2220 con 0x7f3fa007ee80 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.603+0000 7f3f9e000700 1 -- 172.21.15.1:0/582764646 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f3f94022020 con 0x7f3fa007ee80 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.603+0000 7f3fa6c00700 1 -- 172.21.15.1:0/582764646 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa0064a80 con 0x7f3fa007ee80 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.604+0000 7f3f9e000700 1 --2- 172.21.15.1:0/582764646 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3f88039600 0x7f3f8803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.604+0000 7f3f9e000700 1 -- 172.21.15.1:0/582764646 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3f9404d7e0 con 0x7f3fa007ee80 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.604+0000 7f3fa4e00700 1 --2- 172.21.15.1:0/582764646 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3f88039600 0x7f3f8803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:40.510 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.605+0000 7f3fa4e00700 1 --2- 172.21.15.1:0/582764646 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3f88039600 0x7f3f8803bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c006fd0 tx=0x7f3f8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.610+0000 7f3f9e000700 1 -- 172.21.15.1:0/582764646 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3f9400be50 con 0x7f3fa007ee80 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.867+0000 7f3fa6c00700 1 -- 172.21.15.1:0/582764646 --> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f3fa01054d0 con 0x7f3f88039600 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.872+0000 7f3f9e000700 1 -- 172.21.15.1:0/582764646 <== mon.0 v2:172.21.15.1:3300/0 7 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f94018350 con 0x7f3fa007ee80 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.878+0000 7f3f9e000700 1 -- 172.21.15.1:0/582764646 <== mgr.14120 v2:172.21.15.1:6800/3159681858 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3fa01054d0 con 0x7f3f88039600 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.881+0000 7f3fa6c00700 1 -- 172.21.15.1:0/582764646 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3f88039600 msgr2=0x7f3f8803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.881+0000 7f3fa6c00700 1 --2- 172.21.15.1:0/582764646 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3f88039600 0x7f3f8803bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c006fd0 tx=0x7f3f8c006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.881+0000 7f3fa6c00700 1 -- 172.21.15.1:0/582764646 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 msgr2=0x7f3fa019eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.881+0000 7f3fa6c00700 1 --2- 172.21.15.1:0/582764646 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa019eda0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3f94000c00 tx=0x7f3f94011b70 comp rx=0 tx=0).stop 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.882+0000 7f3fa6c00700 1 -- 172.21.15.1:0/582764646 shutdown_connections 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.882+0000 7f3fa6c00700 1 --2- 172.21.15.1:0/582764646 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f3f88039600 0x7f3f8803bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.882+0000 7f3fa6c00700 1 --2- 172.21.15.1:0/582764646 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3fa007ee80 0x7f3fa019eda0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.882+0000 7f3fa6c00700 1 -- 172.21.15.1:0/582764646 >> 172.21.15.1:0/582764646 conn(0x7f3fa0104230 msgr2=0x7f3fa0104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.882+0000 7f3fa6c00700 1 -- 172.21.15.1:0/582764646 shutdown_connections 2024-04-26T21:18:40.511 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:39.882+0000 7f3fa6c00700 1 -- 172.21.15.1:0/582764646 wait complete. 2024-04-26T21:18:41.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:40 smithi001 ceph-mon[28054]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:18:41.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:40 smithi001 ceph-mon[28054]: Saving service mgr spec with placement count:2 2024-04-26T21:18:41.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:40 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:41.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:40 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:41.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:40 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:42.036 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:41 smithi001 ceph-mon[28054]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:18:42.036 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:41 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2856474593' entity='client.admin' 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.121+0000 7fddeec00700 1 Processor -- start 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.121+0000 7fddeec00700 1 -- start start 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.122+0000 7fddeec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde807d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.122+0000 7fddeec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdde807d850 con 0x7fdde807ee20 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.122+0000 7fdde7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde807d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.122+0000 7fdde7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde807d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38900/0 (socket says 172.21.15.1:38900) 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.122+0000 7fdde7e00700 1 -- 172.21.15.1:0/3519968832 learned_addr learned my addr 172.21.15.1:0/3519968832 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.123+0000 7fdde7e00700 1 -- 172.21.15.1:0/3519968832 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde807d9d0 con 0x7fdde807ee20 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.123+0000 7fdde7e00700 1 --2- 172.21.15.1:0/3519968832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde807d280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fddd80098a0 tx=0x7fddd8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6eb4feb29ee19ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.123+0000 7fdde6a00700 1 -- 172.21.15.1:0/3519968832 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddd8004160 con 0x7fdde807ee20 2024-04-26T21:18:42.051 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.123+0000 7fdde6a00700 1 -- 172.21.15.1:0/3519968832 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddd8004300 con 0x7fdde807ee20 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.123+0000 7fdde6a00700 1 -- 172.21.15.1:0/3519968832 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddd8011a20 con 0x7fdde807ee20 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.124+0000 7fddeec00700 1 -- 172.21.15.1:0/3519968832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 msgr2=0x7fdde807d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.124+0000 7fddeec00700 1 --2- 172.21.15.1:0/3519968832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde807d280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fddd80098a0 tx=0x7fddd8009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.124+0000 7fddeec00700 1 -- 172.21.15.1:0/3519968832 shutdown_connections 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.124+0000 7fddeec00700 1 --2- 172.21.15.1:0/3519968832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde807d280 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.124+0000 7fddeec00700 1 -- 172.21.15.1:0/3519968832 >> 172.21.15.1:0/3519968832 conn(0x7fdde8104450 msgr2=0x7fdde8106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.125+0000 7fddeec00700 1 -- 172.21.15.1:0/3519968832 shutdown_connections 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.125+0000 7fddeec00700 1 -- 172.21.15.1:0/3519968832 wait complete. 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.126+0000 7fddeec00700 1 Processor -- start 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.126+0000 7fddeec00700 1 -- start start 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.126+0000 7fddeec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.126+0000 7fddeec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdde810f4b0 con 0x7fdde807ee20 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.126+0000 7fdde7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.126+0000 7fdde7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde819ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38916/0 (socket says 172.21.15.1:38916) 2024-04-26T21:18:42.052 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.126+0000 7fdde7e00700 1 -- 172.21.15.1:0/2856474593 learned_addr learned my addr 172.21.15.1:0/2856474593 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.127+0000 7fdde7e00700 1 -- 172.21.15.1:0/2856474593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde819f440 con 0x7fdde807ee20 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.127+0000 7fdde7e00700 1 --2- 172.21.15.1:0/2856474593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde819ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fddd8004660 tx=0x7fddd8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.127+0000 7fdde4c00700 1 -- 172.21.15.1:0/2856474593 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddd8003aa0 con 0x7fdde807ee20 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.127+0000 7fdde4c00700 1 -- 172.21.15.1:0/2856474593 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddd801b8f0 con 0x7fdde807ee20 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.127+0000 7fdde4c00700 1 -- 172.21.15.1:0/2856474593 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddd8023900 con 0x7fdde807ee20 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.128+0000 7fddeec00700 1 -- 172.21.15.1:0/2856474593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde819f6d0 con 0x7fdde807ee20 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.128+0000 7fddeec00700 1 -- 172.21.15.1:0/2856474593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde81a23c0 con 0x7fdde807ee20 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.129+0000 7fdde4c00700 1 -- 172.21.15.1:0/2856474593 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fddd8023b70 con 0x7fdde807ee20 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.129+0000 7fddeec00700 1 -- 172.21.15.1:0/2856474593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdde8064a80 con 0x7fdde807ee20 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.130+0000 7fdde4c00700 1 --2- 172.21.15.1:0/2856474593 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fddcc039600 0x7fddcc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.130+0000 7fdde4c00700 1 -- 172.21.15.1:0/2856474593 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fddd804ded0 con 0x7fdde807ee20 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.130+0000 7fdde7400700 1 --2- 172.21.15.1:0/2856474593 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fddcc039600 0x7fddcc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.131+0000 7fdde7400700 1 --2- 172.21.15.1:0/2856474593 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fddcc039600 0x7fddcc03bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fddd4006fd0 tx=0x7fddd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.135+0000 7fdde4c00700 1 -- 172.21.15.1:0/2856474593 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fddd801f030 con 0x7fdde807ee20 2024-04-26T21:18:42.053 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.367+0000 7fddeec00700 1 -- 172.21.15.1:0/2856474593 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fdde81a2590 con 0x7fdde807ee20 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.371+0000 7fdde4c00700 1 -- 172.21.15.1:0/2856474593 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fddd80292a0 con 0x7fdde807ee20 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.374+0000 7fddeec00700 1 -- 172.21.15.1:0/2856474593 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fddcc039600 msgr2=0x7fddcc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.374+0000 7fddeec00700 1 --2- 172.21.15.1:0/2856474593 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fddcc039600 0x7fddcc03bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fddd4006fd0 tx=0x7fddd4006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.374+0000 7fddeec00700 1 -- 172.21.15.1:0/2856474593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 msgr2=0x7fdde819ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.374+0000 7fddeec00700 1 --2- 172.21.15.1:0/2856474593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde819ef00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fddd8004660 tx=0x7fddd8011aa0 comp rx=0 tx=0).stop 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.374+0000 7fddeec00700 1 -- 172.21.15.1:0/2856474593 shutdown_connections 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.375+0000 7fddeec00700 1 --2- 172.21.15.1:0/2856474593 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7fddcc039600 0x7fddcc03bac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.375+0000 7fddeec00700 1 --2- 172.21.15.1:0/2856474593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdde807ee20 0x7fdde819ef00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.375+0000 7fddeec00700 1 -- 172.21.15.1:0/2856474593 >> 172.21.15.1:0/2856474593 conn(0x7fdde8104450 msgr2=0x7fdde8104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.375+0000 7fddeec00700 1 -- 172.21.15.1:0/2856474593 shutdown_connections 2024-04-26T21:18:42.054 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:41.375+0000 7fddeec00700 1 -- 172.21.15.1:0/2856474593 wait complete. 2024-04-26T21:18:43.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.670+0000 7f5e99000700 1 Processor -- start 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.670+0000 7f5e99000700 1 -- start start 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.671+0000 7f5e99000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e94112060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.671+0000 7f5e99000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5e94112630 con 0x7f5e9410fc70 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.671+0000 7f5e91e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e94112060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.671+0000 7f5e91e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e94112060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38928/0 (socket says 172.21.15.1:38928) 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.671+0000 7f5e91e00700 1 -- 172.21.15.1:0/3972414754 learned_addr learned my addr 172.21.15.1:0/3972414754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.672+0000 7f5e91e00700 1 -- 172.21.15.1:0/3972414754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e941127b0 con 0x7f5e9410fc70 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.672+0000 7f5e91e00700 1 --2- 172.21.15.1:0/3972414754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e94112060 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5e840098a0 tx=0x7f5e84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6918b53c0821f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.672+0000 7f5e90a00700 1 -- 172.21.15.1:0/3972414754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e84004160 con 0x7f5e9410fc70 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.673+0000 7f5e90a00700 1 -- 172.21.15.1:0/3972414754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e84004300 con 0x7f5e9410fc70 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.673+0000 7f5e90a00700 1 -- 172.21.15.1:0/3972414754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e84011a20 con 0x7f5e9410fc70 2024-04-26T21:18:43.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.673+0000 7f5e99000700 1 -- 172.21.15.1:0/3972414754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 msgr2=0x7f5e94112060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.673+0000 7f5e99000700 1 --2- 172.21.15.1:0/3972414754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e94112060 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5e840098a0 tx=0x7f5e84009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.673+0000 7f5e99000700 1 -- 172.21.15.1:0/3972414754 shutdown_connections 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.673+0000 7f5e99000700 1 --2- 172.21.15.1:0/3972414754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e94112060 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.673+0000 7f5e99000700 1 -- 172.21.15.1:0/3972414754 >> 172.21.15.1:0/3972414754 conn(0x7f5e9407e460 msgr2=0x7f5e94080880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.673+0000 7f5e99000700 1 -- 172.21.15.1:0/3972414754 shutdown_connections 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.673+0000 7f5e99000700 1 -- 172.21.15.1:0/3972414754 wait complete. 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.674+0000 7f5e99000700 1 Processor -- start 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.675+0000 7f5e99000700 1 -- start start 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.675+0000 7f5e99000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e941a3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.675+0000 7f5e99000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5e941137b0 con 0x7f5e9410fc70 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.675+0000 7f5e91e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e941a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.675+0000 7f5e91e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e941a3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38932/0 (socket says 172.21.15.1:38932) 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.675+0000 7f5e91e00700 1 -- 172.21.15.1:0/1038270080 learned_addr learned my addr 172.21.15.1:0/1038270080 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.676+0000 7f5e91e00700 1 -- 172.21.15.1:0/1038270080 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e941a3780 con 0x7f5e9410fc70 2024-04-26T21:18:43.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.676+0000 7f5e91e00700 1 --2- 172.21.15.1:0/1038270080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e941a3240 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e84000c00 tx=0x7f5e84011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.676+0000 7f5e8aa00700 1 -- 172.21.15.1:0/1038270080 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e84003aa0 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.676+0000 7f5e8aa00700 1 -- 172.21.15.1:0/1038270080 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e8401b8f0 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.676+0000 7f5e99000700 1 -- 172.21.15.1:0/1038270080 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e941a3a70 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.676+0000 7f5e8aa00700 1 -- 172.21.15.1:0/1038270080 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e84023870 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.676+0000 7f5e99000700 1 -- 172.21.15.1:0/1038270080 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e941a6700 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.678+0000 7f5e8aa00700 1 -- 172.21.15.1:0/1038270080 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f5e8401b440 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.678+0000 7f5e99000700 1 -- 172.21.15.1:0/1038270080 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e94064a80 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.678+0000 7f5e8aa00700 1 --2- 172.21.15.1:0/1038270080 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5e78039600 0x7f5e7803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.678+0000 7f5e8aa00700 1 -- 172.21.15.1:0/1038270080 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5e8404ddd0 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.678+0000 7f5e91400700 1 --2- 172.21.15.1:0/1038270080 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5e78039600 0x7f5e7803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.679+0000 7f5e91400700 1 --2- 172.21.15.1:0/1038270080 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5e78039600 0x7f5e7803bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5e80006fd0 tx=0x7f5e80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.684+0000 7f5e8aa00700 1 -- 172.21.15.1:0/1038270080 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5e84051030 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.920+0000 7f5e99000700 1 -- 172.21.15.1:0/1038270080 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f5e941a68d0 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.923+0000 7f5e8aa00700 1 -- 172.21.15.1:0/1038270080 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f5e8401f030 con 0x7f5e9410fc70 2024-04-26T21:18:44.000 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.926+0000 7f5e99000700 1 -- 172.21.15.1:0/1038270080 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5e78039600 msgr2=0x7f5e7803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:44.001 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.926+0000 7f5e99000700 1 --2- 172.21.15.1:0/1038270080 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5e78039600 0x7f5e7803bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5e80006fd0 tx=0x7f5e80006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:44.001 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.926+0000 7f5e99000700 1 -- 172.21.15.1:0/1038270080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 msgr2=0x7f5e941a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:44.001 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.926+0000 7f5e99000700 1 --2- 172.21.15.1:0/1038270080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e941a3240 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e84000c00 tx=0x7f5e84011aa0 comp rx=0 tx=0).stop 2024-04-26T21:18:44.001 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.927+0000 7f5e99000700 1 -- 172.21.15.1:0/1038270080 shutdown_connections 2024-04-26T21:18:44.001 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.927+0000 7f5e99000700 1 --2- 172.21.15.1:0/1038270080 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f5e78039600 0x7f5e7803bac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:44.001 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.927+0000 7f5e99000700 1 --2- 172.21.15.1:0/1038270080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e9410fc70 0x7f5e941a3240 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:44.001 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.927+0000 7f5e99000700 1 -- 172.21.15.1:0/1038270080 >> 172.21.15.1:0/1038270080 conn(0x7f5e9407e460 msgr2=0x7f5e9407ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:44.001 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.927+0000 7f5e99000700 1 -- 172.21.15.1:0/1038270080 shutdown_connections 2024-04-26T21:18:44.001 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:42.927+0000 7f5e99000700 1 -- 172.21.15.1:0/1038270080 wait complete. 2024-04-26T21:18:44.001 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling the dashboard module... 2024-04-26T21:18:44.292 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:43 smithi001 ceph-mon[28054]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:18:44.292 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:43 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1038270080' entity='client.admin' 2024-04-26T21:18:44.292 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:43 smithi001 ceph-mon[28054]: from='mgr.14120 172.21.15.1:0/475536812' entity='mgr.x' 2024-04-26T21:18:46.240 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:45 smithi001 ceph-mon[28054]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:18:46.240 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:45 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1061104686' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T21:18:46.240 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: ignoring --setuser ceph since I am not root 2024-04-26T21:18:46.241 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: ignoring --setgroup ceph since I am not root 2024-04-26T21:18:46.241 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:46.237+0000 7fa2162c8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:18:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.592+0000 7faab1400700 1 Processor -- start 2024-04-26T21:18:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.593+0000 7faab1400700 1 -- start start 2024-04-26T21:18:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.593+0000 7faab1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac07d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.593+0000 7faab1400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faaac07d850 con 0x7faaac07ee20 2024-04-26T21:18:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.594+0000 7faaaa800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac07d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.594+0000 7faaaa800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac07d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38946/0 (socket says 172.21.15.1:38946) 2024-04-26T21:18:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.594+0000 7faaaa800700 1 -- 172.21.15.1:0/1908543240 learned_addr learned my addr 172.21.15.1:0/1908543240 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.594+0000 7faaaa800700 1 -- 172.21.15.1:0/1908543240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaac07d9d0 con 0x7faaac07ee20 2024-04-26T21:18:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.595+0000 7faaaa800700 1 --2- 172.21.15.1:0/1908543240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac07d280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0098a0 tx=0x7faa9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d929e31935e26e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.595+0000 7faaa9400700 1 -- 172.21.15.1:0/1908543240 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa9c004160 con 0x7faaac07ee20 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.595+0000 7faaa9400700 1 -- 172.21.15.1:0/1908543240 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa9c004300 con 0x7faaac07ee20 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.595+0000 7faaa9400700 1 -- 172.21.15.1:0/1908543240 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa9c011a20 con 0x7faaac07ee20 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.596+0000 7faab1400700 1 -- 172.21.15.1:0/1908543240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 msgr2=0x7faaac07d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.596+0000 7faab1400700 1 --2- 172.21.15.1:0/1908543240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac07d280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faa9c0098a0 tx=0x7faa9c009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.596+0000 7faab1400700 1 -- 172.21.15.1:0/1908543240 shutdown_connections 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.596+0000 7faab1400700 1 --2- 172.21.15.1:0/1908543240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac07d280 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.596+0000 7faab1400700 1 -- 172.21.15.1:0/1908543240 >> 172.21.15.1:0/1908543240 conn(0x7faaac104450 msgr2=0x7faaac106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.596+0000 7faab1400700 1 -- 172.21.15.1:0/1908543240 shutdown_connections 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.596+0000 7faab1400700 1 -- 172.21.15.1:0/1908543240 wait complete. 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.597+0000 7faab1400700 1 Processor -- start 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.597+0000 7faab1400700 1 -- start start 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.598+0000 7faab1400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.598+0000 7faab1400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faaac10f4b0 con 0x7faaac07ee20 2024-04-26T21:18:46.666 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.598+0000 7faaaa800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.598+0000 7faaaa800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac19ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38962/0 (socket says 172.21.15.1:38962) 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.598+0000 7faaaa800700 1 -- 172.21.15.1:0/1061104686 learned_addr learned my addr 172.21.15.1:0/1061104686 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.599+0000 7faaaa800700 1 -- 172.21.15.1:0/1061104686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaac19f460 con 0x7faaac07ee20 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.599+0000 7faaaa800700 1 --2- 172.21.15.1:0/1061104686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac19ef20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faa9c004660 tx=0x7faa9c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.599+0000 7faaa3400700 1 -- 172.21.15.1:0/1061104686 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa9c003aa0 con 0x7faaac07ee20 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.599+0000 7faaa3400700 1 -- 172.21.15.1:0/1061104686 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa9c01b8f0 con 0x7faaac07ee20 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.599+0000 7faab1400700 1 -- 172.21.15.1:0/1061104686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaac19f6f0 con 0x7faaac07ee20 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.599+0000 7faaa3400700 1 -- 172.21.15.1:0/1061104686 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa9c023900 con 0x7faaac07ee20 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.599+0000 7faab1400700 1 -- 172.21.15.1:0/1061104686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaac1a23e0 con 0x7faaac07ee20 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.601+0000 7faab1400700 1 -- 172.21.15.1:0/1061104686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaac064a80 con 0x7faaac07ee20 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.602+0000 7faaa3400700 1 -- 172.21.15.1:0/1061104686 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7faa9c023bc0 con 0x7faaac07ee20 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.602+0000 7faaa3400700 1 --2- 172.21.15.1:0/1061104686 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7faa900395b0 0x7faa9003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:46.667 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.602+0000 7faaa3400700 1 -- 172.21.15.1:0/1061104686 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faa9c04e480 con 0x7faaac07ee20 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.602+0000 7faaa9e00700 1 --2- 172.21.15.1:0/1061104686 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7faa900395b0 0x7faa9003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.603+0000 7faaa9e00700 1 --2- 172.21.15.1:0/1061104686 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7faa900395b0 0x7faa9003ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faa98006fd0 tx=0x7faa98006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.612+0000 7faaa3400700 1 -- 172.21.15.1:0/1061104686 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faa9c01f030 con 0x7faaac07ee20 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:44.949+0000 7faab1400700 1 -- 172.21.15.1:0/1061104686 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7faaac1a25b0 con 0x7faaac07ee20 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.932+0000 7faaa3400700 1 -- 172.21.15.1:0/1061104686 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7faa9c029c90 con 0x7faaac07ee20 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faaa3400700 1 -- 172.21.15.1:0/1061104686 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7faa9c04d630 con 0x7faaac07ee20 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faab1400700 1 -- 172.21.15.1:0/1061104686 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7faa900395b0 msgr2=0x7faa9003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faab1400700 1 --2- 172.21.15.1:0/1061104686 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7faa900395b0 0x7faa9003ba70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faa98006fd0 tx=0x7faa98006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faab1400700 1 -- 172.21.15.1:0/1061104686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 msgr2=0x7faaac19ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faab1400700 1 --2- 172.21.15.1:0/1061104686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac19ef20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faa9c004660 tx=0x7faa9c011aa0 comp rx=0 tx=0).stop 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faab1400700 1 -- 172.21.15.1:0/1061104686 shutdown_connections 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faab1400700 1 --2- 172.21.15.1:0/1061104686 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7faa900395b0 0x7faa9003ba70 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faa98006fd0 tx=0x7faa98006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:46.668 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faab1400700 1 --2- 172.21.15.1:0/1061104686 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faaac07ee20 0x7faaac19ef20 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faa9c004660 tx=0x7faa9c011aa0 comp rx=0 tx=0).stop 2024-04-26T21:18:46.669 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faab1400700 1 -- 172.21.15.1:0/1061104686 >> 172.21.15.1:0/1061104686 conn(0x7faaac104450 msgr2=0x7faaac104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:46.669 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faab1400700 1 -- 172.21.15.1:0/1061104686 shutdown_connections 2024-04-26T21:18:46.669 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:45.938+0000 7faab1400700 1 -- 172.21.15.1:0/1061104686 wait complete. 2024-04-26T21:18:46.812 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:46.492+0000 7fa2162c8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:18:46.812 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:46.627+0000 7fa2162c8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:18:47.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:46 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1061104686' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T21:18:47.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:46 smithi001 ceph-mon[28054]: mgrmap e9: x(active, since 25s) 2024-04-26T21:18:47.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:47 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:47.041+0000 7fa2162c8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:18:47.456 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:47 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:47.158+0000 7fa2162c8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:18:48.214 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:47 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3874085944' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T21:18:48.224 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:18:48.224 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T21:18:48.225 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T21:18:48.225 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-26T21:18:48.225 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T21:18:48.225 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:18:48.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.246+0000 7f6bad600700 1 Processor -- start 2024-04-26T21:18:48.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.246+0000 7f6bad600700 1 -- start start 2024-04-26T21:18:48.226 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.246+0000 7f6bad600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba81092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:48.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.246+0000 7f6bad600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba81097f0 con 0x7f6ba8108e90 2024-04-26T21:18:48.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.247+0000 7f6ba6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba81092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:48.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.247+0000 7f6ba6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba81092b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39000/0 (socket says 172.21.15.1:39000) 2024-04-26T21:18:48.227 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.247+0000 7f6ba6a00700 1 -- 172.21.15.1:0/3592158330 learned_addr learned my addr 172.21.15.1:0/3592158330 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:48.228 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.247+0000 7f6ba6a00700 1 -- 172.21.15.1:0/3592158330 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba8109ff0 con 0x7f6ba8108e90 2024-04-26T21:18:48.228 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.247+0000 7f6ba6a00700 1 --2- 172.21.15.1:0/3592158330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba81092b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6b980098a0 tx=0x7f6b98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e72b4273f95925 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:48.228 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.248+0000 7f6ba5600700 1 -- 172.21.15.1:0/3592158330 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b98004160 con 0x7f6ba8108e90 2024-04-26T21:18:48.228 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.248+0000 7f6ba5600700 1 -- 172.21.15.1:0/3592158330 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b98004300 con 0x7f6ba8108e90 2024-04-26T21:18:48.229 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.248+0000 7f6ba5600700 1 -- 172.21.15.1:0/3592158330 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b98011a20 con 0x7f6ba8108e90 2024-04-26T21:18:48.229 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.248+0000 7f6bad600700 1 -- 172.21.15.1:0/3592158330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 msgr2=0x7f6ba81092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:48.229 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.248+0000 7f6bad600700 1 --2- 172.21.15.1:0/3592158330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba81092b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6b980098a0 tx=0x7f6b98009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:48.229 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.249+0000 7f6bad600700 1 -- 172.21.15.1:0/3592158330 shutdown_connections 2024-04-26T21:18:48.229 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.249+0000 7f6bad600700 1 --2- 172.21.15.1:0/3592158330 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba81092b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:48.230 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.249+0000 7f6bad600700 1 -- 172.21.15.1:0/3592158330 >> 172.21.15.1:0/3592158330 conn(0x7f6ba8104450 msgr2=0x7f6ba8106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:48.230 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.249+0000 7f6bad600700 1 -- 172.21.15.1:0/3592158330 shutdown_connections 2024-04-26T21:18:48.230 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.249+0000 7f6bad600700 1 -- 172.21.15.1:0/3592158330 wait complete. 2024-04-26T21:18:48.230 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.250+0000 7f6bad600700 1 Processor -- start 2024-04-26T21:18:48.230 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.250+0000 7f6bad600700 1 -- start start 2024-04-26T21:18:48.230 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.250+0000 7f6bad600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba819cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:48.230 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.250+0000 7f6bad600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba810d430 con 0x7f6ba8108e90 2024-04-26T21:18:48.231 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.251+0000 7f6ba6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba819cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:48.231 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.251+0000 7f6ba6a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba819cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39004/0 (socket says 172.21.15.1:39004) 2024-04-26T21:18:48.231 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.251+0000 7f6ba6a00700 1 -- 172.21.15.1:0/3874085944 learned_addr learned my addr 172.21.15.1:0/3874085944 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:48.231 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.251+0000 7f6ba6a00700 1 -- 172.21.15.1:0/3874085944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba819d280 con 0x7f6ba8108e90 2024-04-26T21:18:48.231 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.251+0000 7f6ba6a00700 1 --2- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba819cd40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6b98004660 tx=0x7f6b98011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:48.231 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.251+0000 7f6b9f400700 1 -- 172.21.15.1:0/3874085944 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b98003aa0 con 0x7f6ba8108e90 2024-04-26T21:18:48.232 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.251+0000 7f6b9f400700 1 -- 172.21.15.1:0/3874085944 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b9801b8f0 con 0x7f6ba8108e90 2024-04-26T21:18:48.232 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.251+0000 7f6bad600700 1 -- 172.21.15.1:0/3874085944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba819d510 con 0x7f6ba8108e90 2024-04-26T21:18:48.232 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.252+0000 7f6b9f400700 1 -- 172.21.15.1:0/3874085944 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b98023900 con 0x7f6ba8108e90 2024-04-26T21:18:48.232 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.252+0000 7f6bad600700 1 -- 172.21.15.1:0/3874085944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba81a0200 con 0x7f6ba8108e90 2024-04-26T21:18:48.233 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.253+0000 7f6b9f400700 1 -- 172.21.15.1:0/3874085944 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f6b98023bc0 con 0x7f6ba8108e90 2024-04-26T21:18:48.233 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.253+0000 7f6b9f400700 1 --2- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f6b8c039600 0x7f6b8c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:48.233 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.253+0000 7f6b9f400700 1 -- 172.21.15.1:0/3874085944 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6b98052880 con 0x7f6ba8108e90 2024-04-26T21:18:48.233 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.253+0000 7f6bad600700 1 -- 172.21.15.1:0/3874085944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ba810d5b0 con 0x7f6ba8108e90 2024-04-26T21:18:48.234 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.253+0000 7f6ba6000700 1 -- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f6b8c039600 msgr2=0x7f6b8c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3159681858 2024-04-26T21:18:48.234 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.253+0000 7f6ba6000700 1 --2- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f6b8c039600 0x7f6b8c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:18:48.234 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.259+0000 7f6b9f400700 1 -- 172.21.15.1:0/3874085944 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6b9801f030 con 0x7f6ba8108e90 2024-04-26T21:18:48.234 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.454+0000 7f6ba6000700 1 -- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f6b8c039600 msgr2=0x7f6b8c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3159681858 2024-04-26T21:18:48.234 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.454+0000 7f6ba6000700 1 --2- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f6b8c039600 0x7f6b8c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:18:48.235 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.555+0000 7f6bad600700 1 -- 172.21.15.1:0/3874085944 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f6ba81a0720 con 0x7f6ba8108e90 2024-04-26T21:18:48.235 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.556+0000 7f6b9f400700 1 -- 172.21.15.1:0/3874085944 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f6b980198f0 con 0x7f6ba8108e90 2024-04-26T21:18:48.235 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.559+0000 7f6bad600700 1 -- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f6b8c039600 msgr2=0x7f6b8c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:18:48.236 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.559+0000 7f6bad600700 1 --2- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f6b8c039600 0x7f6b8c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:48.236 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.559+0000 7f6bad600700 1 -- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 msgr2=0x7f6ba819cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:48.236 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.559+0000 7f6bad600700 1 --2- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba819cd40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6b98004660 tx=0x7f6b98011aa0 comp rx=0 tx=0).stop 2024-04-26T21:18:48.236 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.559+0000 7f6bad600700 1 -- 172.21.15.1:0/3874085944 shutdown_connections 2024-04-26T21:18:48.237 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.559+0000 7f6bad600700 1 --2- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f6b8c039600 0x7f6b8c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:48.237 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.559+0000 7f6bad600700 1 --2- 172.21.15.1:0/3874085944 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ba8108e90 0x7f6ba819cd40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:48.237 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.559+0000 7f6bad600700 1 -- 172.21.15.1:0/3874085944 >> 172.21.15.1:0/3874085944 conn(0x7f6ba8104450 msgr2=0x7f6ba8104e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:48.238 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.559+0000 7f6bad600700 1 -- 172.21.15.1:0/3874085944 shutdown_connections 2024-04-26T21:18:48.238 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:47.559+0000 7f6bad600700 1 -- 172.21.15.1:0/3874085944 wait complete. 2024-04-26T21:18:48.238 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for the mgr to restart... 2024-04-26T21:18:48.238 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr epoch 9... 2024-04-26T21:18:49.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:48 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:48.805+0000 7fa2162c8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:18:49.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:49.040+0000 7fa2162c8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:18:49.567 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:49.260+0000 7fa2162c8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:18:50.037 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:49.651+0000 7fa2162c8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:18:50.038 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:49.776+0000 7fa2162c8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:18:50.299 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:50 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:50.035+0000 7fa2162c8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:18:50.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:50 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:50.297+0000 7fa2162c8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:18:51.567 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:51.109+0000 7fa2162c8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:18:52.067 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:51.676+0000 7fa2162c8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:18:52.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:51.809+0000 7fa2162c8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:18:52.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:52 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:52.085+0000 7fa2162c8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:18:53.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:52 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:52.760+0000 7fa2162c8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:18:53.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:52 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:52.888+0000 7fa2162c8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:18:53.538 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:53 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:53.221+0000 7fa2162c8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:18:53.538 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:53 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:53.404+0000 7fa2162c8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:18:53.538 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:53 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:53.536+0000 7fa2162c8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:18:53.817 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:53 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:53.691+0000 7fa2162c8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:18:54.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:54 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:54.068+0000 7fa2162c8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:18:55.149 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:54 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:54.852+0000 7fa2162c8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:18:55.567 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:18:55.147+0000 7fa2162c8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:18:55.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-mon[28054]: Active manager daemon x restarted 2024-04-26T21:18:55.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-mon[28054]: Activating manager daemon x 2024-04-26T21:18:55.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-mon[28054]: osdmap e3: 0 total, 0 up, 0 in 2024-04-26T21:18:55.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-mon[28054]: mgrmap e10: x(active, starting, since 0.00978127s) 2024-04-26T21:18:55.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:18:55.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T21:18:55.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T21:18:55.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T21:18:55.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T21:18:55.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:55 smithi001 ceph-mon[28054]: Manager daemon x is now available 2024-04-26T21:18:56.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:18:56.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T21:18:56.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T21:18:56.533 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:56 smithi001 ceph-mon[28054]: mgrmap e11: x(active, since 1.01377s) 2024-04-26T21:18:56.831 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-04-26T21:18:56.839 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T21:18:56.839 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T21:18:56.839 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-04-26T21:18:56.839 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.828+0000 7f73f9a00700 1 Processor -- start 2024-04-26T21:18:56.839 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.829+0000 7f73f9a00700 1 -- start start 2024-04-26T21:18:56.839 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.829+0000 7f73f9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f410bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.829+0000 7f73f9a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f73f407ce10 con 0x7f73f4109930 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.829+0000 7f73f2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f410bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.830+0000 7f73f2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f410bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38306/0 (socket says 172.21.15.1:38306) 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.830+0000 7f73f2a00700 1 -- 172.21.15.1:0/3633759620 learned_addr learned my addr 172.21.15.1:0/3633759620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.830+0000 7f73f2a00700 1 -- 172.21.15.1:0/3633759620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f410c260 con 0x7f73f4109930 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.830+0000 7f73f2a00700 1 --2- 172.21.15.1:0/3633759620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f410bd20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f73e40098a0 tx=0x7f73e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810efb2c78de6ea4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.831+0000 7f73f2000700 1 -- 172.21.15.1:0/3633759620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73e4004160 con 0x7f73f4109930 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.831+0000 7f73f2000700 1 -- 172.21.15.1:0/3633759620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e4004300 con 0x7f73f4109930 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.831+0000 7f73f2000700 1 -- 172.21.15.1:0/3633759620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73e4011a20 con 0x7f73f4109930 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.831+0000 7f73f9a00700 1 -- 172.21.15.1:0/3633759620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 msgr2=0x7f73f410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.831+0000 7f73f9a00700 1 --2- 172.21.15.1:0/3633759620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f410bd20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f73e40098a0 tx=0x7f73e4009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.832+0000 7f73f9a00700 1 -- 172.21.15.1:0/3633759620 shutdown_connections 2024-04-26T21:18:56.840 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.832+0000 7f73f9a00700 1 --2- 172.21.15.1:0/3633759620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f410bd20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.832+0000 7f73f9a00700 1 -- 172.21.15.1:0/3633759620 >> 172.21.15.1:0/3633759620 conn(0x7f73f4103340 msgr2=0x7f73f4105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.832+0000 7f73f9a00700 1 -- 172.21.15.1:0/3633759620 shutdown_connections 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.832+0000 7f73f9a00700 1 -- 172.21.15.1:0/3633759620 wait complete. 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.833+0000 7f73f9a00700 1 Processor -- start 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.833+0000 7f73f9a00700 1 -- start start 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.833+0000 7f73f9a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f419eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.833+0000 7f73f9a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f73f410d260 con 0x7f73f4109930 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.834+0000 7f73f2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f419eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.834+0000 7f73f2a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f419eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38308/0 (socket says 172.21.15.1:38308) 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.834+0000 7f73f2a00700 1 -- 172.21.15.1:0/868783684 learned_addr learned my addr 172.21.15.1:0/868783684 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.834+0000 7f73f2a00700 1 -- 172.21.15.1:0/868783684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f419f3f0 con 0x7f73f4109930 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.834+0000 7f73f2a00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f419eeb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f73e40116e0 tx=0x7f73e4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.834+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73e4003a60 con 0x7f73f4109930 2024-04-26T21:18:56.841 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.835+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e401b8f0 con 0x7f73f4109930 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.835+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f73e4019860 con 0x7f73f4109930 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.835+0000 7f73f9a00700 1 -- 172.21.15.1:0/868783684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f419f680 con 0x7f73f4109930 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.835+0000 7f73f9a00700 1 -- 172.21.15.1:0/868783684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f41a2370 con 0x7f73f4109930 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.836+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f73e401b440 con 0x7f73f4109930 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.836+0000 7f73eb400700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 0x7f73d803bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.836+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 --> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f73d803c1e0 con 0x7f73d8039610 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.836+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f73e404dcf0 con 0x7f73f4109930 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.837+0000 7f73ebe00700 1 -- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 msgr2=0x7f73d803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3159681858 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:48.837+0000 7f73ebe00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 0x7f73d803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:49.038+0000 7f73ebe00700 1 -- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 msgr2=0x7f73d803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3159681858 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:49.038+0000 7f73ebe00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 0x7f73d803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:49.439+0000 7f73ebe00700 1 -- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 msgr2=0x7f73d803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3159681858 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:49.439+0000 7f73ebe00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 0x7f73d803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:50.240+0000 7f73ebe00700 1 -- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 msgr2=0x7f73d803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3159681858 2024-04-26T21:18:56.842 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:50.240+0000 7f73ebe00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 0x7f73d803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:51.842+0000 7f73ebe00700 1 -- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 msgr2=0x7f73d803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3159681858 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:51.842+0000 7f73ebe00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 0x7f73d803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:55.046+0000 7f73ebe00700 1 -- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 msgr2=0x7f73d803bad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/3159681858 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:55.046+0000 7f73ebe00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 0x7f73d803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:55.158+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f73e400fb40 con 0x7f73f4109930 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:55.158+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 msgr2=0x7f73d803bad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:55.158+0000 7f73eb400700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 0x7f73d803bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.159+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f73e4029970 con 0x7f73f4109930 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.160+0000 7f73eb400700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73d803cff0 0x7f73d803f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.160+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f73e4027e50 con 0x7f73d803cff0 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.162+0000 7f73ebe00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73d803cff0 0x7f73d803f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.164+0000 7f73ebe00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73d803cff0 0x7f73d803f3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f73dc003a10 tx=0x7f73dc009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.166+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f73e4027e50 con 0x7f73d803cff0 2024-04-26T21:18:56.843 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.179+0000 7f73f9a00700 1 -- 172.21.15.1:0/868783684 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f73f419fb60 con 0x7f73d803cff0 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.179+0000 7f73eb400700 1 -- 172.21.15.1:0/868783684 <== mgr.14152 v2:172.21.15.1:6800/2786715792 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f73f419fb60 con 0x7f73d803cff0 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 -- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73d803cff0 msgr2=0x7f73d803f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73d803cff0 0x7f73d803f3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f73dc003a10 tx=0x7f73dc009270 comp rx=0 tx=0).stop 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 -- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 msgr2=0x7f73f419eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f419eeb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f73e40116e0 tx=0x7f73e4011aa0 comp rx=0 tx=0).stop 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 -- 172.21.15.1:0/868783684 shutdown_connections 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73d803cff0 0x7f73d803f3e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4109930 0x7f73f419eeb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 --2- 172.21.15.1:0/868783684 >> [v2:172.21.15.1:6800/3159681858,v1:172.21.15.1:6801/3159681858] conn(0x7f73d8039610 0x7f73d803bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 -- 172.21.15.1:0/868783684 >> 172.21.15.1:0/868783684 conn(0x7f73f4103340 msgr2=0x7f73f4103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 -- 172.21.15.1:0/868783684 shutdown_connections 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:56.180+0000 7f73f9a00700 1 -- 172.21.15.1:0/868783684 wait complete. 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:mgr epoch 9 is available 2024-04-26T21:18:56.844 INFO:teuthology.orchestra.run.smithi001.stdout:Generating a dashboard self-signed certificate... 2024-04-26T21:18:57.436 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:57 smithi001 ceph-mon[28054]: [26/Apr/2024:21:18:56] ENGINE Bus STARTING 2024-04-26T21:18:57.436 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:57 smithi001 ceph-mon[28054]: [26/Apr/2024:21:18:56] ENGINE Serving on https://172.21.15.1:7150 2024-04-26T21:18:57.436 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:57 smithi001 ceph-mon[28054]: [26/Apr/2024:21:18:56] ENGINE Serving on http://172.21.15.1:8765 2024-04-26T21:18:57.436 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:57 smithi001 ceph-mon[28054]: [26/Apr/2024:21:18:56] ENGINE Bus STARTED 2024-04-26T21:18:58.465 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:58 smithi001 ceph-mon[28054]: mgrmap e12: x(active, since 2s) 2024-04-26T21:18:58.465 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:58 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:18:59.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T21:18:59.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.066+0000 7ff0f0200700 1 Processor -- start 2024-04-26T21:18:59.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.066+0000 7ff0f0200700 1 -- start start 2024-04-26T21:18:59.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.067+0000 7ff0f0200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e807d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:59.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.067+0000 7ff0f0200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e807d850 con 0x7ff0e807ee20 2024-04-26T21:18:59.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.067+0000 7ff0ed800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e807d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:59.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.067+0000 7ff0ed800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e807d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36466/0 (socket says 172.21.15.1:36466) 2024-04-26T21:18:59.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.068+0000 7ff0ed800700 1 -- 172.21.15.1:0/2669832444 learned_addr learned my addr 172.21.15.1:0/2669832444 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:59.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.068+0000 7ff0ed800700 1 -- 172.21.15.1:0/2669832444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e807d9d0 con 0x7ff0e807ee20 2024-04-26T21:18:59.154 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.068+0000 7ff0ed800700 1 --2- 172.21.15.1:0/2669832444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e807d280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0098a0 tx=0x7ff0dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f9a15081b891d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.069+0000 7ff0e7e00700 1 -- 172.21.15.1:0/2669832444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0dc004160 con 0x7ff0e807ee20 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.069+0000 7ff0e7e00700 1 -- 172.21.15.1:0/2669832444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0dc004300 con 0x7ff0e807ee20 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.069+0000 7ff0e7e00700 1 -- 172.21.15.1:0/2669832444 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0dc011a20 con 0x7ff0e807ee20 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.070+0000 7ff0f0200700 1 -- 172.21.15.1:0/2669832444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 msgr2=0x7ff0e807d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.070+0000 7ff0f0200700 1 --2- 172.21.15.1:0/2669832444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e807d280 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0098a0 tx=0x7ff0dc009bb0 comp rx=0 tx=0).stop 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.070+0000 7ff0f0200700 1 -- 172.21.15.1:0/2669832444 shutdown_connections 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.070+0000 7ff0f0200700 1 --2- 172.21.15.1:0/2669832444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e807d280 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.070+0000 7ff0f0200700 1 -- 172.21.15.1:0/2669832444 >> 172.21.15.1:0/2669832444 conn(0x7ff0e8104450 msgr2=0x7ff0e8106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.070+0000 7ff0f0200700 1 -- 172.21.15.1:0/2669832444 shutdown_connections 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.070+0000 7ff0f0200700 1 -- 172.21.15.1:0/2669832444 wait complete. 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.071+0000 7ff0f0200700 1 Processor -- start 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.071+0000 7ff0f0200700 1 -- start start 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.072+0000 7ff0f0200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:59.155 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.072+0000 7ff0f0200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e810f4b0 con 0x7ff0e807ee20 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.072+0000 7ff0ed800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.072+0000 7ff0ed800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36482/0 (socket says 172.21.15.1:36482) 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.072+0000 7ff0ed800700 1 -- 172.21.15.1:0/3696448956 learned_addr learned my addr 172.21.15.1:0/3696448956 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.072+0000 7ff0ed800700 1 -- 172.21.15.1:0/3696448956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e819f460 con 0x7ff0e807ee20 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.073+0000 7ff0ed800700 1 --2- 172.21.15.1:0/3696448956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e819ef20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0116e0 tx=0x7ff0dc011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.073+0000 7ff0e6000700 1 -- 172.21.15.1:0/3696448956 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0dc003b20 con 0x7ff0e807ee20 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.073+0000 7ff0e6000700 1 -- 172.21.15.1:0/3696448956 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0dc01b8f0 con 0x7ff0e807ee20 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.073+0000 7ff0e6000700 1 -- 172.21.15.1:0/3696448956 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff0dc0238c0 con 0x7ff0e807ee20 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.073+0000 7ff0f0200700 1 -- 172.21.15.1:0/3696448956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e819f6f0 con 0x7ff0e807ee20 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.073+0000 7ff0f0200700 1 -- 172.21.15.1:0/3696448956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e81a23e0 con 0x7ff0e807ee20 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.074+0000 7ff0e6000700 1 -- 172.21.15.1:0/3696448956 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7ff0dc023a60 con 0x7ff0e807ee20 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.075+0000 7ff0e6000700 1 --2- 172.21.15.1:0/3696448956 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0d0039610 0x7ff0d003bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.075+0000 7ff0e6000700 1 -- 172.21.15.1:0/3696448956 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff0dc04e360 con 0x7ff0e807ee20 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.075+0000 7ff0f0200700 1 -- 172.21.15.1:0/3696448956 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0e8064a80 con 0x7ff0e807ee20 2024-04-26T21:18:59.156 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.075+0000 7ff0ece00700 1 --2- 172.21.15.1:0/3696448956 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0d0039610 0x7ff0d003bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.076+0000 7ff0ece00700 1 --2- 172.21.15.1:0/3696448956 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0d0039610 0x7ff0d003bad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4006fd0 tx=0x7ff0d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.081+0000 7ff0e6000700 1 -- 172.21.15.1:0/3696448956 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff0dc01f030 con 0x7ff0e807ee20 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.337+0000 7ff0f0200700 1 -- 172.21.15.1:0/3696448956 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7ff0e81055b0 con 0x7ff0d0039610 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.460+0000 7ff0e6000700 1 -- 172.21.15.1:0/3696448956 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7ff0e81055b0 con 0x7ff0d0039610 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.464+0000 7ff0f0200700 1 -- 172.21.15.1:0/3696448956 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0d0039610 msgr2=0x7ff0d003bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.464+0000 7ff0f0200700 1 --2- 172.21.15.1:0/3696448956 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0d0039610 0x7ff0d003bad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4006fd0 tx=0x7ff0d4006e40 comp rx=0 tx=0).stop 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.464+0000 7ff0f0200700 1 -- 172.21.15.1:0/3696448956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 msgr2=0x7ff0e819ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.464+0000 7ff0f0200700 1 --2- 172.21.15.1:0/3696448956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e819ef20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0116e0 tx=0x7ff0dc011aa0 comp rx=0 tx=0).stop 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.464+0000 7ff0f0200700 1 -- 172.21.15.1:0/3696448956 shutdown_connections 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.465+0000 7ff0f0200700 1 --2- 172.21.15.1:0/3696448956 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0d0039610 0x7ff0d003bad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.465+0000 7ff0f0200700 1 --2- 172.21.15.1:0/3696448956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0e807ee20 0x7ff0e819ef20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.465+0000 7ff0f0200700 1 -- 172.21.15.1:0/3696448956 >> 172.21.15.1:0/3696448956 conn(0x7ff0e8104450 msgr2=0x7ff0e8104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:18:59.157 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.465+0000 7ff0f0200700 1 -- 172.21.15.1:0/3696448956 shutdown_connections 2024-04-26T21:18:59.158 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:58.465+0000 7ff0f0200700 1 -- 172.21.15.1:0/3696448956 wait complete. 2024-04-26T21:18:59.158 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial admin user... 2024-04-26T21:18:59.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:59 smithi001 ceph-mon[28054]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:18:59.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:59 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:18:59.717 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:18:59 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:00.969 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$n/TDiJsO1.Al8WO2Ebkm5.cz2NjtUisarcakgI9C.qMx6ZejLSkNO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714166340, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T21:19:00.969 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.727+0000 7fcd88e00700 1 Processor -- start 2024-04-26T21:19:00.969 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.728+0000 7fcd88e00700 1 -- start start 2024-04-26T21:19:00.969 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.728+0000 7fcd88e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8410bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:00.969 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.728+0000 7fcd88e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8407cd90 con 0x7fcd841099b0 2024-04-26T21:19:00.969 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.728+0000 7fcd81e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8410bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:00.969 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.728+0000 7fcd81e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8410bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36498/0 (socket says 172.21.15.1:36498) 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.728+0000 7fcd81e00700 1 -- 172.21.15.1:0/701549453 learned_addr learned my addr 172.21.15.1:0/701549453 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.729+0000 7fcd81e00700 1 -- 172.21.15.1:0/701549453 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8410c2e0 con 0x7fcd841099b0 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.729+0000 7fcd81e00700 1 --2- 172.21.15.1:0/701549453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8410bda0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcd740098a0 tx=0x7fcd74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696721febe36c36d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.729+0000 7fcd80a00700 1 -- 172.21.15.1:0/701549453 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd74004160 con 0x7fcd841099b0 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.729+0000 7fcd80a00700 1 -- 172.21.15.1:0/701549453 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd74004300 con 0x7fcd841099b0 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.729+0000 7fcd80a00700 1 -- 172.21.15.1:0/701549453 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd74011a20 con 0x7fcd841099b0 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.730+0000 7fcd88e00700 1 -- 172.21.15.1:0/701549453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 msgr2=0x7fcd8410bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.730+0000 7fcd88e00700 1 --2- 172.21.15.1:0/701549453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8410bda0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcd740098a0 tx=0x7fcd74009bb0 comp rx=0 tx=0).stop 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.730+0000 7fcd88e00700 1 -- 172.21.15.1:0/701549453 shutdown_connections 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.730+0000 7fcd88e00700 1 --2- 172.21.15.1:0/701549453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8410bda0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.730+0000 7fcd88e00700 1 -- 172.21.15.1:0/701549453 >> 172.21.15.1:0/701549453 conn(0x7fcd84103340 msgr2=0x7fcd84105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.730+0000 7fcd88e00700 1 -- 172.21.15.1:0/701549453 shutdown_connections 2024-04-26T21:19:00.970 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.730+0000 7fcd88e00700 1 -- 172.21.15.1:0/701549453 wait complete. 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.731+0000 7fcd88e00700 1 Processor -- start 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.731+0000 7fcd88e00700 1 -- start start 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.732+0000 7fcd88e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8419ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.732+0000 7fcd88e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8410d230 con 0x7fcd841099b0 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.732+0000 7fcd81e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8419ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.732+0000 7fcd81e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8419ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36514/0 (socket says 172.21.15.1:36514) 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.732+0000 7fcd81e00700 1 -- 172.21.15.1:0/3272952196 learned_addr learned my addr 172.21.15.1:0/3272952196 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.734+0000 7fcd81e00700 1 -- 172.21.15.1:0/3272952196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd8419d210 con 0x7fcd841099b0 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.734+0000 7fcd81e00700 1 --2- 172.21.15.1:0/3272952196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8419ccd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcd740116e0 tx=0x7fcd74011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.735+0000 7fcd7aa00700 1 -- 172.21.15.1:0/3272952196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd74003a60 con 0x7fcd841099b0 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.735+0000 7fcd7aa00700 1 -- 172.21.15.1:0/3272952196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd7401b8f0 con 0x7fcd841099b0 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.735+0000 7fcd7aa00700 1 -- 172.21.15.1:0/3272952196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd74019860 con 0x7fcd841099b0 2024-04-26T21:19:00.971 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.735+0000 7fcd88e00700 1 -- 172.21.15.1:0/3272952196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd8419d4a0 con 0x7fcd841099b0 2024-04-26T21:19:00.974 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.735+0000 7fcd88e00700 1 -- 172.21.15.1:0/3272952196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd841a0180 con 0x7fcd841099b0 2024-04-26T21:19:00.974 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.736+0000 7fcd7aa00700 1 -- 172.21.15.1:0/3272952196 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fcd74019b60 con 0x7fcd841099b0 2024-04-26T21:19:00.974 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.736+0000 7fcd88e00700 1 -- 172.21.15.1:0/3272952196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd84064a80 con 0x7fcd841099b0 2024-04-26T21:19:00.974 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.736+0000 7fcd7aa00700 1 --2- 172.21.15.1:0/3272952196 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcd680395b0 0x7fcd6803ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:00.974 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.737+0000 7fcd7aa00700 1 -- 172.21.15.1:0/3272952196 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcd7404df00 con 0x7fcd841099b0 2024-04-26T21:19:00.974 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.737+0000 7fcd81400700 1 --2- 172.21.15.1:0/3272952196 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcd680395b0 0x7fcd6803ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:00.974 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.737+0000 7fcd81400700 1 --2- 172.21.15.1:0/3272952196 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcd680395b0 0x7fcd6803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcd70006fd0 tx=0x7fcd70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:00.974 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.742+0000 7fcd7aa00700 1 -- 172.21.15.1:0/3272952196 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcd7401f030 con 0x7fcd841099b0 2024-04-26T21:19:00.974 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:18:59.993+0000 7fcd88e00700 1 -- 172.21.15.1:0/3272952196 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fcd841a0350 con 0x7fcd680395b0 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.247+0000 7fcd7aa00700 1 -- 172.21.15.1:0/3272952196 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fcd841a0350 con 0x7fcd680395b0 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.250+0000 7fcd88e00700 1 -- 172.21.15.1:0/3272952196 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcd680395b0 msgr2=0x7fcd6803ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.250+0000 7fcd88e00700 1 --2- 172.21.15.1:0/3272952196 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcd680395b0 0x7fcd6803ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcd70006fd0 tx=0x7fcd70006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.250+0000 7fcd88e00700 1 -- 172.21.15.1:0/3272952196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 msgr2=0x7fcd8419ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.250+0000 7fcd88e00700 1 --2- 172.21.15.1:0/3272952196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8419ccd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcd740116e0 tx=0x7fcd74011aa0 comp rx=0 tx=0).stop 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.250+0000 7fcd88e00700 1 -- 172.21.15.1:0/3272952196 shutdown_connections 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.250+0000 7fcd88e00700 1 --2- 172.21.15.1:0/3272952196 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcd680395b0 0x7fcd6803ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.250+0000 7fcd88e00700 1 --2- 172.21.15.1:0/3272952196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd841099b0 0x7fcd8419ccd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.250+0000 7fcd88e00700 1 -- 172.21.15.1:0/3272952196 >> 172.21.15.1:0/3272952196 conn(0x7fcd84103340 msgr2=0x7fcd84103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.250+0000 7fcd88e00700 1 -- 172.21.15.1:0/3272952196 shutdown_connections 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:00.250+0000 7fcd88e00700 1 -- 172.21.15.1:0/3272952196 wait complete. 2024-04-26T21:19:00.975 INFO:teuthology.orchestra.run.smithi001.stdout:Fetching dashboard port number... 2024-04-26T21:19:01.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:01 smithi001 ceph-mon[28054]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:19:01.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:01 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:01.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:01 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:02.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T21:19:02.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.914+0000 7fc875e00700 1 Processor -- start 2024-04-26T21:19:02.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.915+0000 7fc875e00700 1 -- start start 2024-04-26T21:19:02.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.915+0000 7fc875e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87007d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:02.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.915+0000 7fc875e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc87007d850 con 0x7fc87007ee20 2024-04-26T21:19:02.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.916+0000 7fc86ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87007d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:02.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.916+0000 7fc86ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87007d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36518/0 (socket says 172.21.15.1:36518) 2024-04-26T21:19:02.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.916+0000 7fc86ea00700 1 -- 172.21.15.1:0/1389871514 learned_addr learned my addr 172.21.15.1:0/1389871514 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.916+0000 7fc86ea00700 1 -- 172.21.15.1:0/1389871514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87007d9d0 con 0x7fc87007ee20 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.917+0000 7fc86ea00700 1 --2- 172.21.15.1:0/1389871514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87007d280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc8600098a0 tx=0x7fc860009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f913e0a72bcb729 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.917+0000 7fc86d600700 1 -- 172.21.15.1:0/1389871514 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc860004160 con 0x7fc87007ee20 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.917+0000 7fc86d600700 1 -- 172.21.15.1:0/1389871514 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc860004300 con 0x7fc87007ee20 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.917+0000 7fc86d600700 1 -- 172.21.15.1:0/1389871514 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc860011a20 con 0x7fc87007ee20 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.917+0000 7fc875e00700 1 -- 172.21.15.1:0/1389871514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 msgr2=0x7fc87007d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.917+0000 7fc875e00700 1 --2- 172.21.15.1:0/1389871514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87007d280 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc8600098a0 tx=0x7fc860009bb0 comp rx=0 tx=0).stop 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.918+0000 7fc875e00700 1 -- 172.21.15.1:0/1389871514 shutdown_connections 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.918+0000 7fc875e00700 1 --2- 172.21.15.1:0/1389871514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87007d280 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.918+0000 7fc875e00700 1 -- 172.21.15.1:0/1389871514 >> 172.21.15.1:0/1389871514 conn(0x7fc870104410 msgr2=0x7fc870106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.918+0000 7fc875e00700 1 -- 172.21.15.1:0/1389871514 shutdown_connections 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.918+0000 7fc875e00700 1 -- 172.21.15.1:0/1389871514 wait complete. 2024-04-26T21:19:02.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.919+0000 7fc875e00700 1 Processor -- start 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.919+0000 7fc875e00700 1 -- start start 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.919+0000 7fc875e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.919+0000 7fc875e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc87010f470 con 0x7fc87007ee20 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.920+0000 7fc86ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.920+0000 7fc86ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87019eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36524/0 (socket says 172.21.15.1:36524) 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.920+0000 7fc86ea00700 1 -- 172.21.15.1:0/3034196308 learned_addr learned my addr 172.21.15.1:0/3034196308 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.920+0000 7fc86ea00700 1 -- 172.21.15.1:0/3034196308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87019f3f0 con 0x7fc87007ee20 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.920+0000 7fc86ea00700 1 --2- 172.21.15.1:0/3034196308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87019eeb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc860000c00 tx=0x7fc860011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.921+0000 7fc874a00700 1 -- 172.21.15.1:0/3034196308 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc860003aa0 con 0x7fc87007ee20 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.921+0000 7fc874a00700 1 -- 172.21.15.1:0/3034196308 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc86001b8f0 con 0x7fc87007ee20 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.921+0000 7fc874a00700 1 -- 172.21.15.1:0/3034196308 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc860023870 con 0x7fc87007ee20 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.921+0000 7fc875e00700 1 -- 172.21.15.1:0/3034196308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87019f680 con 0x7fc87007ee20 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.921+0000 7fc875e00700 1 -- 172.21.15.1:0/3034196308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8701a2370 con 0x7fc87007ee20 2024-04-26T21:19:02.881 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.922+0000 7fc874a00700 1 -- 172.21.15.1:0/3034196308 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fc86001b440 con 0x7fc87007ee20 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.922+0000 7fc875e00700 1 -- 172.21.15.1:0/3034196308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc870064a80 con 0x7fc87007ee20 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.923+0000 7fc874a00700 1 --2- 172.21.15.1:0/3034196308 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc854039660 0x7fc85403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.923+0000 7fc874a00700 1 -- 172.21.15.1:0/3034196308 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc86004df60 con 0x7fc87007ee20 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.923+0000 7fc86e000700 1 --2- 172.21.15.1:0/3034196308 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc854039660 0x7fc85403bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.924+0000 7fc86e000700 1 --2- 172.21.15.1:0/3034196308 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc854039660 0x7fc85403bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc85c006fd0 tx=0x7fc85c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:01.928+0000 7fc874a00700 1 -- 172.21.15.1:0/3034196308 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc86001f030 con 0x7fc87007ee20 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.038+0000 7fc874a00700 1 -- 172.21.15.1:0/3034196308 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fc86000b3e0 con 0x7fc87007ee20 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.175+0000 7fc875e00700 1 -- 172.21.15.1:0/3034196308 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fc8701a2540 con 0x7fc87007ee20 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.176+0000 7fc874a00700 1 -- 172.21.15.1:0/3034196308 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fc86002a2f0 con 0x7fc87007ee20 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.179+0000 7fc875e00700 1 -- 172.21.15.1:0/3034196308 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc854039660 msgr2=0x7fc85403bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.179+0000 7fc875e00700 1 --2- 172.21.15.1:0/3034196308 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc854039660 0x7fc85403bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc85c006fd0 tx=0x7fc85c006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.179+0000 7fc875e00700 1 -- 172.21.15.1:0/3034196308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 msgr2=0x7fc87019eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:02.882 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.179+0000 7fc875e00700 1 --2- 172.21.15.1:0/3034196308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87019eeb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc860000c00 tx=0x7fc860011aa0 comp rx=0 tx=0).stop 2024-04-26T21:19:02.883 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.179+0000 7fc875e00700 1 -- 172.21.15.1:0/3034196308 shutdown_connections 2024-04-26T21:19:02.883 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.179+0000 7fc875e00700 1 --2- 172.21.15.1:0/3034196308 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc854039660 0x7fc85403bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:02.883 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.179+0000 7fc875e00700 1 --2- 172.21.15.1:0/3034196308 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87007ee20 0x7fc87019eeb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:02.883 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.179+0000 7fc875e00700 1 -- 172.21.15.1:0/3034196308 >> 172.21.15.1:0/3034196308 conn(0x7fc870104410 msgr2=0x7fc870104de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:02.883 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.179+0000 7fc875e00700 1 -- 172.21.15.1:0/3034196308 shutdown_connections 2024-04-26T21:19:02.883 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:02.179+0000 7fc875e00700 1 -- 172.21.15.1:0/3034196308 wait complete. 2024-04-26T21:19:02.913 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-26T21:19:02.914 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-04-26T21:19:02.940 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-26T21:19:02.940 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-04-26T21:19:02.940 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-04-26T21:19:02.940 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T21:19:02.944 INFO:teuthology.orchestra.run.smithi001.stdout:Ceph Dashboard is now available at: 2024-04-26T21:19:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:02.944 INFO:teuthology.orchestra.run.smithi001.stdout: URL: https://smithi001.front.sepia.ceph.com:8443/ 2024-04-26T21:19:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: User: admin 2024-04-26T21:19:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: Password: ak9jxwb9lx 2024-04-26T21:19:02.945 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:02.945 INFO:teuthology.orchestra.run.smithi001.stdout:Saving cluster configuration to /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config directory 2024-04-26T21:19:02.946 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling autotune for osd_memory_target 2024-04-26T21:19:03.188 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:03 smithi001 ceph-mon[28054]: mgrmap e13: x(active, since 6s) 2024-04-26T21:19:03.188 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:03 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3034196308' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T21:19:05.015 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.982+0000 7f97a1800700 1 Processor -- start 2024-04-26T21:19:05.015 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.982+0000 7f97a1800700 1 -- start start 2024-04-26T21:19:05.015 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.982+0000 7f97a1800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c10b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:05.016 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.982+0000 7f97a1800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f979c10ba40 con 0x7f979c10b050 2024-04-26T21:19:05.016 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.983+0000 7f979aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c10b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:05.016 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.983+0000 7f979aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c10b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36532/0 (socket says 172.21.15.1:36532) 2024-04-26T21:19:05.017 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.983+0000 7f979aa00700 1 -- 172.21.15.1:0/3050409615 learned_addr learned my addr 172.21.15.1:0/3050409615 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:05.017 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.983+0000 7f979aa00700 1 -- 172.21.15.1:0/3050409615 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f979c10c250 con 0x7f979c10b050 2024-04-26T21:19:05.017 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.984+0000 7f979aa00700 1 --2- 172.21.15.1:0/3050409615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c10b470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f978c0098a0 tx=0x7f978c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4ea61481b8e02a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:05.017 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.984+0000 7f9799600700 1 -- 172.21.15.1:0/3050409615 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f978c004160 con 0x7f979c10b050 2024-04-26T21:19:05.017 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.984+0000 7f9799600700 1 -- 172.21.15.1:0/3050409615 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f978c004300 con 0x7f979c10b050 2024-04-26T21:19:05.018 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.984+0000 7f9799600700 1 -- 172.21.15.1:0/3050409615 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f978c011a20 con 0x7f979c10b050 2024-04-26T21:19:05.018 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.985+0000 7f97a1800700 1 -- 172.21.15.1:0/3050409615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 msgr2=0x7f979c10b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:05.019 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.985+0000 7f97a1800700 1 --2- 172.21.15.1:0/3050409615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c10b470 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f978c0098a0 tx=0x7f978c009bb0 comp rx=0 tx=0).stop 2024-04-26T21:19:05.019 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.985+0000 7f97a1800700 1 -- 172.21.15.1:0/3050409615 shutdown_connections 2024-04-26T21:19:05.019 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.985+0000 7f97a1800700 1 --2- 172.21.15.1:0/3050409615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c10b470 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:05.020 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.985+0000 7f97a1800700 1 -- 172.21.15.1:0/3050409615 >> 172.21.15.1:0/3050409615 conn(0x7f979c07e460 msgr2=0x7f979c07e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:05.020 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.985+0000 7f97a1800700 1 -- 172.21.15.1:0/3050409615 shutdown_connections 2024-04-26T21:19:05.021 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.985+0000 7f97a1800700 1 -- 172.21.15.1:0/3050409615 wait complete. 2024-04-26T21:19:05.021 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.986+0000 7f97a1800700 1 Processor -- start 2024-04-26T21:19:05.021 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.986+0000 7f97a1800700 1 -- start start 2024-04-26T21:19:05.021 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.986+0000 7f97a1800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:05.021 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.986+0000 7f97a1800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f979c10f600 con 0x7f979c10b050 2024-04-26T21:19:05.021 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.987+0000 7f979aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:05.023 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.987+0000 7f979aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36534/0 (socket says 172.21.15.1:36534) 2024-04-26T21:19:05.023 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.987+0000 7f979aa00700 1 -- 172.21.15.1:0/3558903790 learned_addr learned my addr 172.21.15.1:0/3558903790 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:05.023 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.987+0000 7f979aa00700 1 -- 172.21.15.1:0/3558903790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f979c19f400 con 0x7f979c10b050 2024-04-26T21:19:05.023 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.988+0000 7f979aa00700 1 --2- 172.21.15.1:0/3558903790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c19eec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f978c0116e0 tx=0x7f978c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:05.023 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.988+0000 7f9793400700 1 -- 172.21.15.1:0/3558903790 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f978c0039e0 con 0x7f979c10b050 2024-04-26T21:19:05.023 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.988+0000 7f9793400700 1 -- 172.21.15.1:0/3558903790 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f978c01b8f0 con 0x7f979c10b050 2024-04-26T21:19:05.023 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.988+0000 7f9793400700 1 -- 172.21.15.1:0/3558903790 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f978c023760 con 0x7f979c10b050 2024-04-26T21:19:05.024 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.988+0000 7f97a1800700 1 -- 172.21.15.1:0/3558903790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f979c19f690 con 0x7f979c10b050 2024-04-26T21:19:05.024 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.988+0000 7f97a1800700 1 -- 172.21.15.1:0/3558903790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f979c1a2380 con 0x7f979c10b050 2024-04-26T21:19:05.024 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.989+0000 7f9793400700 1 -- 172.21.15.1:0/3558903790 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f978c01b440 con 0x7f979c10b050 2024-04-26T21:19:05.024 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.989+0000 7f97a1800700 1 -- 172.21.15.1:0/3558903790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f979c064a80 con 0x7f979c10b050 2024-04-26T21:19:05.024 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.990+0000 7f9793400700 1 --2- 172.21.15.1:0/3558903790 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97800396b0 0x7f978003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:05.024 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.990+0000 7f9793400700 1 -- 172.21.15.1:0/3558903790 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f978c04e160 con 0x7f979c10b050 2024-04-26T21:19:05.024 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.990+0000 7f979a000700 1 --2- 172.21.15.1:0/3558903790 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97800396b0 0x7f978003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:05.024 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.991+0000 7f979a000700 1 --2- 172.21.15.1:0/3558903790 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97800396b0 0x7f978003bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9788006fd0 tx=0x7f9788006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:05.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:03.996+0000 7f9793400700 1 -- 172.21.15.1:0/3558903790 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f978c01f030 con 0x7f979c10b050 2024-04-26T21:19:05.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.230+0000 7f97a1800700 1 -- 172.21.15.1:0/3558903790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f979c1a2550 con 0x7f979c10b050 2024-04-26T21:19:05.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.231+0000 7f9793400700 1 -- 172.21.15.1:0/3558903790 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f978c01a3d0 con 0x7f979c10b050 2024-04-26T21:19:05.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.234+0000 7f97a1800700 1 -- 172.21.15.1:0/3558903790 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97800396b0 msgr2=0x7f978003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:05.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.234+0000 7f97a1800700 1 --2- 172.21.15.1:0/3558903790 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97800396b0 0x7f978003bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9788006fd0 tx=0x7f9788006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:05.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.234+0000 7f97a1800700 1 -- 172.21.15.1:0/3558903790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 msgr2=0x7f979c19eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:05.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.234+0000 7f97a1800700 1 --2- 172.21.15.1:0/3558903790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c19eec0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f978c0116e0 tx=0x7f978c011aa0 comp rx=0 tx=0).stop 2024-04-26T21:19:05.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.234+0000 7f97a1800700 1 -- 172.21.15.1:0/3558903790 shutdown_connections 2024-04-26T21:19:05.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.234+0000 7f97a1800700 1 --2- 172.21.15.1:0/3558903790 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97800396b0 0x7f978003bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:05.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.234+0000 7f97a1800700 1 --2- 172.21.15.1:0/3558903790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f979c10b050 0x7f979c19eec0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:05.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.234+0000 7f97a1800700 1 -- 172.21.15.1:0/3558903790 >> 172.21.15.1:0/3558903790 conn(0x7f979c07e460 msgr2=0x7f979c108340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:05.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.234+0000 7f97a1800700 1 -- 172.21.15.1:0/3558903790 shutdown_connections 2024-04-26T21:19:05.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:04.234+0000 7f97a1800700 1 -- 172.21.15.1:0/3558903790 wait complete. 2024-04-26T21:19:06.696 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.655+0000 7f5e74800700 1 Processor -- start 2024-04-26T21:19:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.656+0000 7f5e74800700 1 -- start start 2024-04-26T21:19:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.656+0000 7f5e74800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c107b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.656+0000 7f5e74800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c07cdf0 con 0x7f5e6c107740 2024-04-26T21:19:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.656+0000 7f5e71e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c107b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.657+0000 7f5e71e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c107b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36544/0 (socket says 172.21.15.1:36544) 2024-04-26T21:19:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.657+0000 7f5e71e00700 1 -- 172.21.15.1:0/1522469581 learned_addr learned my addr 172.21.15.1:0/1522469581 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:06.698 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.657+0000 7f5e71e00700 1 -- 172.21.15.1:0/1522469581 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6c1080a0 con 0x7f5e6c107740 2024-04-26T21:19:06.700 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.657+0000 7f5e71e00700 1 --2- 172.21.15.1:0/1522469581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c107b60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5e600098a0 tx=0x7f5e60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99644c7eaa5fb8d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:06.700 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.658+0000 7f5e70a00700 1 -- 172.21.15.1:0/1522469581 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e60004160 con 0x7f5e6c107740 2024-04-26T21:19:06.700 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.658+0000 7f5e70a00700 1 -- 172.21.15.1:0/1522469581 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e60004300 con 0x7f5e6c107740 2024-04-26T21:19:06.700 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.658+0000 7f5e70a00700 1 -- 172.21.15.1:0/1522469581 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e60011a20 con 0x7f5e6c107740 2024-04-26T21:19:06.700 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.658+0000 7f5e74800700 1 -- 172.21.15.1:0/1522469581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 msgr2=0x7f5e6c107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:06.700 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.658+0000 7f5e74800700 1 --2- 172.21.15.1:0/1522469581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c107b60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5e600098a0 tx=0x7f5e60009bb0 comp rx=0 tx=0).stop 2024-04-26T21:19:06.701 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.659+0000 7f5e74800700 1 -- 172.21.15.1:0/1522469581 shutdown_connections 2024-04-26T21:19:06.701 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.659+0000 7f5e74800700 1 --2- 172.21.15.1:0/1522469581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c107b60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:06.701 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.659+0000 7f5e74800700 1 -- 172.21.15.1:0/1522469581 >> 172.21.15.1:0/1522469581 conn(0x7f5e6c103340 msgr2=0x7f5e6c105760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:06.701 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.659+0000 7f5e74800700 1 -- 172.21.15.1:0/1522469581 shutdown_connections 2024-04-26T21:19:06.701 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.659+0000 7f5e74800700 1 -- 172.21.15.1:0/1522469581 wait complete. 2024-04-26T21:19:06.701 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.660+0000 7f5e74800700 1 Processor -- start 2024-04-26T21:19:06.706 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.660+0000 7f5e74800700 1 -- start start 2024-04-26T21:19:06.706 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.660+0000 7f5e74800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:06.706 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.660+0000 7f5e74800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c109110 con 0x7f5e6c107740 2024-04-26T21:19:06.706 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.661+0000 7f5e71e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:06.706 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.661+0000 7f5e71e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36560/0 (socket says 172.21.15.1:36560) 2024-04-26T21:19:06.706 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.661+0000 7f5e71e00700 1 -- 172.21.15.1:0/3709969167 learned_addr learned my addr 172.21.15.1:0/3709969167 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:06.706 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.661+0000 7f5e71e00700 1 -- 172.21.15.1:0/3709969167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6c19f400 con 0x7f5e6c107740 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.661+0000 7f5e71e00700 1 --2- 172.21.15.1:0/3709969167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c19eec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5e600116e0 tx=0x7f5e60011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.662+0000 7f5e66a00700 1 -- 172.21.15.1:0/3709969167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e60003a60 con 0x7f5e6c107740 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.662+0000 7f5e66a00700 1 -- 172.21.15.1:0/3709969167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e6001b8f0 con 0x7f5e6c107740 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.662+0000 7f5e66a00700 1 -- 172.21.15.1:0/3709969167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5e60019860 con 0x7f5e6c107740 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.662+0000 7f5e74800700 1 -- 172.21.15.1:0/3709969167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e6c19f690 con 0x7f5e6c107740 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.662+0000 7f5e74800700 1 -- 172.21.15.1:0/3709969167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e6c1a2380 con 0x7f5e6c107740 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.663+0000 7f5e66a00700 1 -- 172.21.15.1:0/3709969167 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f5e6001b440 con 0x7f5e6c107740 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.663+0000 7f5e74800700 1 -- 172.21.15.1:0/3709969167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e6c064a80 con 0x7f5e6c107740 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.664+0000 7f5e66a00700 1 --2- 172.21.15.1:0/3709969167 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e540396b0 0x7f5e5403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.664+0000 7f5e66a00700 1 -- 172.21.15.1:0/3709969167 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5e6004de90 con 0x7f5e6c107740 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.664+0000 7f5e71400700 1 --2- 172.21.15.1:0/3709969167 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e540396b0 0x7f5e5403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.665+0000 7f5e71400700 1 --2- 172.21.15.1:0/3709969167 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e540396b0 0x7f5e5403bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c006fd0 tx=0x7f5e5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:06.707 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:05.669+0000 7f5e66a00700 1 -- 172.21.15.1:0/3709969167 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5e6001f030 con 0x7f5e6c107740 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.023+0000 7f5e74800700 1 -- 172.21.15.1:0/3709969167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f5e6c02e620 con 0x7f5e6c107740 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.028+0000 7f5e66a00700 1 -- 172.21.15.1:0/3709969167 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f5e600223b0 con 0x7f5e6c107740 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.031+0000 7f5e74800700 1 -- 172.21.15.1:0/3709969167 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e540396b0 msgr2=0x7f5e5403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.032+0000 7f5e74800700 1 --2- 172.21.15.1:0/3709969167 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e540396b0 0x7f5e5403bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c006fd0 tx=0x7f5e5c006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.032+0000 7f5e74800700 1 -- 172.21.15.1:0/3709969167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 msgr2=0x7f5e6c19eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.032+0000 7f5e74800700 1 --2- 172.21.15.1:0/3709969167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c19eec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5e600116e0 tx=0x7f5e60011aa0 comp rx=0 tx=0).stop 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.032+0000 7f5e74800700 1 -- 172.21.15.1:0/3709969167 shutdown_connections 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.032+0000 7f5e74800700 1 --2- 172.21.15.1:0/3709969167 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e540396b0 0x7f5e5403bb70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.032+0000 7f5e74800700 1 --2- 172.21.15.1:0/3709969167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e6c107740 0x7f5e6c19eec0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.032+0000 7f5e74800700 1 -- 172.21.15.1:0/3709969167 >> 172.21.15.1:0/3709969167 conn(0x7f5e6c103340 msgr2=0x7f5e6c195b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.032+0000 7f5e74800700 1 -- 172.21.15.1:0/3709969167 shutdown_connections 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-04-26T21:19:06.032+0000 7f5e74800700 1 -- 172.21.15.1:0/3709969167 wait complete. 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T21:19:06.708 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: ceph telemetry on 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout:For more information see: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:06.709 INFO:teuthology.orchestra.run.smithi001.stdout:Bootstrap complete. 2024-04-26T21:19:06.803 INFO:tasks.cephadm:Fetching config... 2024-04-26T21:19:06.803 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:19:06.803 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T21:19:06.851 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T21:19:06.851 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:19:06.851 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T21:19:06.933 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T21:19:06.933 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:19:06.934 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T21:19:07.056 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T21:19:07.057 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:19:07.057 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T21:19:07.152 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T21:19:07.152 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCePt3ZKYGfsJpFkEgl6aRIKOdWMdEvJyJbwOefZ4nno7pSW2D3F1hKbgwvuuYk1lCyJjgseVOa9NWmWdEca54tzKrCVlva9tk7EbhjkTNGHPKID58mJotEku2LIFqoCNfxvzVroj2+mNyKaIKctSnS1dl/U6pGrWDQQyfObCXDiZ+9oYPTLG8tlCE92JBz+q4KXKAqyxqNqHqjhUJDOnw1EXInSUcFNMd15AZpc1Fi/lXycteCcBMXK24zyuUiu1vwvgDvuHhFRKGbJo7O1pnPeUVuGA0+iby0qkPNBVxwuXMeYMNyJOqTzYwm8qBS5UUMWaEjUt62RpCFllD+8I8cw9DdXgfn4O7YqX7WMpQ14GsefERhgvYPv4s+h+9rUgFXOJsolkCzHLnvmTwsRWTOOBMS1TN0Ku6uknmfaYWnxEFvDCWE9aiD8rwxjjKPTHzPZmRn7DDFuU+N3zLXXKv2109sp/QTiKb+gSrTNjD9VVneIEkilh6MsiIh2RsxtrU= ceph-44ef3d6e-0412-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T21:19:07.263 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:07 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3709969167' entity='client.admin' 2024-04-26T21:19:07.271 INFO:teuthology.orchestra.run.smithi001.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCePt3ZKYGfsJpFkEgl6aRIKOdWMdEvJyJbwOefZ4nno7pSW2D3F1hKbgwvuuYk1lCyJjgseVOa9NWmWdEca54tzKrCVlva9tk7EbhjkTNGHPKID58mJotEku2LIFqoCNfxvzVroj2+mNyKaIKctSnS1dl/U6pGrWDQQyfObCXDiZ+9oYPTLG8tlCE92JBz+q4KXKAqyxqNqHqjhUJDOnw1EXInSUcFNMd15AZpc1Fi/lXycteCcBMXK24zyuUiu1vwvgDvuHhFRKGbJo7O1pnPeUVuGA0+iby0qkPNBVxwuXMeYMNyJOqTzYwm8qBS5UUMWaEjUt62RpCFllD+8I8cw9DdXgfn4O7YqX7WMpQ14GsefERhgvYPv4s+h+9rUgFXOJsolkCzHLnvmTwsRWTOOBMS1TN0Ku6uknmfaYWnxEFvDCWE9aiD8rwxjjKPTHzPZmRn7DDFuU+N3zLXXKv2109sp/QTiKb+gSrTNjD9VVneIEkilh6MsiIh2RsxtrU= ceph-44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:19:07.303 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCePt3ZKYGfsJpFkEgl6aRIKOdWMdEvJyJbwOefZ4nno7pSW2D3F1hKbgwvuuYk1lCyJjgseVOa9NWmWdEca54tzKrCVlva9tk7EbhjkTNGHPKID58mJotEku2LIFqoCNfxvzVroj2+mNyKaIKctSnS1dl/U6pGrWDQQyfObCXDiZ+9oYPTLG8tlCE92JBz+q4KXKAqyxqNqHqjhUJDOnw1EXInSUcFNMd15AZpc1Fi/lXycteCcBMXK24zyuUiu1vwvgDvuHhFRKGbJo7O1pnPeUVuGA0+iby0qkPNBVxwuXMeYMNyJOqTzYwm8qBS5UUMWaEjUt62RpCFllD+8I8cw9DdXgfn4O7YqX7WMpQ14GsefERhgvYPv4s+h+9rUgFXOJsolkCzHLnvmTwsRWTOOBMS1TN0Ku6uknmfaYWnxEFvDCWE9aiD8rwxjjKPTHzPZmRn7DDFuU+N3zLXXKv2109sp/QTiKb+gSrTNjD9VVneIEkilh6MsiIh2RsxtrU= ceph-44ef3d6e-0412-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T21:19:07.345 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCePt3ZKYGfsJpFkEgl6aRIKOdWMdEvJyJbwOefZ4nno7pSW2D3F1hKbgwvuuYk1lCyJjgseVOa9NWmWdEca54tzKrCVlva9tk7EbhjkTNGHPKID58mJotEku2LIFqoCNfxvzVroj2+mNyKaIKctSnS1dl/U6pGrWDQQyfObCXDiZ+9oYPTLG8tlCE92JBz+q4KXKAqyxqNqHqjhUJDOnw1EXInSUcFNMd15AZpc1Fi/lXycteCcBMXK24zyuUiu1vwvgDvuHhFRKGbJo7O1pnPeUVuGA0+iby0qkPNBVxwuXMeYMNyJOqTzYwm8qBS5UUMWaEjUt62RpCFllD+8I8cw9DdXgfn4O7YqX7WMpQ14GsefERhgvYPv4s+h+9rUgFXOJsolkCzHLnvmTwsRWTOOBMS1TN0Ku6uknmfaYWnxEFvDCWE9aiD8rwxjjKPTHzPZmRn7DDFuU+N3zLXXKv2109sp/QTiKb+gSrTNjD9VVneIEkilh6MsiIh2RsxtrU= ceph-44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:19:07.357 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCePt3ZKYGfsJpFkEgl6aRIKOdWMdEvJyJbwOefZ4nno7pSW2D3F1hKbgwvuuYk1lCyJjgseVOa9NWmWdEca54tzKrCVlva9tk7EbhjkTNGHPKID58mJotEku2LIFqoCNfxvzVroj2+mNyKaIKctSnS1dl/U6pGrWDQQyfObCXDiZ+9oYPTLG8tlCE92JBz+q4KXKAqyxqNqHqjhUJDOnw1EXInSUcFNMd15AZpc1Fi/lXycteCcBMXK24zyuUiu1vwvgDvuHhFRKGbJo7O1pnPeUVuGA0+iby0qkPNBVxwuXMeYMNyJOqTzYwm8qBS5UUMWaEjUt62RpCFllD+8I8cw9DdXgfn4O7YqX7WMpQ14GsefERhgvYPv4s+h+9rUgFXOJsolkCzHLnvmTwsRWTOOBMS1TN0Ku6uknmfaYWnxEFvDCWE9aiD8rwxjjKPTHzPZmRn7DDFuU+N3zLXXKv2109sp/QTiKb+gSrTNjD9VVneIEkilh6MsiIh2RsxtrU= ceph-44ef3d6e-0412-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T21:19:07.396 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCePt3ZKYGfsJpFkEgl6aRIKOdWMdEvJyJbwOefZ4nno7pSW2D3F1hKbgwvuuYk1lCyJjgseVOa9NWmWdEca54tzKrCVlva9tk7EbhjkTNGHPKID58mJotEku2LIFqoCNfxvzVroj2+mNyKaIKctSnS1dl/U6pGrWDQQyfObCXDiZ+9oYPTLG8tlCE92JBz+q4KXKAqyxqNqHqjhUJDOnw1EXInSUcFNMd15AZpc1Fi/lXycteCcBMXK24zyuUiu1vwvgDvuHhFRKGbJo7O1pnPeUVuGA0+iby0qkPNBVxwuXMeYMNyJOqTzYwm8qBS5UUMWaEjUt62RpCFllD+8I8cw9DdXgfn4O7YqX7WMpQ14GsefERhgvYPv4s+h+9rUgFXOJsolkCzHLnvmTwsRWTOOBMS1TN0Ku6uknmfaYWnxEFvDCWE9aiD8rwxjjKPTHzPZmRn7DDFuU+N3zLXXKv2109sp/QTiKb+gSrTNjD9VVneIEkilh6MsiIh2RsxtrU= ceph-44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:19:07.409 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T21:19:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:19:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.249+0000 7f00ae000700 1 -- 172.21.15.1:0/3424691577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00a8103910 msgr2=0x7f00a8105d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.249+0000 7f00ae000700 1 --2- 172.21.15.1:0/3424691577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00a8103910 0x7f00a8105d00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0098009910 tx=0x7f0098009c20 comp rx=0 tx=0).stop 2024-04-26T21:19:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.249+0000 7f00ae000700 1 -- 172.21.15.1:0/3424691577 shutdown_connections 2024-04-26T21:19:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.249+0000 7f00ae000700 1 --2- 172.21.15.1:0/3424691577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00a8103910 0x7f00a8105d00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.249+0000 7f00ae000700 1 -- 172.21.15.1:0/3424691577 >> 172.21.15.1:0/3424691577 conn(0x7f00a80fd320 msgr2=0x7f00a80ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.249+0000 7f00ae000700 1 -- 172.21.15.1:0/3424691577 shutdown_connections 2024-04-26T21:19:10.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.249+0000 7f00ae000700 1 -- 172.21.15.1:0/3424691577 wait complete. 2024-04-26T21:19:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.250+0000 7f00ae000700 1 Processor -- start 2024-04-26T21:19:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.250+0000 7f00ae000700 1 -- start start 2024-04-26T21:19:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.250+0000 7f00ae000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00a8103910 0x7f00a819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.250+0000 7f00ae000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f00a8108d60 con 0x7f00a8103910 2024-04-26T21:19:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.251+0000 7f00a7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00a8103910 0x7f00a819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.251+0000 7f00a7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00a8103910 0x7f00a819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32996/0 (socket says 172.21.15.1:32996) 2024-04-26T21:19:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.251+0000 7f00a7400700 1 -- 172.21.15.1:0/838458316 learned_addr learned my addr 172.21.15.1:0/838458316 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.251+0000 7f00a7400700 1 -- 172.21.15.1:0/838458316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a819ab90 con 0x7f00a8103910 2024-04-26T21:19:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.252+0000 7f00a7400700 1 --2- 172.21.15.1:0/838458316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00a8103910 0x7f00a819a650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0098005190 tx=0x7f0098004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:10.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.252+0000 7f009fe00700 1 -- 172.21.15.1:0/838458316 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0098027070 con 0x7f00a8103910 2024-04-26T21:19:10.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.252+0000 7f009fe00700 1 -- 172.21.15.1:0/838458316 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0098022470 con 0x7f00a8103910 2024-04-26T21:19:10.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.252+0000 7f009fe00700 1 -- 172.21.15.1:0/838458316 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f009800f5f0 con 0x7f00a8103910 2024-04-26T21:19:10.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.252+0000 7f00ae000700 1 -- 172.21.15.1:0/838458316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a819ae80 con 0x7f00a8103910 2024-04-26T21:19:10.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.252+0000 7f00ae000700 1 -- 172.21.15.1:0/838458316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a819db10 con 0x7f00a8103910 2024-04-26T21:19:10.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.254+0000 7f009fe00700 1 -- 172.21.15.1:0/838458316 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f009802a070 con 0x7f00a8103910 2024-04-26T21:19:10.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.254+0000 7f00ae000700 1 -- 172.21.15.1:0/838458316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a81094e0 con 0x7f00a8103910 2024-04-26T21:19:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.254+0000 7f009fe00700 1 --2- 172.21.15.1:0/838458316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f008c039700 0x7f008c03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.254+0000 7f009fe00700 1 -- 172.21.15.1:0/838458316 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f009801e030 con 0x7f00a8103910 2024-04-26T21:19:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.255+0000 7f00a6a00700 1 --2- 172.21.15.1:0/838458316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f008c039700 0x7f008c03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:10.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.255+0000 7f00a6a00700 1 --2- 172.21.15.1:0/838458316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f008c039700 0x7f008c03bbc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0094006fd0 tx=0x7f0094006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:10.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.261+0000 7f009fe00700 1 -- 172.21.15.1:0/838458316 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0098013070 con 0x7f00a8103910 2024-04-26T21:19:10.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.491+0000 7f00ae000700 1 -- 172.21.15.1:0/838458316 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f00a802e6e0 con 0x7f00a8103910 2024-04-26T21:19:10.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.496+0000 7f009fe00700 1 -- 172.21.15.1:0/838458316 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f00980310c0 con 0x7f00a8103910 2024-04-26T21:19:10.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.499+0000 7f00ae000700 1 -- 172.21.15.1:0/838458316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f008c039700 msgr2=0x7f008c03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:10.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.500+0000 7f00ae000700 1 --2- 172.21.15.1:0/838458316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f008c039700 0x7f008c03bbc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0094006fd0 tx=0x7f0094006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:10.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.500+0000 7f00ae000700 1 -- 172.21.15.1:0/838458316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00a8103910 msgr2=0x7f00a819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:10.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.500+0000 7f00ae000700 1 --2- 172.21.15.1:0/838458316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00a8103910 0x7f00a819a650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0098005190 tx=0x7f0098004f80 comp rx=0 tx=0).stop 2024-04-26T21:19:10.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.500+0000 7f00ae000700 1 -- 172.21.15.1:0/838458316 shutdown_connections 2024-04-26T21:19:10.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.500+0000 7f00ae000700 1 --2- 172.21.15.1:0/838458316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f008c039700 0x7f008c03bbc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:10.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.500+0000 7f00ae000700 1 --2- 172.21.15.1:0/838458316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00a8103910 0x7f00a819a650 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:10.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.500+0000 7f00ae000700 1 -- 172.21.15.1:0/838458316 >> 172.21.15.1:0/838458316 conn(0x7f00a80fd320 msgr2=0x7f00a80ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:10.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.500+0000 7f00ae000700 1 -- 172.21.15.1:0/838458316 shutdown_connections 2024-04-26T21:19:10.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:10.500+0000 7f00ae000700 1 -- 172.21.15.1:0/838458316 wait complete. 2024-04-26T21:19:11.156 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:11.157 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:11.157 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi001", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:19:11.157 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:11.157 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:19:11.157 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:11.157 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:11 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/838458316' entity='client.admin' 2024-04-26T21:19:11.157 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:19:11.157 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:11.238 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T21:19:11.238 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T21:19:11.678 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:19:13.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.654+0000 7f4a32600700 1 -- 172.21.15.1:0/3027096674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a2c104e80 msgr2=0x7f4a2c1052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:13.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.654+0000 7f4a32600700 1 --2- 172.21.15.1:0/3027096674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a2c104e80 0x7f4a2c1052a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a20009910 tx=0x7f4a20009c20 comp rx=0 tx=0).stop 2024-04-26T21:19:13.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.654+0000 7f4a32600700 1 -- 172.21.15.1:0/3027096674 shutdown_connections 2024-04-26T21:19:13.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.654+0000 7f4a32600700 1 --2- 172.21.15.1:0/3027096674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a2c104e80 0x7f4a2c1052a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:13.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.654+0000 7f4a32600700 1 -- 172.21.15.1:0/3027096674 >> 172.21.15.1:0/3027096674 conn(0x7f4a2c100470 msgr2=0x7f4a2c1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:13.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.655+0000 7f4a32600700 1 -- 172.21.15.1:0/3027096674 shutdown_connections 2024-04-26T21:19:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.655+0000 7f4a32600700 1 -- 172.21.15.1:0/3027096674 wait complete. 2024-04-26T21:19:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.656+0000 7f4a32600700 1 Processor -- start 2024-04-26T21:19:13.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.656+0000 7f4a32600700 1 -- start start 2024-04-26T21:19:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.656+0000 7f4a32600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a2c104e80 0x7f4a2c19a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.656+0000 7f4a32600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c10ad80 con 0x7f4a2c104e80 2024-04-26T21:19:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.656+0000 7f4a31200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a2c104e80 0x7f4a2c19a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.656+0000 7f4a31200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a2c104e80 0x7f4a2c19a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33010/0 (socket says 172.21.15.1:33010) 2024-04-26T21:19:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.657+0000 7f4a31200700 1 -- 172.21.15.1:0/872882304 learned_addr learned my addr 172.21.15.1:0/872882304 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.657+0000 7f4a31200700 1 -- 172.21.15.1:0/872882304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2c19aa70 con 0x7f4a2c104e80 2024-04-26T21:19:13.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.657+0000 7f4a31200700 1 --2- 172.21.15.1:0/872882304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a2c104e80 0x7f4a2c19a530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4a20005800 tx=0x7f4a20009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.657+0000 7f4a29600700 1 -- 172.21.15.1:0/872882304 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a20027070 con 0x7f4a2c104e80 2024-04-26T21:19:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.658+0000 7f4a29600700 1 -- 172.21.15.1:0/872882304 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a2000be10 con 0x7f4a2c104e80 2024-04-26T21:19:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.658+0000 7f4a32600700 1 -- 172.21.15.1:0/872882304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2c19ad00 con 0x7f4a2c104e80 2024-04-26T21:19:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.658+0000 7f4a29600700 1 -- 172.21.15.1:0/872882304 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a2000f460 con 0x7f4a2c104e80 2024-04-26T21:19:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.658+0000 7f4a32600700 1 -- 172.21.15.1:0/872882304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2c19d9f0 con 0x7f4a2c104e80 2024-04-26T21:19:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.659+0000 7f4a29600700 1 -- 172.21.15.1:0/872882304 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f4a2002a070 con 0x7f4a2c104e80 2024-04-26T21:19:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.659+0000 7f4a32600700 1 -- 172.21.15.1:0/872882304 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a2c10b500 con 0x7f4a2c104e80 2024-04-26T21:19:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.659+0000 7f4a29600700 1 --2- 172.21.15.1:0/872882304 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4a140396b0 0x7f4a1403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.660+0000 7f4a29600700 1 -- 172.21.15.1:0/872882304 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4a2001e030 con 0x7f4a2c104e80 2024-04-26T21:19:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.660+0000 7f4a2be00700 1 --2- 172.21.15.1:0/872882304 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4a140396b0 0x7f4a1403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.661+0000 7f4a2be00700 1 --2- 172.21.15.1:0/872882304 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4a140396b0 0x7f4a1403bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4a18006fd0 tx=0x7f4a18006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:13.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.666+0000 7f4a29600700 1 -- 172.21.15.1:0/872882304 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4a20013070 con 0x7f4a2c104e80 2024-04-26T21:19:13.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.899+0000 7f4a32600700 1 -- 172.21.15.1:0/872882304 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f4a2c02e5a0 con 0x7f4a140396b0 2024-04-26T21:19:13.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.907+0000 7f4a29600700 1 -- 172.21.15.1:0/872882304 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4a2c02e5a0 con 0x7f4a140396b0 2024-04-26T21:19:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.910+0000 7f4a32600700 1 -- 172.21.15.1:0/872882304 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4a140396b0 msgr2=0x7f4a1403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.910+0000 7f4a32600700 1 --2- 172.21.15.1:0/872882304 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4a140396b0 0x7f4a1403bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4a18006fd0 tx=0x7f4a18006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.910+0000 7f4a32600700 1 -- 172.21.15.1:0/872882304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a2c104e80 msgr2=0x7f4a2c19a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.910+0000 7f4a32600700 1 --2- 172.21.15.1:0/872882304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a2c104e80 0x7f4a2c19a530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4a20005800 tx=0x7f4a20009f60 comp rx=0 tx=0).stop 2024-04-26T21:19:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.911+0000 7f4a32600700 1 -- 172.21.15.1:0/872882304 shutdown_connections 2024-04-26T21:19:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.911+0000 7f4a32600700 1 --2- 172.21.15.1:0/872882304 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4a140396b0 0x7f4a1403bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.911+0000 7f4a32600700 1 --2- 172.21.15.1:0/872882304 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a2c104e80 0x7f4a2c19a530 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.911+0000 7f4a32600700 1 -- 172.21.15.1:0/872882304 >> 172.21.15.1:0/872882304 conn(0x7f4a2c100470 msgr2=0x7f4a2c100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.911+0000 7f4a32600700 1 -- 172.21.15.1:0/872882304 shutdown_connections 2024-04-26T21:19:13.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:13.911+0000 7f4a32600700 1 -- 172.21.15.1:0/872882304 wait complete. 2024-04-26T21:19:14.667 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi038 2024-04-26T21:19:14.667 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:19:14.667 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/etc/ceph/ceph.conf 2024-04-26T21:19:14.687 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:19:14.687 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:19:14.745 INFO:tasks.cephadm:Adding host smithi038 to orchestrator... 2024-04-26T21:19:14.746 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch host add smithi038 2024-04-26T21:19:15.175 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:19:15.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:14 smithi001 ceph-mon[28054]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:19:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:14 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:14 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:19:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:14 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:19:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:14 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:19:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:14 smithi001 ceph-mon[28054]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T21:19:16.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:15 smithi001 ceph-mon[28054]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:19:16.991 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:16 smithi001 ceph-mon[28054]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:16.991 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:16 smithi001 ceph-mon[28054]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:19:16.991 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:16 smithi001 ceph-mon[28054]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:19:16.991 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:16.991 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:16.991 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:17.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.169+0000 7f70dec00700 1 -- 172.21.15.1:0/489471783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70d8108ec0 msgr2=0x7f70d81092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:17.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.169+0000 7f70dec00700 1 --2- 172.21.15.1:0/489471783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70d8108ec0 0x7f70d81092a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f70c8009910 tx=0x7f70c8009c20 comp rx=0 tx=0).stop 2024-04-26T21:19:17.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.169+0000 7f70dec00700 1 -- 172.21.15.1:0/489471783 shutdown_connections 2024-04-26T21:19:17.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.169+0000 7f70dec00700 1 --2- 172.21.15.1:0/489471783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70d8108ec0 0x7f70d81092a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:17.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.169+0000 7f70dec00700 1 -- 172.21.15.1:0/489471783 >> 172.21.15.1:0/489471783 conn(0x7f70d8077880 msgr2=0x7f70d8077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:17.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.169+0000 7f70dec00700 1 -- 172.21.15.1:0/489471783 shutdown_connections 2024-04-26T21:19:17.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.169+0000 7f70dec00700 1 -- 172.21.15.1:0/489471783 wait complete. 2024-04-26T21:19:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.171+0000 7f70dec00700 1 Processor -- start 2024-04-26T21:19:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.171+0000 7f70dec00700 1 -- start start 2024-04-26T21:19:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.171+0000 7f70dec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70d8108ec0 0x7f70d819e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.171+0000 7f70dec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f70d810d300 con 0x7f70d8108ec0 2024-04-26T21:19:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.171+0000 7f70d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70d8108ec0 0x7f70d819e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:17.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.172+0000 7f70d7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70d8108ec0 0x7f70d819e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33018/0 (socket says 172.21.15.1:33018) 2024-04-26T21:19:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.172+0000 7f70d7e00700 1 -- 172.21.15.1:0/3694998767 learned_addr learned my addr 172.21.15.1:0/3694998767 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.172+0000 7f70d7e00700 1 -- 172.21.15.1:0/3694998767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d819eb60 con 0x7f70d8108ec0 2024-04-26T21:19:17.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.172+0000 7f70d7e00700 1 --2- 172.21.15.1:0/3694998767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70d8108ec0 0x7f70d819e620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f70c8006010 tx=0x7f70c8005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:17.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.173+0000 7f70d5600700 1 -- 172.21.15.1:0/3694998767 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70c8026070 con 0x7f70d8108ec0 2024-04-26T21:19:17.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.173+0000 7f70d5600700 1 -- 172.21.15.1:0/3694998767 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70c800bba0 con 0x7f70d8108ec0 2024-04-26T21:19:17.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.173+0000 7f70d5600700 1 -- 172.21.15.1:0/3694998767 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70c8003910 con 0x7f70d8108ec0 2024-04-26T21:19:17.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.173+0000 7f70dec00700 1 -- 172.21.15.1:0/3694998767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d819ee50 con 0x7f70d8108ec0 2024-04-26T21:19:17.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.173+0000 7f70dec00700 1 -- 172.21.15.1:0/3694998767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d81987a0 con 0x7f70d8108ec0 2024-04-26T21:19:17.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.174+0000 7f70d5600700 1 -- 172.21.15.1:0/3694998767 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f70c8029070 con 0x7f70d8108ec0 2024-04-26T21:19:17.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.174+0000 7f70dec00700 1 -- 172.21.15.1:0/3694998767 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d8104570 con 0x7f70d8108ec0 2024-04-26T21:19:17.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.174+0000 7f70d5600700 1 --2- 172.21.15.1:0/3694998767 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70bc039650 0x7f70bc03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:17.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.174+0000 7f70d5600700 1 -- 172.21.15.1:0/3694998767 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f70c801e070 con 0x7f70d8108ec0 2024-04-26T21:19:17.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.175+0000 7f70cfe00700 1 --2- 172.21.15.1:0/3694998767 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70bc039650 0x7f70bc03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:17.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.176+0000 7f70cfe00700 1 --2- 172.21.15.1:0/3694998767 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70bc039650 0x7f70bc03bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f70c0006fd0 tx=0x7f70c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:17.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.180+0000 7f70d5600700 1 -- 172.21.15.1:0/3694998767 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f70c8013070 con 0x7f70d8108ec0 2024-04-26T21:19:17.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:17.420+0000 7f70dec00700 1 -- 172.21.15.1:0/3694998767 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "target": ["mon-mgr", ""]}) -- 0x7f70d8064ae0 con 0x7f70bc039650 2024-04-26T21:19:19.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:18 smithi001 ceph-mon[28054]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:19.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:18 smithi001 ceph-mon[28054]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:19:20.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:19 smithi001 ceph-mon[28054]: Deploying cephadm binary to smithi038 2024-04-26T21:19:21.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:20 smithi001 ceph-mon[28054]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.290+0000 7f70d5600700 1 -- 172.21.15.1:0/3694998767 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f70d8064ae0 con 0x7f70bc039650 2024-04-26T21:19:22.292 INFO:teuthology.orchestra.run.smithi001.stdout:Added host 'smithi038' with addr '172.21.15.38' 2024-04-26T21:19:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.294+0000 7f70dec00700 1 -- 172.21.15.1:0/3694998767 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70bc039650 msgr2=0x7f70bc03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.294+0000 7f70dec00700 1 --2- 172.21.15.1:0/3694998767 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70bc039650 0x7f70bc03bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f70c0006fd0 tx=0x7f70c0006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.294+0000 7f70dec00700 1 -- 172.21.15.1:0/3694998767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70d8108ec0 msgr2=0x7f70d819e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:22.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.294+0000 7f70dec00700 1 --2- 172.21.15.1:0/3694998767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70d8108ec0 0x7f70d819e620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f70c8006010 tx=0x7f70c8005780 comp rx=0 tx=0).stop 2024-04-26T21:19:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.294+0000 7f70dec00700 1 -- 172.21.15.1:0/3694998767 shutdown_connections 2024-04-26T21:19:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.294+0000 7f70dec00700 1 --2- 172.21.15.1:0/3694998767 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70bc039650 0x7f70bc03bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.294+0000 7f70dec00700 1 --2- 172.21.15.1:0/3694998767 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70d8108ec0 0x7f70d819e620 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.294+0000 7f70dec00700 1 -- 172.21.15.1:0/3694998767 >> 172.21.15.1:0/3694998767 conn(0x7f70d8077880 msgr2=0x7f70d80ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.294+0000 7f70dec00700 1 -- 172.21.15.1:0/3694998767 shutdown_connections 2024-04-26T21:19:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:22.294+0000 7f70dec00700 1 -- 172.21.15.1:0/3694998767 wait complete. 2024-04-26T21:19:23.038 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T21:19:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:22 smithi001 ceph-mon[28054]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:22 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:22 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:19:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:22 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:23.501 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:19:24.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:23 smithi001 ceph-mon[28054]: Added host smithi038 2024-04-26T21:19:24.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:25.214 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:24 smithi001 ceph-mon[28054]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:25.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.537+0000 7fe749c00700 1 -- 172.21.15.1:0/807563026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7441052a0 msgr2=0x7fe744105680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.537+0000 7fe749c00700 1 --2- 172.21.15.1:0/807563026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7441052a0 0x7fe744105680 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe734009910 tx=0x7fe734009c20 comp rx=0 tx=0).stop 2024-04-26T21:19:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.537+0000 7fe749c00700 1 -- 172.21.15.1:0/807563026 shutdown_connections 2024-04-26T21:19:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.537+0000 7fe749c00700 1 --2- 172.21.15.1:0/807563026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7441052a0 0x7fe744105680 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.537+0000 7fe749c00700 1 -- 172.21.15.1:0/807563026 >> 172.21.15.1:0/807563026 conn(0x7fe744100b30 msgr2=0x7fe744102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.537+0000 7fe749c00700 1 -- 172.21.15.1:0/807563026 shutdown_connections 2024-04-26T21:19:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.538+0000 7fe749c00700 1 -- 172.21.15.1:0/807563026 wait complete. 2024-04-26T21:19:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.539+0000 7fe749c00700 1 Processor -- start 2024-04-26T21:19:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.539+0000 7fe749c00700 1 -- start start 2024-04-26T21:19:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.539+0000 7fe749c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7441052a0 0x7fe7441a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.539+0000 7fe749c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe74410f520 con 0x7fe7441052a0 2024-04-26T21:19:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.540+0000 7fe742a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7441052a0 0x7fe7441a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.540+0000 7fe742a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7441052a0 0x7fe7441a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52788/0 (socket says 172.21.15.1:52788) 2024-04-26T21:19:25.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.540+0000 7fe742a00700 1 -- 172.21.15.1:0/4074791755 learned_addr learned my addr 172.21.15.1:0/4074791755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:25.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.540+0000 7fe742a00700 1 -- 172.21.15.1:0/4074791755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7441a0c60 con 0x7fe7441052a0 2024-04-26T21:19:25.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.541+0000 7fe742a00700 1 --2- 172.21.15.1:0/4074791755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7441052a0 0x7fe7441a0720 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe734006010 tx=0x7fe734005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:25.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.541+0000 7fe73b400700 1 -- 172.21.15.1:0/4074791755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe734026070 con 0x7fe7441052a0 2024-04-26T21:19:25.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.541+0000 7fe73b400700 1 -- 172.21.15.1:0/4074791755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe73400bba0 con 0x7fe7441052a0 2024-04-26T21:19:25.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.541+0000 7fe749c00700 1 -- 172.21.15.1:0/4074791755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7441a0f50 con 0x7fe7441052a0 2024-04-26T21:19:25.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.541+0000 7fe73b400700 1 -- 172.21.15.1:0/4074791755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe734003910 con 0x7fe7441052a0 2024-04-26T21:19:25.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.541+0000 7fe749c00700 1 -- 172.21.15.1:0/4074791755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74419a8e0 con 0x7fe7441052a0 2024-04-26T21:19:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.542+0000 7fe749c00700 1 -- 172.21.15.1:0/4074791755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe744106780 con 0x7fe7441052a0 2024-04-26T21:19:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.542+0000 7fe73b400700 1 -- 172.21.15.1:0/4074791755 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fe734029070 con 0x7fe7441052a0 2024-04-26T21:19:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.543+0000 7fe73b400700 1 --2- 172.21.15.1:0/4074791755 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe728039650 0x7fe72803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:25.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.543+0000 7fe73b400700 1 -- 172.21.15.1:0/4074791755 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe73401e070 con 0x7fe7441052a0 2024-04-26T21:19:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.543+0000 7fe742000700 1 --2- 172.21.15.1:0/4074791755 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe728039650 0x7fe72803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:25.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.544+0000 7fe742000700 1 --2- 172.21.15.1:0/4074791755 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe728039650 0x7fe72803bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe730006fd0 tx=0x7fe730006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:25.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.549+0000 7fe73b400700 1 -- 172.21.15.1:0/4074791755 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe734013070 con 0x7fe7441052a0 2024-04-26T21:19:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.786+0000 7fe749c00700 1 -- 172.21.15.1:0/4074791755 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe74419be10 con 0x7fe728039650 2024-04-26T21:19:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.788+0000 7fe73b400700 1 -- 172.21.15.1:0/4074791755 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+158 (secure 0 0 0) 0x7fe74419be10 con 0x7fe728039650 2024-04-26T21:19:25.789 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:25.789 INFO:teuthology.orchestra.run.smithi001.stdout:[{"addr": "172.21.15.1", "hostname": "smithi001", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}] 2024-04-26T21:19:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.792+0000 7fe749c00700 1 -- 172.21.15.1:0/4074791755 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe728039650 msgr2=0x7fe72803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.792+0000 7fe749c00700 1 --2- 172.21.15.1:0/4074791755 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe728039650 0x7fe72803bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe730006fd0 tx=0x7fe730006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.792+0000 7fe749c00700 1 -- 172.21.15.1:0/4074791755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7441052a0 msgr2=0x7fe7441a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.792+0000 7fe749c00700 1 --2- 172.21.15.1:0/4074791755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7441052a0 0x7fe7441a0720 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe734006010 tx=0x7fe734005780 comp rx=0 tx=0).stop 2024-04-26T21:19:25.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.792+0000 7fe749c00700 1 -- 172.21.15.1:0/4074791755 shutdown_connections 2024-04-26T21:19:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.792+0000 7fe749c00700 1 --2- 172.21.15.1:0/4074791755 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe728039650 0x7fe72803bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.792+0000 7fe749c00700 1 --2- 172.21.15.1:0/4074791755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7441052a0 0x7fe7441a0720 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.792+0000 7fe749c00700 1 -- 172.21.15.1:0/4074791755 >> 172.21.15.1:0/4074791755 conn(0x7fe744100b30 msgr2=0x7fe744106c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.793+0000 7fe749c00700 1 -- 172.21.15.1:0/4074791755 shutdown_connections 2024-04-26T21:19:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:25.793+0000 7fe749c00700 1 -- 172.21.15.1:0/4074791755 wait complete. 2024-04-26T21:19:26.463 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:26 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:26.463 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:26 smithi001 ceph-mon[28054]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:26.463 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:26 smithi001 ceph-mon[28054]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:19:26.537 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-04-26T21:19:26.537 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:19:26.537 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-04-26T21:19:26.555 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:19:26.555 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:19:26.613 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-04-26T21:19:26.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch host add smithi186 2024-04-26T21:19:27.065 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:19:28.539 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:28 smithi001 ceph-mon[28054]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:28.540 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:28 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:28.540 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:28 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.037+0000 7f719a400700 1 -- 172.21.15.1:0/3124814465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7194075850 msgr2=0x7f7194075c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.037+0000 7f719a400700 1 --2- 172.21.15.1:0/3124814465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7194075850 0x7f7194075c30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7188009910 tx=0x7f7188009c20 comp rx=0 tx=0).stop 2024-04-26T21:19:29.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.037+0000 7f719a400700 1 -- 172.21.15.1:0/3124814465 shutdown_connections 2024-04-26T21:19:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.037+0000 7f719a400700 1 --2- 172.21.15.1:0/3124814465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7194075850 0x7f7194075c30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.037+0000 7f719a400700 1 -- 172.21.15.1:0/3124814465 >> 172.21.15.1:0/3124814465 conn(0x7f71940fe780 msgr2=0x7f7194100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.037+0000 7f719a400700 1 -- 172.21.15.1:0/3124814465 shutdown_connections 2024-04-26T21:19:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.037+0000 7f719a400700 1 -- 172.21.15.1:0/3124814465 wait complete. 2024-04-26T21:19:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.039+0000 7f719a400700 1 Processor -- start 2024-04-26T21:19:29.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.039+0000 7f719a400700 1 -- start start 2024-04-26T21:19:29.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.040+0000 7f719a400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7194075850 0x7f71941a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:29.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.040+0000 7f719a400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f719410f3d0 con 0x7f7194075850 2024-04-26T21:19:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.040+0000 7f7199000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7194075850 0x7f71941a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.040+0000 7f7199000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7194075850 0x7f71941a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39198/0 (socket says 172.21.15.1:39198) 2024-04-26T21:19:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.040+0000 7f7199000700 1 -- 172.21.15.1:0/1151781867 learned_addr learned my addr 172.21.15.1:0/1151781867 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.040+0000 7f7199000700 1 -- 172.21.15.1:0/1151781867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71941a0bc0 con 0x7f7194075850 2024-04-26T21:19:29.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.041+0000 7f7199000700 1 --2- 172.21.15.1:0/1151781867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7194075850 0x7f71941a0680 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7188006010 tx=0x7f7188005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:29.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.041+0000 7f7191600700 1 -- 172.21.15.1:0/1151781867 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7188026070 con 0x7f7194075850 2024-04-26T21:19:29.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.041+0000 7f7191600700 1 -- 172.21.15.1:0/1151781867 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f718800bc00 con 0x7f7194075850 2024-04-26T21:19:29.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.041+0000 7f719a400700 1 -- 172.21.15.1:0/1151781867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71941a0e50 con 0x7f7194075850 2024-04-26T21:19:29.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.041+0000 7f7191600700 1 -- 172.21.15.1:0/1151781867 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7188003910 con 0x7f7194075850 2024-04-26T21:19:29.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.041+0000 7f719a400700 1 -- 172.21.15.1:0/1151781867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f719419a750 con 0x7f7194075850 2024-04-26T21:19:29.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.042+0000 7f719a400700 1 -- 172.21.15.1:0/1151781867 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7194076f00 con 0x7f7194075850 2024-04-26T21:19:29.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.043+0000 7f7191600700 1 -- 172.21.15.1:0/1151781867 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f7188029070 con 0x7f7194075850 2024-04-26T21:19:29.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.044+0000 7f7191600700 1 --2- 172.21.15.1:0/1151781867 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f717c039700 0x7f717c03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:29.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.044+0000 7f7191600700 1 -- 172.21.15.1:0/1151781867 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f718801e070 con 0x7f7194075850 2024-04-26T21:19:29.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.044+0000 7f7193e00700 1 --2- 172.21.15.1:0/1151781867 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f717c039700 0x7f717c03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:29.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.045+0000 7f7193e00700 1 --2- 172.21.15.1:0/1151781867 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f717c039700 0x7f717c03bbc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7180006fd0 tx=0x7f7180006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:29.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.050+0000 7f7191600700 1 -- 172.21.15.1:0/1151781867 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7188013070 con 0x7f7194075850 2024-04-26T21:19:29.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:29.302+0000 7f719a400700 1 -- 172.21.15.1:0/1151781867 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}) -- 0x7f719419bc80 con 0x7f717c039700 2024-04-26T21:19:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:19:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:19:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:19:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:29 smithi001 ceph-mon[28054]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T21:19:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:29 smithi001 ceph-mon[28054]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:19:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:29 smithi001 ceph-mon[28054]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:19:30.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:30 smithi001 ceph-mon[28054]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:19:30.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:30 smithi001 ceph-mon[28054]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:30.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:30 smithi001 ceph-mon[28054]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:19:30.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:30 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:30.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:30 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:30.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:30 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:30.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:30 smithi001 ceph-mon[28054]: Deploying cephadm binary to smithi186 2024-04-26T21:19:32.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:32 smithi001 ceph-mon[28054]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:34.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.136+0000 7f7191600700 1 -- 172.21.15.1:0/1151781867 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f719419bc80 con 0x7f717c039700 2024-04-26T21:19:34.137 INFO:teuthology.orchestra.run.smithi001.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-04-26T21:19:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.140+0000 7f719a400700 1 -- 172.21.15.1:0/1151781867 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f717c039700 msgr2=0x7f717c03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.140+0000 7f719a400700 1 --2- 172.21.15.1:0/1151781867 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f717c039700 0x7f717c03bbc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7180006fd0 tx=0x7f7180006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.140+0000 7f719a400700 1 -- 172.21.15.1:0/1151781867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7194075850 msgr2=0x7f71941a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.140+0000 7f719a400700 1 --2- 172.21.15.1:0/1151781867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7194075850 0x7f71941a0680 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7188006010 tx=0x7f7188005780 comp rx=0 tx=0).stop 2024-04-26T21:19:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.140+0000 7f719a400700 1 -- 172.21.15.1:0/1151781867 shutdown_connections 2024-04-26T21:19:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.140+0000 7f719a400700 1 --2- 172.21.15.1:0/1151781867 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f717c039700 0x7f717c03bbc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.140+0000 7f719a400700 1 --2- 172.21.15.1:0/1151781867 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7194075850 0x7f71941a0680 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.140+0000 7f719a400700 1 -- 172.21.15.1:0/1151781867 >> 172.21.15.1:0/1151781867 conn(0x7f71940fe780 msgr2=0x7f7194106f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:34.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.140+0000 7f719a400700 1 -- 172.21.15.1:0/1151781867 shutdown_connections 2024-04-26T21:19:34.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:34.140+0000 7f719a400700 1 -- 172.21.15.1:0/1151781867 wait complete. 2024-04-26T21:19:34.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:34 smithi001 ceph-mon[28054]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:34.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:34.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:19:34.855 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T21:19:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:19:35.747 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:35 smithi001 ceph-mon[28054]: Added host smithi186 2024-04-26T21:19:35.747 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:35.747 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:36.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:36 smithi001 ceph-mon[28054]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.278+0000 7f2407a00700 1 -- 172.21.15.1:0/2218814706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2400108ec0 msgr2=0x7f24001092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.278+0000 7f2407a00700 1 --2- 172.21.15.1:0/2218814706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2400108ec0 0x7f24001092a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009910 tx=0x7f23f4009c20 comp rx=0 tx=0).stop 2024-04-26T21:19:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.278+0000 7f2407a00700 1 -- 172.21.15.1:0/2218814706 shutdown_connections 2024-04-26T21:19:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.278+0000 7f2407a00700 1 --2- 172.21.15.1:0/2218814706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2400108ec0 0x7f24001092a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.278+0000 7f2407a00700 1 -- 172.21.15.1:0/2218814706 >> 172.21.15.1:0/2218814706 conn(0x7f2400077880 msgr2=0x7f2400077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.279+0000 7f2407a00700 1 -- 172.21.15.1:0/2218814706 shutdown_connections 2024-04-26T21:19:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.279+0000 7f2407a00700 1 -- 172.21.15.1:0/2218814706 wait complete. 2024-04-26T21:19:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.280+0000 7f2407a00700 1 Processor -- start 2024-04-26T21:19:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.281+0000 7f2407a00700 1 -- start start 2024-04-26T21:19:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.281+0000 7f2407a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2400108ec0 0x7f24001a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.281+0000 7f2407a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f240010d300 con 0x7f2400108ec0 2024-04-26T21:19:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.281+0000 7f2405000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2400108ec0 0x7f24001a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.282+0000 7f2405000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2400108ec0 0x7f24001a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39218/0 (socket says 172.21.15.1:39218) 2024-04-26T21:19:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.282+0000 7f2405000700 1 -- 172.21.15.1:0/1087806513 learned_addr learned my addr 172.21.15.1:0/1087806513 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.282+0000 7f2405000700 1 -- 172.21.15.1:0/1087806513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24001a0c20 con 0x7f2400108ec0 2024-04-26T21:19:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.282+0000 7f2405000700 1 --2- 172.21.15.1:0/1087806513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2400108ec0 0x7f24001a06e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f23f40056d0 tx=0x7f23f40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.283+0000 7f23fd600700 1 -- 172.21.15.1:0/1087806513 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f23f4026070 con 0x7f2400108ec0 2024-04-26T21:19:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.283+0000 7f23fd600700 1 -- 172.21.15.1:0/1087806513 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23f400bba0 con 0x7f2400108ec0 2024-04-26T21:19:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.283+0000 7f2407a00700 1 -- 172.21.15.1:0/1087806513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24001a0f10 con 0x7f2400108ec0 2024-04-26T21:19:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.283+0000 7f23fd600700 1 -- 172.21.15.1:0/1087806513 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f23f4003920 con 0x7f2400108ec0 2024-04-26T21:19:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.283+0000 7f2407a00700 1 -- 172.21.15.1:0/1087806513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f240019a8a0 con 0x7f2400108ec0 2024-04-26T21:19:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.284+0000 7f23fd600700 1 -- 172.21.15.1:0/1087806513 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f23f4029070 con 0x7f2400108ec0 2024-04-26T21:19:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.284+0000 7f2407a00700 1 -- 172.21.15.1:0/1087806513 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2400104570 con 0x7f2400108ec0 2024-04-26T21:19:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.284+0000 7f23fd600700 1 --2- 172.21.15.1:0/1087806513 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f23e8039650 0x7f23e803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.285+0000 7f23fd600700 1 -- 172.21.15.1:0/1087806513 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f23f401e070 con 0x7f2400108ec0 2024-04-26T21:19:37.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.285+0000 7f23ffe00700 1 --2- 172.21.15.1:0/1087806513 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f23e8039650 0x7f23e803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:37.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.286+0000 7f23ffe00700 1 --2- 172.21.15.1:0/1087806513 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f23e8039650 0x7f23e803bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f23ec006fd0 tx=0x7f23ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:37.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.290+0000 7f23fd600700 1 -- 172.21.15.1:0/1087806513 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f23f4013070 con 0x7f2400108ec0 2024-04-26T21:19:37.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.527+0000 7f2407a00700 1 -- 172.21.15.1:0/1087806513 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f240019be40 con 0x7f23e8039650 2024-04-26T21:19:37.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.530+0000 7f23fd600700 1 -- 172.21.15.1:0/1087806513 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+238 (secure 0 0 0) 0x7f240019be40 con 0x7f23e8039650 2024-04-26T21:19:37.531 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:19:37.532 INFO:teuthology.orchestra.run.smithi001.stdout:[{"addr": "172.21.15.1", "hostname": "smithi001", "labels": [], "status": ""}, {"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-04-26T21:19:37.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.533+0000 7f2407a00700 1 -- 172.21.15.1:0/1087806513 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f23e8039650 msgr2=0x7f23e803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:37.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.533+0000 7f2407a00700 1 --2- 172.21.15.1:0/1087806513 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f23e8039650 0x7f23e803bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f23ec006fd0 tx=0x7f23ec006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:37.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.533+0000 7f2407a00700 1 -- 172.21.15.1:0/1087806513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2400108ec0 msgr2=0x7f24001a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:37.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.533+0000 7f2407a00700 1 --2- 172.21.15.1:0/1087806513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2400108ec0 0x7f24001a06e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f23f40056d0 tx=0x7f23f40057b0 comp rx=0 tx=0).stop 2024-04-26T21:19:37.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.534+0000 7f2407a00700 1 -- 172.21.15.1:0/1087806513 shutdown_connections 2024-04-26T21:19:37.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.534+0000 7f2407a00700 1 --2- 172.21.15.1:0/1087806513 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f23e8039650 0x7f23e803bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:37.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.534+0000 7f2407a00700 1 --2- 172.21.15.1:0/1087806513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2400108ec0 0x7f24001a06e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:37.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.534+0000 7f2407a00700 1 -- 172.21.15.1:0/1087806513 >> 172.21.15.1:0/1087806513 conn(0x7f2400077880 msgr2=0x7f24001052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:37.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.534+0000 7f2407a00700 1 -- 172.21.15.1:0/1087806513 shutdown_connections 2024-04-26T21:19:37.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:37.534+0000 7f2407a00700 1 -- 172.21.15.1:0/1087806513 wait complete. 2024-04-26T21:19:38.212 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:38.281 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-26T21:19:38.281 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-26T21:19:38.745 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:19:39.157 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:38 smithi001 ceph-mon[28054]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:39.158 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:38 smithi001 ceph-mon[28054]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:19:40.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.857+0000 7f235f000700 1 -- 172.21.15.1:0/1464798782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2358107d60 msgr2=0x7f235806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:40.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.857+0000 7f235f000700 1 --2- 172.21.15.1:0/1464798782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2358107d60 0x7f235806a840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2348009910 tx=0x7f2348009c20 comp rx=0 tx=0).stop 2024-04-26T21:19:40.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.857+0000 7f235f000700 1 -- 172.21.15.1:0/1464798782 shutdown_connections 2024-04-26T21:19:40.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.857+0000 7f235f000700 1 --2- 172.21.15.1:0/1464798782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2358107d60 0x7f235806a840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:40.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.857+0000 7f235f000700 1 -- 172.21.15.1:0/1464798782 >> 172.21.15.1:0/1464798782 conn(0x7f2358077880 msgr2=0x7f2358077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:40.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.857+0000 7f235f000700 1 -- 172.21.15.1:0/1464798782 shutdown_connections 2024-04-26T21:19:40.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.857+0000 7f235f000700 1 -- 172.21.15.1:0/1464798782 wait complete. 2024-04-26T21:19:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.859+0000 7f235f000700 1 Processor -- start 2024-04-26T21:19:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.859+0000 7f235f000700 1 -- start start 2024-04-26T21:19:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.860+0000 7f235f000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2358107d60 0x7f235819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.860+0000 7f235f000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2358113990 con 0x7f2358107d60 2024-04-26T21:19:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.860+0000 7f2357e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2358107d60 0x7f235819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:40.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.860+0000 7f2357e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2358107d60 0x7f235819ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51974/0 (socket says 172.21.15.1:51974) 2024-04-26T21:19:40.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.860+0000 7f2357e00700 1 -- 172.21.15.1:0/1768569476 learned_addr learned my addr 172.21.15.1:0/1768569476 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:19:40.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.861+0000 7f2357e00700 1 -- 172.21.15.1:0/1768569476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f235819f4b0 con 0x7f2358107d60 2024-04-26T21:19:40.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.861+0000 7f2357e00700 1 --2- 172.21.15.1:0/1768569476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2358107d60 0x7f235819ef70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f23480056d0 tx=0x7f23480057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:40.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.861+0000 7f2355600700 1 -- 172.21.15.1:0/1768569476 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2348026070 con 0x7f2358107d60 2024-04-26T21:19:40.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.861+0000 7f2355600700 1 -- 172.21.15.1:0/1768569476 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f234800bc00 con 0x7f2358107d60 2024-04-26T21:19:40.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.861+0000 7f235f000700 1 -- 172.21.15.1:0/1768569476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f235819f740 con 0x7f2358107d60 2024-04-26T21:19:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.861+0000 7f2355600700 1 -- 172.21.15.1:0/1768569476 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2348003920 con 0x7f2358107d60 2024-04-26T21:19:40.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.861+0000 7f235f000700 1 -- 172.21.15.1:0/1768569476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f235819fba0 con 0x7f2358107d60 2024-04-26T21:19:40.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.863+0000 7f235f000700 1 -- 172.21.15.1:0/1768569476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2358102350 con 0x7f2358107d60 2024-04-26T21:19:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.864+0000 7f2355600700 1 -- 172.21.15.1:0/1768569476 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f2348029070 con 0x7f2358107d60 2024-04-26T21:19:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.864+0000 7f2355600700 1 --2- 172.21.15.1:0/1768569476 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f233c0396b0 0x7f233c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.864+0000 7f2355600700 1 -- 172.21.15.1:0/1768569476 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f234801e070 con 0x7f2358107d60 2024-04-26T21:19:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.864+0000 7f234fe00700 1 --2- 172.21.15.1:0/1768569476 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f233c0396b0 0x7f233c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:40.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.865+0000 7f234fe00700 1 --2- 172.21.15.1:0/1768569476 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f233c0396b0 0x7f233c03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2340006fd0 tx=0x7f2340006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:40.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:40.870+0000 7f2355600700 1 -- 172.21.15.1:0/1768569476 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2348013070 con 0x7f2358107d60 2024-04-26T21:19:41.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:40 smithi001 ceph-mon[28054]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:41.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:41.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:41.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:41.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:41.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:19:41.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:19:41.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:19:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.112+0000 7f235f000700 1 -- 172.21.15.1:0/1768569476 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f23581a02d0 con 0x7f2358107d60 2024-04-26T21:19:41.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.898+0000 7f2355600700 1 -- 172.21.15.1:0/1768569476 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f234800fe20 con 0x7f2358107d60 2024-04-26T21:19:41.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.904+0000 7f235f000700 1 -- 172.21.15.1:0/1768569476 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f233c0396b0 msgr2=0x7f233c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.904+0000 7f235f000700 1 --2- 172.21.15.1:0/1768569476 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f233c0396b0 0x7f233c03bb70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2340006fd0 tx=0x7f2340006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.904+0000 7f235f000700 1 -- 172.21.15.1:0/1768569476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2358107d60 msgr2=0x7f235819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.904+0000 7f235f000700 1 --2- 172.21.15.1:0/1768569476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2358107d60 0x7f235819ef70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f23480056d0 tx=0x7f23480057b0 comp rx=0 tx=0).stop 2024-04-26T21:19:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.904+0000 7f235f000700 1 -- 172.21.15.1:0/1768569476 shutdown_connections 2024-04-26T21:19:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.904+0000 7f235f000700 1 --2- 172.21.15.1:0/1768569476 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f233c0396b0 0x7f233c03bb70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.904+0000 7f235f000700 1 --2- 172.21.15.1:0/1768569476 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2358107d60 0x7f235819ef70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.904+0000 7f235f000700 1 -- 172.21.15.1:0/1768569476 >> 172.21.15.1:0/1768569476 conn(0x7f2358077880 msgr2=0x7f23580fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.905+0000 7f235f000700 1 -- 172.21.15.1:0/1768569476 shutdown_connections 2024-04-26T21:19:41.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:19:41.905+0000 7f235f000700 1 -- 172.21.15.1:0/1768569476 wait complete. 2024-04-26T21:19:41.907 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-04-26T21:19:42.286 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:41 smithi001 ceph-mon[28054]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T21:19:42.287 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:41 smithi001 ceph-mon[28054]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:19:42.287 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:41 smithi001 ceph-mon[28054]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:19:42.287 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:41 smithi001 ceph-mon[28054]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:19:42.287 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:41 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1768569476' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-26T21:19:42.287 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:42.287 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:42.287 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:42.615 INFO:tasks.cephadm:Adding mon.a on smithi001 2024-04-26T21:19:42.615 INFO:tasks.cephadm:Adding mon.b on smithi038 2024-04-26T21:19:42.615 INFO:tasks.cephadm:Adding mon.c on smithi186 2024-04-26T21:19:42.615 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c' 2024-04-26T21:19:42.854 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:19:43.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:42 smithi001 ceph-mon[28054]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:43.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:42 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1768569476' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-26T21:19:43.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:42 smithi001 ceph-mon[28054]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:19:44.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.651+0000 7f7768544700 1 -- 172.21.15.186:0/2289506311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7760107e70 msgr2=0x7f7760108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:44.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.651+0000 7f7768544700 1 --2- 172.21.15.186:0/2289506311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7760107e70 0x7f7760108250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f775c009910 tx=0x7f775c009c20 comp rx=0 tx=0).stop 2024-04-26T21:19:44.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.652+0000 7f7768544700 1 -- 172.21.15.186:0/2289506311 shutdown_connections 2024-04-26T21:19:44.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.652+0000 7f7768544700 1 --2- 172.21.15.186:0/2289506311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7760107e70 0x7f7760108250 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:44.653 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.652+0000 7f7768544700 1 -- 172.21.15.186:0/2289506311 >> 172.21.15.186:0/2289506311 conn(0x7f77600fcfd0 msgr2=0x7f77600ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:44.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.652+0000 7f7768544700 1 -- 172.21.15.186:0/2289506311 shutdown_connections 2024-04-26T21:19:44.654 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.652+0000 7f7768544700 1 -- 172.21.15.186:0/2289506311 wait complete. 2024-04-26T21:19:44.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.652+0000 7f7768544700 1 Processor -- start 2024-04-26T21:19:44.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.652+0000 7f7768544700 1 -- start start 2024-04-26T21:19:44.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.652+0000 7f7768544700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7760107e70 0x7f7760102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:44.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.652+0000 7f7768544700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f776010d480 con 0x7f7760107e70 2024-04-26T21:19:44.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.653+0000 7f77662e0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7760107e70 0x7f7760102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:44.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.653+0000 7f77662e0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7760107e70 0x7f7760102470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.186:33868/0 (socket says 172.21.15.186:33868) 2024-04-26T21:19:44.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.653+0000 7f77662e0700 1 -- 172.21.15.186:0/1076919437 learned_addr learned my addr 172.21.15.186:0/1076919437 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T21:19:44.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.654+0000 7f77662e0700 1 -- 172.21.15.186:0/1076919437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77601042c0 con 0x7f7760107e70 2024-04-26T21:19:44.655 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.654+0000 7f77662e0700 1 --2- 172.21.15.186:0/1076919437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7760107e70 0x7f7760102470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f775c006010 tx=0x7f775c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:44.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.654+0000 7f77577fe700 1 -- 172.21.15.186:0/1076919437 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f775c026070 con 0x7f7760107e70 2024-04-26T21:19:44.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.654+0000 7f7768544700 1 -- 172.21.15.186:0/1076919437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77601029b0 con 0x7f7760107e70 2024-04-26T21:19:44.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.654+0000 7f7768544700 1 -- 172.21.15.186:0/1076919437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7760102db0 con 0x7f7760107e70 2024-04-26T21:19:44.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.655+0000 7f77577fe700 1 -- 172.21.15.186:0/1076919437 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f775c00b750 con 0x7f7760107e70 2024-04-26T21:19:44.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.655+0000 7f77577fe700 1 -- 172.21.15.186:0/1076919437 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f775c030060 con 0x7f7760107e70 2024-04-26T21:19:44.656 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.655+0000 7f7768544700 1 -- 172.21.15.186:0/1076919437 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7744005330 con 0x7f7760107e70 2024-04-26T21:19:44.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.656+0000 7f77577fe700 1 -- 172.21.15.186:0/1076919437 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f775c029070 con 0x7f7760107e70 2024-04-26T21:19:44.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.656+0000 7f77577fe700 1 --2- 172.21.15.186:0/1076919437 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f774c0396b0 0x7f774c03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:44.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.656+0000 7f77577fe700 1 -- 172.21.15.186:0/1076919437 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f775c01e070 con 0x7f7760107e70 2024-04-26T21:19:44.657 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.657+0000 7f7765adf700 1 --2- 172.21.15.186:0/1076919437 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f774c0396b0 0x7f774c03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:44.661 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.658+0000 7f7765adf700 1 --2- 172.21.15.186:0/1076919437 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f774c0396b0 0x7f774c03bb70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7750006fd0 tx=0x7f7750006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:44.674 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.673+0000 7f77577fe700 1 -- 172.21.15.186:0/1076919437 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f775c013070 con 0x7f7760107e70 2024-04-26T21:19:44.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.882+0000 7f7768544700 1 -- 172.21.15.186:0/1076919437 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}) -- 0x7f7744000c70 con 0x7f774c0396b0 2024-04-26T21:19:44.898 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.897+0000 7f77577fe700 1 -- 172.21.15.186:0/1076919437 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7744000c70 con 0x7f774c0396b0 2024-04-26T21:19:44.899 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mon update... 2024-04-26T21:19:44.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.900+0000 7f7768544700 1 -- 172.21.15.186:0/1076919437 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f774c0396b0 msgr2=0x7f774c03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:44.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.900+0000 7f7768544700 1 --2- 172.21.15.186:0/1076919437 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f774c0396b0 0x7f774c03bb70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7750006fd0 tx=0x7f7750006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:44.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.900+0000 7f7768544700 1 -- 172.21.15.186:0/1076919437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7760107e70 msgr2=0x7f7760102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:44.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.900+0000 7f7768544700 1 --2- 172.21.15.186:0/1076919437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7760107e70 0x7f7760102470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f775c006010 tx=0x7f775c005780 comp rx=0 tx=0).stop 2024-04-26T21:19:44.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.900+0000 7f7768544700 1 -- 172.21.15.186:0/1076919437 shutdown_connections 2024-04-26T21:19:44.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.900+0000 7f7768544700 1 --2- 172.21.15.186:0/1076919437 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f774c0396b0 0x7f774c03bb70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:44.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.900+0000 7f7768544700 1 --2- 172.21.15.186:0/1076919437 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7760107e70 0x7f7760102470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:44.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.900+0000 7f7768544700 1 -- 172.21.15.186:0/1076919437 >> 172.21.15.186:0/1076919437 conn(0x7f77600fcfd0 msgr2=0x7f776010aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:44.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.900+0000 7f7768544700 1 -- 172.21.15.186:0/1076919437 shutdown_connections 2024-04-26T21:19:44.901 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:44.900+0000 7f7768544700 1 -- 172.21.15.186:0/1076919437 wait complete. 2024-04-26T21:19:45.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:44 smithi001 ceph-mon[28054]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:45.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:44 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:45.562 DEBUG:teuthology.orchestra.run.smithi038:mon.b> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T21:19:45.564 DEBUG:teuthology.orchestra.run.smithi186:mon.c> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T21:19:45.566 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T21:19:45.567 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T21:19:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:45 smithi001 ceph-mon[28054]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:19:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:45 smithi001 ceph-mon[28054]: Saving service mon spec with placement smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c;count:3 2024-04-26T21:19:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:19:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:19:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:19:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:19:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:19:46.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:45 smithi001 ceph-mon[28054]: Deploying daemon mon.c on smithi186 2024-04-26T21:19:46.499 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:19:47.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:46 smithi001 ceph-mon[28054]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:49.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:48 smithi001 ceph-mon[28054]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:49.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.541+0000 7f7ca49be700 1 -- 172.21.15.186:0/4146516629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ca0101f90 msgr2=0x7f7ca0102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:49.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.541+0000 7f7ca49be700 1 --2- 172.21.15.186:0/4146516629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ca0101f90 0x7f7ca0102370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009910 tx=0x7f7c90009c20 comp rx=0 tx=0).stop 2024-04-26T21:19:49.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.541+0000 7f7ca49be700 1 -- 172.21.15.186:0/4146516629 shutdown_connections 2024-04-26T21:19:49.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.541+0000 7f7ca49be700 1 --2- 172.21.15.186:0/4146516629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ca0101f90 0x7f7ca0102370 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:49.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.541+0000 7f7ca49be700 1 -- 172.21.15.186:0/4146516629 >> 172.21.15.186:0/4146516629 conn(0x7f7ca00fd840 msgr2=0x7f7ca00ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:49.543 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.541+0000 7f7ca49be700 1 -- 172.21.15.186:0/4146516629 shutdown_connections 2024-04-26T21:19:49.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.541+0000 7f7ca49be700 1 -- 172.21.15.186:0/4146516629 wait complete. 2024-04-26T21:19:49.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.542+0000 7f7ca49be700 1 Processor -- start 2024-04-26T21:19:49.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.542+0000 7f7ca49be700 1 -- start start 2024-04-26T21:19:49.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.542+0000 7f7ca49be700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ca0101f90 0x7f7ca0103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:49.544 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.542+0000 7f7ca49be700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca010c210 con 0x7f7ca0101f90 2024-04-26T21:19:49.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.543+0000 7f7c9e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ca0101f90 0x7f7ca0103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:49.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.543+0000 7f7c9e59c700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ca0101f90 0x7f7ca0103a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.186:33882/0 (socket says 172.21.15.186:33882) 2024-04-26T21:19:49.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.543+0000 7f7c9e59c700 1 -- 172.21.15.186:0/854194874 learned_addr learned my addr 172.21.15.186:0/854194874 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T21:19:49.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.543+0000 7f7c9e59c700 1 -- 172.21.15.186:0/854194874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca0105920 con 0x7f7ca0101f90 2024-04-26T21:19:49.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.544+0000 7f7c9e59c700 1 --2- 172.21.15.186:0/854194874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ca0101f90 0x7f7ca0103a30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c900056d0 tx=0x7f7c900057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:49.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.544+0000 7f7c8f7fe700 1 -- 172.21.15.186:0/854194874 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c90026070 con 0x7f7ca0101f90 2024-04-26T21:19:49.545 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.544+0000 7f7ca49be700 1 -- 172.21.15.186:0/854194874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca0103fd0 con 0x7f7ca0101f90 2024-04-26T21:19:49.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.544+0000 7f7ca49be700 1 -- 172.21.15.186:0/854194874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca0104490 con 0x7f7ca0101f90 2024-04-26T21:19:49.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.544+0000 7f7c8f7fe700 1 -- 172.21.15.186:0/854194874 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c9000bba0 con 0x7f7ca0101f90 2024-04-26T21:19:49.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.544+0000 7f7c8f7fe700 1 -- 172.21.15.186:0/854194874 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c900037f0 con 0x7f7ca0101f90 2024-04-26T21:19:49.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.545+0000 7f7ca49be700 1 -- 172.21.15.186:0/854194874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca0103470 con 0x7f7ca0101f90 2024-04-26T21:19:49.546 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.546+0000 7f7c8f7fe700 1 -- 172.21.15.186:0/854194874 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f7c90029070 con 0x7f7ca0101f90 2024-04-26T21:19:49.547 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.546+0000 7f7c8f7fe700 1 --2- 172.21.15.186:0/854194874 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c88039650 0x7f7c8803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:49.547 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.546+0000 7f7c8f7fe700 1 -- 172.21.15.186:0/854194874 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7c9001e070 con 0x7f7ca0101f90 2024-04-26T21:19:49.547 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.546+0000 7f7c9dd9b700 1 --2- 172.21.15.186:0/854194874 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c88039650 0x7f7c8803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:49.548 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.547+0000 7f7c9dd9b700 1 --2- 172.21.15.186:0/854194874 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c88039650 0x7f7c8803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7c94006fd0 tx=0x7f7c94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:49.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.561+0000 7f7c8f7fe700 1 -- 172.21.15.186:0/854194874 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7c90013070 con 0x7f7ca0101f90 2024-04-26T21:19:49.867 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.866+0000 7f7ca49be700 1 -- 172.21.15.186:0/854194874 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7ca0105000 con 0x7f7ca0101f90 2024-04-26T21:19:49.868 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.867+0000 7f7c8f7fe700 1 -- 172.21.15.186:0/854194874 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+741 (secure 0 0 0) 0x7f7c90034090 con 0x7f7ca0101f90 2024-04-26T21:19:49.869 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:19:49.869 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","modified":"2024-04-26T21:17:42.249569Z","created":"2024-04-26T21:17:42.249569Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T21:19:49.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.870+0000 7f7ca49be700 1 -- 172.21.15.186:0/854194874 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c88039650 msgr2=0x7f7c8803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:49.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.870+0000 7f7ca49be700 1 --2- 172.21.15.186:0/854194874 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c88039650 0x7f7c8803bb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7c94006fd0 tx=0x7f7c94006e40 comp rx=0 tx=0).stop 2024-04-26T21:19:49.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.870+0000 7f7ca49be700 1 -- 172.21.15.186:0/854194874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ca0101f90 msgr2=0x7f7ca0103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:49.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.870+0000 7f7ca49be700 1 --2- 172.21.15.186:0/854194874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ca0101f90 0x7f7ca0103a30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c900056d0 tx=0x7f7c900057b0 comp rx=0 tx=0).stop 2024-04-26T21:19:49.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.870+0000 7f7ca49be700 1 -- 172.21.15.186:0/854194874 shutdown_connections 2024-04-26T21:19:49.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.870+0000 7f7ca49be700 1 --2- 172.21.15.186:0/854194874 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c88039650 0x7f7c8803bb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:49.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.870+0000 7f7ca49be700 1 --2- 172.21.15.186:0/854194874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ca0101f90 0x7f7ca0103a30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:49.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.870+0000 7f7ca49be700 1 -- 172.21.15.186:0/854194874 >> 172.21.15.186:0/854194874 conn(0x7f7ca00fd840 msgr2=0x7f7ca006b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:49.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.870+0000 7f7ca49be700 1 -- 172.21.15.186:0/854194874 shutdown_connections 2024-04-26T21:19:49.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:49.870+0000 7f7ca49be700 1 -- 172.21.15.186:0/854194874 wait complete. 2024-04-26T21:19:49.872 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-04-26T21:19:50.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:49 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/854194874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:19:51.653 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T21:19:51.653 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T21:19:51.902 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:19:55.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:55.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:19:55.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: mon.a calling monitor election 2024-04-26T21:19:55.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:19:55.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:19:55.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:55.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: mon.c calling monitor election 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: monmap epoch 2 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: last_changed 2024-04-26T21:19:49.993441+0000 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: created 2024-04-26T21:17:42.249569+0000 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: min_mon_release 19 (squid) 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: election_strategy: 1 2024-04-26T21:19:55.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T21:19:55.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T21:19:55.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: fsmap 2024-04-26T21:19:55.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:19:55.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: mgrmap e13: x(active, since 59s) 2024-04-26T21:19:55.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: overall HEALTH_OK 2024-04-26T21:19:55.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:55.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:55.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:19:55.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:19:55.572 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:19:56.386 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:19:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:19:56.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.385+0000 7f7207fec700 1 -- 172.21.15.186:0/1036209143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71ec0035c0 msgr2=0x7f71ec005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:56.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.385+0000 7f7207fec700 1 --2- 172.21.15.186:0/1036209143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71ec0035c0 0x7f71ec005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f720006b650 tx=0x7f71f8009e80 comp rx=0 tx=0).stop 2024-04-26T21:19:56.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.385+0000 7f7207fec700 1 -- 172.21.15.186:0/1036209143 shutdown_connections 2024-04-26T21:19:56.387 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.385+0000 7f7207fec700 1 --2- 172.21.15.186:0/1036209143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71ec0035c0 0x7f71ec005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.385+0000 7f7207fec700 1 -- 172.21.15.186:0/1036209143 >> 172.21.15.186:0/1036209143 conn(0x7f72000fd910 msgr2=0x7f72000ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.385+0000 7f7207fec700 1 -- 172.21.15.186:0/1036209143 shutdown_connections 2024-04-26T21:19:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.386+0000 7f7207fec700 1 -- 172.21.15.186:0/1036209143 wait complete. 2024-04-26T21:19:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.386+0000 7f7207fec700 1 Processor -- start 2024-04-26T21:19:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.386+0000 7f7207fec700 1 -- start start 2024-04-26T21:19:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7207fec700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ec0035c0 0x7f72001a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7207fec700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72001a0e30 0x7f720019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7207fec700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f720010ae60 con 0x7f72001a0e30 2024-04-26T21:19:56.388 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7207fec700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f720010ace0 con 0x7f71ec0035c0 2024-04-26T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7205d88700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ec0035c0 0x7f72001a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7205d88700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ec0035c0 0x7f72001a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:55712/0 (socket says 172.21.15.186:55712) 2024-04-26T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7205d88700 1 -- 172.21.15.186:0/1149086446 learned_addr learned my addr 172.21.15.186:0/1149086446 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7205d88700 1 -- 172.21.15.186:0/1149086446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ec0035c0 msgr2=0x7f72001a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7205d88700 1 -- 172.21.15.186:0/1149086446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ec0035c0 msgr2=0x7f72001a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7205d88700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ec0035c0 0x7f72001a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7205d88700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ec0035c0 0x7f72001a08f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.387+0000 7f7205587700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72001a0e30 0x7f720019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.388+0000 7f7205587700 1 -- 172.21.15.186:0/1149086446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ec0035c0 msgr2=0x7f72001a08f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:19:56.389 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.388+0000 7f7205587700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ec0035c0 0x7f72001a08f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:56.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.388+0000 7f7205587700 1 -- 172.21.15.186:0/1149086446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72001a1470 con 0x7f72001a0e30 2024-04-26T21:19:56.390 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.388+0000 7f7205587700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72001a0e30 0x7f720019aab0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f71f000b4d0 tx=0x7f71f000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:56.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.389+0000 7f71f6ffd700 1 -- 172.21.15.186:0/1149086446 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f71f0011630 con 0x7f72001a0e30 2024-04-26T21:19:56.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.389+0000 7f7207fec700 1 -- 172.21.15.186:0/1149086446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f720019aff0 con 0x7f72001a0e30 2024-04-26T21:19:56.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.389+0000 7f71f6ffd700 1 -- 172.21.15.186:0/1149086446 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71f0011c80 con 0x7f72001a0e30 2024-04-26T21:19:56.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.389+0000 7f71f6ffd700 1 -- 172.21.15.186:0/1149086446 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f71f0010da0 con 0x7f72001a0e30 2024-04-26T21:19:56.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.389+0000 7f7207fec700 1 -- 172.21.15.186:0/1149086446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f720019b580 con 0x7f72001a0e30 2024-04-26T21:19:56.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.390+0000 7f7207fec700 1 -- 172.21.15.186:0/1149086446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f720006be50 con 0x7f72001a0e30 2024-04-26T21:19:56.391 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.390+0000 7f71f6ffd700 1 -- 172.21.15.186:0/1149086446 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f71f0018020 con 0x7f72001a0e30 2024-04-26T21:19:56.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.390+0000 7f71f6ffd700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f71e4039410 0x7f71e403b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:19:56.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.390+0000 7f71f6ffd700 1 -- 172.21.15.186:0/1149086446 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f71f004c200 con 0x7f72001a0e30 2024-04-26T21:19:56.392 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.391+0000 7f7205d88700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f71e4039410 0x7f71e403b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:19:56.393 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.392+0000 7f7205d88700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f71e4039410 0x7f71e403b8d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f71f8007590 tx=0x7f71f80074e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:19:56.406 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.406+0000 7f71f6ffd700 1 -- 172.21.15.186:0/1149086446 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f71f001f020 con 0x7f72001a0e30 2024-04-26T21:19:56.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:56 smithi001 ceph-mon[28054]: Deploying daemon mon.b on smithi038 2024-04-26T21:19:56.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:19:56.686 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.685+0000 7f7207fec700 1 -- 172.21.15.186:0/1149086446 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f720019bfe0 con 0x7f72001a0e30 2024-04-26T21:19:56.687 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.686+0000 7f71f6ffd700 1 -- 172.21.15.186:0/1149086446 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7f71f0010410 con 0x7f72001a0e30 2024-04-26T21:19:56.687 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:19:56.687 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","modified":"2024-04-26T21:19:49.993441Z","created":"2024-04-26T21:17:42.249569Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T21:19:56.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 -- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f71e4039410 msgr2=0x7f71e403b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:56.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f71e4039410 0x7f71e403b8d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f71f8007590 tx=0x7f71f80074e0 comp rx=0 tx=0).stop 2024-04-26T21:19:56.689 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 -- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72001a0e30 msgr2=0x7f720019aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:19:56.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72001a0e30 0x7f720019aab0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f71f000b4d0 tx=0x7f71f000b890 comp rx=0 tx=0).stop 2024-04-26T21:19:56.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 -- 172.21.15.186:0/1149086446 shutdown_connections 2024-04-26T21:19:56.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f71e4039410 0x7f71e403b8d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:56.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ec0035c0 0x7f72001a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:56.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 --2- 172.21.15.186:0/1149086446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72001a0e30 0x7f720019aab0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:19:56.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 -- 172.21.15.186:0/1149086446 >> 172.21.15.186:0/1149086446 conn(0x7f72000fd910 msgr2=0x7f72000feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:19:56.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 -- 172.21.15.186:0/1149086446 shutdown_connections 2024-04-26T21:19:56.690 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:19:56.688+0000 7f7207fec700 1 -- 172.21.15.186:0/1149086446 wait complete. 2024-04-26T21:19:56.690 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-04-26T21:19:57.332 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:19:57 smithi186 ceph-mon[32991]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:57.332 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:19:57 smithi186 ceph-mon[32991]: from='client.? 172.21.15.186:0/1149086446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:19:57.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:57 smithi001 ceph-mon[28054]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:19:57.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:19:57 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/1149086446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:19:58.360 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T21:19:58.360 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T21:19:58.614 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:19:59.301 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:19:59 smithi038 systemd[1]: Started Ceph mon.b for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:20:04.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: mon.c calling monitor election 2024-04-26T21:20:04.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:20:04.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: mon.a calling monitor election 2024-04-26T21:20:04.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:20:04.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:04.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: monmap epoch 3 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: last_changed 2024-04-26T21:19:58.986419+0000 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: created 2024-04-26T21:17:42.249569+0000 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: min_mon_release 19 (squid) 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: election_strategy: 1 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T21:20:04.321 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-26T21:20:04.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: fsmap 2024-04-26T21:20:04.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:20:04.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: mgrmap e13: x(active, since 68s) 2024-04-26T21:20:04.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: overall HEALTH_OK 2024-04-26T21:20:04.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:04.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:04.322 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: mon.c calling monitor election 2024-04-26T21:20:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:20:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: mon.a calling monitor election 2024-04-26T21:20:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:20:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: monmap epoch 3 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: last_changed 2024-04-26T21:19:58.986419+0000 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: created 2024-04-26T21:17:42.249569+0000 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: min_mon_release 19 (squid) 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: election_strategy: 1 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: fsmap 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:20:04.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: mgrmap e13: x(active, since 68s) 2024-04-26T21:20:04.457 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: overall HEALTH_OK 2024-04-26T21:20:04.457 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:04.457 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:04.457 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:04.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.487+0000 7fef5ef37700 1 -- 172.21.15.186:0/3501717250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef440057d0 msgr2=0x7fef44007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:04.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.487+0000 7fef5ef37700 1 --2- 172.21.15.186:0/3501717250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef440057d0 0x7fef44007c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fef58078560 tx=0x7fef4000ce70 comp rx=0 tx=0).stop 2024-04-26T21:20:04.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.487+0000 7fef5ef37700 1 -- 172.21.15.186:0/3501717250 shutdown_connections 2024-04-26T21:20:04.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.487+0000 7fef5ef37700 1 --2- 172.21.15.186:0/3501717250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef440057d0 0x7fef44007c60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:04.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.487+0000 7fef5ef37700 1 -- 172.21.15.186:0/3501717250 >> 172.21.15.186:0/3501717250 conn(0x7fef58077180 msgr2=0x7fef58077590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:20:04.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.487+0000 7fef5ef37700 1 -- 172.21.15.186:0/3501717250 shutdown_connections 2024-04-26T21:20:04.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.487+0000 7fef5ef37700 1 -- 172.21.15.186:0/3501717250 wait complete. 2024-04-26T21:20:04.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.488+0000 7fef5ef37700 1 Processor -- start 2024-04-26T21:20:04.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.488+0000 7fef5ef37700 1 -- start start 2024-04-26T21:20:04.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.488+0000 7fef5ef37700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef440057d0 0x7fef5819c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:04.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.488+0000 7fef5ef37700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef5819cb90 0x7fef58196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:04.492 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.488+0000 7fef5ef37700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58196c60 0x7fef58197110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.488+0000 7fef5ef37700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fef58108cb0 con 0x7fef58196c60 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.488+0000 7fef5ef37700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fef58108b30 con 0x7fef440057d0 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.488+0000 7fef5ef37700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef58108e30 con 0x7fef5819cb90 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.488+0000 7fef57fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef5819cb90 0x7fef58196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef57fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef5819cb90 0x7fef58196720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:33522/0 (socket says 172.21.15.186:33522) 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef57fff700 1 -- 172.21.15.186:0/1359182050 learned_addr learned my addr 172.21.15.186:0/1359182050 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5ccd3700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef440057d0 0x7fef5819c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5d4d4700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58196c60 0x7fef58197110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef57fff700 1 -- 172.21.15.186:0/1359182050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef5819cb90 msgr2=0x7fef58196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef57fff700 1 -- 172.21.15.186:0/1359182050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef5819cb90 msgr2=0x7fef58196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef57fff700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef5819cb90 0x7fef58196720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef57fff700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef5819cb90 0x7fef58196720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:20:04.493 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5ccd3700 1 -- 172.21.15.186:0/1359182050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef440057d0 msgr2=0x7fef5819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5ccd3700 1 -- 172.21.15.186:0/1359182050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef440057d0 msgr2=0x7fef5819c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5ccd3700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef440057d0 0x7fef5819c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5ccd3700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef440057d0 0x7fef5819c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5d4d4700 1 -- 172.21.15.186:0/1359182050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef5819cb90 msgr2=0x7fef58196720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5d4d4700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef5819cb90 0x7fef58196720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5d4d4700 1 -- 172.21.15.186:0/1359182050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef440057d0 msgr2=0x7fef5819c650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5d4d4700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef440057d0 0x7fef5819c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5d4d4700 1 -- 172.21.15.186:0/1359182050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef581a3400 con 0x7fef58196c60 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.489+0000 7fef5d4d4700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58196c60 0x7fef58197110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fef5000bd50 tx=0x7fef5000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.490+0000 7fef55ffb700 1 -- 172.21.15.186:0/1359182050 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50019070 con 0x7fef58196c60 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.490+0000 7fef55ffb700 1 -- 172.21.15.186:0/1359182050 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef5000cd10 con 0x7fef58196c60 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.490+0000 7fef5ef37700 1 -- 172.21.15.186:0/1359182050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef581a3690 con 0x7fef58196c60 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.490+0000 7fef55ffb700 1 -- 172.21.15.186:0/1359182050 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef50012700 con 0x7fef58196c60 2024-04-26T21:20:04.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.490+0000 7fef5ef37700 1 -- 172.21.15.186:0/1359182050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef581a3bd0 con 0x7fef58196c60 2024-04-26T21:20:04.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.491+0000 7fef55ffb700 1 -- 172.21.15.186:0/1359182050 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fef5001c070 con 0x7fef58196c60 2024-04-26T21:20:04.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.491+0000 7fef5ef37700 1 -- 172.21.15.186:0/1359182050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef58102cf0 con 0x7fef58196c60 2024-04-26T21:20:04.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.491+0000 7fef55ffb700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fef38039ae0 0x7fef3803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:04.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.491+0000 7fef55ffb700 1 -- 172.21.15.186:0/1359182050 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fef5004c1f0 con 0x7fef58196c60 2024-04-26T21:20:04.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.492+0000 7fef5ccd3700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fef38039ae0 0x7fef3803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:04.495 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.492+0000 7fef5ccd3700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fef38039ae0 0x7fef3803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fef40000c00 tx=0x7fef4000cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:20:04.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.498+0000 7fef55ffb700 1 -- 172.21.15.186:0/1359182050 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fef50007290 con 0x7fef58196c60 2024-04-26T21:20:04.778 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.777+0000 7fef5ef37700 1 -- 172.21.15.186:0/1359182050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fef5802e490 con 0x7fef58196c60 2024-04-26T21:20:04.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.779+0000 7fef55ffb700 1 -- 172.21.15.186:0/1359182050 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1275 (secure 0 0 0) 0x7fef5000e090 con 0x7fef58196c60 2024-04-26T21:20:04.780 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T21:20:04.780 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":3,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","modified":"2024-04-26T21:19:58.986419Z","created":"2024-04-26T21:17:42.249569Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T21:20:04.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 -- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fef38039ae0 msgr2=0x7fef3803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:04.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fef38039ae0 0x7fef3803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fef40000c00 tx=0x7fef4000cf90 comp rx=0 tx=0).stop 2024-04-26T21:20:04.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 -- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58196c60 msgr2=0x7fef58197110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:04.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58196c60 0x7fef58197110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fef5000bd50 tx=0x7fef5000be30 comp rx=0 tx=0).stop 2024-04-26T21:20:04.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 -- 172.21.15.186:0/1359182050 shutdown_connections 2024-04-26T21:20:04.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fef38039ae0 0x7fef3803bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:04.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fef58196c60 0x7fef58197110 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:04.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef440057d0 0x7fef5819c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:04.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 --2- 172.21.15.186:0/1359182050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef5819cb90 0x7fef58196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:04.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 -- 172.21.15.186:0/1359182050 >> 172.21.15.186:0/1359182050 conn(0x7fef58077180 msgr2=0x7fef58106160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:20:04.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 -- 172.21.15.186:0/1359182050 shutdown_connections 2024-04-26T21:20:04.784 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:04.782+0000 7fef5ef37700 1 -- 172.21.15.186:0/1359182050 wait complete. 2024-04-26T21:20:04.784 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 3 2024-04-26T21:20:05.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:05 smithi001 ceph-mon[28054]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:05.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:05.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:05.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:05 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/1359182050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:20:05.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:05.344 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:05 smithi186 ceph-mon[32991]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:05.345 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:05.345 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:05.345 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:05 smithi186 ceph-mon[32991]: from='client.? 172.21.15.186:0/1359182050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:20:05.345 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:05.373 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T21:20:05.374 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T21:20:05.776 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:20:06.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: mon.c calling monitor election 2024-04-26T21:20:06.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:20:06.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: mon.a calling monitor election 2024-04-26T21:20:06.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:06.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:20:06.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: monmap epoch 3 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: last_changed 2024-04-26T21:19:58.986419+0000 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: created 2024-04-26T21:17:42.249569+0000 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: min_mon_release 19 (squid) 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: election_strategy: 1 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T21:20:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: fsmap 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: mgrmap e13: x(active, since 68s) 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: overall HEALTH_OK 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='client.? 172.21.15.186:0/1359182050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T21:20:06.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:06.933 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: mon.b calling monitor election 2024-04-26T21:20:06.933 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:06.933 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: mon.b calling monitor election 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: mon.c calling monitor election 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: mon.a calling monitor election 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: monmap epoch 3 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: last_changed 2024-04-26T21:19:58.986419+0000 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: created 2024-04-26T21:17:42.249569+0000 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: min_mon_release 19 (squid) 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: election_strategy: 1 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T21:20:06.934 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T21:20:06.935 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-26T21:20:06.935 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: fsmap 2024-04-26T21:20:06.935 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:20:06.935 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: mgrmap e13: x(active, since 70s) 2024-04-26T21:20:06.935 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: overall HEALTH_OK 2024-04-26T21:20:06.935 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:06.935 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:06.935 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:06.935 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:06 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:06.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: mon.b calling monitor election 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: mon.b calling monitor election 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: mon.c calling monitor election 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: mon.a calling monitor election 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: monmap epoch 3 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: last_changed 2024-04-26T21:19:58.986419+0000 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: created 2024-04-26T21:17:42.249569+0000 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: min_mon_release 19 (squid) 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: election_strategy: 1 2024-04-26T21:20:06.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T21:20:06.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T21:20:06.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-26T21:20:06.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: fsmap 2024-04-26T21:20:06.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:20:06.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: mgrmap e13: x(active, since 70s) 2024-04-26T21:20:06.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: overall HEALTH_OK 2024-04-26T21:20:06.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:06.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:06.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:06.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:06 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:07.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: mon.b calling monitor election 2024-04-26T21:20:07.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:07.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: mon.b calling monitor election 2024-04-26T21:20:07.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: mon.c calling monitor election 2024-04-26T21:20:07.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: mon.a calling monitor election 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: monmap epoch 3 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: last_changed 2024-04-26T21:19:58.986419+0000 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: created 2024-04-26T21:17:42.249569+0000 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: min_mon_release 19 (squid) 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: election_strategy: 1 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: 0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: 2: [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon.b 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: fsmap 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: osdmap e4: 0 total, 0 up, 0 in 2024-04-26T21:20:07.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: mgrmap e13: x(active, since 70s) 2024-04-26T21:20:07.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: overall HEALTH_OK 2024-04-26T21:20:07.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:07.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:06 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:07.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T21:20:07.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T21:20:07.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T21:20:07.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:20:07.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:20:07.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:20:07.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:07.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.824+0000 7f07c8000700 1 -- 172.21.15.1:0/2043493940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07c0108ec0 msgr2=0x7f07c01092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.824+0000 7f07c8000700 1 --2- 172.21.15.1:0/2043493940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07c0108ec0 0x7f07c01092a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009910 tx=0x7f07b0009c20 comp rx=0 tx=0).stop 2024-04-26T21:20:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.825+0000 7f07c8000700 1 -- 172.21.15.1:0/2043493940 shutdown_connections 2024-04-26T21:20:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.825+0000 7f07c8000700 1 --2- 172.21.15.1:0/2043493940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07c0108ec0 0x7f07c01092a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.825+0000 7f07c8000700 1 -- 172.21.15.1:0/2043493940 >> 172.21.15.1:0/2043493940 conn(0x7f07c0077880 msgr2=0x7f07c0077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:20:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.825+0000 7f07c8000700 1 -- 172.21.15.1:0/2043493940 shutdown_connections 2024-04-26T21:20:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.825+0000 7f07c8000700 1 -- 172.21.15.1:0/2043493940 wait complete. 2024-04-26T21:20:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.826+0000 7f07c8000700 1 Processor -- start 2024-04-26T21:20:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.826+0000 7f07c8000700 1 -- start start 2024-04-26T21:20:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c8000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c0108ec0 0x7f07c0101e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c8000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07c0102380 0x7f07c0102800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c8000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07c01a4ca0 0x7f07c01a7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c8000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f07c010d300 con 0x7f07c0102380 2024-04-26T21:20:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c8000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f07c010d180 con 0x7f07c0108ec0 2024-04-26T21:20:07.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c8000700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f07c010d480 con 0x7f07c01a4ca0 2024-04-26T21:20:07.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c5600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c0108ec0 0x7f07c0101e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:07.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c5600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c0108ec0 0x7f07c0101e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54276/0 (socket says 172.21.15.1:54276) 2024-04-26T21:20:07.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c5600700 1 -- 172.21.15.1:0/818716710 learned_addr learned my addr 172.21.15.1:0/818716710 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:20:07.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c4c00700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07c0102380 0x7f07c0102800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:07.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.827+0000 7f07c6000700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07c01a4ca0 0x7f07c01a7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:07.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.828+0000 7f07c4c00700 1 -- 172.21.15.1:0/818716710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07c01a4ca0 msgr2=0x7f07c01a7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:07.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.828+0000 7f07c4c00700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07c01a4ca0 0x7f07c01a7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:07.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.828+0000 7f07c4c00700 1 -- 172.21.15.1:0/818716710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c0108ec0 msgr2=0x7f07c0101e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:07.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.828+0000 7f07c4c00700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c0108ec0 0x7f07c0101e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:07.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.828+0000 7f07c4c00700 1 -- 172.21.15.1:0/818716710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c01a77c0 con 0x7f07c0102380 2024-04-26T21:20:07.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.828+0000 7f07c4c00700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07c0102380 0x7f07c0102800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f07b400b550 tx=0x7f07b400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:20:07.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.829+0000 7f07be000700 1 -- 172.21.15.1:0/818716710 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b4004230 con 0x7f07c0102380 2024-04-26T21:20:07.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.829+0000 7f07be000700 1 -- 172.21.15.1:0/818716710 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07b40043d0 con 0x7f07c0102380 2024-04-26T21:20:07.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.829+0000 7f07c8000700 1 -- 172.21.15.1:0/818716710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07c01a7ab0 con 0x7f07c0102380 2024-04-26T21:20:07.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.829+0000 7f07c8000700 1 -- 172.21.15.1:0/818716710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07c01a8040 con 0x7f07c0102380 2024-04-26T21:20:07.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.829+0000 7f07be000700 1 -- 172.21.15.1:0/818716710 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b4011a50 con 0x7f07c0102380 2024-04-26T21:20:07.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.830+0000 7f07be000700 1 -- 172.21.15.1:0/818716710 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f07b4011cf0 con 0x7f07c0102380 2024-04-26T21:20:07.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.830+0000 7f07c8000700 1 -- 172.21.15.1:0/818716710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07a4005330 con 0x7f07c0102380 2024-04-26T21:20:07.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.831+0000 7f07be000700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07a8039ae0 0x7f07a803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:07.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.831+0000 7f07be000700 1 -- 172.21.15.1:0/818716710 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f07b404c200 con 0x7f07c0102380 2024-04-26T21:20:07.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.831+0000 7f07c5600700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07a8039ae0 0x7f07a803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:07.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.832+0000 7f07c5600700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07a8039ae0 0x7f07a803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009750 tx=0x7f07b001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:20:07.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:07.837+0000 7f07be000700 1 -- 172.21.15.1:0/818716710 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f07b40202a0 con 0x7f07c0102380 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:07.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.075+0000 7f07c8000700 1 -- 172.21.15.1:0/818716710 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f07a4005620 con 0x7f07c0102380 2024-04-26T21:20:08.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T21:20:08.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T21:20:08.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T21:20:08.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:20:08.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:20:08.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:20:08.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:20:08.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.076+0000 7f07be000700 1 -- 172.21.15.1:0/818716710 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7f07b40151c0 con 0x7f07c0102380 2024-04-26T21:20:08.078 INFO:teuthology.orchestra.run.smithi001.stdout:# minimal ceph.conf for 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:20:08.079 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-04-26T21:20:08.079 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:20:08.079 INFO:teuthology.orchestra.run.smithi001.stdout: mon_host = [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T21:20:08.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.080+0000 7f07c8000700 1 -- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07a8039ae0 msgr2=0x7f07a803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:08.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.080+0000 7f07c8000700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07a8039ae0 0x7f07a803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009750 tx=0x7f07b001a040 comp rx=0 tx=0).stop 2024-04-26T21:20:08.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.080+0000 7f07c8000700 1 -- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07c0102380 msgr2=0x7f07c0102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:08.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.080+0000 7f07c8000700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07c0102380 0x7f07c0102800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f07b400b550 tx=0x7f07b400b860 comp rx=0 tx=0).stop 2024-04-26T21:20:08.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.081+0000 7f07c8000700 1 -- 172.21.15.1:0/818716710 shutdown_connections 2024-04-26T21:20:08.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.081+0000 7f07c8000700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07a8039ae0 0x7f07a803bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:08.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.081+0000 7f07c8000700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07c01a4ca0 0x7f07c01a7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:08.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.081+0000 7f07c8000700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07c0108ec0 0x7f07c0101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:08.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.081+0000 7f07c8000700 1 --2- 172.21.15.1:0/818716710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07c0102380 0x7f07c0102800 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:08.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.081+0000 7f07c8000700 1 -- 172.21.15.1:0/818716710 >> 172.21.15.1:0/818716710 conn(0x7f07c0077880 msgr2=0x7f07c00ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:20:08.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.081+0000 7f07c8000700 1 -- 172.21.15.1:0/818716710 shutdown_connections 2024-04-26T21:20:08.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:08.081+0000 7f07c8000700 1 -- 172.21.15.1:0/818716710 wait complete. 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: Reconfiguring daemon mon.a on smithi001 2024-04-26T21:20:08.921 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:08 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/818716710' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:08.995 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T21:20:08.995 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:20:08.995 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: Reconfiguring daemon mon.a on smithi001 2024-04-26T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:08 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/818716710' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:09.086 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:20:09.086 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:20:09.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:09.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:09.135 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:09.135 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:09.135 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:09.135 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: Reconfiguring mon.a (unknown last config time)... 2024-04-26T21:20:09.135 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:20:09.135 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:20:09.135 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:09.135 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: Reconfiguring daemon mon.a on smithi001 2024-04-26T21:20:09.135 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:08 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/818716710' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:09.208 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:20:09.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T21:20:09.241 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:20:09.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:20:09.311 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:20:09.311 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T21:20:09.337 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:20:09.337 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:20:09.402 INFO:tasks.cephadm:Adding mgr.x on smithi001 2024-04-26T21:20:09.402 INFO:tasks.cephadm:Adding mgr.y on smithi038 2024-04-26T21:20:09.402 INFO:tasks.cephadm:Adding mgr.z on smithi186 2024-04-26T21:20:09.402 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi001=x;smithi038=y;smithi186=z' 2024-04-26T21:20:09.675 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:20:11.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:10 smithi186 ceph-mon[32991]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:11.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:11.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:11.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:20:11.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:20:11.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:11.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:10 smithi038 ceph-mon[32724]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:11.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:11.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:11.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:20:11.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:20:11.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:11.266 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.260+0000 7f6bbf55b700 1 -- 172.21.15.186:0/1555516731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb8101e50 msgr2=0x7f6bb8102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:11.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.260+0000 7f6bbf55b700 1 --2- 172.21.15.186:0/1555516731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb8101e50 0x7f6bb8102270 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4009910 tx=0x7f6bb4009c20 comp rx=0 tx=0).stop 2024-04-26T21:20:11.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.260+0000 7f6bbf55b700 1 -- 172.21.15.186:0/1555516731 shutdown_connections 2024-04-26T21:20:11.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.260+0000 7f6bbf55b700 1 --2- 172.21.15.186:0/1555516731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb8101e50 0x7f6bb8102270 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:11.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.260+0000 7f6bbf55b700 1 -- 172.21.15.186:0/1555516731 >> 172.21.15.186:0/1555516731 conn(0x7f6bb80fd460 msgr2=0x7f6bb80ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:20:11.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.260+0000 7f6bbf55b700 1 -- 172.21.15.186:0/1555516731 shutdown_connections 2024-04-26T21:20:11.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.260+0000 7f6bbf55b700 1 -- 172.21.15.186:0/1555516731 wait complete. 2024-04-26T21:20:11.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbf55b700 1 Processor -- start 2024-04-26T21:20:11.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbf55b700 1 -- start start 2024-04-26T21:20:11.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbf55b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb8101e50 0x7f6bb8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:11.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbf55b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bb8198c50 0x7f6bb819dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:11.268 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbf55b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bb81990d0 0x7f6bb81a0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:11.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbf55b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb8107d50 con 0x7f6bb8101e50 2024-04-26T21:20:11.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbf55b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb8107bd0 con 0x7f6bb8198c50 2024-04-26T21:20:11.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbf55b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb8107ed0 con 0x7f6bb81990d0 2024-04-26T21:20:11.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbdaf8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bb81990d0 0x7f6bb81a0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:11.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbdaf8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bb81990d0 0x7f6bb81a0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:33538/0 (socket says 172.21.15.186:33538) 2024-04-26T21:20:11.269 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbdaf8700 1 -- 172.21.15.186:0/2519831331 learned_addr learned my addr 172.21.15.186:0/2519831331 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T21:20:11.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbcaf6700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bb8198c50 0x7f6bb819dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:11.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbd2f7700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb8101e50 0x7f6bb8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:11.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbdaf8700 1 -- 172.21.15.186:0/2519831331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bb8198c50 msgr2=0x7f6bb819dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:11.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbdaf8700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bb8198c50 0x7f6bb819dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:11.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.261+0000 7f6bbdaf8700 1 -- 172.21.15.186:0/2519831331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb8101e50 msgr2=0x7f6bb8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:11.270 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.262+0000 7f6bbdaf8700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb8101e50 0x7f6bb8198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:11.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.262+0000 7f6bbdaf8700 1 -- 172.21.15.186:0/2519831331 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb81a0b20 con 0x7f6bb81990d0 2024-04-26T21:20:11.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.262+0000 7f6bbdaf8700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bb81990d0 0x7f6bb81a0250 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000ba50 tx=0x7f6bb000bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:20:11.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.262+0000 7f6bae7fc700 1 -- 172.21.15.186:0/2519831331 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0019070 con 0x7f6bb81990d0 2024-04-26T21:20:11.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.262+0000 7f6bae7fc700 1 -- 172.21.15.186:0/2519831331 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bb0014440 con 0x7f6bb81990d0 2024-04-26T21:20:11.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.262+0000 7f6bbf55b700 1 -- 172.21.15.186:0/2519831331 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bb81a0e10 con 0x7f6bb81990d0 2024-04-26T21:20:11.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.262+0000 7f6bae7fc700 1 -- 172.21.15.186:0/2519831331 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb000ce60 con 0x7f6bb81990d0 2024-04-26T21:20:11.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.262+0000 7f6bbf55b700 1 -- 172.21.15.186:0/2519831331 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bb81a12f0 con 0x7f6bb81990d0 2024-04-26T21:20:11.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.263+0000 7f6bae7fc700 1 -- 172.21.15.186:0/2519831331 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7f6bb001c070 con 0x7f6bb81990d0 2024-04-26T21:20:11.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.263+0000 7f6bbf55b700 1 -- 172.21.15.186:0/2519831331 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bb8064a80 con 0x7f6bb81990d0 2024-04-26T21:20:11.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.263+0000 7f6bae7fc700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6ba4039ae0 0x7f6ba403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:11.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.263+0000 7f6bae7fc700 1 -- 172.21.15.186:0/2519831331 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6bb004be80 con 0x7f6bb81990d0 2024-04-26T21:20:11.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.264+0000 7f6bbd2f7700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6ba4039ae0 0x7f6ba403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:11.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.265+0000 7f6bbd2f7700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6ba4039ae0 0x7f6ba403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4005e00 tx=0x7f6bb4005260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:20:11.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.268+0000 7f6bae7fc700 1 -- 172.21.15.186:0/2519831331 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6bb0012500 con 0x7f6bb81990d0 2024-04-26T21:20:11.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:10 smithi001 ceph-mon[28054]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:11.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:11.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:11.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:20:11.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:20:11.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:11.480 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.479+0000 7f6bbf55b700 1 -- 172.21.15.186:0/2519831331 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=x;smithi038=y;smithi186=z", "target": ["mon-mgr", ""]}) -- 0x7f6bb8063860 con 0x7f6ba4039ae0 2024-04-26T21:20:11.494 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.493+0000 7f6bae7fc700 1 -- 172.21.15.186:0/2519831331 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6bb8063860 con 0x7f6ba4039ae0 2024-04-26T21:20:11.495 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mgr update... 2024-04-26T21:20:11.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 -- 172.21.15.186:0/2519831331 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6ba4039ae0 msgr2=0x7f6ba403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:11.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6ba4039ae0 0x7f6ba403bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4005e00 tx=0x7f6bb4005260 comp rx=0 tx=0).stop 2024-04-26T21:20:11.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 -- 172.21.15.186:0/2519831331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bb81990d0 msgr2=0x7f6bb81a0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:11.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bb81990d0 0x7f6bb81a0250 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000ba50 tx=0x7f6bb000bd60 comp rx=0 tx=0).stop 2024-04-26T21:20:11.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 -- 172.21.15.186:0/2519831331 shutdown_connections 2024-04-26T21:20:11.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6ba4039ae0 0x7f6ba403bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:11.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bb81990d0 0x7f6bb81a0250 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:11.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bb8101e50 0x7f6bb8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:11.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 --2- 172.21.15.186:0/2519831331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bb8198c50 0x7f6bb819dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:11.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 -- 172.21.15.186:0/2519831331 >> 172.21.15.186:0/2519831331 conn(0x7f6bb80fd460 msgr2=0x7f6bb806bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:20:11.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 -- 172.21.15.186:0/2519831331 shutdown_connections 2024-04-26T21:20:11.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:20:11.495+0000 7f6bbf55b700 1 -- 172.21.15.186:0/2519831331 wait complete. 2024-04-26T21:20:12.036 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:11 smithi038 ceph-mon[32724]: Reconfiguring mon.b (monmap changed)... 2024-04-26T21:20:12.036 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:11 smithi038 ceph-mon[32724]: Reconfiguring daemon mon.b on smithi038 2024-04-26T21:20:12.036 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:12.094 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:11 smithi186 ceph-mon[32991]: Reconfiguring mon.b (monmap changed)... 2024-04-26T21:20:12.094 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:11 smithi186 ceph-mon[32991]: Reconfiguring daemon mon.b on smithi038 2024-04-26T21:20:12.094 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:12.122 DEBUG:teuthology.orchestra.run.smithi038:mgr.y> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T21:20:12.125 DEBUG:teuthology.orchestra.run.smithi186:mgr.z> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.z.service 2024-04-26T21:20:12.127 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T21:20:12.127 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:20:12.127 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T21:20:12.184 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T21:20:12.184 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-04-26T21:20:12.234 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T21:20:12.234 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:12.234 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-26T21:20:12.234 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:12.234 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:12.234 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-04-26 21:19:08.333524500 +0000 2024-04-26T21:20:12.235 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-04-26 21:13:54.204984657 +0000 2024-04-26T21:20:12.235 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-04-26 21:13:54.204984657 +0000 2024-04-26T21:20:12.235 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-04-26 21:13:54.204984657 +0000 2024-04-26T21:20:12.235 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T21:20:12.306 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:11 smithi001 ceph-mon[28054]: Reconfiguring mon.b (monmap changed)... 2024-04-26T21:20:12.306 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:11 smithi001 ceph-mon[28054]: Reconfiguring daemon mon.b on smithi038 2024-04-26T21:20:12.306 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:12.317 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-04-26T21:20:12.317 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-04-26T21:20:12.317 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000276191 s, 1.9 MB/s 2024-04-26T21:20:12.321 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T21:20:12.380 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-04-26T21:20:12.435 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T21:20:12.436 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:12.436 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-26T21:20:12.436 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:12.436 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:12.436 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-04-26 21:19:08.431523168 +0000 2024-04-26T21:20:12.436 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-04-26 21:13:54.782976623 +0000 2024-04-26T21:20:12.436 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-04-26 21:13:54.782976623 +0000 2024-04-26T21:20:12.436 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-04-26 21:13:54.782976623 +0000 2024-04-26T21:20:12.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T21:20:12.524 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-04-26T21:20:12.525 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-04-26T21:20:12.525 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000300037 s, 1.7 MB/s 2024-04-26T21:20:12.528 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T21:20:12.592 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-04-26T21:20:12.650 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T21:20:12.650 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:12.650 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-04-26T21:20:12.650 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:12.650 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:12.650 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-04-26 21:19:08.498522258 +0000 2024-04-26T21:20:12.650 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-04-26 21:13:55.362968562 +0000 2024-04-26T21:20:12.650 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-04-26 21:13:55.362968562 +0000 2024-04-26T21:20:12.650 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-04-26 21:13:55.362968562 +0000 2024-04-26T21:20:12.651 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T21:20:12.740 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-04-26T21:20:12.740 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-04-26T21:20:12.740 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000285818 s, 1.8 MB/s 2024-04-26T21:20:12.745 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T21:20:12.805 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-04-26T21:20:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T21:20:12.859 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:12.859 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 6h/6d Inode: 691 Links: 1 2024-04-26T21:20:12.859 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:12.859 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:12.859 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-04-26 21:19:08.571521266 +0000 2024-04-26T21:20:12.859 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-04-26 21:13:55.971960097 +0000 2024-04-26T21:20:12.859 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-04-26 21:13:55.971960097 +0000 2024-04-26T21:20:12.860 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-04-26 21:13:55.971960097 +0000 2024-04-26T21:20:12.860 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T21:20:12.936 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-04-26T21:20:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-04-26T21:20:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000287081 s, 1.8 MB/s 2024-04-26T21:20:12.940 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T21:20:12.997 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-04-26T21:20:12.997 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T21:20:13.017 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T21:20:13.017 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-04-26T21:20:13.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:12 smithi186 ceph-mon[32991]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:13.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:12 smithi186 ceph-mon[32991]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=x;smithi038=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:20:13.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:12 smithi186 ceph-mon[32991]: Saving service mgr spec with placement smithi001=x;smithi038=y;smithi186=z;count:3 2024-04-26T21:20:13.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:13.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:13.023 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:20:13.024 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:20:13.024 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:13.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:12 smithi001 ceph-mon[28054]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:13.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:12 smithi001 ceph-mon[28054]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=x;smithi038=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:20:13.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:12 smithi001 ceph-mon[28054]: Saving service mgr spec with placement smithi001=x;smithi038=y;smithi186=z;count:3 2024-04-26T21:20:13.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:13.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:13.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:20:13.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:20:13.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:13.076 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T21:20:13.076 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:13.076 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-26T21:20:13.077 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:13.077 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:13.077 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-26 21:19:58.368320551 +0000 2024-04-26T21:20:13.077 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-26 21:13:02.215738220 +0000 2024-04-26T21:20:13.077 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-26 21:13:02.215738220 +0000 2024-04-26T21:20:13.077 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-26 21:13:02.215738220 +0000 2024-04-26T21:20:13.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T21:20:13.143 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:12 smithi038 ceph-mon[32724]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:13.143 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:12 smithi038 ceph-mon[32724]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi001=x;smithi038=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:20:13.143 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:12 smithi038 ceph-mon[32724]: Saving service mgr spec with placement smithi001=x;smithi038=y;smithi186=z;count:3 2024-04-26T21:20:13.143 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:13.143 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:13.143 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T21:20:13.144 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T21:20:13.144 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:13.146 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-26T21:20:13.146 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-26T21:20:13.146 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000314126 s, 1.6 MB/s 2024-04-26T21:20:13.147 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T21:20:13.206 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-04-26T21:20:13.264 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T21:20:13.264 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:13.264 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-26T21:20:13.265 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:13.265 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:13.265 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-26 21:19:58.368320551 +0000 2024-04-26T21:20:13.265 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-26 21:13:02.575731201 +0000 2024-04-26T21:20:13.265 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-26 21:13:02.575731201 +0000 2024-04-26T21:20:13.265 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-26 21:13:02.575731201 +0000 2024-04-26T21:20:13.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T21:20:13.331 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-26T21:20:13.331 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-26T21:20:13.331 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000295906 s, 1.7 MB/s 2024-04-26T21:20:13.332 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T21:20:13.390 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-04-26T21:20:13.447 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T21:20:13.447 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:13.447 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-26T21:20:13.447 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:13.448 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:13.448 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-26 21:19:58.369320532 +0000 2024-04-26T21:20:13.448 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-26 21:13:02.927724337 +0000 2024-04-26T21:20:13.448 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-26 21:13:02.927724337 +0000 2024-04-26T21:20:13.448 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-26 21:13:02.927724337 +0000 2024-04-26T21:20:13.448 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T21:20:13.513 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-26T21:20:13.513 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-26T21:20:13.513 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000290366 s, 1.8 MB/s 2024-04-26T21:20:13.515 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T21:20:13.573 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-04-26T21:20:13.630 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T21:20:13.630 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:13.630 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-26T21:20:13.630 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:13.630 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:13.630 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-04-26 21:19:58.369320532 +0000 2024-04-26T21:20:13.630 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-04-26 21:13:03.287717318 +0000 2024-04-26T21:20:13.630 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-04-26 21:13:03.287717318 +0000 2024-04-26T21:20:13.630 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-04-26 21:13:03.287717318 +0000 2024-04-26T21:20:13.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T21:20:13.695 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-04-26T21:20:13.696 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-04-26T21:20:13.696 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000289433 s, 1.8 MB/s 2024-04-26T21:20:13.697 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T21:20:13.754 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T21:20:13.754 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T21:20:13.769 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T21:20:13.770 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-04-26T21:20:13.826 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T21:20:13.826 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:13.826 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-26T21:20:13.826 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:13.826 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:13.826 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 21:19:48.786948374 +0000 2024-04-26T21:20:13.826 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 21:12:41.334679381 +0000 2024-04-26T21:20:13.826 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 21:12:41.334679381 +0000 2024-04-26T21:20:13.826 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 21:12:41.334679381 +0000 2024-04-26T21:20:13.827 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T21:20:13.889 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T21:20:13.889 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T21:20:13.890 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000191968 s, 2.7 MB/s 2024-04-26T21:20:13.891 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T21:20:13.947 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-04-26T21:20:14.001 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T21:20:14.002 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:14.002 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-26T21:20:14.002 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:14.002 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:14.002 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 21:19:48.787948356 +0000 2024-04-26T21:20:14.002 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 21:12:41.674672950 +0000 2024-04-26T21:20:14.002 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 21:12:41.674672950 +0000 2024-04-26T21:20:14.002 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 21:12:41.674672950 +0000 2024-04-26T21:20:14.002 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T21:20:14.064 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T21:20:14.064 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T21:20:14.064 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000202303 s, 2.5 MB/s 2024-04-26T21:20:14.065 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T21:20:14.121 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-04-26T21:20:14.177 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T21:20:14.177 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:14.177 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-26T21:20:14.177 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:14.177 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:14.177 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 21:19:48.787948356 +0000 2024-04-26T21:20:14.177 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 21:12:42.006666669 +0000 2024-04-26T21:20:14.177 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 21:12:42.006666669 +0000 2024-04-26T21:20:14.177 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 21:12:42.006666669 +0000 2024-04-26T21:20:14.178 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T21:20:14.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: Reconfiguring mon.c (monmap changed)... 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: Reconfiguring daemon mon.c on smithi186 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:20:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:14.228 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T21:20:14.228 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T21:20:14.228 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000204437 s, 2.5 MB/s 2024-04-26T21:20:14.230 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T21:20:14.285 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-04-26T21:20:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: Reconfiguring mon.c (monmap changed)... 2024-04-26T21:20:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: Reconfiguring daemon mon.c on smithi186 2024-04-26T21:20:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:14.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:20:14.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:20:14.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:20:14.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: Reconfiguring mon.c (monmap changed)... 2024-04-26T21:20:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: Reconfiguring daemon mon.c on smithi186 2024-04-26T21:20:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:20:14.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:20:14.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:20:14.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:14.340 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T21:20:14.340 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T21:20:14.340 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-26T21:20:14.340 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:20:14.341 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T21:20:14.341 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 21:19:48.788948338 +0000 2024-04-26T21:20:14.341 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 21:12:42.349660181 +0000 2024-04-26T21:20:14.341 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 21:12:42.349660181 +0000 2024-04-26T21:20:14.341 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-26 21:12:42.349660181 +0000 2024-04-26T21:20:14.341 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T21:20:14.403 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T21:20:14.403 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T21:20:14.403 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000212608 s, 2.4 MB/s 2024-04-26T21:20:14.404 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T21:20:14.460 INFO:tasks.cephadm:Deploying osd.0 on smithi001 with /dev/vg_nvme/lv_4... 2024-04-26T21:20:14.460 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T21:20:14.918 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:20:15.110 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:14 smithi001 ceph-mon[28054]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:15.110 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:14 smithi001 ceph-mon[28054]: Deploying daemon mgr.z on smithi186 2024-04-26T21:20:15.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:14 smithi186 ceph-mon[32991]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:15.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:14 smithi186 ceph-mon[32991]: Deploying daemon mgr.z on smithi186 2024-04-26T21:20:15.229 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:14 smithi038 ceph-mon[32724]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:15.229 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:14 smithi038 ceph-mon[32724]: Deploying daemon mgr.z on smithi186 2024-04-26T21:20:15.894 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:15 smithi186 systemd[1]: Starting Ceph mgr.z for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:20:16.165 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:15 smithi186 podman[34844]: 2024-04-26 21:20:15.893461034 +0000 UTC m=+0.111991108 container create f6020480e4a4231b260effb0cc74e5681acbc7e2b9aca89c25aeec6cbf868a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:20:16.165 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:15 smithi186 podman[34844]: 2024-04-26 21:20:15.799466539 +0000 UTC m=+0.017996641 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:20:16.165 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:16 smithi186 podman[34844]: 2024-04-26 21:20:16.137682995 +0000 UTC m=+0.356213072 container init f6020480e4a4231b260effb0cc74e5681acbc7e2b9aca89c25aeec6cbf868a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-26T21:20:16.165 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:16 smithi186 podman[34844]: 2024-04-26 21:20:16.140256901 +0000 UTC m=+0.358786980 container start f6020480e4a4231b260effb0cc74e5681acbc7e2b9aca89c25aeec6cbf868a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-26T21:20:16.165 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-mgr[34863]: -- 172.21.15.186:0/1891255817 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559f63f905a0 con 0x559f63f65000 2024-04-26T21:20:16.455 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:16 smithi186 bash[34844]: f6020480e4a4231b260effb0cc74e5681acbc7e2b9aca89c25aeec6cbf868a31 2024-04-26T21:20:16.455 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:16 smithi186 systemd[1]: Started Ceph mgr.z for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:20:16.455 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:16.292+0000 7fa7b2ab4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:20:16.849 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:16.476+0000 7fa7b2ab4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:20:16.849 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:16.577+0000 7fa7b2ab4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:20:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:16 smithi001 ceph-mon[28054]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:17.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:17.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:17.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:20:17.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:20:17.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:20:17.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:17.172 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:16 smithi038 ceph-mon[32724]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:17.172 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:17.172 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:17.172 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:17.172 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:20:17.172 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:20:17.173 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:20:17.173 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:17.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-mon[32991]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:20:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T21:20:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:20:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:17.205 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:16.884+0000 7fa7b2ab4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:20:17.205 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:16.969+0000 7fa7b2ab4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:20:17.702 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T21:20:17.703 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T21:20:17.703 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-04-26T21:20:17.703 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-04-26T21:20:17.703 INFO:teuthology.orchestra.run.smithi001.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0388701 s, 270 MB/s 2024-04-26T21:20:17.703 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 2024-04-26T21:20:17.703 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-04-26T21:20:18.052 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:17 smithi038 ceph-mon[32724]: Deploying daemon mgr.y on smithi038 2024-04-26T21:20:18.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:17 smithi001 ceph-mon[28054]: Deploying daemon mgr.y on smithi038 2024-04-26T21:20:18.165 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:17 smithi186 ceph-mon[32991]: Deploying daemon mgr.y on smithi038 2024-04-26T21:20:18.327 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:18 smithi038 systemd[1]: Starting Ceph mgr.y for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:20:18.454 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:18 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:18.164+0000 7fa7b2ab4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:20:18.455 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:18 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:18.333+0000 7fa7b2ab4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:20:18.592 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_4 2024-04-26T21:20:18.740 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:18 smithi038 podman[34101]: 2024-04-26 21:20:18.351891392 +0000 UTC m=+0.018242070 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:20:18.741 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:18 smithi038 podman[34101]: 2024-04-26 21:20:18.460871349 +0000 UTC m=+0.127222028 container create 3a51d5f39ccf3a4d3c165b56e283539be6d1644ff13bc2156d0cb71bff48803d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-26T21:20:18.741 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:18 smithi038 podman[34101]: 2024-04-26 21:20:18.656944955 +0000 UTC m=+0.323295633 container init 3a51d5f39ccf3a4d3c165b56e283539be6d1644ff13bc2156d0cb71bff48803d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T21:20:18.741 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:18 smithi038 podman[34101]: 2024-04-26 21:20:18.659471015 +0000 UTC m=+0.325821693 container start 3a51d5f39ccf3a4d3c165b56e283539be6d1644ff13bc2156d0cb71bff48803d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T21:20:18.741 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:18 smithi038 ceph-mgr[34120]: -- 172.21.15.38:0/3664955954 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5579f089c5a0 con 0x5579f0870800 2024-04-26T21:20:18.741 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:18 smithi038 bash[34101]: 3a51d5f39ccf3a4d3c165b56e283539be6d1644ff13bc2156d0cb71bff48803d 2024-04-26T21:20:18.793 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:18 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:18.506+0000 7fa7b2ab4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:20:19.009 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:18 smithi038 ceph-mon[32724]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:19.010 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:18 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:19.010 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:18 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:19.010 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:18 smithi038 systemd[1]: Started Ceph mgr.y for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:20:19.010 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:18 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:18.815+0000 7fd5f8bff200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:20:19.010 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:19 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:19.008+0000 7fd5f8bff200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:20:19.039 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:20:19.093 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:18 smithi186 ceph-mon[32991]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:19.094 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:18 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:19.094 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:18 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:19.094 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:18 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:18.792+0000 7fa7b2ab4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:20:19.094 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:18 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:18.886+0000 7fa7b2ab4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:20:19.189 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:18 smithi001 ceph-mon[28054]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:19.189 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:18 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:19.189 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:18 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:19.327 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:19 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:19.115+0000 7fd5f8bff200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:20:19.454 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:19 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:19.092+0000 7fa7b2ab4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:20:19.455 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:19 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:19.291+0000 7fa7b2ab4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:20:19.827 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:19 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:19.447+0000 7fd5f8bff200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:20:19.827 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:19 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:19.536+0000 7fd5f8bff200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:20:20.199 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:19 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:20.199 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:19 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:20.199 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:19 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:20.199 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:19 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:19.927+0000 7fa7b2ab4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:20:20.203 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:19 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:20.203 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:19 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:20.203 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:19 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:20.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:19 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:20.394 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:19 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:20.394 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:19 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:20.454 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:20.358+0000 7fa7b2ab4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:20:20.881 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:20.463+0000 7fa7b2ab4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:20:20.882 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:20.665+0000 7fa7b2ab4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:20:21.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:21.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:21.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:21.040 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.041 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:20:21.041 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:20:21.041 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:21.192 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:20.838+0000 7fd5f8bff200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:20:21.192 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:21.014+0000 7fd5f8bff200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:20:21.192 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:21.190+0000 7fd5f8bff200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:20:21.193 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:21.194 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:21.194 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.194 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.194 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.194 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.194 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.194 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:21.194 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:21.194 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:21.195 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T21:20:21.195 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T21:20:21.195 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.213+0000 7fe86d200700 1 -- 172.21.15.1:0/1579310839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe868106840 msgr2=0x7fe868108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.213+0000 7fe86d200700 1 --2- 172.21.15.1:0/1579310839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe868106840 0x7fe868108c30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe854009990 tx=0x7fe854009ca0 comp rx=0 tx=0).stop 2024-04-26T21:20:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.213+0000 7fe86d200700 1 -- 172.21.15.1:0/1579310839 shutdown_connections 2024-04-26T21:20:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.213+0000 7fe86d200700 1 --2- 172.21.15.1:0/1579310839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe868109170 0x7fe86810b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.213+0000 7fe86d200700 1 --2- 172.21.15.1:0/1579310839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe868103f10 0x7fe868106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.213+0000 7fe86d200700 1 --2- 172.21.15.1:0/1579310839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe868106840 0x7fe868108c30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.213+0000 7fe86d200700 1 -- 172.21.15.1:0/1579310839 >> 172.21.15.1:0/1579310839 conn(0x7fe8680fd920 msgr2=0x7fe8680ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:20:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.213+0000 7fe86d200700 1 -- 172.21.15.1:0/1579310839 shutdown_connections 2024-04-26T21:20:21.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.213+0000 7fe86d200700 1 -- 172.21.15.1:0/1579310839 wait complete. 2024-04-26T21:20:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.214+0000 7fe86d200700 1 Processor -- start 2024-04-26T21:20:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.214+0000 7fe86d200700 1 -- start start 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe86d200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe868103f10 0x7fe86819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe86d200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe868106840 0x7fe86819bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe86d200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe868109170 0x7fe8681a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe86d200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe86810ef50 con 0x7fe868103f10 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe86d200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe86810edd0 con 0x7fe868109170 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe86d200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe86810f0d0 con 0x7fe868106840 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe865400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe868106840 0x7fe86819bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe866800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe868109170 0x7fe8681a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe865400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe868106840 0x7fe86819bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:38638/0 (socket says 172.21.15.1:38638) 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe865e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe868103f10 0x7fe86819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.215+0000 7fe865400700 1 -- 172.21.15.1:0/3945985333 learned_addr learned my addr 172.21.15.1:0/3945985333 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe865400700 1 -- 172.21.15.1:0/3945985333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe868109170 msgr2=0x7fe8681a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe865400700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe868109170 0x7fe8681a2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:21.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe865400700 1 -- 172.21.15.1:0/3945985333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe868103f10 msgr2=0x7fe86819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:21.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe865400700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe868103f10 0x7fe86819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:21.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe865400700 1 -- 172.21.15.1:0/3945985333 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8681a3500 con 0x7fe868106840 2024-04-26T21:20:21.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe865400700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe868106840 0x7fe86819bde0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe854000c00 tx=0x7fe8540057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:20:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe85ea00700 1 -- 172.21.15.1:0/3945985333 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854026070 con 0x7fe868106840 2024-04-26T21:20:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe85ea00700 1 -- 172.21.15.1:0/3945985333 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe854021410 con 0x7fe868106840 2024-04-26T21:20:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe85ea00700 1 -- 172.21.15.1:0/3945985333 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854017650 con 0x7fe868106840 2024-04-26T21:20:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe86d200700 1 -- 172.21.15.1:0/3945985333 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8681a3790 con 0x7fe868106840 2024-04-26T21:20:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.216+0000 7fe86d200700 1 -- 172.21.15.1:0/3945985333 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8681a3c40 con 0x7fe868106840 2024-04-26T21:20:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.218+0000 7fe85ea00700 1 -- 172.21.15.1:0/3945985333 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 13) ==== 46613+0+0 (secure 0 0 0) 0x7fe854029070 con 0x7fe868106840 2024-04-26T21:20:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.218+0000 7fe86d200700 1 -- 172.21.15.1:0/3945985333 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86806a110 con 0x7fe868106840 2024-04-26T21:20:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.218+0000 7fe85ea00700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe84c039890 0x7fe84c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:21.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.218+0000 7fe85ea00700 1 -- 172.21.15.1:0/3945985333 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe85404d910 con 0x7fe868106840 2024-04-26T21:20:21.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.219+0000 7fe865e00700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe84c039890 0x7fe84c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:21.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.219+0000 7fe865e00700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe84c039890 0x7fe84c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe858009c60 tx=0x7fe858009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:20:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.224+0000 7fe85ea00700 1 -- 172.21.15.1:0/3945985333 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe854013070 con 0x7fe868106840 2024-04-26T21:20:21.454 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:21.191+0000 7fa7b2ab4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:20:21.455 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:21.283+0000 7fa7b2ab4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:20:21.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:21.471+0000 7fe86d200700 1 -- 172.21.15.1:0/3945985333 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe868102670 con 0x7fe84c039890 2024-04-26T21:20:21.796 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:21.488+0000 7fd5f8bff200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:20:21.796 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:21.583+0000 7fd5f8bff200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:20:21.796 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:21.794+0000 7fd5f8bff200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:20:21.865 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:21.523+0000 7fa7b2ab4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:20:21.865 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:21.655+0000 7fa7b2ab4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:20:21.865 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:21.748+0000 7fa7b2ab4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:20:21.865 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:21.863+0000 7fa7b2ab4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:20:22.077 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:21.993+0000 7fd5f8bff200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:20:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-mon[32724]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T21:20:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-mon[32724]: Reconfiguring daemon mgr.x on smithi001 2024-04-26T21:20:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:20:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:20:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:21 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:22.143 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:21 smithi001 ceph-mon[28054]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T21:20:22.143 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:21 smithi001 ceph-mon[28054]: Reconfiguring daemon mgr.x on smithi001 2024-04-26T21:20:22.144 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:21 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:20:22.144 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:21 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:20:22.144 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:21 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:22.170 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-mon[32991]: Reconfiguring mgr.x (unknown last config time)... 2024-04-26T21:20:22.170 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-mon[32991]: Reconfiguring daemon mgr.x on smithi001 2024-04-26T21:20:22.170 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:20:22.170 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:20:22.170 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:21 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:22.454 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:22 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:22.169+0000 7fa7b2ab4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:20:22.891 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:22 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:22.622+0000 7fd5f8bff200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:20:23.160 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:22 smithi038 ceph-mon[32724]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:23.160 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:22 smithi038 ceph-mon[32724]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:20:23.161 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:23.058+0000 7fd5f8bff200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:20:23.161 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:23.158+0000 7fd5f8bff200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:20:23.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:22 smithi186 ceph-mon[32991]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:22 smithi186 ceph-mon[32991]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:20:23.205 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:22 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:22.759+0000 7fa7b2ab4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:20:23.205 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:20:22 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:20:22.973+0000 7fa7b2ab4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:20:23.250 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:22 smithi001 ceph-mon[28054]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:23.250 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:22 smithi001 ceph-mon[28054]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:20:23.577 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:23.367+0000 7fd5f8bff200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:20:23.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:23.934+0000 7fe85ea00700 1 -- 172.21.15.1:0/3945985333 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mgrmap(e 14) ==== 92601+0+0 (secure 0 0 0) 0x7fe854009d10 con 0x7fe868106840 2024-04-26T21:20:24.201 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: Standby manager daemon z started 2024-04-26T21:20:24.201 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T21:20:24.201 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:20:24.201 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T21:20:24.202 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:20:24.202 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:24.202 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:24.202 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:24.202 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:24.202 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:24.202 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: Standby manager daemon z started 2024-04-26T21:20:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T21:20:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:20:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T21:20:24.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:20:24.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:24.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:24.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:24.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:24.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:24.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:24.256 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:23.909+0000 7fd5f8bff200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:20:24.256 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:24.004+0000 7fd5f8bff200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:20:24.256 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: Standby manager daemon z started 2024-04-26T21:20:24.256 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T21:20:24.256 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:20:24.256 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T21:20:24.256 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.186:0/1435117251' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:20:24.257 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:24.257 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:24.257 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:24.257 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:24.257 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:24.257 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:24.577 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:24.254+0000 7fd5f8bff200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:20:24.577 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:24.391+0000 7fd5f8bff200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:20:24.577 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:24.489+0000 7fd5f8bff200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:20:24.920 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:24.608+0000 7fd5f8bff200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:20:24.920 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:24.918+0000 7fd5f8bff200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:20:25.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:24 smithi001 ceph-mon[28054]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:25.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:24 smithi001 ceph-mon[28054]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-26T21:20:25.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:24 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T21:20:25.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:24 smithi186 ceph-mon[32991]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:25.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:24 smithi186 ceph-mon[32991]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-26T21:20:25.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:24 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T21:20:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:24 smithi038 ceph-mon[32724]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:24 smithi038 ceph-mon[32724]: mgrmap e14: x(active, since 88s), standbys: z 2024-04-26T21:20:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:24 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T21:20:25.827 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:25 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:25.543+0000 7fd5f8bff200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:20:25.827 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:20:25 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:20:25.765+0000 7fd5f8bff200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:20:25.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:25.944+0000 7fe85ea00700 1 -- 172.21.15.1:0/3945985333 <== mon.1 v2:172.21.15.186:3300/0 8 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7fe85400fce0 con 0x7fe868106840 2024-04-26T21:20:26.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:25 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T21:20:26.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:25 smithi186 ceph-mon[32991]: Standby manager daemon y started 2024-04-26T21:20:26.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:25 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:20:26.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:25 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T21:20:26.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:25 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:20:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:25 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T21:20:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:25 smithi001 ceph-mon[28054]: Standby manager daemon y started 2024-04-26T21:20:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:25 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:20:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:25 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T21:20:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:25 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:20:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:25 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T21:20:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:25 smithi038 ceph-mon[32724]: Standby manager daemon y started 2024-04-26T21:20:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:25 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:20:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:25 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T21:20:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:25 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.38:0/1402860269' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:20:27.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:26 smithi186 ceph-mon[32991]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:27.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:26 smithi186 ceph-mon[32991]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-04-26T21:20:27.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:26 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T21:20:27.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:26 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3326424198' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a"}]: dispatch 2024-04-26T21:20:27.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:26 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a"}]: dispatch 2024-04-26T21:20:27.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:26 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a"}]': finished 2024-04-26T21:20:27.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:26 smithi186 ceph-mon[32991]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T21:20:27.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:26 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:27.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:26 smithi001 ceph-mon[28054]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:27.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:26 smithi001 ceph-mon[28054]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-04-26T21:20:27.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:26 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T21:20:27.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:26 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3326424198' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a"}]: dispatch 2024-04-26T21:20:27.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:26 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a"}]: dispatch 2024-04-26T21:20:27.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:26 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a"}]': finished 2024-04-26T21:20:27.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:26 smithi001 ceph-mon[28054]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T21:20:27.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:26 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:26 smithi038 ceph-mon[32724]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:26 smithi038 ceph-mon[32724]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-04-26T21:20:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:26 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T21:20:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:26 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3326424198' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a"}]: dispatch 2024-04-26T21:20:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:26 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a"}]: dispatch 2024-04-26T21:20:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:26 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a"}]': finished 2024-04-26T21:20:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:26 smithi038 ceph-mon[32724]: osdmap e5: 1 total, 0 up, 1 in 2024-04-26T21:20:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:26 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:28.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:27 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3222595193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:20:28.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:27 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3222595193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:20:28.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:27 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3222595193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:20:29.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:28 smithi186 ceph-mon[32991]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:29.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:28 smithi001 ceph-mon[28054]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:29.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:28 smithi038 ceph-mon[32724]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:31.283 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:30 smithi001 ceph-mon[28054]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:31.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:30 smithi038 ceph-mon[32724]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:31.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:30 smithi186 ceph-mon[32991]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:33.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:32 smithi001 ceph-mon[28054]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:33.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:32 smithi038 ceph-mon[32724]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:33.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:32 smithi186 ceph-mon[32991]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:35.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:34 smithi001 ceph-mon[28054]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:35.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:34 smithi038 ceph-mon[32724]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:35.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:34 smithi186 ceph-mon[32991]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:36.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T21:20:36.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:36.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:35 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T21:20:36.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:35 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:35 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T21:20:36.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:35 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:37.283 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:36 smithi001 ceph-mon[28054]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:37.283 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:36 smithi001 ceph-mon[28054]: Deploying daemon osd.0 on smithi001 2024-04-26T21:20:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:36 smithi038 ceph-mon[32724]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:36 smithi038 ceph-mon[32724]: Deploying daemon osd.0 on smithi001 2024-04-26T21:20:37.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:36 smithi186 ceph-mon[32991]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:37.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:36 smithi186 ceph-mon[32991]: Deploying daemon osd.0 on smithi001 2024-04-26T21:20:39.262 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:38 smithi001 ceph-mon[28054]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:39.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:38 smithi038 ceph-mon[32724]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:39.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:38 smithi186 ceph-mon[32991]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:41.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:40 smithi001 ceph-mon[28054]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:41.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:40 smithi038 ceph-mon[32724]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:41.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:40 smithi186 ceph-mon[32991]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:43.124 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:42 smithi001 ceph-mon[28054]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:43.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:42 smithi038 ceph-mon[32724]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:43.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:42 smithi186 ceph-mon[32991]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:45.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:45 smithi001 ceph-mon[28054]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:45.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:45.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:45.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:45.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:45 smithi038 ceph-mon[32724]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:45.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:45.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:45.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:45.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:45 smithi186 ceph-mon[32991]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:45.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:45.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:45.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:47.315 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:47 smithi001 ceph-mon[28054]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:47.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:47 smithi038 ceph-mon[32724]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:47.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:47 smithi186 ceph-mon[32991]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:48.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:48 smithi001 ceph-mon[28054]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:48.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:48 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:48.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:48 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:48.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:48 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:48.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:48 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:48.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:48 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:48.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:48 smithi001 ceph-mon[28054]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T21:20:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:48 smithi038 ceph-mon[32724]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:48 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:48 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:48 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:48 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:48 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:48 smithi038 ceph-mon[32724]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T21:20:48.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:48 smithi186 ceph-mon[32991]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:48.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:48 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:48.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:48 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:48.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:48 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:48.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:48 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:48.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:48 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:48.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:48 smithi186 ceph-mon[32991]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T21:20:48.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.971+0000 7fe85ea00700 1 -- 172.21.15.1:0/3945985333 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe868102670 con 0x7fe84c039890 2024-04-26T21:20:48.972 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 0 on host 'smithi001' 2024-04-26T21:20:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 -- 172.21.15.1:0/3945985333 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe84c039890 msgr2=0x7fe84c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe84c039890 0x7fe84c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe858009c60 tx=0x7fe858009260 comp rx=0 tx=0).stop 2024-04-26T21:20:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 -- 172.21.15.1:0/3945985333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe868106840 msgr2=0x7fe86819bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe868106840 0x7fe86819bde0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe854000c00 tx=0x7fe8540057c0 comp rx=0 tx=0).stop 2024-04-26T21:20:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 -- 172.21.15.1:0/3945985333 shutdown_connections 2024-04-26T21:20:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe84c039890 0x7fe84c03bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:48.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe868109170 0x7fe8681a2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe868103f10 0x7fe86819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 --2- 172.21.15.1:0/3945985333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe868106840 0x7fe86819bde0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 -- 172.21.15.1:0/3945985333 >> 172.21.15.1:0/3945985333 conn(0x7fe8680fd920 msgr2=0x7fe868101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:20:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 -- 172.21.15.1:0/3945985333 shutdown_connections 2024-04-26T21:20:48.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:20:48.975+0000 7fe86d200700 1 -- 172.21.15.1:0/3945985333 wait complete. 2024-04-26T21:20:49.763 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:49 smithi001 ceph-mon[28054]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T21:20:49.763 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:49 smithi001 ceph-mon[28054]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T21:20:49.763 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:49 smithi001 ceph-mon[28054]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:20:49.763 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:49 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:49.763 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:49 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:49.763 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:49 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:49.763 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:49 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:49.816 DEBUG:teuthology.orchestra.run.smithi001:osd.0> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T21:20:49.820 INFO:tasks.cephadm:Deploying osd.1 on smithi038 with /dev/vg_nvme/lv_4... 2024-04-26T21:20:49.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T21:20:49.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:49 smithi186 ceph-mon[32991]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T21:20:49.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:49 smithi186 ceph-mon[32991]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T21:20:49.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:49 smithi186 ceph-mon[32991]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:20:49.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:49 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:49.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:49 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:49.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:49 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:49.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:49 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:50.075 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:20:50.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:49 smithi038 ceph-mon[32724]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T21:20:50.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:49 smithi038 ceph-mon[32724]: osdmap e6: 1 total, 0 up, 1 in 2024-04-26T21:20:50.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:49 smithi038 ceph-mon[32724]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:20:50.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:49 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:50.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:49 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:50.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:49 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:20:50.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:49 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:50.382 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:20:50 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0[40550]: 2024-04-26T21:20:50.351+0000 7f092d000700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T21:20:50.382 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:20:50 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0[40550]: 2024-04-26T21:20:50.380+0000 7f0922e00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:20:50.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:50 smithi001 ceph-mon[28054]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:50.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:50 smithi001 ceph-mon[28054]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:20:50.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:50 smithi001 ceph-mon[28054]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T21:20:50.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:50.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:50.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:50 smithi001 ceph-mon[28054]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' 2024-04-26T21:20:50.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:50 smithi038 ceph-mon[32724]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:50 smithi038 ceph-mon[32724]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:20:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:50 smithi038 ceph-mon[32724]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T21:20:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:50 smithi038 ceph-mon[32724]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' 2024-04-26T21:20:50.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:50 smithi186 ceph-mon[32991]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:50 smithi186 ceph-mon[32991]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:20:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:50 smithi186 ceph-mon[32991]: osdmap e7: 1 total, 0 up, 1 in 2024-04-26T21:20:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:50 smithi186 ceph-mon[32991]: from='osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555]' entity='osd.0' 2024-04-26T21:20:51.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:51 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:20:51.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:51 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:20:51.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:51 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:51.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:51 smithi001 ceph-mon[28054]: osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] boot 2024-04-26T21:20:51.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:51 smithi001 ceph-mon[28054]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T21:20:51.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:51 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:51.833 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T21:20:51.833 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T21:20:51.834 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-26T21:20:51.834 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-26T21:20:51.834 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144266 s, 727 MB/s 2024-04-26T21:20:51.834 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-26T21:20:51.834 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:51 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:20:51.834 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:51 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:20:51.834 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:51 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:51.834 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:51 smithi038 ceph-mon[32724]: osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] boot 2024-04-26T21:20:51.834 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:51 smithi038 ceph-mon[32724]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T21:20:51.834 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:51 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:51.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:51 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:20:51.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:51 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:20:51.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:51 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:51.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:51 smithi186 ceph-mon[32991]: osd.0 [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] boot 2024-04-26T21:20:51.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:51 smithi186 ceph-mon[32991]: osdmap e8: 1 total, 1 up, 1 in 2024-04-26T21:20:51.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:51 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:20:52.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_4 2024-04-26T21:20:52.666 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:20:52.783 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:52 smithi038 ceph-mon[32724]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:52.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:52 smithi186 ceph-mon[32991]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:53.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:52 smithi001 ceph-mon[28054]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T21:20:53.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:53 smithi186 ceph-mon[32991]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T21:20:53.972 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:53 smithi038 ceph-mon[32724]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T21:20:54.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:53 smithi001 ceph-mon[28054]: osdmap e9: 1 total, 1 up, 1 in 2024-04-26T21:20:54.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.239+0000 7f091ebc4700 1 -- 172.21.15.38:0/2229510661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0918103f30 msgr2=0x7f091810a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:54.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.239+0000 7f091ebc4700 1 --2- 172.21.15.38:0/2229510661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0918103f30 0x7f091810a7e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f091000b0d0 tx=0x7f091000b3e0 comp rx=0 tx=0).stop 2024-04-26T21:20:54.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.239+0000 7f091ebc4700 1 -- 172.21.15.38:0/2229510661 shutdown_connections 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.239+0000 7f091ebc4700 1 --2- 172.21.15.38:0/2229510661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0918102390 0x7f09181027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.239+0000 7f091ebc4700 1 --2- 172.21.15.38:0/2229510661 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0918103f30 0x7f091810a7e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.239+0000 7f091ebc4700 1 --2- 172.21.15.38:0/2229510661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0918103590 0x7f09181039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.239+0000 7f091ebc4700 1 -- 172.21.15.38:0/2229510661 >> 172.21.15.38:0/2229510661 conn(0x7f09180fd950 msgr2=0x7f09180ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.239+0000 7f091ebc4700 1 -- 172.21.15.38:0/2229510661 shutdown_connections 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.239+0000 7f091ebc4700 1 -- 172.21.15.38:0/2229510661 wait complete. 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091ebc4700 1 Processor -- start 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091ebc4700 1 -- start start 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091ebc4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0918102390 0x7f0918106320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091ebc4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0918103590 0x7f0918104970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091ebc4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0918103f30 0x7f0918104eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:54.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091ebc4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f091810ded0 con 0x7f0918103590 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091ebc4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f091810dd50 con 0x7f0918102390 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091ebc4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f091810e050 con 0x7f0918103f30 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091c960700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0918102390 0x7f0918106320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091c960700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0918102390 0x7f0918106320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59938/0 (socket says 172.21.15.38:59938) 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.240+0000 7f091c960700 1 -- 172.21.15.38:0/113664699 learned_addr learned my addr 172.21.15.38:0/113664699 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f091d161700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0918103f30 0x7f0918104eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f091c960700 1 -- 172.21.15.38:0/113664699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0918103f30 msgr2=0x7f0918104eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f091c960700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0918103f30 0x7f0918104eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f091c960700 1 -- 172.21.15.38:0/113664699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0918103590 msgr2=0x7f0918104970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f091c960700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0918103590 0x7f0918104970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f091c960700 1 -- 172.21.15.38:0/113664699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0918105760 con 0x7f0918102390 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f091d161700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0918103f30 0x7f0918104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:20:54.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f091c960700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0918102390 0x7f0918106320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f090c00c990 tx=0x7f090c00cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:20:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f0915ffb700 1 -- 172.21.15.38:0/113664699 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c007b00 con 0x7f0918102390 2024-04-26T21:20:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f091ebc4700 1 -- 172.21.15.38:0/113664699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09181a3780 con 0x7f0918102390 2024-04-26T21:20:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f0915ffb700 1 -- 172.21.15.38:0/113664699 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f090c007ca0 con 0x7f0918102390 2024-04-26T21:20:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.241+0000 7f0915ffb700 1 -- 172.21.15.38:0/113664699 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f090c018660 con 0x7f0918102390 2024-04-26T21:20:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.242+0000 7f091ebc4700 1 -- 172.21.15.38:0/113664699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09181a3d40 con 0x7f0918102390 2024-04-26T21:20:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.243+0000 7f091ebc4700 1 -- 172.21.15.38:0/113664699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f091806a110 con 0x7f0918102390 2024-04-26T21:20:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.244+0000 7f0915ffb700 1 -- 172.21.15.38:0/113664699 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7f090c00f450 con 0x7f0918102390 2024-04-26T21:20:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.244+0000 7f0915ffb700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09080a3c30 0x7f09080a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:20:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.244+0000 7f0915ffb700 1 -- 172.21.15.38:0/113664699 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f090c0ce8e0 con 0x7f0918102390 2024-04-26T21:20:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.245+0000 7f0917fff700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09080a3c30 0x7f09080a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:20:54.246 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.246+0000 7f0917fff700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09080a3c30 0x7f09080a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09040097b0 tx=0x7f0904006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:20:54.249 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.249+0000 7f0915ffb700 1 -- 172.21.15.38:0/113664699 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f090c014030 con 0x7f0918102390 2024-04-26T21:20:54.465 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:20:54.465+0000 7f091ebc4700 1 -- 172.21.15.38:0/113664699 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f09181a3fa0 con 0x7f09080a3c30 2024-04-26T21:20:54.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:54 smithi186 ceph-mon[32991]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:20:54.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:20:54.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:20:54.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:55.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:54 smithi001 ceph-mon[28054]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:20:55.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:20:55.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:20:55.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:55.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:54 smithi038 ceph-mon[32724]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:20:55.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:20:55.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:20:55.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:55.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:55 smithi038 ceph-mon[32724]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:20:55.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:55 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi001 2024-04-26T21:20:55.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:55.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:55.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:20:55.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:55 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi001 to 17094M 2024-04-26T21:20:55.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:55.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:55.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:55.797 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:55 smithi186 ceph-mon[32991]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:20:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:55 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi001 2024-04-26T21:20:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:20:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:55 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi001 to 17094M 2024-04-26T21:20:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:55 smithi001 ceph-mon[28054]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:20:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:55 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi001 2024-04-26T21:20:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:20:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:55 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi001 to 17094M 2024-04-26T21:20:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:20:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:20:56.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:20:57.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:56 smithi001 ceph-mon[28054]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:20:57.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:56 smithi038 ceph-mon[32724]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:20:57.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:56 smithi186 ceph-mon[32991]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:20:58.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:57 smithi001 ceph-mon[28054]: from='client.? 172.21.15.38:0/1566619301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf91a985-daf1-49bb-9ad9-d5eaabbe6896"}]: dispatch 2024-04-26T21:20:58.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:57 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf91a985-daf1-49bb-9ad9-d5eaabbe6896"}]: dispatch 2024-04-26T21:20:58.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:57 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf91a985-daf1-49bb-9ad9-d5eaabbe6896"}]': finished 2024-04-26T21:20:58.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:57 smithi001 ceph-mon[28054]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T21:20:58.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:20:58.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:57 smithi038 ceph-mon[32724]: from='client.? 172.21.15.38:0/1566619301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf91a985-daf1-49bb-9ad9-d5eaabbe6896"}]: dispatch 2024-04-26T21:20:58.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:57 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf91a985-daf1-49bb-9ad9-d5eaabbe6896"}]: dispatch 2024-04-26T21:20:58.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:57 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf91a985-daf1-49bb-9ad9-d5eaabbe6896"}]': finished 2024-04-26T21:20:58.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:57 smithi038 ceph-mon[32724]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T21:20:58.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:20:58.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:57 smithi186 ceph-mon[32991]: from='client.? 172.21.15.38:0/1566619301' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf91a985-daf1-49bb-9ad9-d5eaabbe6896"}]: dispatch 2024-04-26T21:20:58.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:57 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bf91a985-daf1-49bb-9ad9-d5eaabbe6896"}]: dispatch 2024-04-26T21:20:58.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:57 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bf91a985-daf1-49bb-9ad9-d5eaabbe6896"}]': finished 2024-04-26T21:20:58.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:57 smithi186 ceph-mon[32991]: osdmap e10: 2 total, 1 up, 2 in 2024-04-26T21:20:58.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:20:59.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:58 smithi001 ceph-mon[28054]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:20:59.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:20:58 smithi001 ceph-mon[28054]: from='client.? 172.21.15.38:0/3741709290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:20:59.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:58 smithi038 ceph-mon[32724]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:20:59.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:20:58 smithi038 ceph-mon[32724]: from='client.? 172.21.15.38:0/3741709290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:20:59.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:58 smithi186 ceph-mon[32991]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:20:59.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:20:58 smithi186 ceph-mon[32991]: from='client.? 172.21.15.38:0/3741709290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:21:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:00 smithi001 ceph-mon[28054]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:01.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:00 smithi038 ceph-mon[32724]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:01.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:00 smithi186 ceph-mon[32991]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:03.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:02 smithi001 ceph-mon[28054]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:03.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:02 smithi038 ceph-mon[32724]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:03.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:02 smithi186 ceph-mon[32991]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:04 smithi001 ceph-mon[28054]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T21:21:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:05.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:04 smithi038 ceph-mon[32724]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:05.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T21:21:05.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:05.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:04 smithi186 ceph-mon[32991]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:05.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T21:21:05.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:06.033 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:05 smithi038 ceph-mon[32724]: Deploying daemon osd.1 on smithi038 2024-04-26T21:21:06.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:05 smithi001 ceph-mon[28054]: Deploying daemon osd.1 on smithi038 2024-04-26T21:21:06.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:05 smithi186 ceph-mon[32991]: Deploying daemon osd.1 on smithi038 2024-04-26T21:21:06.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:06 smithi001 ceph-mon[28054]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:06.963 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:06 smithi038 ceph-mon[32724]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:07.106 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:06 smithi186 ceph-mon[32991]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:09.004 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:08 smithi038 ceph-mon[32724]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:09.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:08 smithi001 ceph-mon[28054]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:09.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:08 smithi186 ceph-mon[32991]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:11.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:10 smithi186 ceph-mon[32991]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:11.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:11.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:11.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:11.297 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:10 smithi038 ceph-mon[32724]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:11.297 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:11.297 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:11.297 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:11.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:10 smithi001 ceph-mon[28054]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:11.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:11.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:11.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:13.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:12 smithi186 ceph-mon[32991]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:13.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:13.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:13.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:13.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:13.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:13.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:12 smithi001 ceph-mon[28054]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:13.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:13.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:13.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:13.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:13.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:13.323 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:12 smithi038 ceph-mon[32724]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:13.323 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:13.323 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:13.323 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:13.323 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:13.323 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:13.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.702+0000 7f0915ffb700 1 -- 172.21.15.38:0/113664699 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f09181a3fa0 con 0x7f09080a3c30 2024-04-26T21:21:13.704 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 1 on host 'smithi038' 2024-04-26T21:21:13.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 -- 172.21.15.38:0/113664699 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09080a3c30 msgr2=0x7f09080a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:21:13.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09080a3c30 0x7f09080a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f09040097b0 tx=0x7f0904006d20 comp rx=0 tx=0).stop 2024-04-26T21:21:13.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 -- 172.21.15.38:0/113664699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0918102390 msgr2=0x7f0918106320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:21:13.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0918102390 0x7f0918106320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f090c00c990 tx=0x7f090c00cd50 comp rx=0 tx=0).stop 2024-04-26T21:21:13.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 -- 172.21.15.38:0/113664699 shutdown_connections 2024-04-26T21:21:13.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09080a3c30 0x7f09080a60f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:13.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0918103f30 0x7f0918104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:13.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0918102390 0x7f0918106320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:13.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 --2- 172.21.15.38:0/113664699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0918103590 0x7f0918104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:13.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 -- 172.21.15.38:0/113664699 >> 172.21.15.38:0/113664699 conn(0x7f09180fd950 msgr2=0x7f09180ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:21:13.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 -- 172.21.15.38:0/113664699 shutdown_connections 2024-04-26T21:21:13.707 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:21:13.704+0000 7f091ebc4700 1 -- 172.21.15.38:0/113664699 wait complete. 2024-04-26T21:21:14.404 DEBUG:teuthology.orchestra.run.smithi038:osd.1> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T21:21:14.407 INFO:tasks.cephadm:Deploying osd.2 on smithi186 with /dev/vg_nvme/lv_4... 2024-04-26T21:21:14.407 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T21:21:14.650 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:21:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:14 smithi186 ceph-mon[32991]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:14 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:14 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:14 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:14 smithi186 ceph-mon[32991]: from='osd.1 [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:21:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:14 smithi186 ceph-mon[32991]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:21:14.947 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:14 smithi038 ceph-mon[32724]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:14.947 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:14 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:14.947 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:14 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:14.947 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:14 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:14.947 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:14 smithi038 ceph-mon[32724]: from='osd.1 [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:21:14.947 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:14 smithi038 ceph-mon[32724]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:21:15.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:14 smithi001 ceph-mon[28054]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:15.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:14 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:15.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:14 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:15.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:14 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:15.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:14 smithi001 ceph-mon[28054]: from='osd.1 [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:21:15.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:14 smithi001 ceph-mon[28054]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T21:21:15.950 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:15 smithi186 ceph-mon[32991]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T21:21:15.951 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:15 smithi186 ceph-mon[32991]: from='osd.1 [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:21:15.951 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:15 smithi186 ceph-mon[32991]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T21:21:15.951 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:15 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:15.951 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:15 smithi186 ceph-mon[32991]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:21:15.988 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:15 smithi001 ceph-mon[28054]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T21:21:15.988 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:15 smithi001 ceph-mon[28054]: from='osd.1 [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:21:15.988 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:15 smithi001 ceph-mon[28054]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T21:21:15.988 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:15 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:15.988 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:15 smithi001 ceph-mon[28054]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:21:16.023 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:15 smithi038 ceph-mon[32724]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T21:21:16.023 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:15 smithi038 ceph-mon[32724]: from='osd.1 [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:21:16.023 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:15 smithi038 ceph-mon[32724]: osdmap e11: 2 total, 1 up, 2 in 2024-04-26T21:21:16.023 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:15 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:16.024 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:15 smithi038 ceph-mon[32724]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:21:16.024 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:21:15 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1[37337]: 2024-04-26T21:21:15.670+0000 7f6e1a5a9700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T21:21:16.024 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:21:15 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1[37337]: 2024-04-26T21:21:15.679+0000 7f6e1639c700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:21:16.594 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T21:21:16.594 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T21:21:16.594 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T21:21:16.594 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T21:21:16.594 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011467 s, 914 MB/s 2024-04-26T21:21:16.594 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T21:21:16.909 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:16 smithi186 ceph-mon[32991]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:16.909 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:16 smithi186 ceph-mon[32991]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:21:16.909 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:16 smithi186 ceph-mon[32991]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T21:21:16.910 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:16.910 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:16 smithi001 ceph-mon[28054]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:16 smithi001 ceph-mon[28054]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:21:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:16 smithi001 ceph-mon[28054]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T21:21:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:16 smithi038 ceph-mon[32724]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:16 smithi038 ceph-mon[32724]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:21:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:16 smithi038 ceph-mon[32724]: osdmap e12: 2 total, 1 up, 2 in 2024-04-26T21:21:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:17.186 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_4 2024-04-26T21:21:17.434 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: osd.1 [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] boot 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:17.788 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:17.789 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:21:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:21:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: osd.1 [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] boot 2024-04-26T21:21:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T21:21:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:17.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:17.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:17.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:21:17.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:17.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:17.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:18.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:21:18.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:21:18.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:18.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: osd.1 [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] boot 2024-04-26T21:21:18.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: osdmap e13: 2 total, 2 up, 2 in 2024-04-26T21:21:18.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:21:18.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:18.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:18.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:21:18.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:18.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:18.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:18.816 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:18 smithi186 ceph-mon[32991]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:18.816 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:18 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi038 2024-04-26T21:21:18.816 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:18 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-26T21:21:18.816 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:18 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:18.816 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:18 smithi186 ceph-mon[32991]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T21:21:19.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.031+0000 7f1fbdf21700 1 -- 172.21.15.186:0/842167646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb8103480 msgr2=0x7f1fb81038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:21:19.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.031+0000 7f1fbdf21700 1 --2- 172.21.15.186:0/842167646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb8103480 0x7f1fb81038a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009990 tx=0x7f1fa8009ca0 comp rx=0 tx=0).stop 2024-04-26T21:21:19.040 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.031+0000 7f1fbdf21700 1 -- 172.21.15.186:0/842167646 shutdown_connections 2024-04-26T21:21:19.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.031+0000 7f1fbdf21700 1 --2- 172.21.15.186:0/842167646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb8103480 0x7f1fb81038a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:19.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.031+0000 7f1fbdf21700 1 --2- 172.21.15.186:0/842167646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fb8105020 0x7f1fb810b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:19.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.031+0000 7f1fbdf21700 1 --2- 172.21.15.186:0/842167646 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fb8104680 0x7f1fb8104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:19.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.031+0000 7f1fbdf21700 1 -- 172.21.15.186:0/842167646 >> 172.21.15.186:0/842167646 conn(0x7f1fb80fea20 msgr2=0x7f1fb8100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:21:19.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.031+0000 7f1fbdf21700 1 -- 172.21.15.186:0/842167646 shutdown_connections 2024-04-26T21:21:19.041 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.031+0000 7f1fbdf21700 1 -- 172.21.15.186:0/842167646 wait complete. 2024-04-26T21:21:19.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.031+0000 7f1fbdf21700 1 Processor -- start 2024-04-26T21:21:19.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fbdf21700 1 -- start start 2024-04-26T21:21:19.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fbdf21700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb8103480 0x7f1fb81996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:21:19.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fbdf21700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fb8104680 0x7f1fb8199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:21:19.042 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fbdf21700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fb8105020 0x7f1fb81a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:21:19.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fbdf21700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb810efd0 con 0x7f1fb8104680 2024-04-26T21:21:19.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fbdf21700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb810ee50 con 0x7f1fb8105020 2024-04-26T21:21:19.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fbdf21700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb810f150 con 0x7f1fb8103480 2024-04-26T21:21:19.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb8103480 0x7f1fb81996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:21:19.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb77fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb8103480 0x7f1fb81996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:52918/0 (socket says 172.21.15.186:52918) 2024-04-26T21:21:19.043 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb77fe700 1 -- 172.21.15.186:0/1449966999 learned_addr learned my addr 172.21.15.186:0/1449966999 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T21:21:19.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb7fff700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fb8105020 0x7f1fb81a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:21:19.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb77fe700 1 -- 172.21.15.186:0/1449966999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fb8105020 msgr2=0x7f1fb81a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:21:19.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb77fe700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fb8105020 0x7f1fb81a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:19.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb77fe700 1 -- 172.21.15.186:0/1449966999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fb8104680 msgr2=0x7f1fb8199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:21:19.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb77fe700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fb8104680 0x7f1fb8199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:19.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb77fe700 1 -- 172.21.15.186:0/1449966999 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb81a1320 con 0x7f1fb8103480 2024-04-26T21:21:19.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb7fff700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fb8105020 0x7f1fb81a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:21:19.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb77fe700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb8103480 0x7f1fb81996c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80047b0 tx=0x7f1fa8003940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:21:19.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb4ff9700 1 -- 172.21.15.186:0/1449966999 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa8027070 con 0x7f1fb8103480 2024-04-26T21:21:19.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb4ff9700 1 -- 172.21.15.186:0/1449966999 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fa8022470 con 0x7f1fb8103480 2024-04-26T21:21:19.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fbdf21700 1 -- 172.21.15.186:0/1449966999 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb81a15b0 con 0x7f1fb8103480 2024-04-26T21:21:19.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.032+0000 7f1fb4ff9700 1 -- 172.21.15.186:0/1449966999 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa800f6c0 con 0x7f1fb8103480 2024-04-26T21:21:19.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.033+0000 7f1fbdf21700 1 -- 172.21.15.186:0/1449966999 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb81a1a00 con 0x7f1fb8103480 2024-04-26T21:21:19.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.033+0000 7f1fbdf21700 1 -- 172.21.15.186:0/1449966999 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fb806a110 con 0x7f1fb8103480 2024-04-26T21:21:19.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.035+0000 7f1fb4ff9700 1 -- 172.21.15.186:0/1449966999 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7f1fa802a070 con 0x7f1fb8103480 2024-04-26T21:21:19.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.035+0000 7f1fb4ff9700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fa00a3bb0 0x7f1fa00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:21:19.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.035+0000 7f1fb4ff9700 1 -- 172.21.15.186:0/1449966999 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f1fa80cff40 con 0x7f1fb8103480 2024-04-26T21:21:19.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.036+0000 7f1fb6ffd700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fa00a3bb0 0x7f1fa00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:21:19.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.037+0000 7f1fb6ffd700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fa00a3bb0 0x7f1fa00a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0097b0 tx=0x7f1fac006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:21:19.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.039+0000 7f1fb4ff9700 1 -- 172.21.15.186:0/1449966999 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1fa8013070 con 0x7f1fb8103480 2024-04-26T21:21:19.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:18 smithi001 ceph-mon[28054]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:19.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:18 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi038 2024-04-26T21:21:19.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:18 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-26T21:21:19.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:18 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:19.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:18 smithi001 ceph-mon[28054]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T21:21:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:18 smithi038 ceph-mon[32724]: pgmap v74: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-26T21:21:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:18 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi038 2024-04-26T21:21:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:18 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi038 to 17093M 2024-04-26T21:21:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:18 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:18 smithi038 ceph-mon[32724]: osdmap e14: 2 total, 2 up, 2 in 2024-04-26T21:21:19.245 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:19.244+0000 7f1fbdf21700 1 -- 172.21.15.186:0/1449966999 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1fb8063860 con 0x7f1fa00a3bb0 2024-04-26T21:21:19.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:19 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:21:19.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:19 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:21:19.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:19 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:20.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:19 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:21:20.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:19 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:21:20.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:19 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:20.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:19 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:21:20.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:19 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:21:20.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:19 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:20.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:20 smithi186 ceph-mon[32991]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:20.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:20 smithi186 ceph-mon[32991]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:21:21.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:20 smithi001 ceph-mon[28054]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:21.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:20 smithi001 ceph-mon[28054]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:21:21.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:20 smithi038 ceph-mon[32724]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:21.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:20 smithi038 ceph-mon[32724]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:21:23.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:22 smithi001 ceph-mon[28054]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:23.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:22 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/2019349812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02453d9d-0297-4356-a632-9b786e930f63"}]: dispatch 2024-04-26T21:21:23.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:22 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02453d9d-0297-4356-a632-9b786e930f63"}]: dispatch 2024-04-26T21:21:23.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:22 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02453d9d-0297-4356-a632-9b786e930f63"}]': finished 2024-04-26T21:21:23.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:22 smithi001 ceph-mon[28054]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T21:21:23.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:22 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:23.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:22 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/494570290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:21:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:22 smithi038 ceph-mon[32724]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:22 smithi038 ceph-mon[32724]: from='client.? 172.21.15.186:0/2019349812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02453d9d-0297-4356-a632-9b786e930f63"}]: dispatch 2024-04-26T21:21:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:22 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02453d9d-0297-4356-a632-9b786e930f63"}]: dispatch 2024-04-26T21:21:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:22 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02453d9d-0297-4356-a632-9b786e930f63"}]': finished 2024-04-26T21:21:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:22 smithi038 ceph-mon[32724]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T21:21:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:22 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:22 smithi038 ceph-mon[32724]: from='client.? 172.21.15.186:0/494570290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:21:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:22 smithi186 ceph-mon[32991]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:22 smithi186 ceph-mon[32991]: from='client.? 172.21.15.186:0/2019349812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02453d9d-0297-4356-a632-9b786e930f63"}]: dispatch 2024-04-26T21:21:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:22 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "02453d9d-0297-4356-a632-9b786e930f63"}]: dispatch 2024-04-26T21:21:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:22 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "02453d9d-0297-4356-a632-9b786e930f63"}]': finished 2024-04-26T21:21:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:22 smithi186 ceph-mon[32991]: osdmap e15: 3 total, 2 up, 3 in 2024-04-26T21:21:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:22 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:22 smithi186 ceph-mon[32991]: from='client.? 172.21.15.186:0/494570290' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:21:25.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:24 smithi001 ceph-mon[28054]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:25.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:24 smithi038 ceph-mon[32724]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:25.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:24 smithi186 ceph-mon[32991]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:27.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:26 smithi001 ceph-mon[28054]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:27.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:26 smithi038 ceph-mon[32724]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:27.155 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:26 smithi186 ceph-mon[32991]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:29.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:28 smithi001 ceph-mon[28054]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:29.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:28 smithi038 ceph-mon[32724]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:29.163 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:28 smithi186 ceph-mon[32991]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:30.055 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:29 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T21:21:30.055 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:29 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:30.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T21:21:30.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:30.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:29 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T21:21:30.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:29 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:30.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:30 smithi186 ceph-mon[32991]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:30.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:30 smithi186 ceph-mon[32991]: Deploying daemon osd.2 on smithi186 2024-04-26T21:21:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:30 smithi001 ceph-mon[28054]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:30 smithi001 ceph-mon[28054]: Deploying daemon osd.2 on smithi186 2024-04-26T21:21:31.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:30 smithi038 ceph-mon[32724]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:30 smithi038 ceph-mon[32724]: Deploying daemon osd.2 on smithi186 2024-04-26T21:21:32.904 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:32 smithi186 ceph-mon[32991]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:33.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:32 smithi001 ceph-mon[28054]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:33.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:32 smithi038 ceph-mon[32724]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:34.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:34 smithi186 ceph-mon[32991]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:35.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:34 smithi001 ceph-mon[28054]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:35.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:34 smithi038 ceph-mon[32724]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:36.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:35 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:36.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:35 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:36.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:35 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:36.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:36.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:36.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:36.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:35 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:36.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:35 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:36.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:35 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:36.997 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:36 smithi186 ceph-mon[32991]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:37.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:36 smithi001 ceph-mon[28054]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:37.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:36 smithi038 ceph-mon[32724]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:38 smithi186 ceph-mon[32991]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:38.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:38.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:38.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.538+0000 7f1fb4ff9700 1 -- 172.21.15.186:0/1449966999 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1fb8063860 con 0x7f1fa00a3bb0 2024-04-26T21:21:38.539 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 2 on host 'smithi186' 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.539+0000 7f1fbdf21700 1 -- 172.21.15.186:0/1449966999 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fa00a3bb0 msgr2=0x7f1fa00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.539+0000 7f1fbdf21700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fa00a3bb0 0x7f1fa00a6070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1fac0097b0 tx=0x7f1fac006d20 comp rx=0 tx=0).stop 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.540+0000 7f1fbdf21700 1 -- 172.21.15.186:0/1449966999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb8103480 msgr2=0x7f1fb81996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.540+0000 7f1fbdf21700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb8103480 0x7f1fb81996c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1fa80047b0 tx=0x7f1fa8003940 comp rx=0 tx=0).stop 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.540+0000 7f1fbdf21700 1 -- 172.21.15.186:0/1449966999 shutdown_connections 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.540+0000 7f1fbdf21700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fa00a3bb0 0x7f1fa00a6070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.540+0000 7f1fbdf21700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fb8103480 0x7f1fb81996c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.540+0000 7f1fbdf21700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fb8105020 0x7f1fb81a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.540+0000 7f1fbdf21700 1 --2- 172.21.15.186:0/1449966999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fb8104680 0x7f1fb8199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.540+0000 7f1fbdf21700 1 -- 172.21.15.186:0/1449966999 >> 172.21.15.186:0/1449966999 conn(0x7f1fb80fea20 msgr2=0x7f1fb8107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.540+0000 7f1fbdf21700 1 -- 172.21.15.186:0/1449966999 shutdown_connections 2024-04-26T21:21:38.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:21:38.540+0000 7f1fbdf21700 1 -- 172.21.15.186:0/1449966999 wait complete. 2024-04-26T21:21:38.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:38 smithi001 ceph-mon[28054]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:38.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:38.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:38.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:38.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:38.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:38.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:38 smithi038 ceph-mon[32724]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:38.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:38.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:38.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:38.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:38.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:39.218 DEBUG:teuthology.orchestra.run.smithi186:osd.2> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T21:21:39.221 INFO:tasks.cephadm:Deploying osd.3 on smithi001 with /dev/vg_nvme/lv_3... 2024-04-26T21:21:39.221 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T21:21:39.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:39 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:39.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:39 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:39.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:39 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:39.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:39 smithi001 ceph-mon[28054]: from='osd.2 [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:21:39.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:39 smithi001 ceph-mon[28054]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:21:39.678 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:21:39.811 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:39 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:39.811 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:39 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:39.812 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:39 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:39.812 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:39 smithi186 ceph-mon[32991]: from='osd.2 [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:21:39.812 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:39 smithi186 ceph-mon[32991]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:21:39.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:39 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:39.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:39 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:21:39.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:39 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:39.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:39 smithi038 ceph-mon[32724]: from='osd.2 [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:21:39.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:39 smithi038 ceph-mon[32724]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T21:21:40.729 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:40 smithi001 ceph-mon[28054]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:40.729 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:40 smithi001 ceph-mon[28054]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T21:21:40.729 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:40 smithi001 ceph-mon[28054]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T21:21:40.729 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:40.729 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:40 smithi001 ceph-mon[28054]: from='osd.2 [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:21:40.729 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:40 smithi001 ceph-mon[28054]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:21:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:40 smithi038 ceph-mon[32724]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:40 smithi038 ceph-mon[32724]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T21:21:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:40 smithi038 ceph-mon[32724]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T21:21:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:40 smithi038 ceph-mon[32724]: from='osd.2 [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:21:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:40 smithi038 ceph-mon[32724]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:21:40.828 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:40 smithi186 ceph-mon[32991]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:40.828 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:40 smithi186 ceph-mon[32991]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T21:21:40.828 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:40 smithi186 ceph-mon[32991]: osdmap e16: 3 total, 2 up, 3 in 2024-04-26T21:21:40.828 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:40.828 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:40 smithi186 ceph-mon[32991]: from='osd.2 [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:21:40.828 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:40 smithi186 ceph-mon[32991]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:21:40.829 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:21:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2[38145]: 2024-04-26T21:21:40.524+0000 7f0edfff9700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T21:21:40.829 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:21:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2[38145]: 2024-04-26T21:21:40.533+0000 7f0edbdec700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:21:41.779 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:41 smithi001 ceph-mon[28054]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:21:41.779 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:41 smithi001 ceph-mon[28054]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T21:21:41.779 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:41.779 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:41.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:41 smithi038 ceph-mon[32724]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:21:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:41 smithi038 ceph-mon[32724]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T21:21:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:41 smithi186 ceph-mon[32991]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:21:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:41 smithi186 ceph-mon[32991]: osdmap e17: 3 total, 2 up, 3 in 2024-04-26T21:21:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T21:21:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T21:21:42.695 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-04-26T21:21:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-04-26T21:21:42.695 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0347141 s, 302 MB/s 2024-04-26T21:21:42.695 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: osd.2 [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] boot 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:21:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:21:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: osd.2 [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] boot 2024-04-26T21:21:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T21:21:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:21:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: osd.2 [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] boot 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: osdmap e18: 3 total, 3 up, 3 in 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:21:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:43.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_3 2024-04-26T21:21:43.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:43 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi186 2024-04-26T21:21:43.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:43 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-26T21:21:43.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:43 smithi001 ceph-mon[28054]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T21:21:43.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:43 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:21:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:43 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi186 2024-04-26T21:21:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:43 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-26T21:21:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:43 smithi038 ceph-mon[32724]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T21:21:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:43 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:21:43.879 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:21:43.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:43 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi186 2024-04-26T21:21:43.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:43 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-26T21:21:43.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:43 smithi186 ceph-mon[32991]: osdmap e19: 3 total, 3 up, 3 in 2024-04-26T21:21:43.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:43 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:21:44.859 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:44 smithi001 ceph-mon[28054]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:44.860 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:44 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T21:21:44.860 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:44 smithi001 ceph-mon[28054]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T21:21:44.860 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:44 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:21:44.860 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:21:44 smithi001 sudo[44712]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:21:44.860 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:21:44 smithi001 sudo[44712]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:44.860 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:21:44 smithi001 sudo[44712]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:44.860 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:21:44 smithi001 sudo[44712]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:44.918 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:44 smithi038 ceph-mon[32724]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:44.918 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:44 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T21:21:44.918 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:44 smithi038 ceph-mon[32724]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T21:21:44.918 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:44 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:21:44.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:44 smithi186 ceph-mon[32991]: pgmap v93: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:44.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:44 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T21:21:44.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:44 smithi186 ceph-mon[32991]: osdmap e20: 3 total, 3 up, 3 in 2024-04-26T21:21:44.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:44 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:21:45.120 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:21:44 smithi001 sudo[44717]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:21:45.120 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:21:44 smithi001 sudo[44717]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.120 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:21:44 smithi001 sudo[44717]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.120 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:21:44 smithi001 sudo[44717]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.248 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:21:44 smithi038 sudo[41161]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:21:45.248 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:21:44 smithi038 sudo[41161]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.248 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:21:44 smithi038 sudo[41161]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.248 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:21:44 smithi038 sudo[41161]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.248 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:21:44 smithi038 sudo[41164]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:21:45.248 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:21:44 smithi038 sudo[41164]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.248 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:21:44 smithi038 sudo[41164]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.248 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:21:44 smithi038 sudo[41164]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 sudo[44720]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:21:45.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 sudo[44720]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 sudo[44720]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 sudo[44720]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 sudo[44723]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:21:45.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 sudo[44723]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 sudo[44723]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.443 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 sudo[44723]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.454 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42071]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:21:45.455 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42071]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.455 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42071]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.455 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42071]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.455 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42074]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:21:45.455 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42074]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.455 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42074]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.455 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42074]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42077]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:21:45.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42077]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42077]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42077]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42080]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:21:45.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42080]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42080]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 sudo[42080]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.528 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 sudo[41167]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-26T21:21:45.528 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 sudo[41167]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.528 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 sudo[41167]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.528 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 sudo[41167]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.528 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 sudo[41171]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-26T21:21:45.528 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 sudo[41171]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-26T21:21:45.528 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 sudo[41171]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-26T21:21:45.528 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 sudo[41171]: pam_unix(sudo:session): session closed for user root 2024-04-26T21:21:45.783 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:21:45.784 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:21:45.785 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:21:45.785 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:21:45.785 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:21:45.785 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:21:45.785 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:21:45.785 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:21:45.785 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:45 smithi001 ceph-mon[28054]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:21:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:21:45.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:21:45.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:21:45.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:21:45.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:21:45.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:21:45.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:21:45.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:45 smithi038 ceph-mon[32724]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:21:45.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:21:45.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: osdmap e21: 3 total, 3 up, 3 in 2024-04-26T21:21:45.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:45.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:45.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:21:45.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:21:45.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:45 smithi186 ceph-mon[32991]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T21:21:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.959+0000 7f07e8c00700 1 -- 172.21.15.1:0/3509807039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07e4107140 msgr2=0x7f07e410d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:21:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.959+0000 7f07e8c00700 1 --2- 172.21.15.1:0/3509807039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07e4107140 0x7f07e410d9f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00b0a0 tx=0x7f07dc00b3b0 comp rx=0 tx=0).stop 2024-04-26T21:21:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.959+0000 7f07e8c00700 1 -- 172.21.15.1:0/3509807039 shutdown_connections 2024-04-26T21:21:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.959+0000 7f07e8c00700 1 --2- 172.21.15.1:0/3509807039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07e4107140 0x7f07e410d9f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.959+0000 7f07e8c00700 1 --2- 172.21.15.1:0/3509807039 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07e41055a0 0x7f07e41059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.959+0000 7f07e8c00700 1 --2- 172.21.15.1:0/3509807039 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e41067a0 0x7f07e4106c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.960+0000 7f07e8c00700 1 -- 172.21.15.1:0/3509807039 >> 172.21.15.1:0/3509807039 conn(0x7f07e4100b40 msgr2=0x7f07e4102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:21:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.960+0000 7f07e8c00700 1 -- 172.21.15.1:0/3509807039 shutdown_connections 2024-04-26T21:21:45.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.960+0000 7f07e8c00700 1 -- 172.21.15.1:0/3509807039 wait complete. 2024-04-26T21:21:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.961+0000 7f07e8c00700 1 Processor -- start 2024-04-26T21:21:45.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.961+0000 7f07e8c00700 1 -- start start 2024-04-26T21:21:45.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e8c00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07e41055a0 0x7f07e419b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:21:45.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e8c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07e41067a0 0x7f07e419bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:21:45.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e8c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4107140 0x7f07e41a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:21:45.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e8c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4111070 con 0x7f07e4107140 2024-04-26T21:21:45.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e8c00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4110ef0 con 0x7f07e41055a0 2024-04-26T21:21:45.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e8c00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41111f0 con 0x7f07e41067a0 2024-04-26T21:21:45.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e3400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07e41055a0 0x7f07e419b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:21:45.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e3400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07e41055a0 0x7f07e419b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:49516/0 (socket says 172.21.15.1:49516) 2024-04-26T21:21:45.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e3400700 1 -- 172.21.15.1:0/2650707808 learned_addr learned my addr 172.21.15.1:0/2650707808 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:21:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e3e00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4107140 0x7f07e41a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:21:45.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.962+0000 7f07e2a00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07e41067a0 0x7f07e419bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07e3400700 1 -- 172.21.15.1:0/2650707808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07e41067a0 msgr2=0x7f07e419bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07e3400700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07e41067a0 0x7f07e419bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07e3400700 1 -- 172.21.15.1:0/2650707808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4107140 msgr2=0x7f07e41a2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07e3400700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4107140 0x7f07e41a2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07e3400700 1 -- 172.21.15.1:0/2650707808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e41a32e0 con 0x7f07e41055a0 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07e2a00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07e41067a0 0x7f07e419bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07e3400700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07e41055a0 0x7f07e419b680 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f07d400ea50 tx=0x7f07d400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07dbe00700 1 -- 172.21.15.1:0/2650707808 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d400cbd0 con 0x7f07e41055a0 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07dbe00700 1 -- 172.21.15.1:0/2650707808 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07d400cd70 con 0x7f07e41055a0 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07dbe00700 1 -- 172.21.15.1:0/2650707808 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d40186f0 con 0x7f07e41055a0 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.963+0000 7f07e8c00700 1 -- 172.21.15.1:0/2650707808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e41a35d0 con 0x7f07e41055a0 2024-04-26T21:21:45.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.964+0000 7f07e8c00700 1 -- 172.21.15.1:0/2650707808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e41a3ac0 con 0x7f07e41055a0 2024-04-26T21:21:45.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.965+0000 7f07e8c00700 1 -- 172.21.15.1:0/2650707808 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07e406a110 con 0x7f07e41055a0 2024-04-26T21:21:45.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.967+0000 7f07dbe00700 1 -- 172.21.15.1:0/2650707808 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 15) ==== 138589+0+0 (secure 0 0 0) 0x7f07d401f020 con 0x7f07e41055a0 2024-04-26T21:21:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.968+0000 7f07dbe00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07c80a3bb0 0x7f07c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:21:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.968+0000 7f07dbe00700 1 -- 172.21.15.1:0/2650707808 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 3237+0+0 (secure 0 0 0) 0x7f07d40cefd0 con 0x7f07e41055a0 2024-04-26T21:21:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.968+0000 7f07e2a00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07c80a3bb0 0x7f07c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:21:45.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.969+0000 7f07e2a00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07c80a3bb0 0x7f07c80a6070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009cb0 tx=0x7f07d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:21:45.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:45.972+0000 7f07dbe00700 1 -- 172.21.15.1:0/2650707808 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f07d409cdc0 con 0x7f07e41055a0 2024-04-26T21:21:46.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:46.102+0000 7f07dbe00700 1 -- 172.21.15.1:0/2650707808 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f07d4098d30 con 0x7f07e41055a0 2024-04-26T21:21:46.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:21:46.225+0000 7f07e8c00700 1 -- 172.21.15.1:0/2650707808 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f07e4063860 con 0x7f07c80a3bb0 2024-04-26T21:21:46.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:46 smithi038 ceph-mon[32724]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:46.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:46 smithi038 ceph-mon[32724]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T21:21:46.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:46 smithi038 ceph-mon[32724]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T21:21:46.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:46 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:21:46.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:46 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:21:46.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:46 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:46.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:46 smithi186 ceph-mon[32991]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:46.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:46 smithi186 ceph-mon[32991]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T21:21:46.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:46 smithi186 ceph-mon[32991]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T21:21:46.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:46 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:21:46.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:46 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:21:46.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:46 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:47.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:46 smithi001 ceph-mon[28054]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:47.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:46 smithi001 ceph-mon[28054]: osdmap e22: 3 total, 3 up, 3 in 2024-04-26T21:21:47.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:46 smithi001 ceph-mon[28054]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-26T21:21:47.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:46 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:21:47.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:46 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:21:47.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:46 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:47.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:47 smithi001 ceph-mon[28054]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:21:47.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:47 smithi186 ceph-mon[32991]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:21:48.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:47 smithi038 ceph-mon[32724]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:21:48.876 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:48 smithi001 ceph-mon[28054]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:48.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:48 smithi186 ceph-mon[32991]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:49.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:48 smithi038 ceph-mon[32724]: pgmap v98: 1 pgs: 1 unknown; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:50 smithi186 ceph-mon[32991]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:50 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1294514225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf"}]: dispatch 2024-04-26T21:21:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:50 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1294514225' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf"}]': finished 2024-04-26T21:21:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:50 smithi186 ceph-mon[32991]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T21:21:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:21:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:50 smithi001 ceph-mon[28054]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:50 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1294514225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf"}]: dispatch 2024-04-26T21:21:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:50 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1294514225' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf"}]': finished 2024-04-26T21:21:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:50 smithi001 ceph-mon[28054]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T21:21:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:21:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:50 smithi038 ceph-mon[32724]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:50 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1294514225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf"}]: dispatch 2024-04-26T21:21:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:50 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1294514225' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf"}]': finished 2024-04-26T21:21:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:50 smithi038 ceph-mon[32724]: osdmap e23: 4 total, 3 up, 4 in 2024-04-26T21:21:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:21:51.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:51 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2682433563' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:21:52.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:51 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2682433563' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:21:52.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:51 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2682433563' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:21:52.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:52 smithi186 ceph-mon[32991]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:53.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:52 smithi001 ceph-mon[28054]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:53.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:52 smithi038 ceph-mon[32724]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:54.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:54 smithi001 ceph-mon[28054]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:54.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:54 smithi186 ceph-mon[32991]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:55.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:54 smithi038 ceph-mon[32724]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:56.781 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:56 smithi001 ceph-mon[28054]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:56.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:56 smithi038 ceph-mon[32724]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:56.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:56 smithi186 ceph-mon[32991]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:58.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:58 smithi186 ceph-mon[32991]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:59.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:58 smithi001 ceph-mon[28054]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:59.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:58 smithi038 ceph-mon[32724]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:21:59.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:59 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T21:21:59.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:21:59 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:21:59.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:59 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T21:21:59.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:21:59 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:00.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:59 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T21:22:00.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:21:59 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:00.927 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:00 smithi001 ceph-mon[28054]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:00.927 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:00 smithi001 ceph-mon[28054]: Deploying daemon osd.3 on smithi001 2024-04-26T21:22:00.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:00 smithi186 ceph-mon[32991]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:00.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:00 smithi186 ceph-mon[32991]: Deploying daemon osd.3 on smithi001 2024-04-26T21:22:01.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:00 smithi038 ceph-mon[32724]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:01.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:00 smithi038 ceph-mon[32724]: Deploying daemon osd.3 on smithi001 2024-04-26T21:22:02.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:02 smithi001 ceph-mon[28054]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:02.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:02 smithi186 ceph-mon[32991]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:03.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:02 smithi038 ceph-mon[32724]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:04.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:04 smithi186 ceph-mon[32991]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:05.062 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:04 smithi001 ceph-mon[28054]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:05.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:04 smithi038 ceph-mon[32724]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:06.926 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:06 smithi001 ceph-mon[28054]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:06.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:06 smithi186 ceph-mon[32991]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:07.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:06 smithi038 ceph-mon[32724]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:08 smithi001 ceph-mon[28054]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:08.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:08 smithi186 ceph-mon[32991]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:08.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:08.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:08.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:08 smithi038 ceph-mon[32724]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:09.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:10.953 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:10 smithi001 ceph-mon[28054]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:10.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:10 smithi186 ceph-mon[32991]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:11.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:10 smithi038 ceph-mon[32724]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:12.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.825+0000 7f07dbe00700 1 -- 172.21.15.1:0/2650707808 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f07e4063860 con 0x7f07c80a3bb0 2024-04-26T21:22:12.827 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 3 on host 'smithi001' 2024-04-26T21:22:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.828+0000 7f07e8c00700 1 -- 172.21.15.1:0/2650707808 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07c80a3bb0 msgr2=0x7f07c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:22:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.828+0000 7f07e8c00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07c80a3bb0 0x7f07c80a6070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009cb0 tx=0x7f07d0009260 comp rx=0 tx=0).stop 2024-04-26T21:22:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.829+0000 7f07e8c00700 1 -- 172.21.15.1:0/2650707808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07e41055a0 msgr2=0x7f07e419b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:22:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.829+0000 7f07e8c00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07e41055a0 0x7f07e419b680 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f07d400ea50 tx=0x7f07d400ed60 comp rx=0 tx=0).stop 2024-04-26T21:22:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.829+0000 7f07e8c00700 1 -- 172.21.15.1:0/2650707808 shutdown_connections 2024-04-26T21:22:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.829+0000 7f07e8c00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f07c80a3bb0 0x7f07c80a6070 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.829+0000 7f07e8c00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f07e4107140 0x7f07e41a2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.829+0000 7f07e8c00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f07e41055a0 0x7f07e419b680 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.829+0000 7f07e8c00700 1 --2- 172.21.15.1:0/2650707808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07e41067a0 0x7f07e419bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:12.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.829+0000 7f07e8c00700 1 -- 172.21.15.1:0/2650707808 >> 172.21.15.1:0/2650707808 conn(0x7f07e4100b40 msgr2=0x7f07e4109aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:22:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.829+0000 7f07e8c00700 1 -- 172.21.15.1:0/2650707808 shutdown_connections 2024-04-26T21:22:12.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:22:12.829+0000 7f07e8c00700 1 -- 172.21.15.1:0/2650707808 wait complete. 2024-04-26T21:22:13.242 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:12 smithi001 ceph-mon[28054]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:13.242 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.242 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.243 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:12 smithi001 ceph-mon[28054]: from='osd.3 [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T21:22:13.243 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:12 smithi001 ceph-mon[28054]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T21:22:13.243 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.243 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:12 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:12 smithi038 ceph-mon[32724]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:13.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:12 smithi038 ceph-mon[32724]: from='osd.3 [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T21:22:13.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:12 smithi038 ceph-mon[32724]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T21:22:13.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:12 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:12 smithi186 ceph-mon[32991]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:13.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:12 smithi186 ceph-mon[32991]: from='osd.3 [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T21:22:13.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:12 smithi186 ceph-mon[32991]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T21:22:13.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:12 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:13.617 DEBUG:teuthology.orchestra.run.smithi001:osd.3> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T21:22:13.623 INFO:tasks.cephadm:Deploying osd.4 on smithi038 with /dev/vg_nvme/lv_3... 2024-04-26T21:22:13.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T21:22:13.877 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:22:14.310 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:14 smithi038 ceph-mon[32724]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T21:22:14.310 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:14 smithi038 ceph-mon[32724]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T21:22:14.310 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:14 smithi038 ceph-mon[32724]: from='osd.3 [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:22:14.310 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:14 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:14.310 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:14 smithi038 ceph-mon[32724]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:22:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:14 smithi001 ceph-mon[28054]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T21:22:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:14 smithi001 ceph-mon[28054]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T21:22:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:14 smithi001 ceph-mon[28054]: from='osd.3 [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:22:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:14 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:14 smithi001 ceph-mon[28054]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:22:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:14 smithi186 ceph-mon[32991]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T21:22:14.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:14 smithi186 ceph-mon[32991]: osdmap e24: 4 total, 3 up, 4 in 2024-04-26T21:22:14.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:14 smithi186 ceph-mon[32991]: from='osd.3 [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:22:14.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:14 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:14.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:14 smithi186 ceph-mon[32991]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:22:15.021 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:22:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:22:14.736+0000 7f2528800700 -1 osd.3 0 waiting for initial osdmap 2024-04-26T21:22:15.021 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:22:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:22:14.755+0000 7f251e600700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:22:15.294 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:15 smithi038 ceph-mon[32724]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:15.294 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:15 smithi038 ceph-mon[32724]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:22:15.294 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:15 smithi038 ceph-mon[32724]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T21:22:15.294 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:15 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:15.294 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:15 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:15.294 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:15 smithi038 ceph-mon[32724]: from='osd.3 ' entity='osd.3' 2024-04-26T21:22:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:15 smithi001 ceph-mon[28054]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:15 smithi001 ceph-mon[28054]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:22:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:15 smithi001 ceph-mon[28054]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T21:22:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:15 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:15 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:15 smithi001 ceph-mon[28054]: from='osd.3 ' entity='osd.3' 2024-04-26T21:22:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:15 smithi186 ceph-mon[32991]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:15.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:15 smithi186 ceph-mon[32991]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:22:15.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:15 smithi186 ceph-mon[32991]: osdmap e25: 4 total, 3 up, 4 in 2024-04-26T21:22:15.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:15 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:15.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:15 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:15.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:15 smithi186 ceph-mon[32991]: from='osd.3 ' entity='osd.3' 2024-04-26T21:22:15.929 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T21:22:15.929 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T21:22:15.929 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-26T21:22:15.929 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-26T21:22:15.929 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0157711 s, 665 MB/s 2024-04-26T21:22:15.930 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-26T21:22:16.194 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:16 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:22:16.194 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:16 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:22:16.194 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:16.194 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:16 smithi001 ceph-mon[28054]: osd.3 [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] boot 2024-04-26T21:22:16.194 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:16 smithi001 ceph-mon[28054]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T21:22:16.194 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:16.260 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:16 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:22:16.260 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:16 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:22:16.260 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:16.260 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:16 smithi038 ceph-mon[32724]: osd.3 [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] boot 2024-04-26T21:22:16.260 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:16 smithi038 ceph-mon[32724]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T21:22:16.260 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:16 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:22:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:16 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:22:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:16 smithi186 ceph-mon[32991]: osd.3 [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] boot 2024-04-26T21:22:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:16 smithi186 ceph-mon[32991]: osdmap e26: 4 total, 4 up, 4 in 2024-04-26T21:22:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:22:16.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_3 2024-04-26T21:22:16.806 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:22:17.098 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:17 smithi038 ceph-mon[32724]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:17.098 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:17 smithi038 ceph-mon[32724]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T21:22:17.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:17 smithi001 ceph-mon[28054]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:17.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:17 smithi001 ceph-mon[28054]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T21:22:17.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:17 smithi186 ceph-mon[32991]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-04-26T21:22:17.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:17 smithi186 ceph-mon[32991]: osdmap e27: 4 total, 4 up, 4 in 2024-04-26T21:22:18.256 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:18 smithi038 ceph-mon[32724]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T21:22:18.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:18 smithi001 ceph-mon[28054]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T21:22:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:18 smithi186 ceph-mon[32991]: osdmap e28: 4 total, 4 up, 4 in 2024-04-26T21:22:18.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 -- 172.21.15.38:0/3731136758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd28c106840 msgr2=0x7fd28c108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:22:18.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 --2- 172.21.15.38:0/3731136758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd28c106840 0x7fd28c108c30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd274009990 tx=0x7fd274009ca0 comp rx=0 tx=0).stop 2024-04-26T21:22:18.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 -- 172.21.15.38:0/3731136758 shutdown_connections 2024-04-26T21:22:18.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 --2- 172.21.15.38:0/3731136758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd28c103f10 0x7fd28c106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:18.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 --2- 172.21.15.38:0/3731136758 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd28c109170 0x7fd28c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 --2- 172.21.15.38:0/3731136758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd28c106840 0x7fd28c108c30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 -- 172.21.15.38:0/3731136758 >> 172.21.15.38:0/3731136758 conn(0x7fd28c0fd920 msgr2=0x7fd28c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 -- 172.21.15.38:0/3731136758 shutdown_connections 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 -- 172.21.15.38:0/3731136758 wait complete. 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 Processor -- start 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.468+0000 7fd2924a7700 1 -- start start 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd2924a7700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd28c103f10 0x7fd28c103870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd2924a7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd28c106840 0x7fd28c101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd2924a7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd28c109170 0x7fd28c102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd2924a7700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c10efa0 con 0x7fd28c103f10 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd2924a7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c10ee20 con 0x7fd28c109170 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd2924a7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c10f120 con 0x7fd28c106840 2024-04-26T21:22:18.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd290a44700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd28c109170 0x7fd28c102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd290a44700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd28c109170 0x7fd28c102400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43758/0 (socket says 172.21.15.38:43758) 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd290a44700 1 -- 172.21.15.38:0/2502349674 learned_addr learned my addr 172.21.15.38:0/2502349674 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd28b7fe700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd28c106840 0x7fd28c101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd290a44700 1 -- 172.21.15.38:0/2502349674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd28c106840 msgr2=0x7fd28c101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd290a44700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd28c106840 0x7fd28c101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd290a44700 1 -- 172.21.15.38:0/2502349674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd28c103f10 msgr2=0x7fd28c103870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd290a44700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd28c103f10 0x7fd28c103870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd290a44700 1 -- 172.21.15.38:0/2502349674 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c102cb0 con 0x7fd28c109170 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd28b7fe700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd28c106840 0x7fd28c101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd290a44700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd28c109170 0x7fd28c102400 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd28400b9e0 tx=0x7fd28400bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.469+0000 7fd2897fa700 1 -- 172.21.15.38:0/2502349674 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284019070 con 0x7fd28c109170 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.470+0000 7fd2897fa700 1 -- 172.21.15.38:0/2502349674 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd284014440 con 0x7fd28c109170 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.470+0000 7fd2924a7700 1 -- 172.21.15.38:0/2502349674 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28c1a3790 con 0x7fd28c109170 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.470+0000 7fd2897fa700 1 -- 172.21.15.38:0/2502349674 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28400ce60 con 0x7fd28c109170 2024-04-26T21:22:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.470+0000 7fd2924a7700 1 -- 172.21.15.38:0/2502349674 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28c1a3cd0 con 0x7fd28c109170 2024-04-26T21:22:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.471+0000 7fd2924a7700 1 -- 172.21.15.38:0/2502349674 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd28c06a110 con 0x7fd28c109170 2024-04-26T21:22:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.472+0000 7fd2897fa700 1 -- 172.21.15.38:0/2502349674 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd28401c070 con 0x7fd28c109170 2024-04-26T21:22:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.472+0000 7fd2897fa700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd2780a3c40 0x7fd2780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:22:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.472+0000 7fd2897fa700 1 -- 172.21.15.38:0/2502349674 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7fd2840ce590 con 0x7fd28c109170 2024-04-26T21:22:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.473+0000 7fd28bfff700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd2780a3c40 0x7fd2780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:22:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.474+0000 7fd28bfff700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd2780a3c40 0x7fd2780a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00a990 tx=0x7fd27c005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:22:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.477+0000 7fd2897fa700 1 -- 172.21.15.38:0/2502349674 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd28409c190 con 0x7fd28c109170 2024-04-26T21:22:18.696 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:18.695+0000 7fd2924a7700 1 -- 172.21.15.38:0/2502349674 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd28c063860 con 0x7fd2780a3c40 2024-04-26T21:22:19.347 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:19 smithi038 ceph-mon[32724]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:19.347 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:19 smithi038 ceph-mon[32724]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T21:22:19.347 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:19 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:22:19.347 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:19 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:22:19.347 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:19 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:19.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:19 smithi186 ceph-mon[32991]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:19.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:19 smithi186 ceph-mon[32991]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T21:22:19.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:19 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:22:19.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:19 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:22:19.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:19 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:19 smithi001 ceph-mon[28054]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:19 smithi001 ceph-mon[28054]: osdmap e29: 4 total, 4 up, 4 in 2024-04-26T21:22:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:19 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:22:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:19 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:22:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:19 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:20.350 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:20.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:20.351 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:20.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:20.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:20.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:22:20.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: osdmap e30: 4 total, 4 up, 4 in 2024-04-26T21:22:20.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:21.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:21 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi001 2024-04-26T21:22:21.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:21 smithi186 ceph-mon[32991]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:22:21.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:21 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi001 to 8547M 2024-04-26T21:22:21.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:21 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi001 2024-04-26T21:22:21.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:21 smithi001 ceph-mon[28054]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:22:21.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:21 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi001 to 8547M 2024-04-26T21:22:21.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:21 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi001 2024-04-26T21:22:21.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:21 smithi038 ceph-mon[32724]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:22:21.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:21 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi001 to 8547M 2024-04-26T21:22:22.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:22 smithi001 ceph-mon[28054]: from='client.? 172.21.15.38:0/3738524387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f"}]: dispatch 2024-04-26T21:22:22.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:22 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f"}]: dispatch 2024-04-26T21:22:22.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:22 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f"}]': finished 2024-04-26T21:22:22.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:22 smithi001 ceph-mon[28054]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T21:22:22.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:22 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:22.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:22 smithi038 ceph-mon[32724]: from='client.? 172.21.15.38:0/3738524387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f"}]: dispatch 2024-04-26T21:22:22.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:22 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f"}]: dispatch 2024-04-26T21:22:22.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:22 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f"}]': finished 2024-04-26T21:22:22.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:22 smithi038 ceph-mon[32724]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T21:22:22.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:22 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:22 smithi186 ceph-mon[32991]: from='client.? 172.21.15.38:0/3738524387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f"}]: dispatch 2024-04-26T21:22:22.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:22 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f"}]: dispatch 2024-04-26T21:22:22.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:22 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f"}]': finished 2024-04-26T21:22:22.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:22 smithi186 ceph-mon[32991]: osdmap e31: 5 total, 4 up, 5 in 2024-04-26T21:22:22.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:22 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:23.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:23 smithi186 ceph-mon[32991]: pgmap v123: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-26T21:22:23.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:23 smithi186 ceph-mon[32991]: from='client.? 172.21.15.38:0/1795133110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:22:23.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:23 smithi001 ceph-mon[28054]: pgmap v123: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-26T21:22:23.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:23 smithi001 ceph-mon[28054]: from='client.? 172.21.15.38:0/1795133110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:22:23.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:23 smithi038 ceph-mon[32724]: pgmap v123: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-04-26T21:22:23.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:23 smithi038 ceph-mon[32724]: from='client.? 172.21.15.38:0/1795133110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:22:25.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:25 smithi186 ceph-mon[32991]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:22:25.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:25 smithi001 ceph-mon[28054]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:22:25.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:25 smithi038 ceph-mon[32724]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:22:27.305 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:27 smithi038 ceph-mon[32724]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:27.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:27 smithi186 ceph-mon[32991]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:27.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:27 smithi001 ceph-mon[28054]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:28 smithi186 ceph-mon[32991]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:28.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:28 smithi001 ceph-mon[28054]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:28.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:28 smithi038 ceph-mon[32724]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:29.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:29 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T21:22:29.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:29 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:29.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:29 smithi038 ceph-mon[32724]: Deploying daemon osd.4 on smithi038 2024-04-26T21:22:29.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:29 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T21:22:29.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:29 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:29.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:29 smithi186 ceph-mon[32991]: Deploying daemon osd.4 on smithi038 2024-04-26T21:22:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T21:22:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:29 smithi001 ceph-mon[28054]: Deploying daemon osd.4 on smithi038 2024-04-26T21:22:30.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:30 smithi038 ceph-mon[32724]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:30 smithi186 ceph-mon[32991]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:30 smithi001 ceph-mon[28054]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:32.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:32 smithi038 ceph-mon[32724]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:32.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:32 smithi001 ceph-mon[28054]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:32.707 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:32 smithi186 ceph-mon[32991]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:34.414 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:34 smithi038 ceph-mon[32724]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:34.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:34 smithi001 ceph-mon[28054]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:34.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:34 smithi186 ceph-mon[32991]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:35.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:35 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:35.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:35 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:35.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:35 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:36.058 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:36.058 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:36.058 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:36.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:35 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:36.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:35 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:36.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:35 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:36.931 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:36 smithi038 ceph-mon[32724]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:36.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:36 smithi186 ceph-mon[32991]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:37.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:36 smithi001 ceph-mon[28054]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:38.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.387+0000 7fd2897fa700 1 -- 172.21.15.38:0/2502349674 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd28c063860 con 0x7fd2780a3c40 2024-04-26T21:22:38.388 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 4 on host 'smithi038' 2024-04-26T21:22:38.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 -- 172.21.15.38:0/2502349674 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd2780a3c40 msgr2=0x7fd2780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd2780a3c40 0x7fd2780a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd27c00a990 tx=0x7fd27c005c80 comp rx=0 tx=0).stop 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 -- 172.21.15.38:0/2502349674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd28c109170 msgr2=0x7fd28c102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd28c109170 0x7fd28c102400 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd28400b9e0 tx=0x7fd28400bda0 comp rx=0 tx=0).stop 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 -- 172.21.15.38:0/2502349674 shutdown_connections 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd2780a3c40 0x7fd2780a6100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd28c109170 0x7fd28c102400 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd28c103f10 0x7fd28c103870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 --2- 172.21.15.38:0/2502349674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd28c106840 0x7fd28c101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 -- 172.21.15.38:0/2502349674 >> 172.21.15.38:0/2502349674 conn(0x7fd28c0fd920 msgr2=0x7fd28c104850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 -- 172.21.15.38:0/2502349674 shutdown_connections 2024-04-26T21:22:38.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:22:38.391+0000 7fd2924a7700 1 -- 172.21.15.38:0/2502349674 wait complete. 2024-04-26T21:22:38.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:38 smithi186 ceph-mon[32991]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:38.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:38.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:38.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:38.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:38.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:38 smithi186 ceph-mon[32991]: from='osd.4 [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:22:38.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:38 smithi186 ceph-mon[32991]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:22:39.034 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:38 smithi038 ceph-mon[32724]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:39.034 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:39.034 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:39.034 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:39.034 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:39.034 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:38 smithi038 ceph-mon[32724]: from='osd.4 [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:22:39.034 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:38 smithi038 ceph-mon[32724]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:22:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:38 smithi001 ceph-mon[28054]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:38 smithi001 ceph-mon[28054]: from='osd.4 [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:22:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:38 smithi001 ceph-mon[28054]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T21:22:39.073 DEBUG:teuthology.orchestra.run.smithi038:osd.4> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T21:22:39.075 INFO:tasks.cephadm:Deploying osd.5 on smithi186 with /dev/vg_nvme/lv_3... 2024-04-26T21:22:39.075 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T21:22:39.321 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:22:40.713 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:40 smithi186 ceph-mon[32991]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:40.713 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:40 smithi186 ceph-mon[32991]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T21:22:40.713 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:40 smithi186 ceph-mon[32991]: from='osd.4 [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:22:40.713 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:40 smithi186 ceph-mon[32991]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T21:22:40.713 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:40.713 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:40 smithi186 ceph-mon[32991]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:22:40.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:40 smithi001 ceph-mon[28054]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:40.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:40 smithi001 ceph-mon[28054]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T21:22:40.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:40 smithi001 ceph-mon[28054]: from='osd.4 [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:22:40.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:40 smithi001 ceph-mon[28054]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T21:22:40.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:40.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:40 smithi001 ceph-mon[28054]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:22:40.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:40 smithi038 ceph-mon[32724]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:40 smithi038 ceph-mon[32724]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T21:22:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:40 smithi038 ceph-mon[32724]: from='osd.4 [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:22:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:40 smithi038 ceph-mon[32724]: osdmap e32: 5 total, 4 up, 5 in 2024-04-26T21:22:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:40 smithi038 ceph-mon[32724]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:22:40.827 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:22:40 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4[43989]: 2024-04-26T21:22:40.454+0000 7f67e3ef2700 -1 osd.4 0 waiting for initial osdmap 2024-04-26T21:22:40.827 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:22:40 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4[43989]: 2024-04-26T21:22:40.460+0000 7f67dfce5700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:22:41.478 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T21:22:41.478 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T21:22:41.478 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T21:22:41.478 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T21:22:41.478 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116322 s, 901 MB/s 2024-04-26T21:22:41.478 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T21:22:41.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:41 smithi038 ceph-mon[32724]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:22:41.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:41 smithi038 ceph-mon[32724]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T21:22:41.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:41.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:41.779 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:41 smithi186 ceph-mon[32991]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:22:41.779 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:41 smithi186 ceph-mon[32991]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T21:22:41.779 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:41.779 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:41.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:41 smithi001 ceph-mon[28054]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:22:41.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:41 smithi001 ceph-mon[28054]: osdmap e33: 5 total, 4 up, 5 in 2024-04-26T21:22:41.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:41.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:42.205 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_3 2024-04-26T21:22:42.445 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:22:42.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:22:42.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:22:42.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:42.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: osd.4 [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] boot 2024-04-26T21:22:42.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T21:22:42.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:42.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi038 2024-04-26T21:22:42.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.749 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:42.750 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:22:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: osd.4 [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] boot 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi038 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:42.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.820 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:42.820 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:42.820 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:42.820 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-04-26T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: osd.4 [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] boot 2024-04-26T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: osdmap e34: 5 total, 5 up, 5 in 2024-04-26T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi038 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi038 to 8546M 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:22:43.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:43 smithi001 ceph-mon[28054]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T21:22:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:43 smithi038 ceph-mon[32724]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T21:22:43.836 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:43 smithi186 ceph-mon[32991]: osdmap e35: 5 total, 5 up, 5 in 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.040+0000 7f664b4d4700 1 -- 172.21.15.186:0/3623441571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6644104b40 msgr2=0x7f6644104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.040+0000 7f664b4d4700 1 --2- 172.21.15.186:0/3623441571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6644104b40 0x7f6644104f60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6640009990 tx=0x7f6640009ca0 comp rx=0 tx=0).stop 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.040+0000 7f664b4d4700 1 -- 172.21.15.186:0/3623441571 shutdown_connections 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.040+0000 7f664b4d4700 1 --2- 172.21.15.186:0/3623441571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6644104b40 0x7f6644104f60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.040+0000 7f664b4d4700 1 --2- 172.21.15.186:0/3623441571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6644101f80 0x7f664410bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.040+0000 7f664b4d4700 1 --2- 172.21.15.186:0/3623441571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66441015c0 0x7f6644101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.040+0000 7f664b4d4700 1 -- 172.21.15.186:0/3623441571 >> 172.21.15.186:0/3623441571 conn(0x7f66440fd160 msgr2=0x7f66440ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f664b4d4700 1 -- 172.21.15.186:0/3623441571 shutdown_connections 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f664b4d4700 1 -- 172.21.15.186:0/3623441571 wait complete. 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f664b4d4700 1 Processor -- start 2024-04-26T21:22:44.044 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f664b4d4700 1 -- start start 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f664b4d4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66441015c0 0x7f66441996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f664b4d4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6644101f80 0x7f6644199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f664b4d4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6644104b40 0x7f66441a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f664b4d4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f664410f080 con 0x7f6644101f80 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f664b4d4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f664410ef00 con 0x7f66441015c0 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f664b4d4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f664410f200 con 0x7f6644104b40 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f6649a71700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6644104b40 0x7f66441a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f6649a71700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6644104b40 0x7f66441a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:48878/0 (socket says 172.21.15.186:48878) 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f6649a71700 1 -- 172.21.15.186:0/904294873 learned_addr learned my addr 172.21.15.186:0/904294873 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.041+0000 7f6649270700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66441015c0 0x7f66441996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f6649a71700 1 -- 172.21.15.186:0/904294873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66441015c0 msgr2=0x7f66441996a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f6649a71700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66441015c0 0x7f66441996a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:44.045 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f6649a71700 1 -- 172.21.15.186:0/904294873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6644101f80 msgr2=0x7f6644199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:22:44.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f6649a71700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6644101f80 0x7f6644199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:22:44.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f6649a71700 1 -- 172.21.15.186:0/904294873 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66441a1300 con 0x7f6644104b40 2024-04-26T21:22:44.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f6649a71700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6644104b40 0x7f66441a0c60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f663c00b9a0 tx=0x7f663c00bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:22:44.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f663a7fc700 1 -- 172.21.15.186:0/904294873 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c019070 con 0x7f6644104b40 2024-04-26T21:22:44.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f663a7fc700 1 -- 172.21.15.186:0/904294873 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f663c00cde0 con 0x7f6644104b40 2024-04-26T21:22:44.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f664b4d4700 1 -- 172.21.15.186:0/904294873 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66441a15f0 con 0x7f6644104b40 2024-04-26T21:22:44.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f663a7fc700 1 -- 172.21.15.186:0/904294873 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c007370 con 0x7f6644104b40 2024-04-26T21:22:44.046 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.042+0000 7f664b4d4700 1 -- 172.21.15.186:0/904294873 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66441a1ae0 con 0x7f6644104b40 2024-04-26T21:22:44.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.043+0000 7f664b4d4700 1 -- 172.21.15.186:0/904294873 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f664406a110 con 0x7f6644104b40 2024-04-26T21:22:44.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.044+0000 7f663a7fc700 1 -- 172.21.15.186:0/904294873 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f663c01c070 con 0x7f6644104b40 2024-04-26T21:22:44.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.045+0000 7f663a7fc700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f66300a3c40 0x7f66300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:22:44.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.045+0000 7f663a7fc700 1 -- 172.21.15.186:0/904294873 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f663c0ce940 con 0x7f6644104b40 2024-04-26T21:22:44.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.045+0000 7f6649270700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f66300a3c40 0x7f66300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:22:44.047 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.046+0000 7f6649270700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f66300a3c40 0x7f66300a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66400038a0 tx=0x7f6640005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:22:44.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.050+0000 7f663a7fc700 1 -- 172.21.15.186:0/904294873 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f663c09c390 con 0x7f6644104b40 2024-04-26T21:22:44.257 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:22:44.256+0000 7f664b4d4700 1 -- 172.21.15.186:0/904294873 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6644063860 con 0x7f66300a3c40 2024-04-26T21:22:44.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:44 smithi186 ceph-mon[32991]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:44.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:44 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:22:44.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:44 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:22:44.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:44 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:44.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:44 smithi001 ceph-mon[28054]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:44.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:44 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:22:44.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:44 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:22:44.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:44 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:44.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:44 smithi038 ceph-mon[32724]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:44.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:44 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:22:44.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:44 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:22:44.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:44 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:45.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:45 smithi001 ceph-mon[28054]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:22:45.820 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:45 smithi186 ceph-mon[32991]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:22:45.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:45 smithi038 ceph-mon[32724]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:22:46.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:46 smithi186 ceph-mon[32991]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:46.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:46 smithi001 ceph-mon[28054]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:46.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:46 smithi038 ceph-mon[32724]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:47.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:47 smithi186 ceph-mon[32991]: from='client.? 172.21.15.186:0/2775839125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb2ab549-8bab-42e1-8a6a-3a41b28301c8"}]: dispatch 2024-04-26T21:22:47.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:47 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb2ab549-8bab-42e1-8a6a-3a41b28301c8"}]: dispatch 2024-04-26T21:22:47.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:47 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb2ab549-8bab-42e1-8a6a-3a41b28301c8"}]': finished 2024-04-26T21:22:47.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:47 smithi186 ceph-mon[32991]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T21:22:47.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:22:47.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:47 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/2775839125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb2ab549-8bab-42e1-8a6a-3a41b28301c8"}]: dispatch 2024-04-26T21:22:47.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:47 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb2ab549-8bab-42e1-8a6a-3a41b28301c8"}]: dispatch 2024-04-26T21:22:47.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:47 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb2ab549-8bab-42e1-8a6a-3a41b28301c8"}]': finished 2024-04-26T21:22:47.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:47 smithi001 ceph-mon[28054]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T21:22:47.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:22:47.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:47 smithi038 ceph-mon[32724]: from='client.? 172.21.15.186:0/2775839125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb2ab549-8bab-42e1-8a6a-3a41b28301c8"}]: dispatch 2024-04-26T21:22:47.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:47 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb2ab549-8bab-42e1-8a6a-3a41b28301c8"}]: dispatch 2024-04-26T21:22:47.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:47 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb2ab549-8bab-42e1-8a6a-3a41b28301c8"}]': finished 2024-04-26T21:22:47.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:47 smithi038 ceph-mon[32724]: osdmap e36: 6 total, 5 up, 6 in 2024-04-26T21:22:47.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:22:48.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:48 smithi001 ceph-mon[28054]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:48.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:48 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/916366524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:22:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:48 smithi038 ceph-mon[32724]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:48 smithi038 ceph-mon[32724]: from='client.? 172.21.15.186:0/916366524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:22:48.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:48 smithi186 ceph-mon[32991]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:48.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:48 smithi186 ceph-mon[32991]: from='client.? 172.21.15.186:0/916366524' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:22:50.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:50 smithi001 ceph-mon[28054]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:50.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:50 smithi038 ceph-mon[32724]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:50.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:50 smithi186 ceph-mon[32991]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:52.814 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:52 smithi186 ceph-mon[32991]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:52.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:52 smithi001 ceph-mon[28054]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:52.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:52 smithi038 ceph-mon[32724]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:54 smithi186 ceph-mon[32991]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T21:22:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:54.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:54 smithi001 ceph-mon[28054]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:54.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T21:22:54.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:54.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:54 smithi038 ceph-mon[32724]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:54.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T21:22:54.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:22:55.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:55 smithi186 ceph-mon[32991]: Deploying daemon osd.5 on smithi186 2024-04-26T21:22:55.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:55 smithi001 ceph-mon[28054]: Deploying daemon osd.5 on smithi186 2024-04-26T21:22:55.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:55 smithi038 ceph-mon[32724]: Deploying daemon osd.5 on smithi186 2024-04-26T21:22:56.690 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:56 smithi186 ceph-mon[32991]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:56.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:56 smithi001 ceph-mon[28054]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:56.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:56 smithi038 ceph-mon[32724]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:58.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:22:58 smithi186 ceph-mon[32991]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:58.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:22:58 smithi001 ceph-mon[28054]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:22:58.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:22:58 smithi038 ceph-mon[32724]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:00.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:00 smithi001 ceph-mon[28054]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:00.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:00 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:00.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:00 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:00.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:00 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:00.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:00 smithi038 ceph-mon[32724]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:00.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:00 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:00.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:00 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:00.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:00 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:00.834 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:00 smithi186 ceph-mon[32991]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:00.834 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:00 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:00.834 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:00 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:00.834 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:00 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:02.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:02 smithi001 ceph-mon[28054]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:02.826 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:02 smithi186 ceph-mon[32991]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:02.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:02 smithi038 ceph-mon[32724]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:04.100 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.099+0000 7f663a7fc700 1 -- 172.21.15.186:0/904294873 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6644063860 con 0x7f66300a3c40 2024-04-26T21:23:04.101 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 5 on host 'smithi186' 2024-04-26T21:23:04.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 -- 172.21.15.186:0/904294873 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f66300a3c40 msgr2=0x7f66300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:23:04.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f66300a3c40 0x7f66300a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66400038a0 tx=0x7f6640005fb0 comp rx=0 tx=0).stop 2024-04-26T21:23:04.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 -- 172.21.15.186:0/904294873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6644104b40 msgr2=0x7f66441a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:23:04.103 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6644104b40 0x7f66441a0c60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f663c00b9a0 tx=0x7f663c00bcb0 comp rx=0 tx=0).stop 2024-04-26T21:23:04.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 -- 172.21.15.186:0/904294873 shutdown_connections 2024-04-26T21:23:04.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f66300a3c40 0x7f66300a6100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:04.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6644104b40 0x7f66441a0c60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:04.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66441015c0 0x7f66441996a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:04.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 --2- 172.21.15.186:0/904294873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6644101f80 0x7f6644199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:04.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 -- 172.21.15.186:0/904294873 >> 172.21.15.186:0/904294873 conn(0x7f66440fd160 msgr2=0x7f6644107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:23:04.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 -- 172.21.15.186:0/904294873 shutdown_connections 2024-04-26T21:23:04.104 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:23:04.102+0000 7f664b4d4700 1 -- 172.21.15.186:0/904294873 wait complete. 2024-04-26T21:23:04.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:04 smithi186 ceph-mon[32991]: from='osd.5 [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:23:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:04 smithi186 ceph-mon[32991]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:23:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:04.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:04 smithi001 ceph-mon[28054]: from='osd.5 [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:23:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:04 smithi001 ceph-mon[28054]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:23:04.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:04.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:04 smithi038 ceph-mon[32724]: from='osd.5 [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:23:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:04 smithi038 ceph-mon[32724]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T21:23:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:04.818 DEBUG:teuthology.orchestra.run.smithi186:osd.5> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T21:23:04.820 INFO:tasks.cephadm:Deploying osd.6 on smithi001 with /dev/vg_nvme/lv_2... 2024-04-26T21:23:04.821 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T21:23:05.294 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:05 smithi186 ceph-mon[32991]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:05.294 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:05 smithi186 ceph-mon[32991]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T21:23:05.294 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:05 smithi186 ceph-mon[32991]: from='osd.5 [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:23:05.294 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:05 smithi186 ceph-mon[32991]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T21:23:05.295 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:05.295 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:05 smithi186 ceph-mon[32991]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:23:05.295 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:23:05.295 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:05 smithi001 ceph-mon[28054]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:05.296 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:05 smithi001 ceph-mon[28054]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T21:23:05.296 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:05 smithi001 ceph-mon[28054]: from='osd.5 [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:23:05.296 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:05 smithi001 ceph-mon[28054]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T21:23:05.296 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:05.296 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:05 smithi001 ceph-mon[28054]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:23:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:05 smithi038 ceph-mon[32724]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:05 smithi038 ceph-mon[32724]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T21:23:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:05 smithi038 ceph-mon[32724]: from='osd.5 [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:23:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:05 smithi038 ceph-mon[32724]: osdmap e37: 6 total, 5 up, 6 in 2024-04-26T21:23:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:05 smithi038 ceph-mon[32724]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:23:06.078 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:23:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:23:05.814+0000 7f6e81766700 -1 osd.5 0 waiting for initial osdmap 2024-04-26T21:23:06.078 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:23:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:23:05.822+0000 7f6e78d50700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:23:06.418 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:06 smithi186 ceph-mon[32991]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:23:06.418 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:06 smithi186 ceph-mon[32991]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T21:23:06.418 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:06 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:06.418 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:06 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:06.418 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:06 smithi186 ceph-mon[32991]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:06.419 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:06 smithi186 ceph-mon[32991]: from='osd.5 ' entity='osd.5' 2024-04-26T21:23:06.457 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:06 smithi001 ceph-mon[28054]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:23:06.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:06 smithi001 ceph-mon[28054]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T21:23:06.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:06 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:06.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:06 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:06.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:06 smithi001 ceph-mon[28054]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:06.458 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:06 smithi001 ceph-mon[28054]: from='osd.5 ' entity='osd.5' 2024-04-26T21:23:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:06 smithi038 ceph-mon[32724]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:23:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:06 smithi038 ceph-mon[32724]: osdmap e38: 6 total, 5 up, 6 in 2024-04-26T21:23:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:06 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:06 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:06 smithi038 ceph-mon[32724]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-26T21:23:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:06 smithi038 ceph-mon[32724]: from='osd.5 ' entity='osd.5' 2024-04-26T21:23:07.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:07 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:23:07.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:07 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:23:07.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:07.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:07 smithi001 ceph-mon[28054]: osd.5 [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] boot 2024-04-26T21:23:07.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:07 smithi001 ceph-mon[28054]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T21:23:07.289 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:07.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:07 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:23:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:07 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:23:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:07 smithi186 ceph-mon[32991]: osd.5 [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] boot 2024-04-26T21:23:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:07 smithi186 ceph-mon[32991]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T21:23:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:07.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:07 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:23:07.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:07 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:23:07.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:07.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:07 smithi038 ceph-mon[32724]: osd.5 [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] boot 2024-04-26T21:23:07.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:07 smithi038 ceph-mon[32724]: osdmap e39: 6 total, 6 up, 6 in 2024-04-26T21:23:07.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:23:08.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T21:23:08.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:08.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi186 2024-04-26T21:23:08.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:08.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:08.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-26T21:23:08.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:08.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:08.320 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi186 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:08.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: osdmap e40: 6 total, 6 up, 6 in 2024-04-26T21:23:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi186 2024-04-26T21:23:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:08.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:08.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-26T21:23:08.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:08.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:08.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:08.597 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T21:23:08.597 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T21:23:08.597 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-04-26T21:23:08.597 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-04-26T21:23:08.597 INFO:teuthology.orchestra.run.smithi001.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.033112 s, 317 MB/s 2024-04-26T21:23:08.597 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 2024-04-26T21:23:08.597 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-04-26T21:23:09.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:09 smithi001 ceph-mon[28054]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T21:23:09.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:09 smithi186 ceph-mon[32991]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T21:23:09.483 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_2 2024-04-26T21:23:09.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:09 smithi038 ceph-mon[32724]: osdmap e41: 6 total, 6 up, 6 in 2024-04-26T21:23:09.946 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:23:10.493 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:10 smithi001 ceph-mon[28054]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T21:23:10.493 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:10 smithi001 ceph-mon[28054]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:10.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:10 smithi038 ceph-mon[32724]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T21:23:10.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:10 smithi038 ceph-mon[32724]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:10.655 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:10 smithi186 ceph-mon[32991]: osdmap e42: 6 total, 6 up, 6 in 2024-04-26T21:23:10.655 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:10 smithi186 ceph-mon[32991]: pgmap v159: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:11.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:11 smithi001 ceph-mon[28054]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T21:23:11.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:11 smithi038 ceph-mon[32724]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T21:23:11.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:11 smithi186 ceph-mon[32991]: osdmap e43: 6 total, 6 up, 6 in 2024-04-26T21:23:12.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.348+0000 7eff51800700 1 -- 172.21.15.1:0/1469664726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff4c1056c0 msgr2=0x7eff4c105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:23:12.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.348+0000 7eff51800700 1 --2- 172.21.15.1:0/1469664726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff4c1056c0 0x7eff4c105ae0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0099c0 tx=0x7eff3c009cd0 comp rx=0 tx=0).stop 2024-04-26T21:23:12.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.348+0000 7eff51800700 1 -- 172.21.15.1:0/1469664726 shutdown_connections 2024-04-26T21:23:12.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.348+0000 7eff51800700 1 --2- 172.21.15.1:0/1469664726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c107260 0x7eff4c10db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:12.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.348+0000 7eff51800700 1 --2- 172.21.15.1:0/1469664726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff4c1056c0 0x7eff4c105ae0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:12.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.348+0000 7eff51800700 1 --2- 172.21.15.1:0/1469664726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff4c1068c0 0x7eff4c106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:12.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.348+0000 7eff51800700 1 -- 172.21.15.1:0/1469664726 >> 172.21.15.1:0/1469664726 conn(0x7eff4c100c80 msgr2=0x7eff4c1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:23:12.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.348+0000 7eff51800700 1 -- 172.21.15.1:0/1469664726 shutdown_connections 2024-04-26T21:23:12.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.348+0000 7eff51800700 1 -- 172.21.15.1:0/1469664726 wait complete. 2024-04-26T21:23:12.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.350+0000 7eff51800700 1 Processor -- start 2024-04-26T21:23:12.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.350+0000 7eff51800700 1 -- start start 2024-04-26T21:23:12.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.350+0000 7eff51800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c1056c0 0x7eff4c19b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:23:12.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.350+0000 7eff51800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff4c1068c0 0x7eff4c19bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:23:12.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.350+0000 7eff51800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff4c107260 0x7eff4c1a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:23:12.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.350+0000 7eff51800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c111140 con 0x7eff4c107260 2024-04-26T21:23:12.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.350+0000 7eff51800700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c110fc0 con 0x7eff4c1068c0 2024-04-26T21:23:12.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.350+0000 7eff51800700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7eff4c1112c0 con 0x7eff4c1056c0 2024-04-26T21:23:12.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.351+0000 7eff4a000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff4c1068c0 0x7eff4c19bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:23:12.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.351+0000 7eff4b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff4c107260 0x7eff4c1a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:23:12.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.351+0000 7eff4a000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff4c1068c0 0x7eff4c19bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36456/0 (socket says 172.21.15.1:36456) 2024-04-26T21:23:12.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.351+0000 7eff4a000700 1 -- 172.21.15.1:0/1113830993 learned_addr learned my addr 172.21.15.1:0/1113830993 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:23:12.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.351+0000 7eff4aa00700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c1056c0 0x7eff4c19b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:23:12.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.351+0000 7eff4a000700 1 -- 172.21.15.1:0/1113830993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c1056c0 msgr2=0x7eff4c19b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:23:12.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.351+0000 7eff4a000700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c1056c0 0x7eff4c19b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:12.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.351+0000 7eff4a000700 1 -- 172.21.15.1:0/1113830993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff4c107260 msgr2=0x7eff4c1a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:23:12.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.351+0000 7eff4a000700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff4c107260 0x7eff4c1a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:12.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.351+0000 7eff4a000700 1 -- 172.21.15.1:0/1113830993 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4c1a3410 con 0x7eff4c1068c0 2024-04-26T21:23:12.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.352+0000 7eff4a000700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff4c1068c0 0x7eff4c19bcf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7eff3800e9a0 tx=0x7eff3800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:23:12.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.352+0000 7eff43400700 1 -- 172.21.15.1:0/1113830993 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3800ca10 con 0x7eff4c1068c0 2024-04-26T21:23:12.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.352+0000 7eff43400700 1 -- 172.21.15.1:0/1113830993 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff3800cbb0 con 0x7eff4c1068c0 2024-04-26T21:23:12.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.352+0000 7eff43400700 1 -- 172.21.15.1:0/1113830993 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38018680 con 0x7eff4c1068c0 2024-04-26T21:23:12.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.352+0000 7eff51800700 1 -- 172.21.15.1:0/1113830993 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff4c1a3700 con 0x7eff4c1068c0 2024-04-26T21:23:12.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.352+0000 7eff51800700 1 -- 172.21.15.1:0/1113830993 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff4c1a3bf0 con 0x7eff4c1068c0 2024-04-26T21:23:12.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.353+0000 7eff51800700 1 -- 172.21.15.1:0/1113830993 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff4c06a110 con 0x7eff4c1068c0 2024-04-26T21:23:12.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.356+0000 7eff43400700 1 -- 172.21.15.1:0/1113830993 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7eff38010430 con 0x7eff4c1068c0 2024-04-26T21:23:12.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.357+0000 7eff43400700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7eff300a3c40 0x7eff300a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:23:12.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.357+0000 7eff43400700 1 -- 172.21.15.1:0/1113830993 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4533+0+0 (secure 0 0 0) 0x7eff380cf740 con 0x7eff4c1068c0 2024-04-26T21:23:12.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.357+0000 7eff4aa00700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7eff300a3c40 0x7eff300a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:23:12.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.358+0000 7eff4aa00700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7eff300a3c40 0x7eff300a6100 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0036f0 tx=0x7eff3c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:23:12.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.362+0000 7eff43400700 1 -- 172.21.15.1:0/1113830993 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7eff3809cf40 con 0x7eff4c1068c0 2024-04-26T21:23:12.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:12 smithi001 ceph-mon[28054]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:12.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:12 smithi038 ceph-mon[32724]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:12.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:12.609+0000 7eff51800700 1 -- 172.21.15.1:0/1113830993 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7eff4c063860 con 0x7eff300a3c40 2024-04-26T21:23:12.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:12 smithi186 ceph-mon[32991]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:13.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:13 smithi001 ceph-mon[28054]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:23:13.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:23:13.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:23:13.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:13.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:13 smithi038 ceph-mon[32724]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:23:13.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:23:13.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:23:13.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:13 smithi186 ceph-mon[32991]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:23:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:23:13.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:23:13.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:14.536 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:14 smithi001 ceph-mon[28054]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:23:14.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:14 smithi038 ceph-mon[32724]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:23:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:14 smithi186 ceph-mon[32991]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:23:16.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:16 smithi001 ceph-mon[28054]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T21:23:16.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:16 smithi038 ceph-mon[32724]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T21:23:16.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:16 smithi186 ceph-mon[32991]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-26T21:23:17.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:17 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3164060965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55da783d-0f5f-479b-b13f-80ba6bd8eb72"}]: dispatch 2024-04-26T21:23:17.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:17 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3164060965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55da783d-0f5f-479b-b13f-80ba6bd8eb72"}]': finished 2024-04-26T21:23:17.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:17 smithi001 ceph-mon[28054]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T21:23:17.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:17.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:17 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3164060965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55da783d-0f5f-479b-b13f-80ba6bd8eb72"}]: dispatch 2024-04-26T21:23:17.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:17 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3164060965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55da783d-0f5f-479b-b13f-80ba6bd8eb72"}]': finished 2024-04-26T21:23:17.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:17 smithi038 ceph-mon[32724]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T21:23:17.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:17.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:17 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3164060965' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55da783d-0f5f-479b-b13f-80ba6bd8eb72"}]: dispatch 2024-04-26T21:23:17.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:17 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3164060965' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55da783d-0f5f-479b-b13f-80ba6bd8eb72"}]': finished 2024-04-26T21:23:17.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:17 smithi186 ceph-mon[32991]: osdmap e44: 7 total, 6 up, 7 in 2024-04-26T21:23:17.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:18.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:18 smithi001 ceph-mon[28054]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:23:18.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:18 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/383653939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:23:18.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:18 smithi038 ceph-mon[32724]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:23:18.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:18 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/383653939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:23:18.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:18 smithi186 ceph-mon[32991]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:23:18.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:18 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/383653939' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:23:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:20 smithi186 ceph-mon[32991]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T21:23:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:20 smithi001 ceph-mon[28054]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T21:23:20.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:20 smithi038 ceph-mon[32724]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-26T21:23:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:22 smithi001 ceph-mon[28054]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T21:23:22.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:22 smithi038 ceph-mon[32724]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T21:23:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:22 smithi186 ceph-mon[32991]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T21:23:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:24 smithi001 ceph-mon[28054]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:24.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:24 smithi038 ceph-mon[32724]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:24.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:24 smithi186 ceph-mon[32991]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:26.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:26 smithi001 ceph-mon[28054]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:26.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:26 smithi038 ceph-mon[32724]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:26.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:26 smithi186 ceph-mon[32991]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:27.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:27 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T21:23:27.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:27 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:27.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:27 smithi038 ceph-mon[32724]: Deploying daemon osd.6 on smithi001 2024-04-26T21:23:27.703 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:27 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T21:23:27.704 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:27 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:27.704 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:27 smithi001 ceph-mon[28054]: Deploying daemon osd.6 on smithi001 2024-04-26T21:23:27.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:27 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T21:23:27.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:27 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:27.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:27 smithi186 ceph-mon[32991]: Deploying daemon osd.6 on smithi001 2024-04-26T21:23:28.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:28 smithi001 ceph-mon[28054]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:28.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:28 smithi186 ceph-mon[32991]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:28.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:28 smithi038 ceph-mon[32724]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:30.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:30 smithi001 ceph-mon[28054]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:30.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:30 smithi186 ceph-mon[32991]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:30.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:30 smithi038 ceph-mon[32724]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:32.594 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:32 smithi001 ceph-mon[28054]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:32.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:32 smithi186 ceph-mon[32991]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:32.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:32 smithi038 ceph-mon[32724]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:34.538 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:34 smithi001 ceph-mon[28054]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:34 smithi186 ceph-mon[32991]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:34.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:34 smithi038 ceph-mon[32724]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:36.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:36 smithi186 ceph-mon[32991]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:36.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:36.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:36.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:36.793 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:36 smithi001 ceph-mon[28054]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:36.794 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:36.794 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:36.794 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:36.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:36 smithi038 ceph-mon[32724]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:36.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:36.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:36.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:38.801 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:38 smithi001 ceph-mon[28054]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:38.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:38 smithi038 ceph-mon[32724]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:38.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:38 smithi186 ceph-mon[32991]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:40.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:40 smithi001 ceph-mon[28054]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:40.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:40.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:40.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:40 smithi001 ceph-mon[28054]: from='osd.6 [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T21:23:40.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:40 smithi001 ceph-mon[28054]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T21:23:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:40 smithi038 ceph-mon[32724]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:40 smithi038 ceph-mon[32724]: from='osd.6 [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T21:23:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:40 smithi038 ceph-mon[32724]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T21:23:40.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.910+0000 7eff43400700 1 -- 172.21.15.1:0/1113830993 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7eff4c063860 con 0x7eff300a3c40 2024-04-26T21:23:40.914 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 6 on host 'smithi001' 2024-04-26T21:23:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.913+0000 7eff51800700 1 -- 172.21.15.1:0/1113830993 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7eff300a3c40 msgr2=0x7eff300a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:23:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.913+0000 7eff51800700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7eff300a3c40 0x7eff300a6100 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0036f0 tx=0x7eff3c005fb0 comp rx=0 tx=0).stop 2024-04-26T21:23:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.913+0000 7eff51800700 1 -- 172.21.15.1:0/1113830993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff4c1068c0 msgr2=0x7eff4c19bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:23:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.913+0000 7eff51800700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff4c1068c0 0x7eff4c19bcf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7eff3800e9a0 tx=0x7eff3800ecb0 comp rx=0 tx=0).stop 2024-04-26T21:23:40.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.913+0000 7eff51800700 1 -- 172.21.15.1:0/1113830993 shutdown_connections 2024-04-26T21:23:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.913+0000 7eff51800700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7eff300a3c40 0x7eff300a6100 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.913+0000 7eff51800700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff4c1056c0 0x7eff4c19b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.913+0000 7eff51800700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff4c107260 0x7eff4c1a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.913+0000 7eff51800700 1 --2- 172.21.15.1:0/1113830993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff4c1068c0 0x7eff4c19bcf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.913+0000 7eff51800700 1 -- 172.21.15.1:0/1113830993 >> 172.21.15.1:0/1113830993 conn(0x7eff4c100c80 msgr2=0x7eff4c109c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:23:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.914+0000 7eff51800700 1 -- 172.21.15.1:0/1113830993 shutdown_connections 2024-04-26T21:23:40.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:23:40.914+0000 7eff51800700 1 -- 172.21.15.1:0/1113830993 wait complete. 2024-04-26T21:23:40.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:40 smithi186 ceph-mon[32991]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:40.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:40.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:40.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:40 smithi186 ceph-mon[32991]: from='osd.6 [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T21:23:40.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:40 smithi186 ceph-mon[32991]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T21:23:41.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:41 smithi001 ceph-mon[28054]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T21:23:41.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:41 smithi001 ceph-mon[28054]: from='osd.6 [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:23:41.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:41 smithi001 ceph-mon[28054]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T21:23:41.799 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:41.800 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:41 smithi001 ceph-mon[28054]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:23:41.800 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:41.800 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:41 smithi038 ceph-mon[32724]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T21:23:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:41 smithi038 ceph-mon[32724]: from='osd.6 [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:23:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:41 smithi038 ceph-mon[32724]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T21:23:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:41 smithi038 ceph-mon[32724]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:23:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:41.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:41.868 DEBUG:teuthology.orchestra.run.smithi001:osd.6> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T21:23:41.873 INFO:tasks.cephadm:Deploying osd.7 on smithi038 with /dev/vg_nvme/lv_2... 2024-04-26T21:23:41.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T21:23:41.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:41 smithi186 ceph-mon[32991]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T21:23:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:41 smithi186 ceph-mon[32991]: from='osd.6 [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:23:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:41 smithi186 ceph-mon[32991]: osdmap e45: 7 total, 6 up, 7 in 2024-04-26T21:23:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:41 smithi186 ceph-mon[32991]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:23:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:42.129 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:23:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:42 smithi001 ceph-mon[28054]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:42 smithi001 ceph-mon[28054]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:23:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:42 smithi001 ceph-mon[28054]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T21:23:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:42.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:42 smithi038 ceph-mon[32724]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:42 smithi038 ceph-mon[32724]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:23:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:42 smithi038 ceph-mon[32724]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T21:23:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:42.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:42 smithi186 ceph-mon[32991]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:42.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:42 smithi186 ceph-mon[32991]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:23:42.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:42 smithi186 ceph-mon[32991]: osdmap e46: 7 total, 6 up, 7 in 2024-04-26T21:23:42.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:42.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:43 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:23:43.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:43 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:23:43.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:43 smithi038 ceph-mon[32724]: osd.6 [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] boot 2024-04-26T21:23:43.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:43 smithi038 ceph-mon[32724]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T21:23:43.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:43 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:43.879 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:43 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:23:43.879 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:43 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:23:43.879 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:43 smithi001 ceph-mon[28054]: osd.6 [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] boot 2024-04-26T21:23:43.879 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:43 smithi001 ceph-mon[28054]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T21:23:43.879 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:43 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:43.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:43 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:23:43.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:43 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:23:43.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:43 smithi186 ceph-mon[32991]: osd.6 [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] boot 2024-04-26T21:23:43.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:43 smithi186 ceph-mon[32991]: osdmap e47: 7 total, 7 up, 7 in 2024-04-26T21:23:43.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:43 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:23:44.154 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T21:23:44.154 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T21:23:44.154 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-26T21:23:44.154 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-26T21:23:44.154 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.014644 s, 716 MB/s 2024-04-26T21:23:44.154 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-26T21:23:44.778 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:44 smithi038 ceph-mon[32724]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:44.778 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:44 smithi038 ceph-mon[32724]: osdmap e48: 7 total, 7 up, 7 in 2024-04-26T21:23:44.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_2 2024-04-26T21:23:44.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:44 smithi186 ceph-mon[32991]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:44.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:44 smithi186 ceph-mon[32991]: osdmap e48: 7 total, 7 up, 7 in 2024-04-26T21:23:45.051 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:23:45.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:44 smithi001 ceph-mon[28054]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-26T21:23:45.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:44 smithi001 ceph-mon[28054]: osdmap e48: 7 total, 7 up, 7 in 2024-04-26T21:23:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.575+0000 7f16f26fb700 1 -- 172.21.15.38:0/137381715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16ec102380 msgr2=0x7f16ec1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:23:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.575+0000 7f16f26fb700 1 --2- 172.21.15.38:0/137381715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16ec102380 0x7f16ec1027a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f16dc0099c0 tx=0x7f16dc009cd0 comp rx=0 tx=0).stop 2024-04-26T21:23:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.575+0000 7f16f26fb700 1 -- 172.21.15.38:0/137381715 shutdown_connections 2024-04-26T21:23:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.575+0000 7f16f26fb700 1 --2- 172.21.15.38:0/137381715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec103f20 0x7f16ec10a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.575+0000 7f16f26fb700 1 --2- 172.21.15.38:0/137381715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16ec102380 0x7f16ec1027a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:46.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.575+0000 7f16f26fb700 1 --2- 172.21.15.38:0/137381715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16ec103580 0x7f16ec1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.575+0000 7f16f26fb700 1 -- 172.21.15.38:0/137381715 >> 172.21.15.38:0/137381715 conn(0x7f16ec0fd920 msgr2=0x7f16ec0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.575+0000 7f16f26fb700 1 -- 172.21.15.38:0/137381715 shutdown_connections 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.575+0000 7f16f26fb700 1 -- 172.21.15.38:0/137381715 wait complete. 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f26fb700 1 Processor -- start 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f26fb700 1 -- start start 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f26fb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16ec102380 0x7f16ec1064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f26fb700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec103580 0x7f16ec104b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f26fb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16ec103f20 0x7f16ec105060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f26fb700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec10de90 con 0x7f16ec103580 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f26fb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec10dd10 con 0x7f16ec103f20 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f26fb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec10e010 con 0x7f16ec102380 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f0c98700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16ec103f20 0x7f16ec105060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f0c98700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16ec103f20 0x7f16ec105060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44654/0 (socket says 172.21.15.38:44654) 2024-04-26T21:23:46.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16f0c98700 1 -- 172.21.15.38:0/4244696502 learned_addr learned my addr 172.21.15.38:0/4244696502 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-26T21:23:46.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.576+0000 7f16ebfff700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16ec102380 0x7f16ec1064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:23:46.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16f0c98700 1 -- 172.21.15.38:0/4244696502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16ec102380 msgr2=0x7f16ec1064d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:23:46.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16f0c98700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16ec102380 0x7f16ec1064d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:46.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16f0c98700 1 -- 172.21.15.38:0/4244696502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec103580 msgr2=0x7f16ec104b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:23:46.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16f0c98700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec103580 0x7f16ec104b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:23:46.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16f0c98700 1 -- 172.21.15.38:0/4244696502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ec105910 con 0x7f16ec103f20 2024-04-26T21:23:46.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16ebfff700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16ec102380 0x7f16ec1064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:23:46.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16f0c98700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16ec103f20 0x7f16ec105060 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16e400b970 tx=0x7f16e400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:23:46.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16e97fa700 1 -- 172.21.15.38:0/4244696502 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e4019070 con 0x7f16ec103f20 2024-04-26T21:23:46.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16e97fa700 1 -- 172.21.15.38:0/4244696502 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16e400cde0 con 0x7f16ec103f20 2024-04-26T21:23:46.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16e97fa700 1 -- 172.21.15.38:0/4244696502 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e4007370 con 0x7f16ec103f20 2024-04-26T21:23:46.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16f26fb700 1 -- 172.21.15.38:0/4244696502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16ec1a3800 con 0x7f16ec103f20 2024-04-26T21:23:46.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.577+0000 7f16f26fb700 1 -- 172.21.15.38:0/4244696502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16ec1a3d40 con 0x7f16ec103f20 2024-04-26T21:23:46.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.578+0000 7f16f26fb700 1 -- 172.21.15.38:0/4244696502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16ec1a3fa0 con 0x7f16ec103f20 2024-04-26T21:23:46.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.579+0000 7f16e97fa700 1 -- 172.21.15.38:0/4244696502 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f16e401c070 con 0x7f16ec103f20 2024-04-26T21:23:46.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.580+0000 7f16e97fa700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f16d40a3cc0 0x7f16d40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:23:46.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.580+0000 7f16e97fa700 1 -- 172.21.15.38:0/4244696502 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f16e40cec60 con 0x7f16ec103f20 2024-04-26T21:23:46.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.581+0000 7f16ebfff700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f16d40a3cc0 0x7f16d40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:23:46.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.582+0000 7f16ebfff700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f16d40a3cc0 0x7f16d40a6180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f16dc000c00 tx=0x7f16dc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:23:46.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.585+0000 7f16e97fa700 1 -- 172.21.15.38:0/4244696502 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f16e409c2b0 con 0x7f16ec103f20 2024-04-26T21:23:46.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:23:46.804+0000 7f16f26fb700 1 -- 172.21.15.38:0/4244696502 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f16ec06b810 con 0x7f16d40a3cc0 2024-04-26T21:23:46.806 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:46 smithi038 ceph-mon[32724]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:46.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:46 smithi001 ceph-mon[28054]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:46.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:46 smithi186 ceph-mon[32991]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:47.618 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:23:47.618 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:23:47.618 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:23:47.618 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:47.618 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi001 2024-04-26T21:23:47.618 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:47.618 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:47.618 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:47.618 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:47.618 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:47.619 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:47.619 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:47.619 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:47.619 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:47.619 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:47.619 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:47.619 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:47.619 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:47 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:47.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:23:47.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:23:47.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:23:47.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:47.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi001 2024-04-26T21:23:47.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:47.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:47.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:47.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:47.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:47.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:47.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:47.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:47.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:47.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:47.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:47.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:47.956 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:47 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:48.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi001 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:48.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:48.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:23:48.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:48.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:23:48.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:47 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:23:48.742 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:48 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi001 to 5698M 2024-04-26T21:23:48.742 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:48 smithi038 ceph-mon[32724]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:48.894 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:48 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi001 to 5698M 2024-04-26T21:23:48.895 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:48 smithi001 ceph-mon[28054]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:48.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:48 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi001 to 5698M 2024-04-26T21:23:48.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:48 smithi186 ceph-mon[32991]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:50 smithi186 ceph-mon[32991]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:50 smithi186 ceph-mon[32991]: from='client.? 172.21.15.38:0/275595700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfdf3d21-894f-4b0d-9055-152108098b66"}]: dispatch 2024-04-26T21:23:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:50 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfdf3d21-894f-4b0d-9055-152108098b66"}]: dispatch 2024-04-26T21:23:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:50 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfdf3d21-894f-4b0d-9055-152108098b66"}]': finished 2024-04-26T21:23:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:50 smithi186 ceph-mon[32991]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T21:23:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:23:50.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:50 smithi186 ceph-mon[32991]: from='client.? 172.21.15.38:0/3694739694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:23:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:50 smithi001 ceph-mon[28054]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:50 smithi001 ceph-mon[28054]: from='client.? 172.21.15.38:0/275595700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfdf3d21-894f-4b0d-9055-152108098b66"}]: dispatch 2024-04-26T21:23:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:50 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfdf3d21-894f-4b0d-9055-152108098b66"}]: dispatch 2024-04-26T21:23:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:50 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfdf3d21-894f-4b0d-9055-152108098b66"}]': finished 2024-04-26T21:23:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:50 smithi001 ceph-mon[28054]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T21:23:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:23:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:50 smithi001 ceph-mon[28054]: from='client.? 172.21.15.38:0/3694739694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:23:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:50 smithi038 ceph-mon[32724]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:50 smithi038 ceph-mon[32724]: from='client.? 172.21.15.38:0/275595700' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfdf3d21-894f-4b0d-9055-152108098b66"}]: dispatch 2024-04-26T21:23:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:50 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dfdf3d21-894f-4b0d-9055-152108098b66"}]: dispatch 2024-04-26T21:23:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:50 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dfdf3d21-894f-4b0d-9055-152108098b66"}]': finished 2024-04-26T21:23:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:50 smithi038 ceph-mon[32724]: osdmap e49: 8 total, 7 up, 8 in 2024-04-26T21:23:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:23:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:50 smithi038 ceph-mon[32724]: from='client.? 172.21.15.38:0/3694739694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:23:52.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:52 smithi186 ceph-mon[32991]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:53.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:52 smithi001 ceph-mon[28054]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:53.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:52 smithi038 ceph-mon[32724]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:54.893 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:54 smithi038 ceph-mon[32724]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:54.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:54 smithi186 ceph-mon[32991]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:55.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:54 smithi001 ceph-mon[28054]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:56.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:56 smithi186 ceph-mon[32991]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:57.054 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:56 smithi038 ceph-mon[32724]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:57.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:56 smithi001 ceph-mon[28054]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:57.770 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T21:23:57.771 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:57.771 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:57 smithi038 ceph-mon[32724]: Deploying daemon osd.7 on smithi038 2024-04-26T21:23:57.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T21:23:57.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:57.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:57 smithi186 ceph-mon[32991]: Deploying daemon osd.7 on smithi038 2024-04-26T21:23:58.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T21:23:58.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:23:58.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:57 smithi001 ceph-mon[28054]: Deploying daemon osd.7 on smithi038 2024-04-26T21:23:58.820 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:23:58 smithi038 ceph-mon[32724]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:58.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:23:58 smithi186 ceph-mon[32991]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:23:59.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:23:58 smithi001 ceph-mon[28054]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:00.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:00 smithi038 ceph-mon[32724]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:00.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:00 smithi186 ceph-mon[32991]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:00 smithi001 ceph-mon[28054]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:02.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:02 smithi186 ceph-mon[32991]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:03.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:02 smithi001 ceph-mon[28054]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:03.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:02 smithi038 ceph-mon[32724]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:04.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:04.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:04.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:04.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:04.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:04.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:04.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:04.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:04.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:05.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:04 smithi186 ceph-mon[32991]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:05.219 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:04 smithi001 ceph-mon[28054]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:05.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:04 smithi038 ceph-mon[32724]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:06.953 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 7 on host 'smithi038' 2024-04-26T21:24:06.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.952+0000 7f16e97fa700 1 -- 172.21.15.38:0/4244696502 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f16ec06b810 con 0x7f16d40a3cc0 2024-04-26T21:24:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.954+0000 7f16f26fb700 1 -- 172.21.15.38:0/4244696502 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f16d40a3cc0 msgr2=0x7f16d40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.954+0000 7f16f26fb700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f16d40a3cc0 0x7f16d40a6180 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f16dc000c00 tx=0x7f16dc005fb0 comp rx=0 tx=0).stop 2024-04-26T21:24:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.954+0000 7f16f26fb700 1 -- 172.21.15.38:0/4244696502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16ec103f20 msgr2=0x7f16ec105060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.954+0000 7f16f26fb700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16ec103f20 0x7f16ec105060 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f16e400b970 tx=0x7f16e400bc80 comp rx=0 tx=0).stop 2024-04-26T21:24:06.955 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.955+0000 7f16f26fb700 1 -- 172.21.15.38:0/4244696502 shutdown_connections 2024-04-26T21:24:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.955+0000 7f16f26fb700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f16d40a3cc0 0x7f16d40a6180 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.955+0000 7f16f26fb700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16ec102380 0x7f16ec1064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.955+0000 7f16f26fb700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16ec103f20 0x7f16ec105060 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.955+0000 7f16f26fb700 1 --2- 172.21.15.38:0/4244696502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16ec103580 0x7f16ec104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.955+0000 7f16f26fb700 1 -- 172.21.15.38:0/4244696502 >> 172.21.15.38:0/4244696502 conn(0x7f16ec0fd920 msgr2=0x7f16ec06b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.955+0000 7f16f26fb700 1 -- 172.21.15.38:0/4244696502 shutdown_connections 2024-04-26T21:24:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:24:06.955+0000 7f16f26fb700 1 -- 172.21.15.38:0/4244696502 wait complete. 2024-04-26T21:24:07.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:07.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:07.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:07.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: from='osd.7 [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:24:07.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:24:07.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:07 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: from='osd.7 [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:24:07.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:24:07.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:07 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: from='osd.7 [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:24:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T21:24:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:07.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:07 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:07.663 DEBUG:teuthology.orchestra.run.smithi038:osd.7> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T21:24:07.665 INFO:tasks.cephadm:Deploying osd.8 on smithi186 with /dev/vg_nvme/lv_2... 2024-04-26T21:24:07.666 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T21:24:07.913 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:24:08.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:08 smithi038 ceph-mon[32724]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T21:24:08.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:08 smithi038 ceph-mon[32724]: from='osd.7 [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:24:08.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:08 smithi038 ceph-mon[32724]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T21:24:08.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:08.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:08 smithi038 ceph-mon[32724]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:24:08.364 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:08 smithi186 ceph-mon[32991]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T21:24:08.364 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:08 smithi186 ceph-mon[32991]: from='osd.7 [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:24:08.364 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:08 smithi186 ceph-mon[32991]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T21:24:08.364 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:08.364 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:08 smithi186 ceph-mon[32991]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:24:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:08 smithi001 ceph-mon[28054]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T21:24:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:08 smithi001 ceph-mon[28054]: from='osd.7 [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:24:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:08 smithi001 ceph-mon[28054]: osdmap e50: 8 total, 7 up, 8 in 2024-04-26T21:24:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:08 smithi001 ceph-mon[28054]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:24:09.074 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:24:08 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:24:08.811+0000 7f066b5c9700 -1 osd.7 0 waiting for initial osdmap 2024-04-26T21:24:09.074 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:24:08 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:24:08.820+0000 7f06633b4700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:24:09.322 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:09 smithi186 ceph-mon[32991]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:09.322 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:09 smithi186 ceph-mon[32991]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:24:09.322 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:09 smithi186 ceph-mon[32991]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T21:24:09.322 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:09 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:09.322 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:09 smithi186 ceph-mon[32991]: osdmap e52: 8 total, 7 up, 8 in 2024-04-26T21:24:09.322 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:09 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:09.322 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:09 smithi186 ceph-mon[32991]: from='osd.7 ' entity='osd.7' 2024-04-26T21:24:09.322 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:09 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:09 smithi038 ceph-mon[32724]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:09 smithi038 ceph-mon[32724]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:24:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:09 smithi038 ceph-mon[32724]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T21:24:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:09 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:09 smithi038 ceph-mon[32724]: osdmap e52: 8 total, 7 up, 8 in 2024-04-26T21:24:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:09 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:09 smithi038 ceph-mon[32724]: from='osd.7 ' entity='osd.7' 2024-04-26T21:24:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:09 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:09 smithi001 ceph-mon[28054]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:09 smithi001 ceph-mon[28054]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:24:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:09 smithi001 ceph-mon[28054]: osdmap e51: 8 total, 7 up, 8 in 2024-04-26T21:24:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:09 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:09 smithi001 ceph-mon[28054]: osdmap e52: 8 total, 7 up, 8 in 2024-04-26T21:24:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:09 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:09 smithi001 ceph-mon[28054]: from='osd.7 ' entity='osd.7' 2024-04-26T21:24:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:09 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:09.914 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T21:24:09.914 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T21:24:09.915 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T21:24:09.915 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T21:24:09.915 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129755 s, 808 MB/s 2024-04-26T21:24:09.915 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T21:24:10.656 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_2 2024-04-26T21:24:10.757 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:10 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:24:10.757 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:10 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:24:10.757 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:10 smithi038 ceph-mon[32724]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:10.757 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:10 smithi038 ceph-mon[32724]: osd.7 [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] boot 2024-04-26T21:24:10.757 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:10 smithi038 ceph-mon[32724]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T21:24:10.757 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:10.904 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:24:10.941 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:10 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:24:10.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:10 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:24:10.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:10 smithi186 ceph-mon[32991]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:10.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:10 smithi186 ceph-mon[32991]: osd.7 [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] boot 2024-04-26T21:24:10.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:10 smithi186 ceph-mon[32991]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T21:24:10.942 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:11.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:10 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:24:11.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:10 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:24:11.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:10 smithi001 ceph-mon[28054]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-26T21:24:11.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:10 smithi001 ceph-mon[28054]: osd.7 [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] boot 2024-04-26T21:24:11.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:10 smithi001 ceph-mon[28054]: osdmap e53: 8 total, 8 up, 8 in 2024-04-26T21:24:11.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:24:11.894 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi038 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:11.895 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:12.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T21:24:12.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi038 2024-04-26T21:24:12.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:12.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:12.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:12.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:12.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:12.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-26T21:24:12.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:12.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:12.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:12.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:12.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: osdmap e54: 8 total, 8 up, 8 in 2024-04-26T21:24:12.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi038 2024-04-26T21:24:12.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:12.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:12.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:12.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:12.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:12.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi038 to 5697M 2024-04-26T21:24:12.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:12.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:12.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:12.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.565+0000 7fc2e4db0700 1 -- 172.21.15.186:0/1646934663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2e0102390 msgr2=0x7fc2e01027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:12.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.565+0000 7fc2e4db0700 1 --2- 172.21.15.186:0/1646934663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2e0102390 0x7fc2e01027b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800b0a0 tx=0x7fc2d800b3b0 comp rx=0 tx=0).stop 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.565+0000 7fc2e4db0700 1 -- 172.21.15.186:0/1646934663 shutdown_connections 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.565+0000 7fc2e4db0700 1 --2- 172.21.15.186:0/1646934663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2e0102390 0x7fc2e01027b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.565+0000 7fc2e4db0700 1 --2- 172.21.15.186:0/1646934663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e0103f30 0x7fc2e010a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.565+0000 7fc2e4db0700 1 --2- 172.21.15.186:0/1646934663 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2e0103590 0x7fc2e01039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.565+0000 7fc2e4db0700 1 -- 172.21.15.186:0/1646934663 >> 172.21.15.186:0/1646934663 conn(0x7fc2e00fd950 msgr2=0x7fc2e00ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.565+0000 7fc2e4db0700 1 -- 172.21.15.186:0/1646934663 shutdown_connections 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.565+0000 7fc2e4db0700 1 -- 172.21.15.186:0/1646934663 wait complete. 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2e4db0700 1 Processor -- start 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2e4db0700 1 -- start start 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2e4db0700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2e0102390 0x7fc2e006c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2e4db0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e0103590 0x7fc2e019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2e4db0700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:12.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2e4db0700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e010dea0 con 0x7fc2e0102390 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2e4db0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e010dd20 con 0x7fc2e0103590 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2e4db0700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2e010e020 con 0x7fc2e0103f30 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2ded9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2ded9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:46712/0 (socket says 172.21.15.186:46712) 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2ded9d700 1 -- 172.21.15.186:0/2869475470 learned_addr learned my addr 172.21.15.186:0/2869475470 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.566+0000 7fc2ddd9b700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e0103590 0x7fc2e019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2ded9d700 1 -- 172.21.15.186:0/2869475470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e0103590 msgr2=0x7fc2e019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2ded9d700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e0103590 0x7fc2e019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2ded9d700 1 -- 172.21.15.186:0/2869475470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2e0102390 msgr2=0x7fc2e006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2ded9d700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2e0102390 0x7fc2e006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2ded9d700 1 -- 172.21.15.186:0/2869475470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2e01a1340 con 0x7fc2e0103f30 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2ded9d700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a0a70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000c970 tx=0x7fc2d000cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:12.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2d77fe700 1 -- 172.21.15.186:0/2869475470 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d000ce60 con 0x7fc2e0103f30 2024-04-26T21:24:12.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2d77fe700 1 -- 172.21.15.186:0/2869475470 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2d0004830 con 0x7fc2e0103f30 2024-04-26T21:24:12.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2e4db0700 1 -- 172.21.15.186:0/2869475470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2e01a1630 con 0x7fc2e0103f30 2024-04-26T21:24:12.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2d77fe700 1 -- 172.21.15.186:0/2869475470 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d00054c0 con 0x7fc2e0103f30 2024-04-26T21:24:12.571 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.567+0000 7fc2e4db0700 1 -- 172.21.15.186:0/2869475470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2e01a1bf0 con 0x7fc2e0103f30 2024-04-26T21:24:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.568+0000 7fc2e4db0700 1 -- 172.21.15.186:0/2869475470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2e006a110 con 0x7fc2e0103f30 2024-04-26T21:24:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.569+0000 7fc2d77fe700 1 -- 172.21.15.186:0/2869475470 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc2d00049e0 con 0x7fc2e0103f30 2024-04-26T21:24:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.569+0000 7fc2d77fe700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2cc0a3c40 0x7fc2cc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.570+0000 7fc2d77fe700 1 -- 172.21.15.186:0/2869475470 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5426+0+0 (secure 0 0 0) 0x7fc2d00cf710 con 0x7fc2e0103f30 2024-04-26T21:24:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.570+0000 7fc2de59c700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2cc0a3c40 0x7fc2cc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:12.572 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.571+0000 7fc2de59c700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2cc0a3c40 0x7fc2cc0a6100 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800b830 tx=0x7fc2d80060c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:12.575 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.574+0000 7fc2d77fe700 1 -- 172.21.15.186:0/2869475470 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc2d0014030 con 0x7fc2e0103f30 2024-04-26T21:24:12.783 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:12.782+0000 7fc2e4db0700 1 -- 172.21.15.186:0/2869475470 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc2e0063860 con 0x7fc2cc0a3c40 2024-04-26T21:24:12.784 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:12 smithi186 ceph-mon[32991]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:12.784 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:12 smithi186 ceph-mon[32991]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T21:24:13.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:12 smithi001 ceph-mon[28054]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:13.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:12 smithi001 ceph-mon[28054]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T21:24:13.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:12 smithi038 ceph-mon[32724]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:13.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:12 smithi038 ceph-mon[32724]: osdmap e55: 8 total, 8 up, 8 in 2024-04-26T21:24:13.676 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:13 smithi186 ceph-mon[32991]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T21:24:13.676 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:13 smithi186 ceph-mon[32991]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:24:13.677 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:24:13.677 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:24:13.677 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:13 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:13.677 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:13 smithi186 ceph-mon[32991]: osdmap e57: 8 total, 8 up, 8 in 2024-04-26T21:24:14.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:13 smithi001 ceph-mon[28054]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T21:24:14.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:13 smithi001 ceph-mon[28054]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:24:14.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:24:14.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:24:14.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:13 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:14.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:13 smithi001 ceph-mon[28054]: osdmap e57: 8 total, 8 up, 8 in 2024-04-26T21:24:14.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:13 smithi038 ceph-mon[32724]: osdmap e56: 8 total, 8 up, 8 in 2024-04-26T21:24:14.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:13 smithi038 ceph-mon[32724]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:24:14.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:24:14.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:24:14.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:13 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:14.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:13 smithi038 ceph-mon[32724]: osdmap e57: 8 total, 8 up, 8 in 2024-04-26T21:24:14.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:14 smithi186 ceph-mon[32991]: pgmap v205: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:15.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:14 smithi001 ceph-mon[28054]: pgmap v205: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:15.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:14 smithi038 ceph-mon[32724]: pgmap v205: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:16.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:16 smithi186 ceph-mon[32991]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:16.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:16 smithi186 ceph-mon[32991]: from='client.? 172.21.15.186:0/4269633174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74279a09-821c-4a4f-94b9-1ac7a56d6cf7"}]: dispatch 2024-04-26T21:24:16.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:16 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74279a09-821c-4a4f-94b9-1ac7a56d6cf7"}]: dispatch 2024-04-26T21:24:16.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:16 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74279a09-821c-4a4f-94b9-1ac7a56d6cf7"}]': finished 2024-04-26T21:24:16.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:16 smithi186 ceph-mon[32991]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T21:24:16.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:16.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:16 smithi186 ceph-mon[32991]: from='client.? 172.21.15.186:0/2925974767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:24:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:16 smithi001 ceph-mon[28054]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:16 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/4269633174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74279a09-821c-4a4f-94b9-1ac7a56d6cf7"}]: dispatch 2024-04-26T21:24:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:16 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74279a09-821c-4a4f-94b9-1ac7a56d6cf7"}]: dispatch 2024-04-26T21:24:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:16 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74279a09-821c-4a4f-94b9-1ac7a56d6cf7"}]': finished 2024-04-26T21:24:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:16 smithi001 ceph-mon[28054]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T21:24:17.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:17.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:16 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/2925974767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:24:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:16 smithi038 ceph-mon[32724]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:16 smithi038 ceph-mon[32724]: from='client.? 172.21.15.186:0/4269633174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74279a09-821c-4a4f-94b9-1ac7a56d6cf7"}]: dispatch 2024-04-26T21:24:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:16 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "74279a09-821c-4a4f-94b9-1ac7a56d6cf7"}]: dispatch 2024-04-26T21:24:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:16 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "74279a09-821c-4a4f-94b9-1ac7a56d6cf7"}]': finished 2024-04-26T21:24:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:16 smithi038 ceph-mon[32724]: osdmap e58: 9 total, 8 up, 9 in 2024-04-26T21:24:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:16 smithi038 ceph-mon[32724]: from='client.? 172.21.15.186:0/2925974767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:24:18.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:18 smithi186 ceph-mon[32991]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:19.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:18 smithi001 ceph-mon[28054]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:18 smithi038 ceph-mon[32724]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:20.923 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:20 smithi186 ceph-mon[32991]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:21.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:20 smithi001 ceph-mon[28054]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:21.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:20 smithi038 ceph-mon[32724]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:22.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:22 smithi186 ceph-mon[32991]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:23.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:22 smithi001 ceph-mon[28054]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:22 smithi038 ceph-mon[32724]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:23.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T21:24:23.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:23.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:23 smithi186 ceph-mon[32991]: Deploying daemon osd.8 on smithi186 2024-04-26T21:24:24.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T21:24:24.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:24.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:23 smithi001 ceph-mon[28054]: Deploying daemon osd.8 on smithi186 2024-04-26T21:24:24.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-26T21:24:24.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:24.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:23 smithi038 ceph-mon[32724]: Deploying daemon osd.8 on smithi186 2024-04-26T21:24:24.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:24 smithi186 ceph-mon[32991]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:25.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:24 smithi001 ceph-mon[28054]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:25.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:24 smithi038 ceph-mon[32724]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:26.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:26 smithi186 ceph-mon[32991]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:27.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:26 smithi001 ceph-mon[28054]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:27.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:26 smithi038 ceph-mon[32724]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:28.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:28 smithi186 ceph-mon[32991]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:29.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:28 smithi001 ceph-mon[28054]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:29.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:28 smithi038 ceph-mon[32724]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:30.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:30.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:30.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:29 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:30.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:29 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:30.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:29 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:30.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:29 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:30.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:29 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:30.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:29 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:30.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:29 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:31.159 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:30 smithi186 ceph-mon[32991]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:31.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:30 smithi001 ceph-mon[28054]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:31.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:30 smithi038 ceph-mon[32724]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:32.421 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.420+0000 7fc2d77fe700 1 -- 172.21.15.186:0/2869475470 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc2e0063860 con 0x7fc2cc0a3c40 2024-04-26T21:24:32.421 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 8 on host 'smithi186' 2024-04-26T21:24:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 -- 172.21.15.186:0/2869475470 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2cc0a3c40 msgr2=0x7fc2cc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2cc0a3c40 0x7fc2cc0a6100 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800b830 tx=0x7fc2d80060c0 comp rx=0 tx=0).stop 2024-04-26T21:24:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 -- 172.21.15.186:0/2869475470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2e0103f30 msgr2=0x7fc2e01a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a0a70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc2d000c970 tx=0x7fc2d000cc80 comp rx=0 tx=0).stop 2024-04-26T21:24:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 -- 172.21.15.186:0/2869475470 shutdown_connections 2024-04-26T21:24:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2cc0a3c40 0x7fc2cc0a6100 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2e0103f30 0x7fc2e01a0a70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2e0102390 0x7fc2e006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 --2- 172.21.15.186:0/2869475470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2e0103590 0x7fc2e019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 -- 172.21.15.186:0/2869475470 >> 172.21.15.186:0/2869475470 conn(0x7fc2e00fd950 msgr2=0x7fc2e0106a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.423+0000 7fc2e4db0700 1 -- 172.21.15.186:0/2869475470 shutdown_connections 2024-04-26T21:24:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:24:32.424+0000 7fc2e4db0700 1 -- 172.21.15.186:0/2869475470 wait complete. 2024-04-26T21:24:32.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:32 smithi186 ceph-mon[32991]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:32.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:32 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:32.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:32 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:32.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:32 smithi186 ceph-mon[32991]: from='osd.8 [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:24:32.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:32 smithi186 ceph-mon[32991]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:24:33.146 DEBUG:teuthology.orchestra.run.smithi186:osd.8> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.8.service 2024-04-26T21:24:33.149 INFO:tasks.cephadm:Deploying osd.9 on smithi001 with /dev/vg_nvme/lv_1... 2024-04-26T21:24:33.149 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T21:24:33.231 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:32 smithi001 ceph-mon[28054]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:33.232 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:32 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:33.232 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:32 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:33.232 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:32 smithi001 ceph-mon[28054]: from='osd.8 [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:24:33.232 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:32 smithi001 ceph-mon[28054]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:24:33.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:32 smithi038 ceph-mon[32724]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:33.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:32 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:33.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:32 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:33.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:32 smithi038 ceph-mon[32724]: from='osd.8 [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:24:33.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:32 smithi038 ceph-mon[32724]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-26T21:24:33.611 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:24:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:34 smithi186 ceph-mon[32991]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:34.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:34.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:34 smithi186 ceph-mon[32991]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T21:24:34.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:34 smithi186 ceph-mon[32991]: from='osd.8 [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:24:34.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:34 smithi186 ceph-mon[32991]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T21:24:34.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:34.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:34 smithi186 ceph-mon[32991]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:24:34.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:34.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:34 smithi001 ceph-mon[28054]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:34.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:34.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:34 smithi001 ceph-mon[28054]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T21:24:34.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:34 smithi001 ceph-mon[28054]: from='osd.8 [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:24:34.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:34 smithi001 ceph-mon[28054]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T21:24:34.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:34.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:34 smithi001 ceph-mon[28054]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:24:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:34 smithi038 ceph-mon[32724]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:34 smithi038 ceph-mon[32724]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-26T21:24:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:34 smithi038 ceph-mon[32724]: from='osd.8 [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:24:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:34 smithi038 ceph-mon[32724]: osdmap e59: 9 total, 8 up, 9 in 2024-04-26T21:24:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:34 smithi038 ceph-mon[32724]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:24:35.168 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:24:35 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:24:35.156+0000 7f182b9a8700 -1 osd.8 0 waiting for initial osdmap 2024-04-26T21:24:35.168 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:24:35 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:24:35.167+0000 7f1823793700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:24:35.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:35 smithi186 ceph-mon[32991]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:24:35.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:35 smithi186 ceph-mon[32991]: osdmap e60: 9 total, 8 up, 9 in 2024-04-26T21:24:35.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:35 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:35.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:35 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:35.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:35 smithi186 ceph-mon[32991]: from='osd.8 ' entity='osd.8' 2024-04-26T21:24:35.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:35 smithi001 ceph-mon[28054]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:24:35.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:35 smithi001 ceph-mon[28054]: osdmap e60: 9 total, 8 up, 9 in 2024-04-26T21:24:35.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:35.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:35 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:35.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:35 smithi001 ceph-mon[28054]: from='osd.8 ' entity='osd.8' 2024-04-26T21:24:35.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:35 smithi038 ceph-mon[32724]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:24:35.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:35 smithi038 ceph-mon[32724]: osdmap e60: 9 total, 8 up, 9 in 2024-04-26T21:24:35.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:35 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:35.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:35 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:35.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:35 smithi038 ceph-mon[32724]: from='osd.8 ' entity='osd.8' 2024-04-26T21:24:36.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:36 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:24:36.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:36 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:24:36.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:36 smithi186 ceph-mon[32991]: pgmap v220: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:36.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:36 smithi186 ceph-mon[32991]: osd.8 [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] boot 2024-04-26T21:24:36.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:36 smithi186 ceph-mon[32991]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T21:24:36.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:36.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:36 smithi186 ceph-mon[32991]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T21:24:36.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:36 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:24:36.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:36 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:24:36.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:36 smithi001 ceph-mon[28054]: pgmap v220: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:36.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:36 smithi001 ceph-mon[28054]: osd.8 [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] boot 2024-04-26T21:24:36.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:36 smithi001 ceph-mon[28054]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T21:24:36.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:36.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:36 smithi001 ceph-mon[28054]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T21:24:36.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:36 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:24:36.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:36 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:24:36.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:36 smithi038 ceph-mon[32724]: pgmap v220: 1 pgs: 1 peering; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-04-26T21:24:36.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:36 smithi038 ceph-mon[32724]: osd.8 [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] boot 2024-04-26T21:24:36.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:36 smithi038 ceph-mon[32724]: osdmap e61: 9 total, 9 up, 9 in 2024-04-26T21:24:36.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:24:36.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:36 smithi038 ceph-mon[32724]: osdmap e62: 9 total, 9 up, 9 in 2024-04-26T21:24:36.858 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T21:24:36.858 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T21:24:36.858 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-04-26T21:24:36.858 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-04-26T21:24:36.858 INFO:teuthology.orchestra.run.smithi001.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0417418 s, 251 MB/s 2024-04-26T21:24:36.858 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-04-26T21:24:37.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_1 2024-04-26T21:24:38.106 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi186 2024-04-26T21:24:38.106 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.106 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.106 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:38.106 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:38.106 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:38.106 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-26T21:24:38.106 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.107 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:38.107 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:38.107 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.107 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:38.107 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:37 smithi186 ceph-mon[32991]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T21:24:38.264 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi186 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:38.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:37 smithi001 ceph-mon[28054]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T21:24:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi186 2024-04-26T21:24:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:24:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-26T21:24:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:38.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:38.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:38.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:24:38.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:37 smithi038 ceph-mon[32724]: osdmap e63: 9 total, 9 up, 9 in 2024-04-26T21:24:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:38 smithi001 ceph-mon[28054]: pgmap v223: 1 pgs: 1 peering; 577 KiB data, 580 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:38 smithi001 ceph-mon[28054]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T21:24:39.121 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:38 smithi186 ceph-mon[32991]: pgmap v223: 1 pgs: 1 peering; 577 KiB data, 580 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:39.122 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:39.122 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:39.122 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:39.122 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:38 smithi186 ceph-mon[32991]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T21:24:39.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:38 smithi038 ceph-mon[32724]: pgmap v223: 1 pgs: 1 peering; 577 KiB data, 580 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:39.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:39.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:24:39.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:24:39.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:38 smithi038 ceph-mon[32724]: osdmap e64: 9 total, 9 up, 9 in 2024-04-26T21:24:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.539+0000 7f019a400700 1 -- 172.21.15.1:0/1580734966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0194104670 msgr2=0x7f0194104ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.539+0000 7f019a400700 1 --2- 172.21.15.1:0/1580734966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0194104670 0x7f0194104ad0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f017c0099c0 tx=0x7f017c009cd0 comp rx=0 tx=0).stop 2024-04-26T21:24:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.539+0000 7f019a400700 1 -- 172.21.15.1:0/1580734966 shutdown_connections 2024-04-26T21:24:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.539+0000 7f019a400700 1 --2- 172.21.15.1:0/1580734966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0194103470 0x7f0194103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.539+0000 7f019a400700 1 --2- 172.21.15.1:0/1580734966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194105010 0x7f019410b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.539+0000 7f019a400700 1 --2- 172.21.15.1:0/1580734966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0194104670 0x7f0194104ad0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.539+0000 7f019a400700 1 -- 172.21.15.1:0/1580734966 >> 172.21.15.1:0/1580734966 conn(0x7f01940fea10 msgr2=0x7f0194100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:24:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.539+0000 7f019a400700 1 -- 172.21.15.1:0/1580734966 shutdown_connections 2024-04-26T21:24:40.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.539+0000 7f019a400700 1 -- 172.21.15.1:0/1580734966 wait complete. 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.540+0000 7f019a400700 1 Processor -- start 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f019a400700 1 -- start start 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f019a400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194103470 0x7f019419b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f019a400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0194104670 0x7f019419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f019a400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0194105010 0x7f01941a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f019a400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f019410ef40 con 0x7f0194103470 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f019a400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f019410edc0 con 0x7f0194105010 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f019a400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f019410f0c0 con 0x7f0194104670 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f0193400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194103470 0x7f019419b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f0193400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194103470 0x7f019419b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40928/0 (socket says 172.21.15.1:40928) 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f0193400700 1 -- 172.21.15.1:0/48466412 learned_addr learned my addr 172.21.15.1:0/48466412 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:24:40.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.541+0000 7f018be00700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0194104670 0x7f019419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:40.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.542+0000 7f0193e00700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0194105010 0x7f01941a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:40.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.542+0000 7f018be00700 1 -- 172.21.15.1:0/48466412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0194105010 msgr2=0x7f01941a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:40.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.542+0000 7f018be00700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0194105010 0x7f01941a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:40.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.542+0000 7f018be00700 1 -- 172.21.15.1:0/48466412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194103470 msgr2=0x7f019419b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:24:40.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.542+0000 7f018be00700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194103470 0x7f019419b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:24:40.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.542+0000 7f018be00700 1 -- 172.21.15.1:0/48466412 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01941a3450 con 0x7f0194104670 2024-04-26T21:24:40.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.542+0000 7f0193400700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194103470 0x7f019419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:24:40.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.542+0000 7f018be00700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0194104670 0x7f019419bd30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f017c0099c0 tx=0x7f017c003af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:40.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.543+0000 7f0190c00700 1 -- 172.21.15.1:0/48466412 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017c0050c0 con 0x7f0194104670 2024-04-26T21:24:40.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.543+0000 7f019a400700 1 -- 172.21.15.1:0/48466412 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01941a36e0 con 0x7f0194104670 2024-04-26T21:24:40.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.543+0000 7f0190c00700 1 -- 172.21.15.1:0/48466412 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f017c00f460 con 0x7f0194104670 2024-04-26T21:24:40.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.543+0000 7f0190c00700 1 -- 172.21.15.1:0/48466412 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f017c017650 con 0x7f0194104670 2024-04-26T21:24:40.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.543+0000 7f019a400700 1 -- 172.21.15.1:0/48466412 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01941a3b90 con 0x7f0194104670 2024-04-26T21:24:40.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.544+0000 7f019a400700 1 -- 172.21.15.1:0/48466412 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f019406a110 con 0x7f0194104670 2024-04-26T21:24:40.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.546+0000 7f0190c00700 1 -- 172.21.15.1:0/48466412 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f017c00f910 con 0x7f0194104670 2024-04-26T21:24:40.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.548+0000 7f0190c00700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01780a3c40 0x7f01780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:24:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.548+0000 7f0190c00700 1 -- 172.21.15.1:0/48466412 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5829+0+0 (secure 0 0 0) 0x7f017c0d07a0 con 0x7f0194104670 2024-04-26T21:24:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.548+0000 7f0193400700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01780a3c40 0x7f01780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:24:40.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.549+0000 7f0193400700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01780a3c40 0x7f01780a6100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0184005950 tx=0x7f0184009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:24:40.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.552+0000 7f0190c00700 1 -- 172.21.15.1:0/48466412 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f017c013070 con 0x7f0194104670 2024-04-26T21:24:40.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:40 smithi186 ceph-mon[32991]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 580 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:40.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:40 smithi186 ceph-mon[32991]: osdmap e65: 9 total, 9 up, 9 in 2024-04-26T21:24:40.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:24:40.796+0000 7f019a400700 1 -- 172.21.15.1:0/48466412 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0194076d70 con 0x7f01780a3c40 2024-04-26T21:24:40.797 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:40 smithi001 ceph-mon[28054]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 580 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:40.797 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:40 smithi001 ceph-mon[28054]: osdmap e65: 9 total, 9 up, 9 in 2024-04-26T21:24:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:40 smithi038 ceph-mon[32724]: pgmap v226: 1 pgs: 1 peering; 577 KiB data, 580 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:40 smithi038 ceph-mon[32724]: osdmap e65: 9 total, 9 up, 9 in 2024-04-26T21:24:41.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:41 smithi186 ceph-mon[32991]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:24:41.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:24:41.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:24:41.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:41.798 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:41 smithi001 ceph-mon[28054]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:24:41.798 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:24:41.798 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:24:41.798 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:41 smithi038 ceph-mon[32724]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:24:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:24:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:24:41.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:42 smithi186 ceph-mon[32991]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-26T21:24:42.813 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:42 smithi001 ceph-mon[28054]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-26T21:24:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:42 smithi038 ceph-mon[32724]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 274 MiB used, 804 GiB / 805 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-26T21:24:44.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:44 smithi186 ceph-mon[32991]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:24:44.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:44 smithi001 ceph-mon[28054]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:24:44.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:44 smithi038 ceph-mon[32724]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T21:24:45.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:45 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/206789160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bd72f3-4a1d-4aca-b57f-9939e36d9786"}]: dispatch 2024-04-26T21:24:45.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:45 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bd72f3-4a1d-4aca-b57f-9939e36d9786"}]: dispatch 2024-04-26T21:24:45.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:45 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7bd72f3-4a1d-4aca-b57f-9939e36d9786"}]': finished 2024-04-26T21:24:45.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:45 smithi186 ceph-mon[32991]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T21:24:45.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:45 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:24:45.792 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:45 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/206789160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bd72f3-4a1d-4aca-b57f-9939e36d9786"}]: dispatch 2024-04-26T21:24:45.792 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:45 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bd72f3-4a1d-4aca-b57f-9939e36d9786"}]: dispatch 2024-04-26T21:24:45.793 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:45 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7bd72f3-4a1d-4aca-b57f-9939e36d9786"}]': finished 2024-04-26T21:24:45.793 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:45 smithi001 ceph-mon[28054]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T21:24:45.793 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:45 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:24:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:45 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/206789160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bd72f3-4a1d-4aca-b57f-9939e36d9786"}]: dispatch 2024-04-26T21:24:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:45 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7bd72f3-4a1d-4aca-b57f-9939e36d9786"}]: dispatch 2024-04-26T21:24:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:45 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7bd72f3-4a1d-4aca-b57f-9939e36d9786"}]': finished 2024-04-26T21:24:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:45 smithi038 ceph-mon[32724]: osdmap e66: 10 total, 9 up, 10 in 2024-04-26T21:24:45.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:45 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:24:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:46 smithi186 ceph-mon[32991]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-26T21:24:46.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:46 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3533490120' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:24:46.728 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:46 smithi038 ceph-mon[32724]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-26T21:24:46.728 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:46 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3533490120' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:24:46.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:46 smithi001 ceph-mon[28054]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-26T21:24:46.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:46 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3533490120' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:24:48.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:48 smithi001 ceph-mon[28054]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:24:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:48 smithi038 ceph-mon[32724]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:24:48.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:48 smithi186 ceph-mon[32991]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T21:24:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:50 smithi001 ceph-mon[28054]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T21:24:50.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:50 smithi186 ceph-mon[32991]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T21:24:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:50 smithi038 ceph-mon[32724]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T21:24:52.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:52 smithi001 ceph-mon[28054]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:52.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:52 smithi038 ceph-mon[32724]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:52.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:52 smithi186 ceph-mon[32991]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:54.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:54 smithi001 ceph-mon[28054]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:54.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:54 smithi038 ceph-mon[32724]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:54.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:54 smithi186 ceph-mon[32991]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:55.813 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T21:24:55.813 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:55.813 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:55 smithi001 ceph-mon[28054]: Deploying daemon osd.9 on smithi001 2024-04-26T21:24:55.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T21:24:55.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:55.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:55 smithi038 ceph-mon[32724]: Deploying daemon osd.9 on smithi001 2024-04-26T21:24:55.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-26T21:24:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:24:55.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:55 smithi186 ceph-mon[32991]: Deploying daemon osd.9 on smithi001 2024-04-26T21:24:56.816 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:56 smithi001 ceph-mon[28054]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:56.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:56 smithi038 ceph-mon[32724]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:56.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:56 smithi186 ceph-mon[32991]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:58.732 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:24:58 smithi001 ceph-mon[28054]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:58.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:24:58 smithi038 ceph-mon[32724]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:24:58.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:24:58 smithi186 ceph-mon[32991]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:00.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:00 smithi001 ceph-mon[28054]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:00.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:00 smithi038 ceph-mon[32724]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:00.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:00 smithi186 ceph-mon[32991]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:02.703 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:02 smithi001 ceph-mon[28054]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:02.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:02 smithi038 ceph-mon[32724]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:02.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:02 smithi186 ceph-mon[32991]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:03.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:03.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:03.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:03.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:03.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:03.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:03.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:03.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:03.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:04 smithi001 ceph-mon[28054]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:04.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:04 smithi038 ceph-mon[32724]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:04.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:04 smithi186 ceph-mon[32991]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:06.775 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:06 smithi001 ceph-mon[28054]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:06.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:06 smithi038 ceph-mon[32724]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:06.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:06 smithi186 ceph-mon[32991]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:08.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.204+0000 7f0190c00700 1 -- 172.21.15.1:0/48466412 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0194076d70 con 0x7f01780a3c40 2024-04-26T21:25:08.206 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 9 on host 'smithi001' 2024-04-26T21:25:08.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.207+0000 7f019a400700 1 -- 172.21.15.1:0/48466412 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01780a3c40 msgr2=0x7f01780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:08.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.207+0000 7f019a400700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01780a3c40 0x7f01780a6100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0184005950 tx=0x7f0184009260 comp rx=0 tx=0).stop 2024-04-26T21:25:08.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.207+0000 7f019a400700 1 -- 172.21.15.1:0/48466412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0194104670 msgr2=0x7f019419bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:08.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.207+0000 7f019a400700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0194104670 0x7f019419bd30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f017c0099c0 tx=0x7f017c003af0 comp rx=0 tx=0).stop 2024-04-26T21:25:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.207+0000 7f019a400700 1 -- 172.21.15.1:0/48466412 shutdown_connections 2024-04-26T21:25:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.207+0000 7f019a400700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01780a3c40 0x7f01780a6100 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.207+0000 7f019a400700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0194105010 0x7f01941a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.207+0000 7f019a400700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0194103470 0x7f019419b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.207+0000 7f019a400700 1 --2- 172.21.15.1:0/48466412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0194104670 0x7f019419bd30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.207+0000 7f019a400700 1 -- 172.21.15.1:0/48466412 >> 172.21.15.1:0/48466412 conn(0x7f01940fea10 msgr2=0x7f0194075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.208+0000 7f019a400700 1 -- 172.21.15.1:0/48466412 shutdown_connections 2024-04-26T21:25:08.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:25:08.208+0000 7f019a400700 1 -- 172.21.15.1:0/48466412 wait complete. 2024-04-26T21:25:08.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:08.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:08 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:08.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:08 smithi001 ceph-mon[28054]: from='osd.9 [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T21:25:08.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:08 smithi001 ceph-mon[28054]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T21:25:08.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:08.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:08 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:08.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:08 smithi038 ceph-mon[32724]: from='osd.9 [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T21:25:08.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:08 smithi038 ceph-mon[32724]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T21:25:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:08 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:08 smithi186 ceph-mon[32991]: from='osd.9 [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T21:25:08.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:08 smithi186 ceph-mon[32991]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-26T21:25:09.142 DEBUG:teuthology.orchestra.run.smithi001:osd.9> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.9.service 2024-04-26T21:25:09.148 INFO:tasks.cephadm:Deploying osd.10 on smithi038 with /dev/vg_nvme/lv_1... 2024-04-26T21:25:09.148 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T21:25:09.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:09 smithi001 ceph-mon[28054]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:09.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:09 smithi001 ceph-mon[28054]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T21:25:09.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:09 smithi001 ceph-mon[28054]: from='osd.9 [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:25:09.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:09 smithi001 ceph-mon[28054]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T21:25:09.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:09 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:09.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:09 smithi001 ceph-mon[28054]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:25:09.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:09 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:09.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:09 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:09 smithi038 ceph-mon[32724]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:09 smithi038 ceph-mon[32724]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T21:25:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:09 smithi038 ceph-mon[32724]: from='osd.9 [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:25:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:09 smithi038 ceph-mon[32724]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T21:25:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:09 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:09 smithi038 ceph-mon[32724]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:25:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:09 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:09 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:09.405 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:25:09.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:09 smithi186 ceph-mon[32991]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:09.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:09 smithi186 ceph-mon[32991]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-26T21:25:09.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:09 smithi186 ceph-mon[32991]: from='osd.9 [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:25:09.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:09 smithi186 ceph-mon[32991]: osdmap e67: 10 total, 9 up, 10 in 2024-04-26T21:25:09.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:09 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:09.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:09 smithi186 ceph-mon[32991]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-04-26T21:25:09.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:09 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:09.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:09 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:10.069 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:25:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:25:09.781+0000 7f88c5e00700 -1 osd.9 0 waiting for initial osdmap 2024-04-26T21:25:10.069 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:25:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:25:09.822+0000 7f88bb200700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:25:10.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:10 smithi038 ceph-mon[32724]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:25:10.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:10 smithi038 ceph-mon[32724]: osdmap e68: 10 total, 9 up, 10 in 2024-04-26T21:25:10.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:10.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:10 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:10.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:10 smithi038 ceph-mon[32724]: from='osd.9 ' entity='osd.9' 2024-04-26T21:25:10.398 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:10 smithi001 ceph-mon[28054]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:25:10.398 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:10 smithi001 ceph-mon[28054]: osdmap e68: 10 total, 9 up, 10 in 2024-04-26T21:25:10.398 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:10.398 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:10 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:10.398 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:10 smithi001 ceph-mon[28054]: from='osd.9 ' entity='osd.9' 2024-04-26T21:25:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:10 smithi186 ceph-mon[32991]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-04-26T21:25:10.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:10 smithi186 ceph-mon[32991]: osdmap e68: 10 total, 9 up, 10 in 2024-04-26T21:25:10.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:10.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:10 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:10.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:10 smithi186 ceph-mon[32991]: from='osd.9 ' entity='osd.9' 2024-04-26T21:25:11.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:11 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:25:11.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:11 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:25:11.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:11 smithi038 ceph-mon[32724]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:11.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:11.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:11 smithi038 ceph-mon[32724]: osd.9 [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] boot 2024-04-26T21:25:11.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:11 smithi038 ceph-mon[32724]: osdmap e69: 10 total, 10 up, 10 in 2024-04-26T21:25:11.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:11 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:11.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:11 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:25:11.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:11 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:25:11.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:11 smithi186 ceph-mon[32991]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:11.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:11.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:11 smithi186 ceph-mon[32991]: osd.9 [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] boot 2024-04-26T21:25:11.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:11 smithi186 ceph-mon[32991]: osdmap e69: 10 total, 10 up, 10 in 2024-04-26T21:25:11.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:11 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:11.530 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:11 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:25:11.531 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:11 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:25:11.531 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:11 smithi001 ceph-mon[28054]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 273 MiB used, 804 GiB / 805 GiB avail 2024-04-26T21:25:11.531 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:11.531 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:11 smithi001 ceph-mon[28054]: osd.9 [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] boot 2024-04-26T21:25:11.531 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:11 smithi001 ceph-mon[28054]: osdmap e69: 10 total, 10 up, 10 in 2024-04-26T21:25:11.531 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:11 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:25:11.665 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T21:25:11.665 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T21:25:11.665 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-04-26T21:25:11.665 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-04-26T21:25:11.665 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115732 s, 906 MB/s 2024-04-26T21:25:11.665 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 2024-04-26T21:25:11.665 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-04-26T21:25:12.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi038:vg_nvme/lv_1 2024-04-26T21:25:12.751 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:25:13.175 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:13 smithi038 ceph-mon[32724]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:13.175 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:13 smithi038 ceph-mon[32724]: osdmap e70: 10 total, 10 up, 10 in 2024-04-26T21:25:13.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:13 smithi186 ceph-mon[32991]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:13.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:13 smithi186 ceph-mon[32991]: osdmap e70: 10 total, 10 up, 10 in 2024-04-26T21:25:13.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:13 smithi001 ceph-mon[28054]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:13.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:13 smithi001 ceph-mon[28054]: osdmap e70: 10 total, 10 up, 10 in 2024-04-26T21:25:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.588+0000 7f8c7a01a700 1 -- 172.21.15.38:0/3194937570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c741035a0 msgr2=0x7f8c74103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.588+0000 7f8c7a01a700 1 --2- 172.21.15.38:0/3194937570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c741035a0 0x7f8c74103a00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8c680099c0 tx=0x7f8c68009cd0 comp rx=0 tx=0).stop 2024-04-26T21:25:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.588+0000 7f8c7a01a700 1 -- 172.21.15.38:0/3194937570 shutdown_connections 2024-04-26T21:25:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.588+0000 7f8c7a01a700 1 --2- 172.21.15.38:0/3194937570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74103f40 0x7f8c7410a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.588+0000 7f8c7a01a700 1 --2- 172.21.15.38:0/3194937570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c741023a0 0x7f8c741027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.588+0000 7f8c7a01a700 1 --2- 172.21.15.38:0/3194937570 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c741035a0 0x7f8c74103a00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.588+0000 7f8c7a01a700 1 -- 172.21.15.38:0/3194937570 >> 172.21.15.38:0/3194937570 conn(0x7f8c740fd940 msgr2=0x7f8c740ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.588+0000 7f8c7a01a700 1 -- 172.21.15.38:0/3194937570 shutdown_connections 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.588+0000 7f8c7a01a700 1 -- 172.21.15.38:0/3194937570 wait complete. 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c7a01a700 1 Processor -- start 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c7a01a700 1 -- start start 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c7a01a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c741023a0 0x7f8c74199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c7a01a700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c741035a0 0x7f8c74199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c7a01a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74103f40 0x7f8c741a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c7a01a700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7410df20 con 0x7f8c741035a0 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c7a01a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7410dda0 con 0x7f8c741023a0 2024-04-26T21:25:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c7a01a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c7410e0a0 con 0x7f8c74103f40 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c737fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c741023a0 0x7f8c74199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c737fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c741023a0 0x7f8c74199700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47380/0 (socket says 172.21.15.38:47380) 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c737fe700 1 -- 172.21.15.38:0/991398455 learned_addr learned my addr 172.21.15.38:0/991398455 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c73fff700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74103f40 0x7f8c741a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c737fe700 1 -- 172.21.15.38:0/991398455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74103f40 msgr2=0x7f8c741a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c737fe700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74103f40 0x7f8c741a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c737fe700 1 -- 172.21.15.38:0/991398455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c741035a0 msgr2=0x7f8c74199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c737fe700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c741035a0 0x7f8c74199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.589+0000 7f8c737fe700 1 -- 172.21.15.38:0/991398455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c741a12f0 con 0x7f8c741023a0 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.590+0000 7f8c737fe700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c741023a0 0x7f8c74199700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400b500 tx=0x7f8c6400b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.590+0000 7f8c70ff9700 1 -- 172.21.15.38:0/991398455 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64015020 con 0x7f8c741023a0 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.590+0000 7f8c70ff9700 1 -- 172.21.15.38:0/991398455 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c64011ce0 con 0x7f8c741023a0 2024-04-26T21:25:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.590+0000 7f8c7a01a700 1 -- 172.21.15.38:0/991398455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c741a15e0 con 0x7f8c741023a0 2024-04-26T21:25:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.590+0000 7f8c70ff9700 1 -- 172.21.15.38:0/991398455 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c64010d50 con 0x7f8c741023a0 2024-04-26T21:25:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.590+0000 7f8c7a01a700 1 -- 172.21.15.38:0/991398455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c741a1b20 con 0x7f8c741023a0 2024-04-26T21:25:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.591+0000 7f8c7a01a700 1 -- 172.21.15.38:0/991398455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c7406a110 con 0x7f8c741023a0 2024-04-26T21:25:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.592+0000 7f8c70ff9700 1 -- 172.21.15.38:0/991398455 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f8c64018020 con 0x7f8c741023a0 2024-04-26T21:25:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.593+0000 7f8c70ff9700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8c5c0a3c40 0x7f8c5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.593+0000 7f8c70ff9700 1 -- 172.21.15.38:0/991398455 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6261+0+0 (secure 0 0 0) 0x7f8c640cf910 con 0x7f8c741023a0 2024-04-26T21:25:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.593+0000 7f8c72ffd700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8c5c0a3c40 0x7f8c5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.594+0000 7f8c72ffd700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8c5c0a3c40 0x7f8c5c0a6100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8c68003790 tx=0x7f8c68009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.596+0000 7f8c70ff9700 1 -- 172.21.15.38:0/991398455 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8c6401f020 con 0x7f8c741023a0 2024-04-26T21:25:14.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:14.814+0000 7f8c7a01a700 1 -- 172.21.15.38:0/991398455 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f8c740ffb20 con 0x7f8c5c0a3c40 2024-04-26T21:25:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:15 smithi001 ceph-mon[28054]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:15 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:25:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:15 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:25:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:15 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:15 smithi186 ceph-mon[32991]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:15 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:25:15.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:15 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:25:15.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:15 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:15.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:15 smithi038 ceph-mon[32724]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:15.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:15 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:25:15.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:15 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:25:15.488 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:15 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi001 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi001 to 4273M 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:16.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:16.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:16.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:16.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:16.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:16.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi001 2024-04-26T21:25:16.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.492 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi001 to 4273M 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:16.493 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:16.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi001 2024-04-26T21:25:16.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi001 to 4273M 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:16.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:17.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:17 smithi186 ceph-mon[32991]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:17.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:17 smithi001 ceph-mon[28054]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:17.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:17 smithi038 ceph-mon[32724]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:18.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:18 smithi001 ceph-mon[28054]: from='client.? 172.21.15.38:0/3788176204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03733579-57ef-431d-a55f-d01a168480a2"}]: dispatch 2024-04-26T21:25:18.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:18 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03733579-57ef-431d-a55f-d01a168480a2"}]: dispatch 2024-04-26T21:25:18.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:18 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03733579-57ef-431d-a55f-d01a168480a2"}]': finished 2024-04-26T21:25:18.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:18 smithi001 ceph-mon[28054]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T21:25:18.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:18 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:18 smithi186 ceph-mon[32991]: from='client.? 172.21.15.38:0/3788176204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03733579-57ef-431d-a55f-d01a168480a2"}]: dispatch 2024-04-26T21:25:18.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:18 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03733579-57ef-431d-a55f-d01a168480a2"}]: dispatch 2024-04-26T21:25:18.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:18 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03733579-57ef-431d-a55f-d01a168480a2"}]': finished 2024-04-26T21:25:18.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:18 smithi186 ceph-mon[32991]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T21:25:18.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:18 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:18.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:18 smithi038 ceph-mon[32724]: from='client.? 172.21.15.38:0/3788176204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03733579-57ef-431d-a55f-d01a168480a2"}]: dispatch 2024-04-26T21:25:18.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:18 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "03733579-57ef-431d-a55f-d01a168480a2"}]: dispatch 2024-04-26T21:25:18.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:18 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "03733579-57ef-431d-a55f-d01a168480a2"}]': finished 2024-04-26T21:25:18.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:18 smithi038 ceph-mon[32724]: osdmap e71: 11 total, 10 up, 11 in 2024-04-26T21:25:18.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:18 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:19.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:19 smithi186 ceph-mon[32991]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:19.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:19 smithi186 ceph-mon[32991]: from='client.? 172.21.15.38:0/4261543932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:25:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:19 smithi001 ceph-mon[28054]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:19 smithi001 ceph-mon[28054]: from='client.? 172.21.15.38:0/4261543932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:25:19.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:19 smithi038 ceph-mon[32724]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:19.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:19 smithi038 ceph-mon[32724]: from='client.? 172.21.15.38:0/4261543932' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:25:21.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:21 smithi186 ceph-mon[32991]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:21.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:21 smithi001 ceph-mon[28054]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:21.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:21 smithi038 ceph-mon[32724]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:23.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:23 smithi038 ceph-mon[32724]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:23.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:23 smithi186 ceph-mon[32991]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:23.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:23 smithi001 ceph-mon[28054]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:25.211 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:25 smithi038 ceph-mon[32724]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:25.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:25 smithi186 ceph-mon[32991]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:25.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:25 smithi001 ceph-mon[28054]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:26.205 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:26 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T21:25:26.205 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:26 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:26 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T21:25:26.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:26 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:26 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-26T21:25:26.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:26 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:27.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:27 smithi186 ceph-mon[32991]: Deploying daemon osd.10 on smithi038 2024-04-26T21:25:27.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:27 smithi186 ceph-mon[32991]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:27.496 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:27 smithi038 ceph-mon[32724]: Deploying daemon osd.10 on smithi038 2024-04-26T21:25:27.497 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:27 smithi038 ceph-mon[32724]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:27.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:27 smithi001 ceph-mon[28054]: Deploying daemon osd.10 on smithi038 2024-04-26T21:25:27.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:27 smithi001 ceph-mon[28054]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:28 smithi186 ceph-mon[32991]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:28.457 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:28 smithi038 ceph-mon[32724]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:28.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:28 smithi001 ceph-mon[28054]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:30.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:30 smithi038 ceph-mon[32724]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:30.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:30 smithi186 ceph-mon[32991]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:30.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:30 smithi001 ceph-mon[28054]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:31 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:32.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:31 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:32.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:31 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:32.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:31 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:32.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:31 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:32.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:31 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:32.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:31 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:32.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:31 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:32.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:31 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:33.115 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:32 smithi038 ceph-mon[32724]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:33.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:32 smithi186 ceph-mon[32991]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:33.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:32 smithi001 ceph-mon[28054]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:34.755 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.754+0000 7f8c70ff9700 1 -- 172.21.15.38:0/991398455 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f8c740ffb20 con 0x7f8c5c0a3c40 2024-04-26T21:25:34.756 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 10 on host 'smithi038' 2024-04-26T21:25:34.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.756+0000 7f8c7a01a700 1 -- 172.21.15.38:0/991398455 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8c5c0a3c40 msgr2=0x7f8c5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:34.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.756+0000 7f8c7a01a700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8c5c0a3c40 0x7f8c5c0a6100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8c68003790 tx=0x7f8c68009f10 comp rx=0 tx=0).stop 2024-04-26T21:25:34.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.756+0000 7f8c7a01a700 1 -- 172.21.15.38:0/991398455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c741023a0 msgr2=0x7f8c74199700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:34.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.756+0000 7f8c7a01a700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c741023a0 0x7f8c74199700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400b500 tx=0x7f8c6400b810 comp rx=0 tx=0).stop 2024-04-26T21:25:34.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.756+0000 7f8c7a01a700 1 -- 172.21.15.38:0/991398455 shutdown_connections 2024-04-26T21:25:34.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.756+0000 7f8c7a01a700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8c5c0a3c40 0x7f8c5c0a6100 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:34.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.756+0000 7f8c7a01a700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c74103f40 0x7f8c741a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:34.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.756+0000 7f8c7a01a700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c741023a0 0x7f8c74199700 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:34.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.756+0000 7f8c7a01a700 1 --2- 172.21.15.38:0/991398455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c741035a0 0x7f8c74199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:34.758 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.756+0000 7f8c7a01a700 1 -- 172.21.15.38:0/991398455 >> 172.21.15.38:0/991398455 conn(0x7f8c740fd940 msgr2=0x7f8c740ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:34.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.757+0000 7f8c7a01a700 1 -- 172.21.15.38:0/991398455 shutdown_connections 2024-04-26T21:25:34.759 INFO:teuthology.orchestra.run.smithi038.stderr:2024-04-26T21:25:34.757+0000 7f8c7a01a700 1 -- 172.21.15.38:0/991398455 wait complete. 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: from='osd.10 [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:25:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:34 smithi186 ceph-mon[32991]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:25:35.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:35.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:35.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:35.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:35.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: from='osd.10 [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:25:35.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:34 smithi001 ceph-mon[28054]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: from='osd.10 [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:25:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:34 smithi038 ceph-mon[32724]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-26T21:25:35.437 DEBUG:teuthology.orchestra.run.smithi038:osd.10> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.10.service 2024-04-26T21:25:35.439 INFO:tasks.cephadm:Deploying osd.11 on smithi186 with /dev/vg_nvme/lv_1... 2024-04-26T21:25:35.440 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T21:25:35.687 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:25:36.948 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:36 smithi038 ceph-mon[32724]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:36.948 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:36 smithi038 ceph-mon[32724]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T21:25:36.948 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:36 smithi038 ceph-mon[32724]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T21:25:36.948 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:36.948 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:36 smithi038 ceph-mon[32724]: from='osd.10 [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:25:36.948 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:36 smithi038 ceph-mon[32724]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:25:36.949 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:25:36 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:25:36.743+0000 7fa1d5188700 -1 osd.10 0 waiting for initial osdmap 2024-04-26T21:25:36.949 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:25:36 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:25:36.756+0000 7fa1d0f7b700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:25:37.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:36 smithi001 ceph-mon[28054]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:37.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:36 smithi001 ceph-mon[28054]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T21:25:37.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:36 smithi001 ceph-mon[28054]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T21:25:37.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:37.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:36 smithi001 ceph-mon[28054]: from='osd.10 [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:25:37.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:36 smithi001 ceph-mon[28054]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:25:37.125 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:36 smithi186 ceph-mon[32991]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:37.125 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:36 smithi186 ceph-mon[32991]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-26T21:25:37.125 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:36 smithi186 ceph-mon[32991]: osdmap e72: 11 total, 10 up, 11 in 2024-04-26T21:25:37.125 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:37.125 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:36 smithi186 ceph-mon[32991]: from='osd.10 [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:25:37.125 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:36 smithi186 ceph-mon[32991]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-04-26T21:25:37.862 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T21:25:37.862 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T21:25:37.862 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-26T21:25:37.862 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-26T21:25:37.862 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114393 s, 917 MB/s 2024-04-26T21:25:37.862 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-26T21:25:38.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:37 smithi001 ceph-mon[28054]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:25:38.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:37 smithi001 ceph-mon[28054]: osdmap e73: 11 total, 10 up, 11 in 2024-04-26T21:25:38.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:38.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:38.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:37 smithi038 ceph-mon[32724]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:25:38.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:37 smithi038 ceph-mon[32724]: osdmap e73: 11 total, 10 up, 11 in 2024-04-26T21:25:38.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:38.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:38.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:37 smithi186 ceph-mon[32991]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-04-26T21:25:38.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:37 smithi186 ceph-mon[32991]: osdmap e73: 11 total, 10 up, 11 in 2024-04-26T21:25:38.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:38.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:38.595 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_1 2024-04-26T21:25:38.750 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:25:38.750 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:25:38.750 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:38.750 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:38.750 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: osd.10 [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] boot 2024-04-26T21:25:38.846 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:25:38.875 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:25:38.875 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:25:38.875 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:38.875 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: osd.10 [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] boot 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:38.876 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:38 smithi186 ceph-mon[32991]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T21:25:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:25:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:25:39.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: osd.10 [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] boot 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:39.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:39.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:38 smithi001 ceph-mon[28054]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T21:25:39.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: osdmap e74: 11 total, 11 up, 11 in 2024-04-26T21:25:39.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:25:39.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:39.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:39.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:39.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:39.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:39.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:25:39.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:39.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:39.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:25:39.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:39.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:38 smithi038 ceph-mon[32724]: osdmap e75: 11 total, 11 up, 11 in 2024-04-26T21:25:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:39 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi038 2024-04-26T21:25:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:39 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-26T21:25:40.017 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:39 smithi186 ceph-mon[32991]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T21:25:40.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:39 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi038 2024-04-26T21:25:40.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:39 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-26T21:25:40.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:39 smithi001 ceph-mon[28054]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T21:25:40.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:39 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi038 2024-04-26T21:25:40.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:39 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi038 to 4273M 2024-04-26T21:25:40.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:39 smithi038 ceph-mon[32724]: osdmap e76: 11 total, 11 up, 11 in 2024-04-26T21:25:40.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.548+0000 7fac227b4700 1 -- 172.21.15.186:0/3637722125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac1c101770 msgr2=0x7fac1c101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:40.551 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.548+0000 7fac227b4700 1 --2- 172.21.15.186:0/3637722125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac1c101770 0x7fac1c101b90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fac0c009990 tx=0x7fac0c009ca0 comp rx=0 tx=0).stop 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 -- 172.21.15.186:0/3637722125 shutdown_connections 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 --2- 172.21.15.186:0/3637722125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac1c101770 0x7fac1c101b90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 --2- 172.21.15.186:0/3637722125 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac1c1050f0 0x7fac1c10b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 --2- 172.21.15.186:0/3637722125 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c104750 0x7fac1c104bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 -- 172.21.15.186:0/3637722125 >> 172.21.15.186:0/3637722125 conn(0x7fac1c0fd0d0 msgr2=0x7fac1c0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 -- 172.21.15.186:0/3637722125 shutdown_connections 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 -- 172.21.15.186:0/3637722125 wait complete. 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 Processor -- start 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 -- start start 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac1c101770 0x7fac1c1040b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.549+0000 7fac227b4700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c104750 0x7fac1c102700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac227b4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac1c1050f0 0x7fac1c102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:40.552 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac227b4700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c10efc0 con 0x7fac1c104750 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac227b4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c10ee40 con 0x7fac1c101770 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac227b4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c10f140 con 0x7fac1c1050f0 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac20d51700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac1c1050f0 0x7fac1c102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac20d51700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac1c1050f0 0x7fac1c102c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:34016/0 (socket says 172.21.15.186:34016) 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac20d51700 1 -- 172.21.15.186:0/1041744158 learned_addr learned my addr 172.21.15.186:0/1041744158 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac1bfff700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac1c101770 0x7fac1c1040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac20d51700 1 -- 172.21.15.186:0/1041744158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac1c101770 msgr2=0x7fac1c1040b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac20d51700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac1c101770 0x7fac1c1040b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac20d51700 1 -- 172.21.15.186:0/1041744158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c104750 msgr2=0x7fac1c102700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac20d51700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c104750 0x7fac1c102700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac20d51700 1 -- 172.21.15.186:0/1041744158 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1c1034f0 con 0x7fac1c1050f0 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac20d51700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac1c1050f0 0x7fac1c102c40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac14007ee0 tx=0x7fac1400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac197fa700 1 -- 172.21.15.186:0/1041744158 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1400dbd0 con 0x7fac1c1050f0 2024-04-26T21:25:40.553 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac227b4700 1 -- 172.21.15.186:0/1041744158 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac1c1a37c0 con 0x7fac1c1050f0 2024-04-26T21:25:40.554 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac197fa700 1 -- 172.21.15.186:0/1041744158 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac1400dd70 con 0x7fac1c1050f0 2024-04-26T21:25:40.554 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac197fa700 1 -- 172.21.15.186:0/1041744158 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14005990 con 0x7fac1c1050f0 2024-04-26T21:25:40.554 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.550+0000 7fac227b4700 1 -- 172.21.15.186:0/1041744158 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac1c1a3d50 con 0x7fac1c1050f0 2024-04-26T21:25:40.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.551+0000 7fac227b4700 1 -- 172.21.15.186:0/1041744158 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac1c06a110 con 0x7fac1c1050f0 2024-04-26T21:25:40.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.552+0000 7fac197fa700 1 -- 172.21.15.186:0/1041744158 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fac1401a020 con 0x7fac1c1050f0 2024-04-26T21:25:40.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.553+0000 7fac197fa700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fac040a3c40 0x7fac040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:25:40.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.553+0000 7fac197fa700 1 -- 172.21.15.186:0/1041744158 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6723+0+0 (secure 0 0 0) 0x7fac140cf560 con 0x7fac1c1050f0 2024-04-26T21:25:40.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.553+0000 7fac1bfff700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fac040a3c40 0x7fac040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:25:40.555 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.554+0000 7fac1bfff700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fac040a3c40 0x7fac040a6100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0038a0 tx=0x7fac0c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:25:40.558 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.558+0000 7fac197fa700 1 -- 172.21.15.186:0/1041744158 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fac14010070 con 0x7fac1c1050f0 2024-04-26T21:25:40.767 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:25:40.767+0000 7fac227b4700 1 -- 172.21.15.186:0/1041744158 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fac1c063920 con 0x7fac040a3c40 2024-04-26T21:25:41.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:40 smithi001 ceph-mon[28054]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:41.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:40 smithi001 ceph-mon[28054]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T21:25:41.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:40 smithi038 ceph-mon[32724]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:41.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:40 smithi038 ceph-mon[32724]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T21:25:41.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:40 smithi186 ceph-mon[32991]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:41.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:40 smithi186 ceph-mon[32991]: osdmap e77: 11 total, 11 up, 11 in 2024-04-26T21:25:41.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:41 smithi186 ceph-mon[32991]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:25:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:25:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:41 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:41.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:41 smithi186 ceph-mon[32991]: osdmap e78: 11 total, 11 up, 11 in 2024-04-26T21:25:42.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:41 smithi001 ceph-mon[28054]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:42.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:25:42.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:25:42.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:41 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:42.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:41 smithi001 ceph-mon[28054]: osdmap e78: 11 total, 11 up, 11 in 2024-04-26T21:25:42.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:41 smithi038 ceph-mon[32724]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:25:42.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T21:25:42.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T21:25:42.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:41 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:42.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:41 smithi038 ceph-mon[32724]: osdmap e78: 11 total, 11 up, 11 in 2024-04-26T21:25:43.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:42 smithi001 ceph-mon[28054]: pgmap v270: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:25:43.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:42 smithi038 ceph-mon[32724]: pgmap v270: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:25:43.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:42 smithi186 ceph-mon[32991]: pgmap v270: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-26T21:25:44.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:43 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/2190011434' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb81b189-ee25-44fd-bd06-ba73245f065c"}]: dispatch 2024-04-26T21:25:44.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:43 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb81b189-ee25-44fd-bd06-ba73245f065c"}]: dispatch 2024-04-26T21:25:44.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:43 smithi001 ceph-mon[28054]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb81b189-ee25-44fd-bd06-ba73245f065c"}]': finished 2024-04-26T21:25:44.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:43 smithi001 ceph-mon[28054]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T21:25:44.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:43 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:25:44.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:43 smithi038 ceph-mon[32724]: from='client.? 172.21.15.186:0/2190011434' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb81b189-ee25-44fd-bd06-ba73245f065c"}]: dispatch 2024-04-26T21:25:44.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:43 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb81b189-ee25-44fd-bd06-ba73245f065c"}]: dispatch 2024-04-26T21:25:44.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:43 smithi038 ceph-mon[32724]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb81b189-ee25-44fd-bd06-ba73245f065c"}]': finished 2024-04-26T21:25:44.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:43 smithi038 ceph-mon[32724]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T21:25:44.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:43 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:25:44.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:43 smithi186 ceph-mon[32991]: from='client.? 172.21.15.186:0/2190011434' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb81b189-ee25-44fd-bd06-ba73245f065c"}]: dispatch 2024-04-26T21:25:44.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:43 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb81b189-ee25-44fd-bd06-ba73245f065c"}]: dispatch 2024-04-26T21:25:44.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:43 smithi186 ceph-mon[32991]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb81b189-ee25-44fd-bd06-ba73245f065c"}]': finished 2024-04-26T21:25:44.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:43 smithi186 ceph-mon[32991]: osdmap e79: 12 total, 11 up, 12 in 2024-04-26T21:25:44.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:43 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:25:45.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:44 smithi001 ceph-mon[28054]: pgmap v272: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-26T21:25:45.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:44 smithi001 ceph-mon[28054]: from='client.? 172.21.15.186:0/1762432595' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:25:45.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:44 smithi038 ceph-mon[32724]: pgmap v272: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-26T21:25:45.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:44 smithi038 ceph-mon[32724]: from='client.? 172.21.15.186:0/1762432595' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:25:45.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:44 smithi186 ceph-mon[32991]: pgmap v272: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-04-26T21:25:45.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:44 smithi186 ceph-mon[32991]: from='client.? 172.21.15.186:0/1762432595' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T21:25:47.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:46 smithi001 ceph-mon[28054]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-26T21:25:47.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:46 smithi038 ceph-mon[32724]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-26T21:25:47.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:46 smithi186 ceph-mon[32991]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-26T21:25:48.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:48 smithi186 ceph-mon[32991]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:49.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:48 smithi001 ceph-mon[28054]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:49.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:48 smithi038 ceph-mon[32724]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:51.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:50 smithi001 ceph-mon[28054]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:50 smithi038 ceph-mon[32724]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:51.197 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:50 smithi186 ceph-mon[32991]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:52.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:51 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T21:25:52.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:51 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:52.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:51 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T21:25:52.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:51 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:52.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:51 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-26T21:25:52.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:51 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:25:53.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:52 smithi001 ceph-mon[28054]: Deploying daemon osd.11 on smithi186 2024-04-26T21:25:53.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:52 smithi001 ceph-mon[28054]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:53.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:52 smithi038 ceph-mon[32724]: Deploying daemon osd.11 on smithi186 2024-04-26T21:25:53.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:52 smithi038 ceph-mon[32724]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:53.080 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:52 smithi186 ceph-mon[32991]: Deploying daemon osd.11 on smithi186 2024-04-26T21:25:53.081 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:52 smithi186 ceph-mon[32991]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:54.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:54 smithi186 ceph-mon[32991]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:55.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:54 smithi001 ceph-mon[28054]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:55.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:54 smithi038 ceph-mon[32724]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:56.978 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:56 smithi186 ceph-mon[32991]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:57.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:56 smithi038 ceph-mon[32724]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:57.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:56 smithi001 ceph-mon[28054]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:58 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:58.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:58 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:58.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:58 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:58 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:58.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:58 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:58.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:58 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:58.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:58 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:58.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:58 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:25:58.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:58 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:25:59.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:25:59 smithi186 ceph-mon[32991]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:59.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:25:59 smithi001 ceph-mon[28054]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:25:59.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:25:59 smithi038 ceph-mon[32724]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:26:01.217 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 11 on host 'smithi186' 2024-04-26T21:26:01.218 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.216+0000 7fac197fa700 1 -- 172.21.15.186:0/1041744158 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fac1c063920 con 0x7fac040a3c40 2024-04-26T21:26:01.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.218+0000 7fac227b4700 1 -- 172.21.15.186:0/1041744158 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fac040a3c40 msgr2=0x7fac040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:01.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.218+0000 7fac227b4700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fac040a3c40 0x7fac040a6100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0038a0 tx=0x7fac0c005fb0 comp rx=0 tx=0).stop 2024-04-26T21:26:01.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.218+0000 7fac227b4700 1 -- 172.21.15.186:0/1041744158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac1c1050f0 msgr2=0x7fac1c102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:01.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.218+0000 7fac227b4700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac1c1050f0 0x7fac1c102c40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac14007ee0 tx=0x7fac1400d3b0 comp rx=0 tx=0).stop 2024-04-26T21:26:01.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.218+0000 7fac227b4700 1 -- 172.21.15.186:0/1041744158 shutdown_connections 2024-04-26T21:26:01.219 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.218+0000 7fac227b4700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fac040a3c40 0x7fac040a6100 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:01.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.218+0000 7fac227b4700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac1c1050f0 0x7fac1c102c40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:01.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.218+0000 7fac227b4700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac1c101770 0x7fac1c1040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:01.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.218+0000 7fac227b4700 1 --2- 172.21.15.186:0/1041744158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fac1c104750 0x7fac1c102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:01.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.218+0000 7fac227b4700 1 -- 172.21.15.186:0/1041744158 >> 172.21.15.186:0/1041744158 conn(0x7fac1c0fd0d0 msgr2=0x7fac1c107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:01.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.219+0000 7fac227b4700 1 -- 172.21.15.186:0/1041744158 shutdown_connections 2024-04-26T21:26:01.220 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:01.219+0000 7fac227b4700 1 -- 172.21.15.186:0/1041744158 wait complete. 2024-04-26T21:26:01.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:01 smithi186 ceph-mon[32991]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:26:01.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:01 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:01.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:01 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:01.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:01 smithi186 ceph-mon[32991]: from='osd.11 [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:26:01.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:01 smithi186 ceph-mon[32991]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:26:01.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:01 smithi001 ceph-mon[28054]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:26:01.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:01 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:01.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:01 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:01.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:01 smithi001 ceph-mon[28054]: from='osd.11 [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:26:01.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:01 smithi001 ceph-mon[28054]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:26:01.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:01 smithi038 ceph-mon[32724]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:26:01.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:01 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:01.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:01 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:01.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:01 smithi038 ceph-mon[32724]: from='osd.11 [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:26:01.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:01 smithi038 ceph-mon[32724]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-26T21:26:01.928 DEBUG:teuthology.orchestra.run.smithi186:osd.11> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.11.service 2024-04-26T21:26:01.930 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-26T21:26:01.930 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T21:26:02.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:02 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:02.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:02 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:02.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:02 smithi001 ceph-mon[28054]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:26:02.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:02 smithi001 ceph-mon[28054]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T21:26:02.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:02 smithi001 ceph-mon[28054]: from='osd.11 [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:26:02.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:02 smithi001 ceph-mon[28054]: osdmap e80: 12 total, 11 up, 12 in 2024-04-26T21:26:02.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:02 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:02.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:02 smithi001 ceph-mon[28054]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:26:02.401 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:26:02.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:02 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:02.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:02 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:02.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:02 smithi186 ceph-mon[32991]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:26:02.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:02 smithi186 ceph-mon[32991]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T21:26:02.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:02 smithi186 ceph-mon[32991]: from='osd.11 [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:26:02.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:02 smithi186 ceph-mon[32991]: osdmap e80: 12 total, 11 up, 12 in 2024-04-26T21:26:02.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:02 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:02.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:02 smithi186 ceph-mon[32991]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:26:02.456 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:26:02 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:26:02.327+0000 7f24a3d02700 -1 osd.11 0 waiting for initial osdmap 2024-04-26T21:26:02.456 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:26:02 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:26:02.340+0000 7f249faf5700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T21:26:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:02 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:02 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:02 smithi038 ceph-mon[32724]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:26:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:02 smithi038 ceph-mon[32724]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-26T21:26:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:02 smithi038 ceph-mon[32724]: from='osd.11 [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:26:02.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:02 smithi038 ceph-mon[32724]: osdmap e80: 12 total, 11 up, 12 in 2024-04-26T21:26:02.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:02 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:02.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:02 smithi038 ceph-mon[32724]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-26T21:26:03.424 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:03 smithi186 ceph-mon[32991]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:26:03.424 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:03 smithi186 ceph-mon[32991]: osdmap e81: 12 total, 11 up, 12 in 2024-04-26T21:26:03.424 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:03.424 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:03 smithi038 ceph-mon[32724]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:26:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:03 smithi038 ceph-mon[32724]: osdmap e81: 12 total, 11 up, 12 in 2024-04-26T21:26:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:03.593 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:03 smithi001 ceph-mon[28054]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-26T21:26:03.593 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:03 smithi001 ceph-mon[28054]: osdmap e81: 12 total, 11 up, 12 in 2024-04-26T21:26:03.593 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:03.593 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:04 smithi038 ceph-mon[32724]: purged_snaps scrub starts 2024-04-26T21:26:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:04 smithi038 ceph-mon[32724]: purged_snaps scrub ok 2024-04-26T21:26:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:04 smithi038 ceph-mon[32724]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:26:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:04 smithi038 ceph-mon[32724]: osd.11 [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] boot 2024-04-26T21:26:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:04 smithi038 ceph-mon[32724]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T21:26:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:04 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:04.601 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:04 smithi186 ceph-mon[32991]: purged_snaps scrub starts 2024-04-26T21:26:04.601 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:04 smithi186 ceph-mon[32991]: purged_snaps scrub ok 2024-04-26T21:26:04.601 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:04 smithi186 ceph-mon[32991]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:26:04.601 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:04.601 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:04 smithi186 ceph-mon[32991]: osd.11 [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] boot 2024-04-26T21:26:04.601 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:04 smithi186 ceph-mon[32991]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T21:26:04.601 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:04 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:04.644 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:04 smithi001 ceph-mon[28054]: purged_snaps scrub starts 2024-04-26T21:26:04.644 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:04 smithi001 ceph-mon[28054]: purged_snaps scrub ok 2024-04-26T21:26:04.644 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:04 smithi001 ceph-mon[28054]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-26T21:26:04.644 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:04.644 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:04 smithi001 ceph-mon[28054]: osd.11 [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] boot 2024-04-26T21:26:04.644 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:04 smithi001 ceph-mon[28054]: osdmap e82: 12 total, 12 up, 12 in 2024-04-26T21:26:04.644 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:04 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:26:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.817+0000 7f33de600700 1 -- 172.21.15.1:0/2733676233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33d8106840 msgr2=0x7f33d8108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.817+0000 7f33de600700 1 --2- 172.21.15.1:0/2733676233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33d8106840 0x7f33d8108c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f33c00098f0 tx=0x7f33c0009c00 comp rx=0 tx=0).stop 2024-04-26T21:26:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.817+0000 7f33de600700 1 -- 172.21.15.1:0/2733676233 shutdown_connections 2024-04-26T21:26:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.817+0000 7f33de600700 1 --2- 172.21.15.1:0/2733676233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33d8109170 0x7f33d810b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:04.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.817+0000 7f33de600700 1 --2- 172.21.15.1:0/2733676233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33d8103f10 0x7f33d8106300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.817+0000 7f33de600700 1 --2- 172.21.15.1:0/2733676233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33d8106840 0x7f33d8108c30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.817+0000 7f33de600700 1 -- 172.21.15.1:0/2733676233 >> 172.21.15.1:0/2733676233 conn(0x7f33d80fd920 msgr2=0x7f33d80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.817+0000 7f33de600700 1 -- 172.21.15.1:0/2733676233 shutdown_connections 2024-04-26T21:26:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.817+0000 7f33de600700 1 -- 172.21.15.1:0/2733676233 wait complete. 2024-04-26T21:26:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.819+0000 7f33de600700 1 Processor -- start 2024-04-26T21:26:04.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.819+0000 7f33de600700 1 -- start start 2024-04-26T21:26:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.819+0000 7f33de600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33d8103f10 0x7f33d8103870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.819+0000 7f33de600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33d8106840 0x7f33d8101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:04.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.819+0000 7f33de600700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33d8109170 0x7f33d8102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.819+0000 7f33de600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f33d810ef50 con 0x7f33d8103f10 2024-04-26T21:26:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.819+0000 7f33de600700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f33d810edd0 con 0x7f33d8106840 2024-04-26T21:26:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.819+0000 7f33de600700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f33d810f0d0 con 0x7f33d8109170 2024-04-26T21:26:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.820+0000 7f33d7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33d8103f10 0x7f33d8103870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.820+0000 7f33d7400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33d8103f10 0x7f33d8103870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57162/0 (socket says 172.21.15.1:57162) 2024-04-26T21:26:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.820+0000 7f33d7400700 1 -- 172.21.15.1:0/3844958750 learned_addr learned my addr 172.21.15.1:0/3844958750 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.820+0000 7f33cfe00700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33d8106840 0x7f33d8101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:04.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.820+0000 7f33d7e00700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33d8109170 0x7f33d8102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.820+0000 7f33d7e00700 1 -- 172.21.15.1:0/3844958750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33d8106840 msgr2=0x7f33d8101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.820+0000 7f33d7e00700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33d8106840 0x7f33d8101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.820+0000 7f33d7e00700 1 -- 172.21.15.1:0/3844958750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33d8103f10 msgr2=0x7f33d8103870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.820+0000 7f33d7e00700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33d8103f10 0x7f33d8103870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.820+0000 7f33d7e00700 1 -- 172.21.15.1:0/3844958750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d8102cb0 con 0x7f33d8109170 2024-04-26T21:26:04.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.821+0000 7f33d7e00700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33d8109170 0x7f33d8102400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f33d000ef90 tx=0x7f33d000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.821+0000 7f33d4c00700 1 -- 172.21.15.1:0/3844958750 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d00092e0 con 0x7f33d8109170 2024-04-26T21:26:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.821+0000 7f33d4c00700 1 -- 172.21.15.1:0/3844958750 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33d0004bb0 con 0x7f33d8109170 2024-04-26T21:26:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.821+0000 7f33d4c00700 1 -- 172.21.15.1:0/3844958750 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0004020 con 0x7f33d8109170 2024-04-26T21:26:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.821+0000 7f33de600700 1 -- 172.21.15.1:0/3844958750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d81a3810 con 0x7f33d8109170 2024-04-26T21:26:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.821+0000 7f33de600700 1 -- 172.21.15.1:0/3844958750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d81a3cd0 con 0x7f33d8109170 2024-04-26T21:26:04.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.823+0000 7f33de600700 1 -- 172.21.15.1:0/3844958750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33d806a110 con 0x7f33d8109170 2024-04-26T21:26:04.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.824+0000 7f33d4c00700 1 -- 172.21.15.1:0/3844958750 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f33d001b030 con 0x7f33d8109170 2024-04-26T21:26:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.826+0000 7f33d4c00700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33bc0a3c40 0x7f33bc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.826+0000 7f33d4c00700 1 -- 172.21.15.1:0/3844958750 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f33d00cfb30 con 0x7f33d8109170 2024-04-26T21:26:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.826+0000 7f33d7400700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33bc0a3c40 0x7f33bc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:04.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.827+0000 7f33d7400700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33bc0a3c40 0x7f33bc0a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f33c8005d10 tx=0x7f33c8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:04.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:04.830+0000 7f33d4c00700 1 -- 172.21.15.1:0/3844958750 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f33d0022020 con 0x7f33d8109170 2024-04-26T21:26:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.092+0000 7f33de600700 1 -- 172.21.15.1:0/3844958750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f33d8064030 con 0x7f33d8109170 2024-04-26T21:26:05.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.093+0000 7f33d4c00700 1 -- 172.21.15.1:0/3844958750 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v83) ==== 74+0+133 (secure 0 0 0) 0x7f33d0010040 con 0x7f33d8109170 2024-04-26T21:26:05.094 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.095+0000 7f33de600700 1 -- 172.21.15.1:0/3844958750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33bc0a3c40 msgr2=0x7f33bc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.095+0000 7f33de600700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33bc0a3c40 0x7f33bc0a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f33c8005d10 tx=0x7f33c8009310 comp rx=0 tx=0).stop 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.095+0000 7f33de600700 1 -- 172.21.15.1:0/3844958750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33d8109170 msgr2=0x7f33d8102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.095+0000 7f33de600700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33d8109170 0x7f33d8102400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f33d000ef90 tx=0x7f33d000c3a0 comp rx=0 tx=0).stop 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.095+0000 7f33de600700 1 -- 172.21.15.1:0/3844958750 shutdown_connections 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.096+0000 7f33de600700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33bc0a3c40 0x7f33bc0a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.096+0000 7f33de600700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33d8109170 0x7f33d8102400 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.096+0000 7f33de600700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33d8103f10 0x7f33d8103870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.096+0000 7f33de600700 1 --2- 172.21.15.1:0/3844958750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f33d8106840 0x7f33d8101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.096+0000 7f33de600700 1 -- 172.21.15.1:0/3844958750 >> 172.21.15.1:0/3844958750 conn(0x7f33d80fd920 msgr2=0x7f33d8105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.096+0000 7f33de600700 1 -- 172.21.15.1:0/3844958750 shutdown_connections 2024-04-26T21:26:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:05.096+0000 7f33de600700 1 -- 172.21.15.1:0/3844958750 wait complete. 2024-04-26T21:26:05.518 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T21:26:05.518 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: Detected new or changed devices on smithi186 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:05.519 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3844958750' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T21:26:05.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:05.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:05.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:05.520 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: Detected new or changed devices on smithi186 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3844958750' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T21:26:05.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:05.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:05.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:05.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: osdmap e83: 12 total, 12 up, 12 in 2024-04-26T21:26:05.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: Detected new or changed devices on smithi186 2024-04-26T21:26:05.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3844958750' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:05.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:05.927 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1714166763,"num_in_osds":12,"osd_in_since":1714166743,"num_remapped_pgs":0} 2024-04-26T21:26:05.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T21:26:06.351 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:26:07.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:07 smithi001 ceph-mon[28054]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:07 smithi038 ceph-mon[32724]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:07.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:07 smithi186 ceph-mon[32991]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.561+0000 7f9817200700 1 -- 172.21.15.1:0/1952963142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9810103590 msgr2=0x7f9810103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.561+0000 7f9817200700 1 --2- 172.21.15.1:0/1952963142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9810103590 0x7f9810103970 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9804009990 tx=0x7f9804009ca0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.561+0000 7f9817200700 1 -- 172.21.15.1:0/1952963142 shutdown_connections 2024-04-26T21:26:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.561+0000 7f9817200700 1 --2- 172.21.15.1:0/1952963142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9810108670 0x7f9810110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.561+0000 7f9817200700 1 --2- 172.21.15.1:0/1952963142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9810103590 0x7f9810103970 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.561+0000 7f9817200700 1 --2- 172.21.15.1:0/1952963142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9810103f40 0x7f9810107f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.561+0000 7f9817200700 1 -- 172.21.15.1:0/1952963142 >> 172.21.15.1:0/1952963142 conn(0x7f98100fee40 msgr2=0x7f9810101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:08.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.561+0000 7f9817200700 1 -- 172.21.15.1:0/1952963142 shutdown_connections 2024-04-26T21:26:08.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.561+0000 7f9817200700 1 -- 172.21.15.1:0/1952963142 wait complete. 2024-04-26T21:26:08.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.562+0000 7f9817200700 1 Processor -- start 2024-04-26T21:26:08.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.562+0000 7f9817200700 1 -- start start 2024-04-26T21:26:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9817200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9810103590 0x7f98101a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:08.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9817200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9810103f40 0x7f98101a1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9817200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9810108670 0x7f981019ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9817200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9810113500 con 0x7f9810108670 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9817200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9810113380 con 0x7f9810103f40 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9817200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9810113680 con 0x7f9810103590 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9816800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9810108670 0x7f981019ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9815e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9810103590 0x7f98101a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9816800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9810108670 0x7f981019ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45994/0 (socket says 172.21.15.1:45994) 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9816800700 1 -- 172.21.15.1:0/4035376174 learned_addr learned my addr 172.21.15.1:0/4035376174 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.563+0000 7f9815400700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9810103f40 0x7f98101a1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f9815e00700 1 -- 172.21.15.1:0/4035376174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9810103f40 msgr2=0x7f98101a1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:08.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f9815e00700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9810103f40 0x7f98101a1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f9815e00700 1 -- 172.21.15.1:0/4035376174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9810108670 msgr2=0x7f981019ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f9815e00700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9810108670 0x7f981019ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f9815e00700 1 -- 172.21.15.1:0/4035376174 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981019c1c0 con 0x7f9810103590 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f9816800700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9810108670 0x7f981019ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f9815400700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9810103f40 0x7f98101a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f9815e00700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9810103590 0x7f98101a18a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9804006010 tx=0x7f98040057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f980aa00700 1 -- 172.21.15.1:0/4035376174 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804027070 con 0x7f9810103590 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f9817200700 1 -- 172.21.15.1:0/4035376174 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f981019c450 con 0x7f9810103590 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f980aa00700 1 -- 172.21.15.1:0/4035376174 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f980400b610 con 0x7f9810103590 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f980aa00700 1 -- 172.21.15.1:0/4035376174 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9804022e30 con 0x7f9810103590 2024-04-26T21:26:08.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.564+0000 7f9817200700 1 -- 172.21.15.1:0/4035376174 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98101a8770 con 0x7f9810103590 2024-04-26T21:26:08.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.566+0000 7f9817200700 1 -- 172.21.15.1:0/4035376174 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f981006a110 con 0x7f9810103590 2024-04-26T21:26:08.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.567+0000 7f980aa00700 1 -- 172.21.15.1:0/4035376174 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f980402a070 con 0x7f9810103590 2024-04-26T21:26:08.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.569+0000 7f980aa00700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97f80a3cc0 0x7f97f80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:08.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.569+0000 7f980aa00700 1 -- 172.21.15.1:0/4035376174 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f98040d0d30 con 0x7f9810103590 2024-04-26T21:26:08.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.569+0000 7f9815400700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97f80a3cc0 0x7f97f80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:08.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.570+0000 7f9815400700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97f80a3cc0 0x7f97f80a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009680 tx=0x7f97fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:08.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.573+0000 7f980aa00700 1 -- 172.21.15.1:0/4035376174 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9804013070 con 0x7f9810103590 2024-04-26T21:26:08.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.807+0000 7f9817200700 1 -- 172.21.15.1:0/4035376174 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98101a8940 con 0x7f9810103590 2024-04-26T21:26:08.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.808+0000 7f980aa00700 1 -- 172.21.15.1:0/4035376174 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18300 (secure 0 0 0) 0x7f980409dba0 con 0x7f9810103590 2024-04-26T21:26:08.811 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:08.811 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":83,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:26:04.333876+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:26:08.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.812+0000 7f9817200700 1 -- 172.21.15.1:0/4035376174 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97f80a3cc0 msgr2=0x7f97f80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:08.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.812+0000 7f9817200700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97f80a3cc0 0x7f97f80a6180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009680 tx=0x7f97fc009260 comp rx=0 tx=0).stop 2024-04-26T21:26:08.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.813+0000 7f9817200700 1 -- 172.21.15.1:0/4035376174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9810103590 msgr2=0x7f98101a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:08.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.813+0000 7f9817200700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9810103590 0x7f98101a18a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9804006010 tx=0x7f98040057c0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.813+0000 7f9817200700 1 -- 172.21.15.1:0/4035376174 shutdown_connections 2024-04-26T21:26:08.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.813+0000 7f9817200700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f97f80a3cc0 0x7f97f80a6180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.813+0000 7f9817200700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9810103590 0x7f98101a18a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.813+0000 7f9817200700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9810108670 0x7f981019ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.813+0000 7f9817200700 1 --2- 172.21.15.1:0/4035376174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9810103f40 0x7f98101a1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.813+0000 7f9817200700 1 -- 172.21.15.1:0/4035376174 >> 172.21.15.1:0/4035376174 conn(0x7f98100fee40 msgr2=0x7f981010f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.813+0000 7f9817200700 1 -- 172.21.15.1:0/4035376174 shutdown_connections 2024-04-26T21:26:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:08.813+0000 7f9817200700 1 -- 172.21.15.1:0/4035376174 wait complete. 2024-04-26T21:26:09.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:09 smithi001 ceph-mon[28054]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:09.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:09 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4035376174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:26:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:09 smithi038 ceph-mon[32724]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:09 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4035376174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:26:09.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:09 smithi186 ceph-mon[32991]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:09.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:09 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4035376174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:26:09.612 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:26:09.612 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T21:26:10.078 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:26:11.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:11 smithi001 ceph-mon[28054]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:11.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:11 smithi038 ceph-mon[32724]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:11.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:11 smithi186 ceph-mon[32991]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:12.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.241+0000 7fd19e200700 1 -- 172.21.15.1:0/2592147493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1980760e0 msgr2=0x7fd198113e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:12.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.241+0000 7fd19e200700 1 --2- 172.21.15.1:0/2592147493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1980760e0 0x7fd198113e20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd184009990 tx=0x7fd184009ca0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.241+0000 7fd19e200700 1 -- 172.21.15.1:0/2592147493 shutdown_connections 2024-04-26T21:26:12.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.241+0000 7fd19e200700 1 --2- 172.21.15.1:0/2592147493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1980757c0 0x7fd198075ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.241+0000 7fd19e200700 1 --2- 172.21.15.1:0/2592147493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd198114360 0x7fd198116750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.241+0000 7fd19e200700 1 --2- 172.21.15.1:0/2592147493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1980760e0 0x7fd198113e20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.241+0000 7fd19e200700 1 -- 172.21.15.1:0/2592147493 >> 172.21.15.1:0/2592147493 conn(0x7fd1980feea0 msgr2=0x7fd1981012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:12.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.241+0000 7fd19e200700 1 -- 172.21.15.1:0/2592147493 shutdown_connections 2024-04-26T21:26:12.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.241+0000 7fd19e200700 1 -- 172.21.15.1:0/2592147493 wait complete. 2024-04-26T21:26:12.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.242+0000 7fd19e200700 1 Processor -- start 2024-04-26T21:26:12.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.242+0000 7fd19e200700 1 -- start start 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd19e200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1980757c0 0x7fd1981a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd19e200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1980760e0 0x7fd1981a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd19e200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd198114360 0x7fd1981a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd19e200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd198119bb0 con 0x7fd1980757c0 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd19e200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd198119a30 con 0x7fd198114360 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd19e200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd198119d30 con 0x7fd1980760e0 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd19ce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1980757c0 0x7fd1981a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd19ce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1980757c0 0x7fd1981a0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46010/0 (socket says 172.21.15.1:46010) 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd19ce00700 1 -- 172.21.15.1:0/245032178 learned_addr learned my addr 172.21.15.1:0/245032178 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd197e00700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1980760e0 0x7fd1981a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:12.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.243+0000 7fd19d800700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd198114360 0x7fd1981a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:12.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.244+0000 7fd19ce00700 1 -- 172.21.15.1:0/245032178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1980760e0 msgr2=0x7fd1981a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.244+0000 7fd19ce00700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1980760e0 0x7fd1981a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.244+0000 7fd19ce00700 1 -- 172.21.15.1:0/245032178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd198114360 msgr2=0x7fd1981a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.244+0000 7fd19ce00700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd198114360 0x7fd1981a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.244+0000 7fd19ce00700 1 -- 172.21.15.1:0/245032178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1981a5070 con 0x7fd1980757c0 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.244+0000 7fd19d800700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd198114360 0x7fd1981a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.244+0000 7fd19ce00700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1980757c0 0x7fd1981a0060 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd18c00e9c0 tx=0x7fd18c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.244+0000 7fd195600700 1 -- 172.21.15.1:0/245032178 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c00cbd0 con 0x7fd1980757c0 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.244+0000 7fd19e200700 1 -- 172.21.15.1:0/245032178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1981a5360 con 0x7fd1980757c0 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.244+0000 7fd195600700 1 -- 172.21.15.1:0/245032178 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd18c00cd70 con 0x7fd1980757c0 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.245+0000 7fd195600700 1 -- 172.21.15.1:0/245032178 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c005610 con 0x7fd1980757c0 2024-04-26T21:26:12.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.245+0000 7fd19e200700 1 -- 172.21.15.1:0/245032178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1981acbe0 con 0x7fd1980757c0 2024-04-26T21:26:12.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.246+0000 7fd19e200700 1 -- 172.21.15.1:0/245032178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd19806a110 con 0x7fd1980757c0 2024-04-26T21:26:12.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.249+0000 7fd195600700 1 -- 172.21.15.1:0/245032178 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd18c004750 con 0x7fd1980757c0 2024-04-26T21:26:12.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.250+0000 7fd195600700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1800a3cc0 0x7fd1800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:12.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.250+0000 7fd195600700 1 -- 172.21.15.1:0/245032178 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fd18c0d02b0 con 0x7fd1980757c0 2024-04-26T21:26:12.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.250+0000 7fd197e00700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1800a3cc0 0x7fd1800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:12.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.250+0000 7fd197e00700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1800a3cc0 0x7fd1800a6180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd1981015a0 tx=0x7fd18400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:12.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.254+0000 7fd195600700 1 -- 172.21.15.1:0/245032178 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd18c09d090 con 0x7fd1980757c0 2024-04-26T21:26:12.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.482+0000 7fd19e200700 1 -- 172.21.15.1:0/245032178 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1981a13c0 con 0x7fd1980757c0 2024-04-26T21:26:12.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.482+0000 7fd195600700 1 -- 172.21.15.1:0/245032178 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7fd18c09c990 con 0x7fd1980757c0 2024-04-26T21:26:12.484 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:26:12.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.485+0000 7fd19e200700 1 -- 172.21.15.1:0/245032178 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1800a3cc0 msgr2=0x7fd1800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.485+0000 7fd19e200700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1800a3cc0 0x7fd1800a6180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd1981015a0 tx=0x7fd18400b260 comp rx=0 tx=0).stop 2024-04-26T21:26:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.485+0000 7fd19e200700 1 -- 172.21.15.1:0/245032178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1980757c0 msgr2=0x7fd1981a0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.485+0000 7fd19e200700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1980757c0 0x7fd1981a0060 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd18c00e9c0 tx=0x7fd18c00ecd0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.486+0000 7fd19e200700 1 -- 172.21.15.1:0/245032178 shutdown_connections 2024-04-26T21:26:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.486+0000 7fd19e200700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1800a3cc0 0x7fd1800a6180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.486+0000 7fd19e200700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd198114360 0x7fd1981a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.486+0000 7fd19e200700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1980757c0 0x7fd1981a0060 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.486+0000 7fd19e200700 1 --2- 172.21.15.1:0/245032178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1980760e0 0x7fd1981a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:12.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.486+0000 7fd19e200700 1 -- 172.21.15.1:0/245032178 >> 172.21.15.1:0/245032178 conn(0x7fd1980feea0 msgr2=0x7fd198100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:12.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.486+0000 7fd19e200700 1 -- 172.21.15.1:0/245032178 shutdown_connections 2024-04-26T21:26:12.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:12.486+0000 7fd19e200700 1 -- 172.21.15.1:0/245032178 wait complete. 2024-04-26T21:26:13.259 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:13 smithi001 ceph-mon[28054]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:13.260 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:13 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/245032178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:26:13.324 INFO:tasks.cephadm:Adding mds.a on smithi001 2024-04-26T21:26:13.324 INFO:tasks.cephadm:Adding mds.d on smithi001 2024-04-26T21:26:13.324 INFO:tasks.cephadm:Adding mds.g on smithi001 2024-04-26T21:26:13.324 INFO:tasks.cephadm:Adding mds.j on smithi001 2024-04-26T21:26:13.324 INFO:tasks.cephadm:Adding mds.b on smithi038 2024-04-26T21:26:13.324 INFO:tasks.cephadm:Adding mds.e on smithi038 2024-04-26T21:26:13.324 INFO:tasks.cephadm:Adding mds.h on smithi038 2024-04-26T21:26:13.324 INFO:tasks.cephadm:Adding mds.k on smithi038 2024-04-26T21:26:13.325 INFO:tasks.cephadm:Adding mds.c on smithi186 2024-04-26T21:26:13.325 INFO:tasks.cephadm:Adding mds.f on smithi186 2024-04-26T21:26:13.325 INFO:tasks.cephadm:Adding mds.i on smithi186 2024-04-26T21:26:13.325 INFO:tasks.cephadm:Adding mds.l on smithi186 2024-04-26T21:26:13.325 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l' 2024-04-26T21:26:13.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:13 smithi038 ceph-mon[32724]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:13.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:13 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/245032178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:26:13.353 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:13 smithi186 ceph-mon[32991]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:13.353 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:13 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/245032178' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:26:13.574 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:26:15.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.258+0000 7fd203c7b700 1 -- 172.21.15.186:0/331607563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1fc105050 msgr2=0x7fd1fc10b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:15.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.258+0000 7fd203c7b700 1 --2- 172.21.15.186:0/331607563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1fc105050 0x7fd1fc10b900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd1f400b0a0 tx=0x7fd1f400b3b0 comp rx=0 tx=0).stop 2024-04-26T21:26:15.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.258+0000 7fd203c7b700 1 -- 172.21.15.186:0/331607563 shutdown_connections 2024-04-26T21:26:15.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.258+0000 7fd203c7b700 1 --2- 172.21.15.186:0/331607563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1fc105050 0x7fd1fc10b900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:15.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.258+0000 7fd203c7b700 1 --2- 172.21.15.186:0/331607563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1fc1034b0 0x7fd1fc1038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:15.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.258+0000 7fd203c7b700 1 --2- 172.21.15.186:0/331607563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1046b0 0x7fd1fc104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:15.261 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.258+0000 7fd203c7b700 1 -- 172.21.15.186:0/331607563 >> 172.21.15.186:0/331607563 conn(0x7fd1fc0fea70 msgr2=0x7fd1fc100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:15.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.258+0000 7fd203c7b700 1 -- 172.21.15.186:0/331607563 shutdown_connections 2024-04-26T21:26:15.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.258+0000 7fd203c7b700 1 -- 172.21.15.186:0/331607563 wait complete. 2024-04-26T21:26:15.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd203c7b700 1 Processor -- start 2024-04-26T21:26:15.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd203c7b700 1 -- start start 2024-04-26T21:26:15.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd203c7b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1fc1034b0 0x7fd1fc199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:15.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd203c7b700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1046b0 0x7fd1fc199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:15.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd203c7b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1fc105050 0x7fd1fc1a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:15.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd203c7b700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc10efc0 con 0x7fd1fc1046b0 2024-04-26T21:26:15.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd203c7b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc10ee40 con 0x7fd1fc105050 2024-04-26T21:26:15.262 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd203c7b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc10f140 con 0x7fd1fc1034b0 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd201a17700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1fc1034b0 0x7fd1fc199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd201a17700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1fc1034b0 0x7fd1fc199570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:56722/0 (socket says 172.21.15.186:56722) 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd201a17700 1 -- 172.21.15.186:0/4089775819 learned_addr learned my addr 172.21.15.186:0/4089775819 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd202218700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1fc105050 0x7fd1fc1a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd201a17700 1 -- 172.21.15.186:0/4089775819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1fc105050 msgr2=0x7fd1fc1a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd201a17700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1fc105050 0x7fd1fc1a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd201a17700 1 -- 172.21.15.186:0/4089775819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1046b0 msgr2=0x7fd1fc199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd201a17700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1046b0 0x7fd1fc199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd201a17700 1 -- 172.21.15.186:0/4089775819 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1fc075650 con 0x7fd1fc1034b0 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd202218700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1fc105050 0x7fd1fc1a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.259+0000 7fd201a17700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1fc1034b0 0x7fd1fc199570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00d9b0 tx=0x7fd1ec00dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:15.263 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.260+0000 7fd1f2ffd700 1 -- 172.21.15.186:0/4089775819 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec017070 con 0x7fd1fc1034b0 2024-04-26T21:26:15.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.260+0000 7fd203c7b700 1 -- 172.21.15.186:0/4089775819 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1fc075940 con 0x7fd1fc1034b0 2024-04-26T21:26:15.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.260+0000 7fd1f2ffd700 1 -- 172.21.15.186:0/4089775819 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1ec010460 con 0x7fd1fc1034b0 2024-04-26T21:26:15.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.260+0000 7fd1f2ffd700 1 -- 172.21.15.186:0/4089775819 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ec00f620 con 0x7fd1fc1034b0 2024-04-26T21:26:15.264 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.260+0000 7fd203c7b700 1 -- 172.21.15.186:0/4089775819 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1fc075eb0 con 0x7fd1fc1034b0 2024-04-26T21:26:15.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.261+0000 7fd203c7b700 1 -- 172.21.15.186:0/4089775819 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1fc06a110 con 0x7fd1fc1034b0 2024-04-26T21:26:15.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.262+0000 7fd1f2ffd700 1 -- 172.21.15.186:0/4089775819 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd1ec010610 con 0x7fd1fc1034b0 2024-04-26T21:26:15.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.262+0000 7fd1f2ffd700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1e80a3c40 0x7fd1e80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:15.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.262+0000 7fd1f2ffd700 1 -- 172.21.15.186:0/4089775819 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fd1ec0cfb80 con 0x7fd1fc1034b0 2024-04-26T21:26:15.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.263+0000 7fd201216700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1e80a3c40 0x7fd1e80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:15.265 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.264+0000 7fd201216700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1e80a3c40 0x7fd1e80a6100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd1f80097b0 tx=0x7fd1f8006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:15.265 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:15 smithi186 ceph-mon[32991]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:15.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.266+0000 7fd1f2ffd700 1 -- 172.21.15.186:0/4089775819 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd1ec09ca00 con 0x7fd1fc1034b0 2024-04-26T21:26:15.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:15 smithi001 ceph-mon[28054]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:15.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:15 smithi038 ceph-mon[32724]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:15.478 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.477+0000 7fd203c7b700 1 -- 172.21.15.186:0/4089775819 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}) -- 0x7fd1fc10a720 con 0x7fd1e80a3c40 2024-04-26T21:26:15.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.496+0000 7fd1f2ffd700 1 -- 172.21.15.186:0/4089775819 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd1fc10a720 con 0x7fd1e80a3c40 2024-04-26T21:26:15.497 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mds.all update... 2024-04-26T21:26:15.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.498+0000 7fd203c7b700 1 -- 172.21.15.186:0/4089775819 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1e80a3c40 msgr2=0x7fd1e80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:15.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.498+0000 7fd203c7b700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1e80a3c40 0x7fd1e80a6100 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd1f80097b0 tx=0x7fd1f8006d20 comp rx=0 tx=0).stop 2024-04-26T21:26:15.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.498+0000 7fd203c7b700 1 -- 172.21.15.186:0/4089775819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1fc1034b0 msgr2=0x7fd1fc199570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:15.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.498+0000 7fd203c7b700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1fc1034b0 0x7fd1fc199570 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd1ec00d9b0 tx=0x7fd1ec00dd70 comp rx=0 tx=0).stop 2024-04-26T21:26:15.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.498+0000 7fd203c7b700 1 -- 172.21.15.186:0/4089775819 shutdown_connections 2024-04-26T21:26:15.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.498+0000 7fd203c7b700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1e80a3c40 0x7fd1e80a6100 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:15.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.498+0000 7fd203c7b700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1fc1034b0 0x7fd1fc199570 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:15.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.498+0000 7fd203c7b700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1fc105050 0x7fd1fc1a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:15.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.498+0000 7fd203c7b700 1 --2- 172.21.15.186:0/4089775819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1fc1046b0 0x7fd1fc199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:15.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.498+0000 7fd203c7b700 1 -- 172.21.15.186:0/4089775819 >> 172.21.15.186:0/4089775819 conn(0x7fd1fc0fea70 msgr2=0x7fd1fc1079b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:15.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.499+0000 7fd203c7b700 1 -- 172.21.15.186:0/4089775819 shutdown_connections 2024-04-26T21:26:15.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T21:26:15.499+0000 7fd203c7b700 1 -- 172.21.15.186:0/4089775819 wait complete. 2024-04-26T21:26:16.261 DEBUG:teuthology.orchestra.run.smithi001:mds.a> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.a.service 2024-04-26T21:26:16.266 DEBUG:teuthology.orchestra.run.smithi001:mds.d> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.d.service 2024-04-26T21:26:16.313 DEBUG:teuthology.orchestra.run.smithi001:mds.g> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.g.service 2024-04-26T21:26:16.359 DEBUG:teuthology.orchestra.run.smithi001:mds.j> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.j.service 2024-04-26T21:26:16.405 DEBUG:teuthology.orchestra.run.smithi038:mds.b> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.b.service 2024-04-26T21:26:16.408 DEBUG:teuthology.orchestra.run.smithi038:mds.e> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.e.service 2024-04-26T21:26:16.451 DEBUG:teuthology.orchestra.run.smithi038:mds.h> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.h.service 2024-04-26T21:26:16.494 DEBUG:teuthology.orchestra.run.smithi038:mds.k> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.k.service 2024-04-26T21:26:16.537 DEBUG:teuthology.orchestra.run.smithi186:mds.c> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.c.service 2024-04-26T21:26:16.539 DEBUG:teuthology.orchestra.run.smithi186:mds.f> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.f.service 2024-04-26T21:26:16.582 DEBUG:teuthology.orchestra.run.smithi186:mds.i> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.i.service 2024-04-26T21:26:16.625 DEBUG:teuthology.orchestra.run.smithi186:mds.l> sudo journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.l.service 2024-04-26T21:26:16.668 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-26T21:26:16.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:26:16.748 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:16 smithi001 ceph-mon[28054]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:16.749 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:16 smithi001 ceph-mon[28054]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:16.749 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:16 smithi001 ceph-mon[28054]: Saving service mds.all spec with placement smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T21:26:16.749 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:16.749 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:16 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:16.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:16 smithi038 ceph-mon[32724]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:16.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:16 smithi038 ceph-mon[32724]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:16.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:16 smithi038 ceph-mon[32724]: Saving service mds.all spec with placement smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T21:26:16.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:16.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:16 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.904+0000 7f674b200640 1 -- 172.21.15.1:0/3690175351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10a940 msgr2=0x7f674c1111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.904+0000 7f674b200640 1 --2- 172.21.15.1:0/3690175351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10a940 0x7f674c1111d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6744009ca0 tx=0x7f674402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:26:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.904+0000 7f674b200640 1 -- 172.21.15.1:0/3690175351 shutdown_connections 2024-04-26T21:26:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.904+0000 7f674b200640 1 --2- 172.21.15.1:0/3690175351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10a940 0x7f674c1111d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.904+0000 7f674b200640 1 --2- 172.21.15.1:0/3690175351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c109f80 0x7f674c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.904+0000 7f674b200640 1 --2- 172.21.15.1:0/3690175351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f674c108d90 0x7f674c1091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.904+0000 7f674b200640 1 -- 172.21.15.1:0/3690175351 >> 172.21.15.1:0/3690175351 conn(0x7f674c077d30 msgr2=0x7f674c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.904+0000 7f674b200640 1 -- 172.21.15.1:0/3690175351 shutdown_connections 2024-04-26T21:26:16.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.904+0000 7f674b200640 1 -- 172.21.15.1:0/3690175351 wait complete. 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.905+0000 7f674b200640 1 Processor -- start 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.905+0000 7f674b200640 1 -- start start 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.905+0000 7f674b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c108d90 0x7f674c19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.905+0000 7f674b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f674c109f80 0x7f674c19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.905+0000 7f674b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c10a940 0x7f674c1a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.905+0000 7f674b200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f674c1148f0 con 0x7f674c109f80 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.905+0000 7f674b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f674c114770 con 0x7f674c10a940 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.905+0000 7f674b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f674c114a70 con 0x7f674c108d90 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6749e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c108d90 0x7f674c19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6743e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f674c109f80 0x7f674c19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:16.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6749e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c108d90 0x7f674c19ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:60746/0 (socket says 172.21.15.1:60746) 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6749e00640 1 -- 172.21.15.1:0/3813060596 learned_addr learned my addr 172.21.15.1:0/3813060596 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f674a800640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c10a940 0x7f674c1a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6749e00640 1 -- 172.21.15.1:0/3813060596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c10a940 msgr2=0x7f674c1a64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6749e00640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c10a940 0x7f674c1a64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6749e00640 1 -- 172.21.15.1:0/3813060596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f674c109f80 msgr2=0x7f674c19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6749e00640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f674c109f80 0x7f674c19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6749e00640 1 -- 172.21.15.1:0/3813060596 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c1a6b40 con 0x7f674c108d90 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6743e00640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f674c109f80 0x7f674c19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.906+0000 7f6749e00640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c108d90 0x7f674c19ef20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f673c00e9c0 tx=0x7f673c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.907+0000 7f6742a00640 1 -- 172.21.15.1:0/3813060596 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c00cd80 con 0x7f674c108d90 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.907+0000 7f6742a00640 1 -- 172.21.15.1:0/3813060596 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f673c0035f0 con 0x7f674c108d90 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.907+0000 7f6742a00640 1 -- 172.21.15.1:0/3813060596 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c010690 con 0x7f674c108d90 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.907+0000 7f674b200640 1 -- 172.21.15.1:0/3813060596 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c1a6e30 con 0x7f674c108d90 2024-04-26T21:26:16.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.907+0000 7f674b200640 1 -- 172.21.15.1:0/3813060596 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c1a7240 con 0x7f674c108d90 2024-04-26T21:26:16.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.908+0000 7f674b200640 1 -- 172.21.15.1:0/3813060596 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674c10ffe0 con 0x7f674c108d90 2024-04-26T21:26:16.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.910+0000 7f6742a00640 1 -- 172.21.15.1:0/3813060596 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f673c0037a0 con 0x7f674c108d90 2024-04-26T21:26:16.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.911+0000 7f6742a00640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f67300a3cd0 0x7f67300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:16.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.911+0000 7f6742a00640 1 -- 172.21.15.1:0/3813060596 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f673c0d02e0 con 0x7f674c108d90 2024-04-26T21:26:16.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.911+0000 7f6743e00640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f67300a3cd0 0x7f67300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:16.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.912+0000 7f6743e00640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f67300a3cd0 0x7f67300a6190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f674c1a0430 tx=0x7f6734005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:16.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:16.915+0000 7f6742a00640 1 -- 172.21.15.1:0/3813060596 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f673c09d0c0 con 0x7f674c108d90 2024-04-26T21:26:16.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:16 smithi186 ceph-mon[32991]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:16.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:16 smithi186 ceph-mon[32991]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:26:16.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:16 smithi186 ceph-mon[32991]: Saving service mds.all spec with placement smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T21:26:16.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:16.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:16 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:17.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.122+0000 7f674b200640 1 -- 172.21.15.1:0/3813060596 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f674c063630 con 0x7f674c108d90 2024-04-26T21:26:17.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.124+0000 7f6742a00640 1 -- 172.21.15.1:0/3813060596 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18300 (secure 0 0 0) 0x7f673c09c9c0 con 0x7f674c108d90 2024-04-26T21:26:17.125 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:17.125 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":83,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:26:04.333876+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 -- 172.21.15.1:0/3813060596 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f67300a3cd0 msgr2=0x7f67300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f67300a3cd0 0x7f67300a6190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f674c1a0430 tx=0x7f6734005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 -- 172.21.15.1:0/3813060596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c108d90 msgr2=0x7f674c19ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c108d90 0x7f674c19ef20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f673c00e9c0 tx=0x7f673c00ee80 comp rx=0 tx=0).stop 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 -- 172.21.15.1:0/3813060596 shutdown_connections 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f67300a3cd0 0x7f67300a6190 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c10a940 0x7f674c1a64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f674c109f80 0x7f674c19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 --2- 172.21.15.1:0/3813060596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c108d90 0x7f674c19ef20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 -- 172.21.15.1:0/3813060596 >> 172.21.15.1:0/3813060596 conn(0x7f674c077d30 msgr2=0x7f674c10d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:17.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 -- 172.21.15.1:0/3813060596 shutdown_connections 2024-04-26T21:26:17.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.127+0000 7f674b200640 1 -- 172.21.15.1:0/3813060596 wait complete. 2024-04-26T21:26:17.149 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:26:17.149 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:26:17.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.372+0000 7fb712400640 1 -- 172.21.15.1:0/3764670532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb70c103990 msgr2=0x7fb70c103e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:17.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.372+0000 7fb712400640 1 --2- 172.21.15.1:0/3764670532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb70c103990 0x7fb70c103e10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc009860 tx=0x7fb6fc02a340 comp rx=0 tx=0).stop 2024-04-26T21:26:17.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.372+0000 7fb712400640 1 -- 172.21.15.1:0/3764670532 shutdown_connections 2024-04-26T21:26:17.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.372+0000 7fb712400640 1 --2- 172.21.15.1:0/3764670532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb70c104350 0x7fb70c10abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.372+0000 7fb712400640 1 --2- 172.21.15.1:0/3764670532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb70c103990 0x7fb70c103e10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.372+0000 7fb712400640 1 --2- 172.21.15.1:0/3764670532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70c1027a0 0x7fb70c102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.372+0000 7fb712400640 1 -- 172.21.15.1:0/3764670532 >> 172.21.15.1:0/3764670532 conn(0x7fb70c0fdd10 msgr2=0x7fb70c100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:17.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.373+0000 7fb712400640 1 -- 172.21.15.1:0/3764670532 shutdown_connections 2024-04-26T21:26:17.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.373+0000 7fb712400640 1 -- 172.21.15.1:0/3764670532 wait complete. 2024-04-26T21:26:17.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.374+0000 7fb712400640 1 Processor -- start 2024-04-26T21:26:17.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.374+0000 7fb712400640 1 -- start start 2024-04-26T21:26:17.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.374+0000 7fb712400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70c1027a0 0x7fb70c0766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.374+0000 7fb712400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb70c103990 0x7fb70c074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.374+0000 7fb712400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb70c104350 0x7fb70c075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.374+0000 7fb712400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c10e2d0 con 0x7fb70c1027a0 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.374+0000 7fb712400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c10e150 con 0x7fb70c103990 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.374+0000 7fb712400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c10e450 con 0x7fb70c104350 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.374+0000 7fb711000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70c1027a0 0x7fb70c0766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.374+0000 7fb711000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70c1027a0 0x7fb70c0766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46042/0 (socket says 172.21.15.1:46042) 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.375+0000 7fb711000640 1 -- 172.21.15.1:0/2381844647 learned_addr learned my addr 172.21.15.1:0/2381844647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.375+0000 7fb711a00640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb70c104350 0x7fb70c075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.375+0000 7fb70be00640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb70c103990 0x7fb70c074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.375+0000 7fb711000640 1 -- 172.21.15.1:0/2381844647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb70c104350 msgr2=0x7fb70c075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.375+0000 7fb711000640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb70c104350 0x7fb70c075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.375+0000 7fb711000640 1 -- 172.21.15.1:0/2381844647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb70c103990 msgr2=0x7fb70c074d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:17.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.375+0000 7fb711000640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb70c103990 0x7fb70c074d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.375+0000 7fb711000640 1 -- 172.21.15.1:0/2381844647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70c075af0 con 0x7fb70c1027a0 2024-04-26T21:26:17.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.375+0000 7fb711a00640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb70c104350 0x7fb70c075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:17.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.375+0000 7fb711000640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70c1027a0 0x7fb70c0766e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb70000e9f0 tx=0x7fb70000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:17.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.376+0000 7fb709600640 1 -- 172.21.15.1:0/2381844647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70000ccc0 con 0x7fb70c1027a0 2024-04-26T21:26:17.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.376+0000 7fb709600640 1 -- 172.21.15.1:0/2381844647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7000035f0 con 0x7fb70c1027a0 2024-04-26T21:26:17.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.376+0000 7fb709600640 1 -- 172.21.15.1:0/2381844647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70000ce60 con 0x7fb70c1027a0 2024-04-26T21:26:17.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.376+0000 7fb712400640 1 -- 172.21.15.1:0/2381844647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb70c1a2af0 con 0x7fb70c1027a0 2024-04-26T21:26:17.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.376+0000 7fb712400640 1 -- 172.21.15.1:0/2381844647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb70c1a3010 con 0x7fb70c1027a0 2024-04-26T21:26:17.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.377+0000 7fb712400640 1 -- 172.21.15.1:0/2381844647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb70c0506c0 con 0x7fb70c1027a0 2024-04-26T21:26:17.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.379+0000 7fb709600640 1 -- 172.21.15.1:0/2381844647 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fb700004000 con 0x7fb70c1027a0 2024-04-26T21:26:17.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.380+0000 7fb709600640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb6f40a3bb0 0x7fb6f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:17.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.381+0000 7fb709600640 1 -- 172.21.15.1:0/2381844647 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb7000d0310 con 0x7fb70c1027a0 2024-04-26T21:26:17.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.381+0000 7fb70be00640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb6f40a3bb0 0x7fb6f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:17.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.382+0000 7fb70be00640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb6f40a3bb0 0x7fb6f40a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc000c00 tx=0x7fb6fc030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:17.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.383+0000 7fb709600640 1 -- 172.21.15.1:0/2381844647 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb70009d0f0 con 0x7fb70c1027a0 2024-04-26T21:26:17.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.598+0000 7fb712400640 1 -- 172.21.15.1:0/2381844647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb70c02db50 con 0x7fb70c1027a0 2024-04-26T21:26:17.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.599+0000 7fb709600640 1 -- 172.21.15.1:0/2381844647 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7fb70009c9f0 con 0x7fb70c1027a0 2024-04-26T21:26:17.600 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:26:17.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 -- 172.21.15.1:0/2381844647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb6f40a3bb0 msgr2=0x7fb6f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:17.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb6f40a3bb0 0x7fb6f40a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc000c00 tx=0x7fb6fc030470 comp rx=0 tx=0).stop 2024-04-26T21:26:17.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 -- 172.21.15.1:0/2381844647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70c1027a0 msgr2=0x7fb70c0766e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:17.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70c1027a0 0x7fb70c0766e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb70000e9f0 tx=0x7fb70000eeb0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 -- 172.21.15.1:0/2381844647 shutdown_connections 2024-04-26T21:26:17.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb6f40a3bb0 0x7fb6f40a6070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb70c104350 0x7fb70c075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb70c103990 0x7fb70c074d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 --2- 172.21.15.1:0/2381844647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb70c1027a0 0x7fb70c0766e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 -- 172.21.15.1:0/2381844647 >> 172.21.15.1:0/2381844647 conn(0x7fb70c0fdd10 msgr2=0x7fb70c106d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:17.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 -- 172.21.15.1:0/2381844647 shutdown_connections 2024-04-26T21:26:17.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.602+0000 7fb712400640 1 -- 172.21.15.1:0/2381844647 wait complete. 2024-04-26T21:26:17.624 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:17.646 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:17.647 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:17.647 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:17.647 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:17.647 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:17.647 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:17 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:17.647 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:17 smithi186 ceph-mon[32991]: Deploying daemon mds.l on smithi186 2024-04-26T21:26:17.647 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:17 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3813060596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:26:17.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:17.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:17.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:17.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:17.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:17.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:17 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:17.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:17 smithi001 ceph-mon[28054]: Deploying daemon mds.l on smithi186 2024-04-26T21:26:17.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:17 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3813060596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:26:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:26:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:17 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:17 smithi038 ceph-mon[32724]: Deploying daemon mds.l on smithi186 2024-04-26T21:26:17.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:17 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3813060596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:26:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.841+0000 7f99adc00640 1 -- 172.21.15.1:0/1052489238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99a8100fe0 msgr2=0x7f99a8103410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.841+0000 7f99adc00640 1 --2- 172.21.15.1:0/1052489238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99a8100fe0 0x7f99a8103410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9998009be0 tx=0x7f999802a6d0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.841+0000 7f99adc00640 1 -- 172.21.15.1:0/1052489238 shutdown_connections 2024-04-26T21:26:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.841+0000 7f99adc00640 1 --2- 172.21.15.1:0/1052489238 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99a8106300 0x7f99a8106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.841+0000 7f99adc00640 1 --2- 172.21.15.1:0/1052489238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99a8103950 0x7f99a8105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.841+0000 7f99adc00640 1 --2- 172.21.15.1:0/1052489238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99a8100fe0 0x7f99a8103410 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.841+0000 7f99adc00640 1 -- 172.21.15.1:0/1052489238 >> 172.21.15.1:0/1052489238 conn(0x7f99a80fcc00 msgr2=0x7f99a80ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:17.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.841+0000 7f99adc00640 1 -- 172.21.15.1:0/1052489238 shutdown_connections 2024-04-26T21:26:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.841+0000 7f99adc00640 1 -- 172.21.15.1:0/1052489238 wait complete. 2024-04-26T21:26:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.842+0000 7f99adc00640 1 Processor -- start 2024-04-26T21:26:17.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99adc00640 1 -- start start 2024-04-26T21:26:17.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99adc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99a8100fe0 0x7f99a819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99adc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99a8103950 0x7f99a819f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99adc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99a8106300 0x7f99a81a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99adc00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f99a8114890 con 0x7f99a8106300 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99adc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f99a8114710 con 0x7f99a8103950 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99adc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f99a8114a10 con 0x7f99a8100fe0 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99ad200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99a8106300 0x7f99a81a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99a7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99a8103950 0x7f99a819f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99ad200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99a8106300 0x7f99a81a6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46058/0 (socket says 172.21.15.1:46058) 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99ad200640 1 -- 172.21.15.1:0/2212315515 learned_addr learned my addr 172.21.15.1:0/2212315515 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.843+0000 7f99a7e00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99a8100fe0 0x7f99a819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.844+0000 7f99a7e00640 1 -- 172.21.15.1:0/2212315515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99a8103950 msgr2=0x7f99a819f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.844+0000 7f99a7e00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99a8103950 0x7f99a819f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.844+0000 7f99a7e00640 1 -- 172.21.15.1:0/2212315515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99a8106300 msgr2=0x7f99a81a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:17.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.844+0000 7f99a7e00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99a8106300 0x7f99a81a6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.844+0000 7f99a7e00640 1 -- 172.21.15.1:0/2212315515 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a81a6ad0 con 0x7f99a8100fe0 2024-04-26T21:26:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.844+0000 7f99a7400640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99a8103950 0x7f99a819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.844+0000 7f99a7e00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99a8100fe0 0x7f99a819eeb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9998009b30 tx=0x7f9998030d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:17.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.845+0000 7f99a4c00640 1 -- 172.21.15.1:0/2212315515 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998041020 con 0x7f99a8100fe0 2024-04-26T21:26:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.845+0000 7f99a4c00640 1 -- 172.21.15.1:0/2212315515 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99980318f0 con 0x7f99a8100fe0 2024-04-26T21:26:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.845+0000 7f99adc00640 1 -- 172.21.15.1:0/2212315515 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a81a6d60 con 0x7f99a8100fe0 2024-04-26T21:26:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.845+0000 7f99a4c00640 1 -- 172.21.15.1:0/2212315515 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998036040 con 0x7f99a8100fe0 2024-04-26T21:26:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.845+0000 7f99adc00640 1 -- 172.21.15.1:0/2212315515 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a81a71d0 con 0x7f99a8100fe0 2024-04-26T21:26:17.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.846+0000 7f99adc00640 1 -- 172.21.15.1:0/2212315515 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a80fe6f0 con 0x7f99a8100fe0 2024-04-26T21:26:17.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.848+0000 7f99a4c00640 1 -- 172.21.15.1:0/2212315515 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f9998044020 con 0x7f99a8100fe0 2024-04-26T21:26:17.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.850+0000 7f99a4c00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f998c0a3c50 0x7f998c0a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:17.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.850+0000 7f99a4c00640 1 -- 172.21.15.1:0/2212315515 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f99980f11a0 con 0x7f99a8100fe0 2024-04-26T21:26:17.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.850+0000 7f99a7400640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f998c0a3c50 0x7f998c0a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:17.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.851+0000 7f99a7400640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f998c0a3c50 0x7f998c0a6110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f99a81a03c0 tx=0x7f9994005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:17.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:17.852+0000 7f99a4c00640 1 -- 172.21.15.1:0/2212315515 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f999804b020 con 0x7f99a8100fe0 2024-04-26T21:26:18.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.132+0000 7f99adc00640 1 -- 172.21.15.1:0/2212315515 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99a802dad0 con 0x7f99a8100fe0 2024-04-26T21:26:18.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.133+0000 7f99a4c00640 1 -- 172.21.15.1:0/2212315515 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f99980bdf80 con 0x7f99a8100fe0 2024-04-26T21:26:18.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:18.134 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-26T21:26:18.134 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 1 2024-04-26T21:26:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 -- 172.21.15.1:0/2212315515 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f998c0a3c50 msgr2=0x7f998c0a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f998c0a3c50 0x7f998c0a6110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f99a81a03c0 tx=0x7f9994005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 -- 172.21.15.1:0/2212315515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99a8100fe0 msgr2=0x7f99a819eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99a8100fe0 0x7f99a819eeb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9998009b30 tx=0x7f9998030d10 comp rx=0 tx=0).stop 2024-04-26T21:26:18.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 -- 172.21.15.1:0/2212315515 shutdown_connections 2024-04-26T21:26:18.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f998c0a3c50 0x7f998c0a6110 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:18.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99a8106300 0x7f99a81a6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:18.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f99a8103950 0x7f99a819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:18.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 --2- 172.21.15.1:0/2212315515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99a8100fe0 0x7f99a819eeb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:18.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 -- 172.21.15.1:0/2212315515 >> 172.21.15.1:0/2212315515 conn(0x7f99a80fcc00 msgr2=0x7f99a8104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:18.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 -- 172.21.15.1:0/2212315515 shutdown_connections 2024-04-26T21:26:18.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:18.135+0000 7f99adc00640 1 -- 172.21.15.1:0/2212315515 wait complete. 2024-04-26T21:26:18.682 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:18 smithi186 ceph-mon[32991]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:18.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:18 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2381844647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:26:18.730 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:18 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2212315515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:18.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:18 smithi001 ceph-mon[28054]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:18.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:18 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2381844647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:26:18.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:18 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2212315515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:18.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:18 smithi038 ceph-mon[32724]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:18.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:18 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2381844647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:26:18.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:18 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2212315515' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:18.954 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:26:18 smithi186 systemd[1]: Starting Ceph mds.l for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:26:19.340 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:26:19 smithi186 podman[62917]: 2024-04-26 21:26:18.976771468 +0000 UTC m=+0.018044274 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:19.340 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:26:19 smithi186 podman[62917]: 2024-04-26 21:26:19.104372059 +0000 UTC m=+0.145644859 container create 15b4c1050def987490d134da5857b047574c00e786f3ec3810b3989b982b5ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-l, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-26T21:26:19.340 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:26:19 smithi186 podman[62917]: 2024-04-26 21:26:19.340332571 +0000 UTC m=+0.381605371 container init 15b4c1050def987490d134da5857b047574c00e786f3ec3810b3989b982b5ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-26T21:26:19.704 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:26:19 smithi186 podman[62917]: 2024-04-26 21:26:19.343044233 +0000 UTC m=+0.384317034 container start 15b4c1050def987490d134da5857b047574c00e786f3ec3810b3989b982b5ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T21:26:19.705 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:26:19 smithi186 ceph-mds[62936]: -- 172.21.15.186:0/4105016550 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563577d73a40 con 0x563577d46800 2024-04-26T21:26:19.705 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:26:19 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-l[62932]: starting mds.l at 2024-04-26T21:26:19.705 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:26:19 smithi186 bash[62917]: 15b4c1050def987490d134da5857b047574c00e786f3ec3810b3989b982b5ce6 2024-04-26T21:26:19.705 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:26:19 smithi186 systemd[1]: Started Ceph mds.l for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:20.156 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:20.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.381+0000 7f5054c00640 1 -- 172.21.15.1:0/3598179209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5050105b90 msgr2=0x7f5050106010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:20.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.381+0000 7f5054c00640 1 --2- 172.21.15.1:0/3598179209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5050105b90 0x7f5050106010 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f503c009900 tx=0x7f503c02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:26:20.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.382+0000 7f5054c00640 1 -- 172.21.15.1:0/3598179209 shutdown_connections 2024-04-26T21:26:20.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.382+0000 7f5054c00640 1 --2- 172.21.15.1:0/3598179209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5050106550 0x7f505010cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:20.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.382+0000 7f5054c00640 1 --2- 172.21.15.1:0/3598179209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5050105b90 0x7f5050106010 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:20.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.382+0000 7f5054c00640 1 --2- 172.21.15.1:0/3598179209 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50501049a0 0x7f5050104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:20.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.382+0000 7f5054c00640 1 -- 172.21.15.1:0/3598179209 >> 172.21.15.1:0/3598179209 conn(0x7f50500fff10 msgr2=0x7f5050102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:20.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.382+0000 7f5054c00640 1 -- 172.21.15.1:0/3598179209 shutdown_connections 2024-04-26T21:26:20.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.382+0000 7f5054c00640 1 -- 172.21.15.1:0/3598179209 wait complete. 2024-04-26T21:26:20.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.383+0000 7f5054c00640 1 Processor -- start 2024-04-26T21:26:20.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.383+0000 7f5054c00640 1 -- start start 2024-04-26T21:26:20.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.383+0000 7f5054c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50501049a0 0x7f505019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:20.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.383+0000 7f5054c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5050105b90 0x7f505019afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.383+0000 7f5054c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5050106550 0x7f50501a2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.383+0000 7f5054c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5050110520 con 0x7f5050105b90 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.383+0000 7f5054c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f50501103a0 con 0x7f50501049a0 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.383+0000 7f5054c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f50501106a0 con 0x7f5050106550 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50501049a0 0x7f505019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50501049a0 0x7f505019aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34866/0 (socket says 172.21.15.1:34866) 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504f400640 1 -- 172.21.15.1:0/2620480368 learned_addr learned my addr 172.21.15.1:0/2620480368 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504ea00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5050105b90 0x7f505019afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504fe00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5050106550 0x7f50501a2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504f400640 1 -- 172.21.15.1:0/2620480368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5050106550 msgr2=0x7f50501a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504f400640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5050106550 0x7f50501a2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504f400640 1 -- 172.21.15.1:0/2620480368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5050105b90 msgr2=0x7f505019afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504f400640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5050105b90 0x7f505019afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:20.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504f400640 1 -- 172.21.15.1:0/2620480368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50501a26c0 con 0x7f50501049a0 2024-04-26T21:26:20.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f504f400640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50501049a0 0x7f505019aaa0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f504000d770 tx=0x7f504000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:20.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.384+0000 7f5047e00640 1 -- 172.21.15.1:0/2620480368 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040017070 con 0x7f50501049a0 2024-04-26T21:26:20.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.385+0000 7f5047e00640 1 -- 172.21.15.1:0/2620480368 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5040009c30 con 0x7f50501049a0 2024-04-26T21:26:20.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.385+0000 7f5047e00640 1 -- 172.21.15.1:0/2620480368 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040003cc0 con 0x7f50501049a0 2024-04-26T21:26:20.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.385+0000 7f5054c00640 1 -- 172.21.15.1:0/2620480368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50501a29b0 con 0x7f50501049a0 2024-04-26T21:26:20.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.385+0000 7f5054c00640 1 -- 172.21.15.1:0/2620480368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50501a2e80 con 0x7f50501049a0 2024-04-26T21:26:20.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.386+0000 7f5054c00640 1 -- 172.21.15.1:0/2620480368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f505010bc60 con 0x7f50501049a0 2024-04-26T21:26:20.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.388+0000 7f5047e00640 1 -- 172.21.15.1:0/2620480368 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f5040003e60 con 0x7f50501049a0 2024-04-26T21:26:20.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.389+0000 7f5047e00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f50340a3cd0 0x7f50340a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:20.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.389+0000 7f5047e00640 1 -- 172.21.15.1:0/2620480368 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f50400cf910 con 0x7f50501049a0 2024-04-26T21:26:20.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.389+0000 7f504ea00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f50340a3cd0 0x7f50340a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:20.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.390+0000 7f504ea00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f50340a3cd0 0x7f50340a6190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f503c031500 tx=0x7f503c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:20.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.393+0000 7f5047e00640 1 -- 172.21.15.1:0/2620480368 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5040009de0 con 0x7f50501049a0 2024-04-26T21:26:20.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:20.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.186:6832/1838831066,v1:172.21.15.186:6833/1838831066] up:boot 2024-04-26T21:26:20.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: fsmap 1 up:standby 2024-04-26T21:26:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T21:26:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:20.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:20 smithi001 ceph-mon[28054]: Deploying daemon mds.k on smithi038 2024-04-26T21:26:20.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:20.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.186:6832/1838831066,v1:172.21.15.186:6833/1838831066] up:boot 2024-04-26T21:26:20.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: fsmap 1 up:standby 2024-04-26T21:26:20.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T21:26:20.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:20.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:20.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:20.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:20.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:20.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:20.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:20 smithi038 ceph-mon[32724]: Deploying daemon mds.k on smithi038 2024-04-26T21:26:20.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.640+0000 7f5054c00640 1 -- 172.21.15.1:0/2620480368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5050063530 con 0x7f50501049a0 2024-04-26T21:26:20.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.640+0000 7f5047e00640 1 -- 172.21.15.1:0/2620480368 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f50400bc090 con 0x7f50501049a0 2024-04-26T21:26:20.641 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:20.642 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-26T21:26:20.642 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 2 2024-04-26T21:26:20.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.643+0000 7f5054c00640 1 -- 172.21.15.1:0/2620480368 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f50340a3cd0 msgr2=0x7f50340a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:20.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.643+0000 7f5054c00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f50340a3cd0 0x7f50340a6190 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f503c031500 tx=0x7f503c005a20 comp rx=0 tx=0).stop 2024-04-26T21:26:20.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.643+0000 7f5054c00640 1 -- 172.21.15.1:0/2620480368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50501049a0 msgr2=0x7f505019aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:20.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.643+0000 7f5054c00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50501049a0 0x7f505019aaa0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f504000d770 tx=0x7f504000dc30 comp rx=0 tx=0).stop 2024-04-26T21:26:20.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.643+0000 7f5054c00640 1 -- 172.21.15.1:0/2620480368 shutdown_connections 2024-04-26T21:26:20.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.643+0000 7f5054c00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f50340a3cd0 0x7f50340a6190 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:20.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.643+0000 7f5054c00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5050106550 0x7f50501a2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:20.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.643+0000 7f5054c00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5050105b90 0x7f505019afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:20.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.643+0000 7f5054c00640 1 --2- 172.21.15.1:0/2620480368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50501049a0 0x7f505019aaa0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:20.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.643+0000 7f5054c00640 1 -- 172.21.15.1:0/2620480368 >> 172.21.15.1:0/2620480368 conn(0x7f50500fff10 msgr2=0x7f5050108e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:20.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.644+0000 7f5054c00640 1 -- 172.21.15.1:0/2620480368 shutdown_connections 2024-04-26T21:26:20.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:20.644+0000 7f5054c00640 1 -- 172.21.15.1:0/2620480368 wait complete. 2024-04-26T21:26:20.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:20.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.186:6832/1838831066,v1:172.21.15.186:6833/1838831066] up:boot 2024-04-26T21:26:20.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: fsmap 1 up:standby 2024-04-26T21:26:20.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T21:26:20.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:20.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:20.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:20.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:20.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:20.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:20.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:20 smithi186 ceph-mon[32991]: Deploying daemon mds.k on smithi038 2024-04-26T21:26:21.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:21 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2620480368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:21.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:21 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2620480368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:21.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:21 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2620480368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:22.054 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:26:21 smithi038 systemd[1]: Starting Ceph mds.k for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:26:22.310 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:26:22 smithi038 podman[61853]: 2024-04-26 21:26:21.957145639 +0000 UTC m=+0.018340466 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:22.311 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:26:22 smithi038 podman[61853]: 2024-04-26 21:26:22.066019309 +0000 UTC m=+0.127214134 container create aa549981267a2c2abad07c1fbf292ecd3d29d918479dccb5a52ba5e4b043da3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:26:22.311 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:26:22 smithi038 podman[61853]: 2024-04-26 21:26:22.310637361 +0000 UTC m=+0.371832175 container init aa549981267a2c2abad07c1fbf292ecd3d29d918479dccb5a52ba5e4b043da3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-26T21:26:22.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:22 smithi038 ceph-mon[32724]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:22.577 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:26:22 smithi038 podman[61853]: 2024-04-26 21:26:22.313688306 +0000 UTC m=+0.374883129 container start aa549981267a2c2abad07c1fbf292ecd3d29d918479dccb5a52ba5e4b043da3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-k, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-26T21:26:22.577 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:26:22 smithi038 ceph-mds[61872]: -- 172.21.15.38:0/2251962366 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5563ba371a40 con 0x5563ba344800 2024-04-26T21:26:22.577 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:26:22 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-k[61868]: starting mds.k at 2024-04-26T21:26:22.577 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:26:22 smithi038 bash[61853]: aa549981267a2c2abad07c1fbf292ecd3d29d918479dccb5a52ba5e4b043da3d 2024-04-26T21:26:22.577 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:26:22 smithi038 systemd[1]: Started Ceph mds.k for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:22.664 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:22 smithi186 ceph-mon[32991]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:22.727 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:22 smithi001 ceph-mon[28054]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:22.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.857+0000 7fcdd5800640 1 -- 172.21.15.1:0/3022473558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0104330 msgr2=0x7fcdd010abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.857+0000 7fcdd5800640 1 --2- 172.21.15.1:0/3022473558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0104330 0x7fcdd010abc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009be0 tx=0x7fcdc802a6c0 comp rx=0 tx=0).stop 2024-04-26T21:26:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.857+0000 7fcdd5800640 1 -- 172.21.15.1:0/3022473558 shutdown_connections 2024-04-26T21:26:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.857+0000 7fcdd5800640 1 --2- 172.21.15.1:0/3022473558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0104330 0x7fcdd010abc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.857+0000 7fcdd5800640 1 --2- 172.21.15.1:0/3022473558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdd0103970 0x7fcdd0103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.857+0000 7fcdd5800640 1 --2- 172.21.15.1:0/3022473558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdd0102780 0x7fcdd0102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.857+0000 7fcdd5800640 1 -- 172.21.15.1:0/3022473558 >> 172.21.15.1:0/3022473558 conn(0x7fcdd00fdcf0 msgr2=0x7fcdd0100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.857+0000 7fcdd5800640 1 -- 172.21.15.1:0/3022473558 shutdown_connections 2024-04-26T21:26:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.857+0000 7fcdd5800640 1 -- 172.21.15.1:0/3022473558 wait complete. 2024-04-26T21:26:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.858+0000 7fcdd5800640 1 Processor -- start 2024-04-26T21:26:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.858+0000 7fcdd5800640 1 -- start start 2024-04-26T21:26:22.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.858+0000 7fcdd5800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdd0102780 0x7fcdd00766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.858+0000 7fcdd5800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdd0103970 0x7fcdd0074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdd5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0104330 0x7fcdd0075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdd5800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd010e2b0 con 0x7fcdd0102780 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdd5800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd010e130 con 0x7fcdd0103970 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdd5800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd010e430 con 0x7fcdd0104330 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdcfe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdd0102780 0x7fcdd00766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdc7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdd0103970 0x7fcdd0074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdc7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdd0103970 0x7fcdd0074d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34876/0 (socket says 172.21.15.1:34876) 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdc7e00640 1 -- 172.21.15.1:0/3396149492 learned_addr learned my addr 172.21.15.1:0/3396149492 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdd4e00640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0104330 0x7fcdd0075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdc7e00640 1 -- 172.21.15.1:0/3396149492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0104330 msgr2=0x7fcdd0075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:22.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdc7e00640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0104330 0x7fcdd0075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdc7e00640 1 -- 172.21.15.1:0/3396149492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdd0102780 msgr2=0x7fcdd00766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdc7e00640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdd0102780 0x7fcdd00766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.859+0000 7fcdc7e00640 1 -- 172.21.15.1:0/3396149492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd00757a0 con 0x7fcdd0103970 2024-04-26T21:26:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.860+0000 7fcdc7e00640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdd0103970 0x7fcdd0074d20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdb800e9c0 tx=0x7fcdb800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.860+0000 7fcdcd600640 1 -- 172.21.15.1:0/3396149492 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb800cde0 con 0x7fcdd0103970 2024-04-26T21:26:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.860+0000 7fcdcd600640 1 -- 172.21.15.1:0/3396149492 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdb80035f0 con 0x7fcdd0103970 2024-04-26T21:26:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.860+0000 7fcdd5800640 1 -- 172.21.15.1:0/3396149492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd0075a90 con 0x7fcdd0103970 2024-04-26T21:26:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.860+0000 7fcdcd600640 1 -- 172.21.15.1:0/3396149492 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb8010690 con 0x7fcdd0103970 2024-04-26T21:26:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.860+0000 7fcdd5800640 1 -- 172.21.15.1:0/3396149492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd01a7320 con 0x7fcdd0103970 2024-04-26T21:26:22.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.861+0000 7fcdd5800640 1 -- 172.21.15.1:0/3396149492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd00506c0 con 0x7fcdd0103970 2024-04-26T21:26:22.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.863+0000 7fcdcd600640 1 -- 172.21.15.1:0/3396149492 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fcdb8004000 con 0x7fcdd0103970 2024-04-26T21:26:22.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.864+0000 7fcdcd600640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcdb40a3cd0 0x7fcdb40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:22.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.865+0000 7fcdcd600640 1 -- 172.21.15.1:0/3396149492 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fcdb80d0290 con 0x7fcdd0103970 2024-04-26T21:26:22.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.865+0000 7fcdcfe00640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcdb40a3cd0 0x7fcdb40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:22.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.865+0000 7fcdcfe00640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcdb40a3cd0 0x7fcdb40a6190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcdd01037d0 tx=0x7fcdc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:22.869+0000 7fcdcd600640 1 -- 172.21.15.1:0/3396149492 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcdb8009510 con 0x7fcdd0103970 2024-04-26T21:26:23.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.118+0000 7fcdd5800640 1 -- 172.21.15.1:0/3396149492 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdd0109a40 con 0x7fcdd0103970 2024-04-26T21:26:23.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.118+0000 7fcdcd600640 1 -- 172.21.15.1:0/3396149492 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fcdb809d070 con 0x7fcdd0103970 2024-04-26T21:26:23.119 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:23.119 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T21:26:23.120 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 3 2024-04-26T21:26:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.121+0000 7fcdd5800640 1 -- 172.21.15.1:0/3396149492 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcdb40a3cd0 msgr2=0x7fcdb40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.121+0000 7fcdd5800640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcdb40a3cd0 0x7fcdb40a6190 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcdd01037d0 tx=0x7fcdc0005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.121+0000 7fcdd5800640 1 -- 172.21.15.1:0/3396149492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdd0103970 msgr2=0x7fcdd0074d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.121+0000 7fcdd5800640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdd0103970 0x7fcdd0074d20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdb800e9c0 tx=0x7fcdb800ee80 comp rx=0 tx=0).stop 2024-04-26T21:26:23.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.121+0000 7fcdd5800640 1 -- 172.21.15.1:0/3396149492 shutdown_connections 2024-04-26T21:26:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.121+0000 7fcdd5800640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcdb40a3cd0 0x7fcdb40a6190 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.121+0000 7fcdd5800640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0104330 0x7fcdd0075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.121+0000 7fcdd5800640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdd0103970 0x7fcdd0074d20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.121+0000 7fcdd5800640 1 --2- 172.21.15.1:0/3396149492 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdd0102780 0x7fcdd00766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.121+0000 7fcdd5800640 1 -- 172.21.15.1:0/3396149492 >> 172.21.15.1:0/3396149492 conn(0x7fcdd00fdcf0 msgr2=0x7fcdd0106c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.122+0000 7fcdd5800640 1 -- 172.21.15.1:0/3396149492 shutdown_connections 2024-04-26T21:26:23.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:23.122+0000 7fcdd5800640 1 -- 172.21.15.1:0/3396149492 wait complete. 2024-04-26T21:26:23.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.38:6832/1237951568,v1:172.21.15.38:6833/1237951568] up:boot 2024-04-26T21:26:23.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: fsmap 2 up:standby 2024-04-26T21:26:23.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T21:26:23.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:23.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:23.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:23.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:23.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:23.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:23.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: Deploying daemon mds.j on smithi001 2024-04-26T21:26:23.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:23 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3396149492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:23.774 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.38:6832/1237951568,v1:172.21.15.38:6833/1237951568] up:boot 2024-04-26T21:26:23.774 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: fsmap 2 up:standby 2024-04-26T21:26:23.774 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T21:26:23.774 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:23.774 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:23.774 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:23.774 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:23.775 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:23.775 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:23.775 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: Deploying daemon mds.j on smithi001 2024-04-26T21:26:23.775 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:23 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3396149492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:23.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.38:6832/1237951568,v1:172.21.15.38:6833/1237951568] up:boot 2024-04-26T21:26:23.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: fsmap 2 up:standby 2024-04-26T21:26:23.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T21:26:23.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:23.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:23.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:23.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:23.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:23.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:23.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: Deploying daemon mds.j on smithi001 2024-04-26T21:26:23.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:23 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3396149492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:24.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:24 smithi001 ceph-mon[28054]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:24.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:24 smithi038 ceph-mon[32724]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:24.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:24 smithi186 ceph-mon[32991]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:25.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.429+0000 7fc521a00640 1 -- 172.21.15.1:0/4134628187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc51c10f020 msgr2=0x7fc51c1114e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.429+0000 7fc521a00640 1 --2- 172.21.15.1:0/4134628187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc51c10f020 0x7fc51c1114e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc50c008410 tx=0x7fc50c005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:25.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.429+0000 7fc521a00640 1 -- 172.21.15.1:0/4134628187 shutdown_connections 2024-04-26T21:26:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.429+0000 7fc521a00640 1 --2- 172.21.15.1:0/4134628187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc51c10f020 0x7fc51c1114e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.429+0000 7fc521a00640 1 --2- 172.21.15.1:0/4134628187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc51c10c560 0x7fc51c10e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.429+0000 7fc521a00640 1 --2- 172.21.15.1:0/4134628187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc51c109ba0 0x7fc51c10bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.429+0000 7fc521a00640 1 -- 172.21.15.1:0/4134628187 >> 172.21.15.1:0/4134628187 conn(0x7fc51c06f9a0 msgr2=0x7fc51c071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.429+0000 7fc521a00640 1 -- 172.21.15.1:0/4134628187 shutdown_connections 2024-04-26T21:26:25.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.429+0000 7fc521a00640 1 -- 172.21.15.1:0/4134628187 wait complete. 2024-04-26T21:26:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.430+0000 7fc521a00640 1 Processor -- start 2024-04-26T21:26:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc521a00640 1 -- start start 2024-04-26T21:26:25.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc521a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc51c109ba0 0x7fc51c1a34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc521a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc51c10c560 0x7fc51c1a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc521a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc51c10f020 0x7fc51c1aaa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc521a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc51c114ab0 con 0x7fc51c109ba0 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc521a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc51c114930 con 0x7fc51c10c560 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc521a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc51c114c30 con 0x7fc51c10f020 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc51aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc51c109ba0 0x7fc51c1a34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc51aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc51c109ba0 0x7fc51c1a34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59896/0 (socket says 172.21.15.1:59896) 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc51aa00640 1 -- 172.21.15.1:0/703380846 learned_addr learned my addr 172.21.15.1:0/703380846 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc51a000640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc51c10c560 0x7fc51c1a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.431+0000 7fc51b400640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc51c10f020 0x7fc51c1aaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.432+0000 7fc51aa00640 1 -- 172.21.15.1:0/703380846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc51c10f020 msgr2=0x7fc51c1aaa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.432+0000 7fc51aa00640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc51c10f020 0x7fc51c1aaa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:25.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.432+0000 7fc51aa00640 1 -- 172.21.15.1:0/703380846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc51c10c560 msgr2=0x7fc51c1a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.432+0000 7fc51aa00640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc51c10c560 0x7fc51c1a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.432+0000 7fc51aa00640 1 -- 172.21.15.1:0/703380846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51c1ab100 con 0x7fc51c109ba0 2024-04-26T21:26:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.432+0000 7fc51a000640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc51c10c560 0x7fc51c1a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.432+0000 7fc51aa00640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc51c109ba0 0x7fc51c1a34e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc50800e9c0 tx=0x7fc50800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.432+0000 7fc513400640 1 -- 172.21.15.1:0/703380846 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50800ccc0 con 0x7fc51c109ba0 2024-04-26T21:26:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.432+0000 7fc513400640 1 -- 172.21.15.1:0/703380846 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5080054d0 con 0x7fc51c109ba0 2024-04-26T21:26:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.432+0000 7fc513400640 1 -- 172.21.15.1:0/703380846 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc508010430 con 0x7fc51c109ba0 2024-04-26T21:26:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.433+0000 7fc521a00640 1 -- 172.21.15.1:0/703380846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc51c1ab3f0 con 0x7fc51c109ba0 2024-04-26T21:26:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.433+0000 7fc521a00640 1 -- 172.21.15.1:0/703380846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc51c1ab990 con 0x7fc51c109ba0 2024-04-26T21:26:25.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.434+0000 7fc521a00640 1 -- 172.21.15.1:0/703380846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc51c0724a0 con 0x7fc51c109ba0 2024-04-26T21:26:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.437+0000 7fc513400640 1 -- 172.21.15.1:0/703380846 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc508004f80 con 0x7fc51c109ba0 2024-04-26T21:26:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.438+0000 7fc513400640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc5000a3c30 0x7fc5000a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.438+0000 7fc513400640 1 -- 172.21.15.1:0/703380846 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc5080d0380 con 0x7fc51c109ba0 2024-04-26T21:26:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.438+0000 7fc51a000640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc5000a3c30 0x7fc5000a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:25.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.439+0000 7fc51a000640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc5000a3c30 0x7fc5000a60f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc51400bbb0 tx=0x7fc51400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:25.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.448+0000 7fc513400640 1 -- 172.21.15.1:0/703380846 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc50809d160 con 0x7fc51c109ba0 2024-04-26T21:26:25.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.700+0000 7fc521a00640 1 -- 172.21.15.1:0/703380846 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc51c02dad0 con 0x7fc51c109ba0 2024-04-26T21:26:25.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.701+0000 7fc513400640 1 -- 172.21.15.1:0/703380846 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fc50809ca60 con 0x7fc51c109ba0 2024-04-26T21:26:25.702 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:25.702 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T21:26:25.703 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 3 2024-04-26T21:26:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 -- 172.21.15.1:0/703380846 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc5000a3c30 msgr2=0x7fc5000a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc5000a3c30 0x7fc5000a60f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc51400bbb0 tx=0x7fc51400d040 comp rx=0 tx=0).stop 2024-04-26T21:26:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 -- 172.21.15.1:0/703380846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc51c109ba0 msgr2=0x7fc51c1a34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc51c109ba0 0x7fc51c1a34e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc50800e9c0 tx=0x7fc50800ee80 comp rx=0 tx=0).stop 2024-04-26T21:26:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 -- 172.21.15.1:0/703380846 shutdown_connections 2024-04-26T21:26:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc5000a3c30 0x7fc5000a60f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc51c10f020 0x7fc51c1aaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc51c10c560 0x7fc51c1a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 --2- 172.21.15.1:0/703380846 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc51c109ba0 0x7fc51c1a34e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 -- 172.21.15.1:0/703380846 >> 172.21.15.1:0/703380846 conn(0x7fc51c06f9a0 msgr2=0x7fc51c10bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 -- 172.21.15.1:0/703380846 shutdown_connections 2024-04-26T21:26:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:25.704+0000 7fc521a00640 1 -- 172.21.15.1:0/703380846 wait complete. 2024-04-26T21:26:26.319 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:26:26 smithi001 podman[65875]: 2024-04-26 21:26:26.099333906 +0000 UTC m=+0.199441092 container create 31aa4a5f93ef579d4d4fbf9a91be9683795e4e3f0ba6f31b8292c0137b3893a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-26T21:26:26.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:26 smithi001 ceph-mon[28054]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:26.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:26 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/703380846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:26.818 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:26:26 smithi001 podman[65875]: 2024-04-26 21:26:26.462975558 +0000 UTC m=+0.563082790 container init 31aa4a5f93ef579d4d4fbf9a91be9683795e4e3f0ba6f31b8292c0137b3893a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:26:26.818 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:26:26 smithi001 podman[65875]: 2024-04-26 21:26:26.473794971 +0000 UTC m=+0.573902187 container start 31aa4a5f93ef579d4d4fbf9a91be9683795e4e3f0ba6f31b8292c0137b3893a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-j, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T21:26:26.818 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:26:26 smithi001 ceph-mds[65894]: -- 172.21.15.1:0/3004693265 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d516465a40 con 0x55d516439000 2024-04-26T21:26:26.818 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:26:26 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-j[65890]: starting mds.j at 2024-04-26T21:26:26.818 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:26:26 smithi001 bash[65875]: 31aa4a5f93ef579d4d4fbf9a91be9683795e4e3f0ba6f31b8292c0137b3893a0 2024-04-26T21:26:26.818 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:26:26 smithi001 systemd[1]: Started Ceph mds.j for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:26.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:26 smithi186 ceph-mon[32991]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:26.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:26 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/703380846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:27.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:26 smithi038 ceph-mon[32724]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:27.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:26 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/703380846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:27.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:27.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.928+0000 7fe5ebe00640 1 -- 172.21.15.1:0/3625095407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5ec1027a0 msgr2=0x7fe5ec102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:27.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.928+0000 7fe5ebe00640 1 --2- 172.21.15.1:0/3625095407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5ec1027a0 0x7fe5ec102bc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009900 tx=0x7fe5dc02a410 comp rx=0 tx=0).stop 2024-04-26T21:26:27.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:27 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.1:6834/2332941873,v1:172.21.15.1:6835/2332941873] up:boot 2024-04-26T21:26:27.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:27 smithi001 ceph-mon[28054]: fsmap 3 up:standby 2024-04-26T21:26:27.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:27 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T21:26:27.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:27 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:27.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:27 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:27.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:27 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:27.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:27 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:27.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:27 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:27.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:27 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:27.930 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:27 smithi001 ceph-mon[28054]: Deploying daemon mds.d on smithi001 2024-04-26T21:26:27.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.928+0000 7fe5ebe00640 1 -- 172.21.15.1:0/3625095407 shutdown_connections 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.928+0000 7fe5ebe00640 1 --2- 172.21.15.1:0/3625095407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5ec104350 0x7fe5ec10abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.928+0000 7fe5ebe00640 1 --2- 172.21.15.1:0/3625095407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec103990 0x7fe5ec103e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.928+0000 7fe5ebe00640 1 --2- 172.21.15.1:0/3625095407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5ec1027a0 0x7fe5ec102bc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.928+0000 7fe5ebe00640 1 -- 172.21.15.1:0/3625095407 >> 172.21.15.1:0/3625095407 conn(0x7fe5ec0fdd10 msgr2=0x7fe5ec100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.929+0000 7fe5ebe00640 1 -- 172.21.15.1:0/3625095407 shutdown_connections 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.929+0000 7fe5ebe00640 1 -- 172.21.15.1:0/3625095407 wait complete. 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.930+0000 7fe5ebe00640 1 Processor -- start 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.930+0000 7fe5ebe00640 1 -- start start 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.930+0000 7fe5ebe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5ec1027a0 0x7fe5ec19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.930+0000 7fe5ebe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec103990 0x7fe5ec19b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.930+0000 7fe5ebe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5ec104350 0x7fe5ec1a2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:27.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.930+0000 7fe5ebe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec10e2d0 con 0x7fe5ec1027a0 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.930+0000 7fe5ebe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec10e150 con 0x7fe5ec103990 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.930+0000 7fe5ebe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ec10e450 con 0x7fe5ec104350 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5eaa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5ec1027a0 0x7fe5ec19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5eaa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5ec1027a0 0x7fe5ec19aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34546/0 (socket says 172.21.15.1:34546) 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5eaa00640 1 -- 172.21.15.1:0/1417103585 learned_addr learned my addr 172.21.15.1:0/1417103585 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5e3e00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec103990 0x7fe5ec19b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5eb400640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5ec104350 0x7fe5ec1a2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5e3e00640 1 -- 172.21.15.1:0/1417103585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5ec104350 msgr2=0x7fe5ec1a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5e3e00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5ec104350 0x7fe5ec1a2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5e3e00640 1 -- 172.21.15.1:0/1417103585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5ec1027a0 msgr2=0x7fe5ec19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5e3e00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5ec1027a0 0x7fe5ec19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5e3e00640 1 -- 172.21.15.1:0/1417103585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ec1a26f0 con 0x7fe5ec103990 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5eaa00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5ec1027a0 0x7fe5ec19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:27.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.931+0000 7fe5e3e00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec103990 0x7fe5ec19b010 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400e9f0 tx=0x7fe5d400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:27.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.932+0000 7fe5e3400640 1 -- 172.21.15.1:0/1417103585 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d400ccc0 con 0x7fe5ec103990 2024-04-26T21:26:27.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.932+0000 7fe5ebe00640 1 -- 172.21.15.1:0/1417103585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ec1a29e0 con 0x7fe5ec103990 2024-04-26T21:26:27.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.932+0000 7fe5e3400640 1 -- 172.21.15.1:0/1417103585 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5d40108e0 con 0x7fe5ec103990 2024-04-26T21:26:27.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.932+0000 7fe5e3400640 1 -- 172.21.15.1:0/1417103585 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5d4018880 con 0x7fe5ec103990 2024-04-26T21:26:27.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.932+0000 7fe5ebe00640 1 -- 172.21.15.1:0/1417103585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ec1a2f30 con 0x7fe5ec103990 2024-04-26T21:26:27.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.933+0000 7fe5ebe00640 1 -- 172.21.15.1:0/1417103585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ec1a3160 con 0x7fe5ec103990 2024-04-26T21:26:27.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.935+0000 7fe5e3400640 1 -- 172.21.15.1:0/1417103585 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fe5d4010430 con 0x7fe5ec103990 2024-04-26T21:26:27.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.937+0000 7fe5e3400640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe5d00a3c50 0x7fe5d00a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:27.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.938+0000 7fe5e3400640 1 -- 172.21.15.1:0/1417103585 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fe5d40d02f0 con 0x7fe5ec103990 2024-04-26T21:26:27.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.938+0000 7fe5eaa00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe5d00a3c50 0x7fe5d00a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:27.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.939+0000 7fe5eaa00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe5d00a3c50 0x7fe5d00a6110 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc032fd0 tx=0x7fe5dc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:27.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:27.941+0000 7fe5e3400640 1 -- 172.21.15.1:0/1417103585 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe5d409ca80 con 0x7fe5ec103990 2024-04-26T21:26:27.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:27 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.1:6834/2332941873,v1:172.21.15.1:6835/2332941873] up:boot 2024-04-26T21:26:27.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:27 smithi186 ceph-mon[32991]: fsmap 3 up:standby 2024-04-26T21:26:27.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:27 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T21:26:27.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:27 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:27.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:27 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:27.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:27 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:27.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:27 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:27.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:27 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:27.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:27 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:27.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:27 smithi186 ceph-mon[32991]: Deploying daemon mds.d on smithi001 2024-04-26T21:26:28.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:27 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.1:6834/2332941873,v1:172.21.15.1:6835/2332941873] up:boot 2024-04-26T21:26:28.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:27 smithi038 ceph-mon[32724]: fsmap 3 up:standby 2024-04-26T21:26:28.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:27 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T21:26:28.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:27 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:28.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:27 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:28.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:27 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:28.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:27 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:28.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:27 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:28.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:27 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:28.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:27 smithi038 ceph-mon[32724]: Deploying daemon mds.d on smithi001 2024-04-26T21:26:28.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.183+0000 7fe5ebe00640 1 -- 172.21.15.1:0/1417103585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5ec1a3330 con 0x7fe5ec103990 2024-04-26T21:26:28.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.183+0000 7fe5e3400640 1 -- 172.21.15.1:0/1417103585 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7fe5d40a0d80 con 0x7fe5ec103990 2024-04-26T21:26:28.184 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:28.185 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-26T21:26:28.185 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 4 2024-04-26T21:26:28.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 -- 172.21.15.1:0/1417103585 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe5d00a3c50 msgr2=0x7fe5d00a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:28.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe5d00a3c50 0x7fe5d00a6110 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc032fd0 tx=0x7fe5dc0314a0 comp rx=0 tx=0).stop 2024-04-26T21:26:28.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 -- 172.21.15.1:0/1417103585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec103990 msgr2=0x7fe5ec19b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:28.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec103990 0x7fe5ec19b010 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe5d400e9f0 tx=0x7fe5d400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:26:28.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 -- 172.21.15.1:0/1417103585 shutdown_connections 2024-04-26T21:26:28.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe5d00a3c50 0x7fe5d00a6110 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:28.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5ec104350 0x7fe5ec1a2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:28.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ec103990 0x7fe5ec19b010 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:28.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 --2- 172.21.15.1:0/1417103585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5ec1027a0 0x7fe5ec19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:28.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 -- 172.21.15.1:0/1417103585 >> 172.21.15.1:0/1417103585 conn(0x7fe5ec0fdd10 msgr2=0x7fe5ec074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:28.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 -- 172.21.15.1:0/1417103585 shutdown_connections 2024-04-26T21:26:28.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:28.186+0000 7fe5ebe00640 1 -- 172.21.15.1:0/1417103585 wait complete. 2024-04-26T21:26:28.964 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:28 smithi001 ceph-mon[28054]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:28.964 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:28 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1417103585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:29.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:28 smithi038 ceph-mon[32724]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:29.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:28 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1417103585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:29.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:28 smithi186 ceph-mon[32991]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:29.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:28 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1417103585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:30.068 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:26:29 smithi001 systemd[1]: Starting Ceph mds.d for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:26:30.205 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:30.420 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:26:30 smithi001 podman[66215]: 2024-04-26 21:26:30.065196567 +0000 UTC m=+0.033805880 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:30.420 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:26:30 smithi001 podman[66215]: 2024-04-26 21:26:30.181948755 +0000 UTC m=+0.150558107 container create bdead4d57d832ebcef6d0d6942ca488969dba77d50715bd77b16a8d74a35fc0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-26T21:26:30.420 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:26:30 smithi001 podman[66215]: 2024-04-26 21:26:30.41931592 +0000 UTC m=+0.387925288 container init bdead4d57d832ebcef6d0d6942ca488969dba77d50715bd77b16a8d74a35fc0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-26T21:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.431+0000 7fdb25e00640 1 -- 172.21.15.1:0/2654254470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb201031f0 msgr2=0x7fdb201055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.431+0000 7fdb25e00640 1 --2- 172.21.15.1:0/2654254470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb201031f0 0x7fdb201055e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb14009860 tx=0x7fdb1402a340 comp rx=0 tx=0).stop 2024-04-26T21:26:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.431+0000 7fdb25e00640 1 -- 172.21.15.1:0/2654254470 shutdown_connections 2024-04-26T21:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.432+0000 7fdb25e00640 1 --2- 172.21.15.1:0/2654254470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb20108450 0x7fdb2010a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.432+0000 7fdb25e00640 1 --2- 172.21.15.1:0/2654254470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb20105b20 0x7fdb20107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.432+0000 7fdb25e00640 1 --2- 172.21.15.1:0/2654254470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb201031f0 0x7fdb201055e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.432+0000 7fdb25e00640 1 -- 172.21.15.1:0/2654254470 >> 172.21.15.1:0/2654254470 conn(0x7fdb200fcc00 msgr2=0x7fdb200ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.432+0000 7fdb25e00640 1 -- 172.21.15.1:0/2654254470 shutdown_connections 2024-04-26T21:26:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.432+0000 7fdb25e00640 1 -- 172.21.15.1:0/2654254470 wait complete. 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb25e00640 1 Processor -- start 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb25e00640 1 -- start start 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb25e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb201031f0 0x7fdb2019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb25e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb20105b20 0x7fdb2019afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb25e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb20108450 0x7fdb201a2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb25e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2010e2c0 con 0x7fdb20105b20 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb25e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2010e140 con 0x7fdb201031f0 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb25e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2010e440 con 0x7fdb20108450 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb1fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb20105b20 0x7fdb2019afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb24a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb201031f0 0x7fdb2019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.433+0000 7fdb24a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb201031f0 0x7fdb2019aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:55122/0 (socket says 172.21.15.1:55122) 2024-04-26T21:26:30.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.434+0000 7fdb24a00640 1 -- 172.21.15.1:0/1032395279 learned_addr learned my addr 172.21.15.1:0/1032395279 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.434+0000 7fdb25400640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb20108450 0x7fdb201a2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.434+0000 7fdb24a00640 1 -- 172.21.15.1:0/1032395279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb20108450 msgr2=0x7fdb201a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.434+0000 7fdb24a00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb20108450 0x7fdb201a2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.434+0000 7fdb24a00640 1 -- 172.21.15.1:0/1032395279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb20105b20 msgr2=0x7fdb2019afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.434+0000 7fdb24a00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb20105b20 0x7fdb2019afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.434+0000 7fdb24a00640 1 -- 172.21.15.1:0/1032395279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb201a26b0 con 0x7fdb201031f0 2024-04-26T21:26:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.434+0000 7fdb24a00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb201031f0 0x7fdb2019aa90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdb140057c0 tx=0x7fdb14032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:30.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.435+0000 7fdb1d600640 1 -- 172.21.15.1:0/1032395279 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14037040 con 0x7fdb201031f0 2024-04-26T21:26:30.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.435+0000 7fdb25e00640 1 -- 172.21.15.1:0/1032395279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb201a2940 con 0x7fdb201031f0 2024-04-26T21:26:30.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.435+0000 7fdb1d600640 1 -- 172.21.15.1:0/1032395279 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb14030bd0 con 0x7fdb201031f0 2024-04-26T21:26:30.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.435+0000 7fdb1d600640 1 -- 172.21.15.1:0/1032395279 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb14005280 con 0x7fdb201031f0 2024-04-26T21:26:30.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.435+0000 7fdb25e00640 1 -- 172.21.15.1:0/1032395279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb201a2e20 con 0x7fdb201031f0 2024-04-26T21:26:30.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.436+0000 7fdb25e00640 1 -- 172.21.15.1:0/1032395279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb20109fa0 con 0x7fdb201031f0 2024-04-26T21:26:30.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.438+0000 7fdb1d600640 1 -- 172.21.15.1:0/1032395279 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fdb14043020 con 0x7fdb201031f0 2024-04-26T21:26:30.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.440+0000 7fdb1d600640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb080a3c50 0x7fdb080a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:30.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.440+0000 7fdb1d600640 1 -- 172.21.15.1:0/1032395279 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fdb140efbe0 con 0x7fdb201031f0 2024-04-26T21:26:30.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.440+0000 7fdb1fe00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb080a3c50 0x7fdb080a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:30.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.441+0000 7fdb1fe00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb080a3c50 0x7fdb080a6110 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdb2019bfa0 tx=0x7fdb0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:30.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.442+0000 7fdb1d600640 1 -- 172.21.15.1:0/1032395279 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdb140bc9c0 con 0x7fdb201031f0 2024-04-26T21:26:30.737 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:26:30 smithi001 podman[66215]: 2024-04-26 21:26:30.427050673 +0000 UTC m=+0.395659999 container start bdead4d57d832ebcef6d0d6942ca488969dba77d50715bd77b16a8d74a35fc0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:26:30.737 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:26:30 smithi001 ceph-mds[66262]: -- 172.21.15.1:0/2717136642 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c7fc6cba40 con 0x55c7fc69ec00 2024-04-26T21:26:30.737 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:26:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-d[66250]: starting mds.d at 2024-04-26T21:26:30.737 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:26:30 smithi001 bash[66215]: bdead4d57d832ebcef6d0d6942ca488969dba77d50715bd77b16a8d74a35fc0b 2024-04-26T21:26:30.737 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:26:30 smithi001 systemd[1]: Started Ceph mds.d for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:30.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.747+0000 7fdb25e00640 1 -- 172.21.15.1:0/1032395279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb2002dbf0 con 0x7fdb201031f0 2024-04-26T21:26:30.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.748+0000 7fdb1d600640 1 -- 172.21.15.1:0/1032395279 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3862 (secure 0 0 0) 0x7fdb140328c0 con 0x7fdb201031f0 2024-04-26T21:26:30.749 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:30.749 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-26T21:26:30.749 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 5 2024-04-26T21:26:30.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 -- 172.21.15.1:0/1032395279 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb080a3c50 msgr2=0x7fdb080a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:30.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb080a3c50 0x7fdb080a6110 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdb2019bfa0 tx=0x7fdb0c009260 comp rx=0 tx=0).stop 2024-04-26T21:26:30.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 -- 172.21.15.1:0/1032395279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb201031f0 msgr2=0x7fdb2019aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:30.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb201031f0 0x7fdb2019aa90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdb140057c0 tx=0x7fdb14032da0 comp rx=0 tx=0).stop 2024-04-26T21:26:30.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 -- 172.21.15.1:0/1032395279 shutdown_connections 2024-04-26T21:26:30.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb080a3c50 0x7fdb080a6110 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:30.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb20108450 0x7fdb201a2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:30.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb20105b20 0x7fdb2019afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:30.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 --2- 172.21.15.1:0/1032395279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb201031f0 0x7fdb2019aa90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:30.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 -- 172.21.15.1:0/1032395279 >> 172.21.15.1:0/1032395279 conn(0x7fdb200fcc00 msgr2=0x7fdb20102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:30.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 -- 172.21.15.1:0/1032395279 shutdown_connections 2024-04-26T21:26:30.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:30.751+0000 7fdb25e00640 1 -- 172.21.15.1:0/1032395279 wait complete. 2024-04-26T21:26:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:30 smithi001 ceph-mon[28054]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:30 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:30 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:30 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:30 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:30 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:30 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:30 smithi038 ceph-mon[32724]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:30 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:30 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:30 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:30 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:30 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:30 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:31.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:30 smithi186 ceph-mon[32991]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:31.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:30 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:31.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:30 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:31.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:30 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:31.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:30 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:31.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:30 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:31.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:30 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:32.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:31 smithi001 ceph-mon[28054]: Deploying daemon mds.e on smithi038 2024-04-26T21:26:32.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:31 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.1:6836/532288494,v1:172.21.15.1:6837/532288494] up:boot 2024-04-26T21:26:32.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:31 smithi001 ceph-mon[28054]: fsmap 4 up:standby 2024-04-26T21:26:32.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:31 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T21:26:32.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:31 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1032395279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:32.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:31 smithi038 ceph-mon[32724]: Deploying daemon mds.e on smithi038 2024-04-26T21:26:32.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:31 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.1:6836/532288494,v1:172.21.15.1:6837/532288494] up:boot 2024-04-26T21:26:32.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:31 smithi038 ceph-mon[32724]: fsmap 4 up:standby 2024-04-26T21:26:32.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:31 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T21:26:32.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:31 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1032395279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:31 smithi186 ceph-mon[32991]: Deploying daemon mds.e on smithi038 2024-04-26T21:26:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:31 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.1:6836/532288494,v1:172.21.15.1:6837/532288494] up:boot 2024-04-26T21:26:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:31 smithi186 ceph-mon[32991]: fsmap 4 up:standby 2024-04-26T21:26:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:31 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T21:26:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:31 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1032395279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:32.773 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:32.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.997+0000 7f1c08a00640 1 -- 172.21.15.1:0/2209057459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c04108d90 msgr2=0x7f1c041091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:32.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.997+0000 7f1c08a00640 1 --2- 172.21.15.1:0/2209057459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c04108d90 0x7f1c041091b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009860 tx=0x7f1bf402a340 comp rx=0 tx=0).stop 2024-04-26T21:26:32.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.997+0000 7f1c08a00640 1 -- 172.21.15.1:0/2209057459 shutdown_connections 2024-04-26T21:26:32.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.997+0000 7f1c08a00640 1 --2- 172.21.15.1:0/2209057459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c0410a940 0x7f1c041111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:32.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.997+0000 7f1c08a00640 1 --2- 172.21.15.1:0/2209057459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c04109f80 0x7f1c0410a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.997+0000 7f1c08a00640 1 --2- 172.21.15.1:0/2209057459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c04108d90 0x7f1c041091b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.997+0000 7f1c08a00640 1 -- 172.21.15.1:0/2209057459 >> 172.21.15.1:0/2209057459 conn(0x7f1c04077d30 msgr2=0x7f1c0407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.997+0000 7f1c08a00640 1 -- 172.21.15.1:0/2209057459 shutdown_connections 2024-04-26T21:26:32.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.997+0000 7f1c08a00640 1 -- 172.21.15.1:0/2209057459 wait complete. 2024-04-26T21:26:33.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.998+0000 7f1c08a00640 1 Processor -- start 2024-04-26T21:26:33.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.998+0000 7f1c08a00640 1 -- start start 2024-04-26T21:26:33.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1c08a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c04108d90 0x7f1c0419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:33.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1c08a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c04109f80 0x7f1c0419f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:33.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1c08a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0410a940 0x7f1c041a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:33.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1c08a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1c041148b0 con 0x7f1c04109f80 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1c08a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1c04114730 con 0x7f1c0410a940 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1c08a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1c04114a30 con 0x7f1c04108d90 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c04109f80 0x7f1c0419f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1c03200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0410a940 0x7f1c041a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c04109f80 0x7f1c0419f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34586/0 (socket says 172.21.15.1:34586) 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1bfbe00640 1 -- 172.21.15.1:0/3792024804 learned_addr learned my addr 172.21.15.1:0/3792024804 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1c02800640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c04108d90 0x7f1c0419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1bfbe00640 1 -- 172.21.15.1:0/3792024804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c04108d90 msgr2=0x7f1c0419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1bfbe00640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c04108d90 0x7f1c0419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1bfbe00640 1 -- 172.21.15.1:0/3792024804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0410a940 msgr2=0x7f1c041a64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1bfbe00640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0410a940 0x7f1c041a64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1bfbe00640 1 -- 172.21.15.1:0/3792024804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c041a6b50 con 0x7f1c04109f80 2024-04-26T21:26:33.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:32.999+0000 7f1c03200640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0410a940 0x7f1c041a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:33.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.000+0000 7f1bfbe00640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c04109f80 0x7f1c0419f470 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00e9f0 tx=0x7f1bec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:33.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.000+0000 7f1bfb400640 1 -- 172.21.15.1:0/3792024804 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec00cd80 con 0x7f1c04109f80 2024-04-26T21:26:33.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.000+0000 7f1bfb400640 1 -- 172.21.15.1:0/3792024804 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bec0054d0 con 0x7f1c04109f80 2024-04-26T21:26:33.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.000+0000 7f1bfb400640 1 -- 172.21.15.1:0/3792024804 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec010690 con 0x7f1c04109f80 2024-04-26T21:26:33.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.000+0000 7f1c08a00640 1 -- 172.21.15.1:0/3792024804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c041a6e40 con 0x7f1c04109f80 2024-04-26T21:26:33.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.000+0000 7f1c08a00640 1 -- 172.21.15.1:0/3792024804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c041a72b0 con 0x7f1c04109f80 2024-04-26T21:26:33.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.001+0000 7f1c08a00640 1 -- 172.21.15.1:0/3792024804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c041100b0 con 0x7f1c04109f80 2024-04-26T21:26:33.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.003+0000 7f1bfb400640 1 -- 172.21.15.1:0/3792024804 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f1bec004e90 con 0x7f1c04109f80 2024-04-26T21:26:33.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.005+0000 7f1bfb400640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1be80a3cd0 0x7f1be80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:33.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.005+0000 7f1bfb400640 1 -- 172.21.15.1:0/3792024804 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f1bec0d0360 con 0x7f1c04109f80 2024-04-26T21:26:33.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.005+0000 7f1c02800640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1be80a3cd0 0x7f1be80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:33.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.006+0000 7f1c02800640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1be80a3cd0 0x7f1be80a6190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1bf402a930 tx=0x7f1bf4030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:33.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.009+0000 7f1bfb400640 1 -- 172.21.15.1:0/3792024804 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1bec09d140 con 0x7f1c04109f80 2024-04-26T21:26:33.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:32 smithi001 ceph-mon[28054]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:33.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:32 smithi038 ceph-mon[32724]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:33.077 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:26:32 smithi038 systemd[1]: Starting Ceph mds.e for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:26:33.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:32 smithi186 ceph-mon[32991]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:33.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.257+0000 7f1c08a00640 1 -- 172.21.15.1:0/3792024804 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c0402dad0 con 0x7f1c04109f80 2024-04-26T21:26:33.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.259+0000 7f1bfb400640 1 -- 172.21.15.1:0/3792024804 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3862 (secure 0 0 0) 0x7f1bec09ca40 con 0x7f1c04109f80 2024-04-26T21:26:33.260 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:33.260 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-26T21:26:33.260 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 5 2024-04-26T21:26:33.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.262+0000 7f1c08a00640 1 -- 172.21.15.1:0/3792024804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1be80a3cd0 msgr2=0x7f1be80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:33.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.262+0000 7f1c08a00640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1be80a3cd0 0x7f1be80a6190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1bf402a930 tx=0x7f1bf4030470 comp rx=0 tx=0).stop 2024-04-26T21:26:33.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.262+0000 7f1c08a00640 1 -- 172.21.15.1:0/3792024804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c04109f80 msgr2=0x7f1c0419f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:33.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.262+0000 7f1c08a00640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c04109f80 0x7f1c0419f470 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00e9f0 tx=0x7f1bec00edf0 comp rx=0 tx=0).stop 2024-04-26T21:26:33.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.262+0000 7f1c08a00640 1 -- 172.21.15.1:0/3792024804 shutdown_connections 2024-04-26T21:26:33.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.262+0000 7f1c08a00640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1be80a3cd0 0x7f1be80a6190 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:33.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.262+0000 7f1c08a00640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c0410a940 0x7f1c041a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:33.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.262+0000 7f1c08a00640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c04109f80 0x7f1c0419f470 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:33.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.262+0000 7f1c08a00640 1 --2- 172.21.15.1:0/3792024804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c04108d90 0x7f1c0419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:33.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.263+0000 7f1c08a00640 1 -- 172.21.15.1:0/3792024804 >> 172.21.15.1:0/3792024804 conn(0x7f1c04077d30 msgr2=0x7f1c0410d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:33.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.263+0000 7f1c08a00640 1 -- 172.21.15.1:0/3792024804 shutdown_connections 2024-04-26T21:26:33.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:33.263+0000 7f1c08a00640 1 -- 172.21.15.1:0/3792024804 wait complete. 2024-04-26T21:26:33.387 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:26:33 smithi038 podman[62165]: 2024-04-26 21:26:33.022858879 +0000 UTC m=+0.018266400 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:33.387 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:26:33 smithi038 podman[62165]: 2024-04-26 21:26:33.13121655 +0000 UTC m=+0.126624047 container create ef0064fcc81e70980eb15dddbc6e74677cd5d2d770212fb155bf3a4ea87173ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:26:33.387 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:26:33 smithi038 podman[62165]: 2024-04-26 21:26:33.359187467 +0000 UTC m=+0.354594964 container init ef0064fcc81e70980eb15dddbc6e74677cd5d2d770212fb155bf3a4ea87173ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:26:33.387 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:26:33 smithi038 podman[62165]: 2024-04-26 21:26:33.362333475 +0000 UTC m=+0.357740972 container start ef0064fcc81e70980eb15dddbc6e74677cd5d2d770212fb155bf3a4ea87173ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True) 2024-04-26T21:26:33.387 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:26:33 smithi038 ceph-mds[62184]: -- 172.21.15.38:0/1002526855 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f0de9d1a40 con 0x55f0de9a5000 2024-04-26T21:26:33.764 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:26:33 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e[62180]: starting mds.e at 2024-04-26T21:26:33.765 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:26:33 smithi038 bash[62165]: ef0064fcc81e70980eb15dddbc6e74677cd5d2d770212fb155bf3a4ea87173ae 2024-04-26T21:26:33.765 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:26:33 smithi038 systemd[1]: Started Ceph mds.e for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:34.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:33 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3792024804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:34.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:33 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:34.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:33 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:34.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:33 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:34.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:33 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:34.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:33 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:34.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:33 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:33 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3792024804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:33 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:33 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:33 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:33 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:33 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:33 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:34.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:33 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3792024804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:34.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:33 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:34.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:33 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:34.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:33 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:34.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:33 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:34.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:33 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:34.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:33 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:35.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:34 smithi001 ceph-mon[28054]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:35.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:34 smithi001 ceph-mon[28054]: Deploying daemon mds.f on smithi186 2024-04-26T21:26:35.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:34 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.38:6834/3801556553,v1:172.21.15.38:6835/3801556553] up:boot 2024-04-26T21:26:35.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:34 smithi001 ceph-mon[28054]: fsmap 5 up:standby 2024-04-26T21:26:35.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:34 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:26:35.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:34 smithi038 ceph-mon[32724]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:35.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:34 smithi038 ceph-mon[32724]: Deploying daemon mds.f on smithi186 2024-04-26T21:26:35.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:34 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.38:6834/3801556553,v1:172.21.15.38:6835/3801556553] up:boot 2024-04-26T21:26:35.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:34 smithi038 ceph-mon[32724]: fsmap 5 up:standby 2024-04-26T21:26:35.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:34 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:26:35.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:34 smithi186 ceph-mon[32991]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:34 smithi186 ceph-mon[32991]: Deploying daemon mds.f on smithi186 2024-04-26T21:26:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:34 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.38:6834/3801556553,v1:172.21.15.38:6835/3801556553] up:boot 2024-04-26T21:26:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:34 smithi186 ceph-mon[32991]: fsmap 5 up:standby 2024-04-26T21:26:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:34 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:26:35.283 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.506+0000 7f8e32600640 1 -- 172.21.15.1:0/825812260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2c105b80 msgr2=0x7f8e2c106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.506+0000 7f8e32600640 1 --2- 172.21.15.1:0/825812260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2c105b80 0x7f8e2c106000 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009900 tx=0x7f8e1402a410 comp rx=0 tx=0).stop 2024-04-26T21:26:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.506+0000 7f8e32600640 1 -- 172.21.15.1:0/825812260 shutdown_connections 2024-04-26T21:26:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.506+0000 7f8e32600640 1 --2- 172.21.15.1:0/825812260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2c105b80 0x7f8e2c106000 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.506+0000 7f8e32600640 1 --2- 172.21.15.1:0/825812260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e2c106540 0x7f8e2c10cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.506+0000 7f8e32600640 1 --2- 172.21.15.1:0/825812260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e2c104990 0x7f8e2c104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.506+0000 7f8e32600640 1 -- 172.21.15.1:0/825812260 >> 172.21.15.1:0/825812260 conn(0x7f8e2c0fff00 msgr2=0x7f8e2c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.506+0000 7f8e32600640 1 -- 172.21.15.1:0/825812260 shutdown_connections 2024-04-26T21:26:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.506+0000 7f8e32600640 1 -- 172.21.15.1:0/825812260 wait complete. 2024-04-26T21:26:35.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.507+0000 7f8e32600640 1 Processor -- start 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.507+0000 7f8e32600640 1 -- start start 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.507+0000 7f8e32600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2c104990 0x7f8e2c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.507+0000 7f8e32600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e2c105b80 0x7f8e2c19afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.507+0000 7f8e32600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e2c106540 0x7f8e2c1a2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.507+0000 7f8e32600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2c110510 con 0x7f8e2c105b80 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.507+0000 7f8e32600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2c110390 con 0x7f8e2c106540 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.507+0000 7f8e32600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2c110690 con 0x7f8e2c104990 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e31200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2c104990 0x7f8e2c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e23e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e2c105b80 0x7f8e2c19afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e31c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e2c106540 0x7f8e2c1a2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e31200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2c104990 0x7f8e2c19aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46436/0 (socket says 172.21.15.1:46436) 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e31200640 1 -- 172.21.15.1:0/678101349 learned_addr learned my addr 172.21.15.1:0/678101349 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e31200640 1 -- 172.21.15.1:0/678101349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e2c106540 msgr2=0x7f8e2c1a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e31200640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e2c106540 0x7f8e2c1a2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e31200640 1 -- 172.21.15.1:0/678101349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e2c105b80 msgr2=0x7f8e2c19afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e31200640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e2c105b80 0x7f8e2c19afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e31200640 1 -- 172.21.15.1:0/678101349 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e2c1a26c0 con 0x7f8e2c104990 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.508+0000 7f8e31200640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2c104990 0x7f8e2c19aaa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00e9c0 tx=0x7f8e1c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.509+0000 7f8e2a000640 1 -- 172.21.15.1:0/678101349 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c00cd80 con 0x7f8e2c104990 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.509+0000 7f8e2a000640 1 -- 172.21.15.1:0/678101349 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e1c0035f0 con 0x7f8e2c104990 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.509+0000 7f8e2a000640 1 -- 172.21.15.1:0/678101349 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c010690 con 0x7f8e2c104990 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.509+0000 7f8e32600640 1 -- 172.21.15.1:0/678101349 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e2c1a29b0 con 0x7f8e2c104990 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.509+0000 7f8e32600640 1 -- 172.21.15.1:0/678101349 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e2c1a2e80 con 0x7f8e2c104990 2024-04-26T21:26:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.510+0000 7f8e32600640 1 -- 172.21.15.1:0/678101349 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e2c10bbe0 con 0x7f8e2c104990 2024-04-26T21:26:35.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.512+0000 7f8e2a000640 1 -- 172.21.15.1:0/678101349 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f8e1c0037a0 con 0x7f8e2c104990 2024-04-26T21:26:35.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.513+0000 7f8e2a000640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e100a3cd0 0x7f8e100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:35.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.513+0000 7f8e2a000640 1 -- 172.21.15.1:0/678101349 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f8e1c0d0300 con 0x7f8e2c104990 2024-04-26T21:26:35.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.514+0000 7f8e23e00640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e100a3cd0 0x7f8e100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:35.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.514+0000 7f8e23e00640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e100a3cd0 0x7f8e100a6190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8e14031500 tx=0x7f8e14005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:35.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.517+0000 7f8e2a000640 1 -- 172.21.15.1:0/678101349 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8e1c09d0e0 con 0x7f8e2c104990 2024-04-26T21:26:35.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.765+0000 7f8e32600640 1 -- 172.21.15.1:0/678101349 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e2c02dad0 con 0x7f8e2c104990 2024-04-26T21:26:35.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.766+0000 7f8e2a000640 1 -- 172.21.15.1:0/678101349 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4687 (secure 0 0 0) 0x7f8e1c09c9e0 con 0x7f8e2c104990 2024-04-26T21:26:35.767 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:35.767 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T21:26:35.768 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 6 2024-04-26T21:26:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 -- 172.21.15.1:0/678101349 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e100a3cd0 msgr2=0x7f8e100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e100a3cd0 0x7f8e100a6190 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8e14031500 tx=0x7f8e14005a20 comp rx=0 tx=0).stop 2024-04-26T21:26:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 -- 172.21.15.1:0/678101349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2c104990 msgr2=0x7f8e2c19aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2c104990 0x7f8e2c19aaa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00e9c0 tx=0x7f8e1c00ee80 comp rx=0 tx=0).stop 2024-04-26T21:26:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 -- 172.21.15.1:0/678101349 shutdown_connections 2024-04-26T21:26:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e100a3cd0 0x7f8e100a6190 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e2c106540 0x7f8e2c1a2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e2c105b80 0x7f8e2c19afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 --2- 172.21.15.1:0/678101349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2c104990 0x7f8e2c19aaa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:35.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 -- 172.21.15.1:0/678101349 >> 172.21.15.1:0/678101349 conn(0x7f8e2c0fff00 msgr2=0x7f8e2c108db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:35.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 -- 172.21.15.1:0/678101349 shutdown_connections 2024-04-26T21:26:35.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:35.769+0000 7f8e32600640 1 -- 172.21.15.1:0/678101349 wait complete. 2024-04-26T21:26:35.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:35 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/678101349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:36.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:35 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/678101349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:36.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:35 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/678101349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:36.253 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:26:36 smithi186 podman[63224]: 2024-04-26 21:26:36.230044758 +0000 UTC m=+0.363659193 container start 1fbd5fa336a03bf24c191e18ff7f8f61dee295c7fbe0291746603ceedb7950c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:26:36.254 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:26:36 smithi186 ceph-mds[63243]: -- 172.21.15.186:0/4162752123 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cd6f15fa40 con 0x55cd6f133000 2024-04-26T21:26:36.704 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:26:36 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-f[63239]: starting mds.f at 2024-04-26T21:26:36.705 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:26:36 smithi186 bash[63224]: 1fbd5fa336a03bf24c191e18ff7f8f61dee295c7fbe0291746603ceedb7950c5 2024-04-26T21:26:36.705 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:26:36 smithi186 systemd[1]: Started Ceph mds.f for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:37.114 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:36 smithi186 ceph-mon[32991]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:37.114 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:37.114 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:37.114 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:37.114 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:37.114 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:37.114 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:36 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:37.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:36 smithi001 ceph-mon[28054]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:37.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:37.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:37.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:37.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:37.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:37.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:36 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:36 smithi038 ceph-mon[32724]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:36 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:37.794 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.007+0000 7ff91e400640 1 -- 172.21.15.1:0/4038136037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff91810ab60 msgr2=0x7ff9181113f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.007+0000 7ff91e400640 1 --2- 172.21.15.1:0/4038136037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff91810ab60 0x7ff9181113f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff910009ca0 tx=0x7ff91002a8c0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.008+0000 7ff91e400640 1 -- 172.21.15.1:0/4038136037 shutdown_connections 2024-04-26T21:26:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.008+0000 7ff91e400640 1 --2- 172.21.15.1:0/4038136037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91810a1a0 0x7ff91810a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.008+0000 7ff91e400640 1 --2- 172.21.15.1:0/4038136037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff91810ab60 0x7ff9181113f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.008+0000 7ff91e400640 1 --2- 172.21.15.1:0/4038136037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff918108fb0 0x7ff9181093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.008+0000 7ff91e400640 1 -- 172.21.15.1:0/4038136037 >> 172.21.15.1:0/4038136037 conn(0x7ff918077f40 msgr2=0x7ff91807a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.008+0000 7ff91e400640 1 -- 172.21.15.1:0/4038136037 shutdown_connections 2024-04-26T21:26:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.008+0000 7ff91e400640 1 -- 172.21.15.1:0/4038136037 wait complete. 2024-04-26T21:26:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.009+0000 7ff91e400640 1 Processor -- start 2024-04-26T21:26:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.009+0000 7ff91e400640 1 -- start start 2024-04-26T21:26:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.009+0000 7ff91e400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff918108fb0 0x7ff91819f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.009+0000 7ff91e400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91810a1a0 0x7ff91819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.009+0000 7ff91e400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91810ab60 0x7ff9181a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.009+0000 7ff91e400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff918114b40 con 0x7ff918108fb0 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.009+0000 7ff91e400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff9181149c0 con 0x7ff91810a1a0 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.009+0000 7ff91e400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff918114cc0 con 0x7ff91810ab60 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff90fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91810a1a0 0x7ff91819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff90fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91810a1a0 0x7ff91819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51624/0 (socket says 172.21.15.1:51624) 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff90fe00640 1 -- 172.21.15.1:0/81483371 learned_addr learned my addr 172.21.15.1:0/81483371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff917400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff918108fb0 0x7ff91819f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff917e00640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91810ab60 0x7ff9181a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff90fe00640 1 -- 172.21.15.1:0/81483371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91810ab60 msgr2=0x7ff9181a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff90fe00640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91810ab60 0x7ff9181a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff90fe00640 1 -- 172.21.15.1:0/81483371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff918108fb0 msgr2=0x7ff91819f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff90fe00640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff918108fb0 0x7ff91819f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff90fe00640 1 -- 172.21.15.1:0/81483371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9181a6da0 con 0x7ff91810a1a0 2024-04-26T21:26:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.010+0000 7ff90fe00640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91810a1a0 0x7ff91819f6c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff90000e9f0 tx=0x7ff90000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.011+0000 7ff914c00640 1 -- 172.21.15.1:0/81483371 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90000cd80 con 0x7ff91810a1a0 2024-04-26T21:26:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.011+0000 7ff914c00640 1 -- 172.21.15.1:0/81483371 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9000054d0 con 0x7ff91810a1a0 2024-04-26T21:26:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.011+0000 7ff914c00640 1 -- 172.21.15.1:0/81483371 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff900010690 con 0x7ff91810a1a0 2024-04-26T21:26:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.011+0000 7ff91e400640 1 -- 172.21.15.1:0/81483371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9181a7090 con 0x7ff91810a1a0 2024-04-26T21:26:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.011+0000 7ff91e400640 1 -- 172.21.15.1:0/81483371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9181a7590 con 0x7ff91810a1a0 2024-04-26T21:26:38.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.012+0000 7ff91e400640 1 -- 172.21.15.1:0/81483371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff918110270 con 0x7ff91810a1a0 2024-04-26T21:26:38.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.014+0000 7ff914c00640 1 -- 172.21.15.1:0/81483371 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7ff900004e90 con 0x7ff91810a1a0 2024-04-26T21:26:38.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.015+0000 7ff914c00640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8fc0a3bb0 0x7ff8fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:38.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.015+0000 7ff914c00640 1 -- 172.21.15.1:0/81483371 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff9000d03f0 con 0x7ff91810a1a0 2024-04-26T21:26:38.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.015+0000 7ff917400640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8fc0a3bb0 0x7ff8fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:38.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.016+0000 7ff917400640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8fc0a3bb0 0x7ff8fc0a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff91810a000 tx=0x7ff908005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:38.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.020+0000 7ff914c00640 1 -- 172.21.15.1:0/81483371 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff90009d1d0 con 0x7ff91810a1a0 2024-04-26T21:26:38.140 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:37 smithi186 ceph-mon[32991]: Deploying daemon mds.c on smithi186 2024-04-26T21:26:38.140 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:37 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.186:6834/1778623983,v1:172.21.15.186:6835/1778623983] up:boot 2024-04-26T21:26:38.140 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:37 smithi186 ceph-mon[32991]: fsmap 6 up:standby 2024-04-26T21:26:38.140 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:37 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.269+0000 7ff91e400640 1 -- 172.21.15.1:0/81483371 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff91802dad0 con 0x7ff91810a1a0 2024-04-26T21:26:38.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.270+0000 7ff914c00640 1 -- 172.21.15.1:0/81483371 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5515 (secure 0 0 0) 0x7ff90009cad0 con 0x7ff91810a1a0 2024-04-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-26T21:26:38.271 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 7 2024-04-26T21:26:38.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.272+0000 7ff91e400640 1 -- 172.21.15.1:0/81483371 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8fc0a3bb0 msgr2=0x7ff8fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:38.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.272+0000 7ff91e400640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8fc0a3bb0 0x7ff8fc0a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff91810a000 tx=0x7ff908005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.272+0000 7ff91e400640 1 -- 172.21.15.1:0/81483371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91810a1a0 msgr2=0x7ff91819f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.272+0000 7ff91e400640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91810a1a0 0x7ff91819f6c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff90000e9f0 tx=0x7ff90000edf0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.273+0000 7ff91e400640 1 -- 172.21.15.1:0/81483371 shutdown_connections 2024-04-26T21:26:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.273+0000 7ff91e400640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8fc0a3bb0 0x7ff8fc0a6070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.273+0000 7ff91e400640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff91810ab60 0x7ff9181a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.273+0000 7ff91e400640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91810a1a0 0x7ff91819f6c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.273+0000 7ff91e400640 1 --2- 172.21.15.1:0/81483371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff918108fb0 0x7ff91819f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.273+0000 7ff91e400640 1 -- 172.21.15.1:0/81483371 >> 172.21.15.1:0/81483371 conn(0x7ff918077f40 msgr2=0x7ff91810d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.273+0000 7ff91e400640 1 -- 172.21.15.1:0/81483371 shutdown_connections 2024-04-26T21:26:38.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:38.273+0000 7ff91e400640 1 -- 172.21.15.1:0/81483371 wait complete. 2024-04-26T21:26:38.290 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:37 smithi001 ceph-mon[28054]: Deploying daemon mds.c on smithi186 2024-04-26T21:26:38.290 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:37 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.186:6834/1778623983,v1:172.21.15.186:6835/1778623983] up:boot 2024-04-26T21:26:38.290 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:37 smithi001 ceph-mon[28054]: fsmap 6 up:standby 2024-04-26T21:26:38.291 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:37 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:26:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:37 smithi038 ceph-mon[32724]: Deploying daemon mds.c on smithi186 2024-04-26T21:26:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:37 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.186:6834/1778623983,v1:172.21.15.186:6835/1778623983] up:boot 2024-04-26T21:26:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:37 smithi038 ceph-mon[32724]: fsmap 6 up:standby 2024-04-26T21:26:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:37 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:26:38.844 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:26:38 smithi186 systemd[1]: Starting Ceph mds.c for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:26:39.097 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:38 smithi186 ceph-mon[32991]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:39.098 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:38 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/81483371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:39.098 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:26:38 smithi186 podman[63535]: 2024-04-26 21:26:38.8322076 +0000 UTC m=+0.017858797 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:39.098 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:26:38 smithi186 podman[63535]: 2024-04-26 21:26:38.972317769 +0000 UTC m=+0.157968951 container create f83a8c0b275a4eb9fb6025209f200047d6f35df5d4e0458141b694a8b0757e2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:26:39.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:38 smithi001 ceph-mon[28054]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:39.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:38 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/81483371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:39.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:38 smithi038 ceph-mon[32724]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:39.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:38 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/81483371' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:39.455 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:26:39 smithi186 podman[63535]: 2024-04-26 21:26:39.216879345 +0000 UTC m=+0.402530527 container init f83a8c0b275a4eb9fb6025209f200047d6f35df5d4e0458141b694a8b0757e2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:26:39.455 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:26:39 smithi186 podman[63535]: 2024-04-26 21:26:39.219532333 +0000 UTC m=+0.405183518 container start f83a8c0b275a4eb9fb6025209f200047d6f35df5d4e0458141b694a8b0757e2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T21:26:39.455 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:26:39 smithi186 ceph-mds[63554]: -- 172.21.15.186:0/151777309 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562e7edd3a40 con 0x562e7eda7000 2024-04-26T21:26:39.455 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:26:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c[63550]: starting mds.c at 2024-04-26T21:26:39.455 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:26:39 smithi186 bash[63535]: f83a8c0b275a4eb9fb6025209f200047d6f35df5d4e0458141b694a8b0757e2f 2024-04-26T21:26:39.455 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:26:39 smithi186 systemd[1]: Started Ceph mds.c for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:40.294 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:40.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.527+0000 7f6b97000640 1 -- 172.21.15.1:0/3317669564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b900768f0 msgr2=0x7f6b90074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.527+0000 7f6b97000640 1 --2- 172.21.15.1:0/3317669564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b900768f0 0x7f6b90074d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009900 tx=0x7f6b8002a480 comp rx=0 tx=0).stop 2024-04-26T21:26:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.527+0000 7f6b97000640 1 -- 172.21.15.1:0/3317669564 shutdown_connections 2024-04-26T21:26:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.527+0000 7f6b97000640 1 --2- 172.21.15.1:0/3317669564 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b9010a650 0x7f6b9010cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.527+0000 7f6b97000640 1 --2- 172.21.15.1:0/3317669564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b90075310 0x7f6b90075770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.527+0000 7f6b97000640 1 --2- 172.21.15.1:0/3317669564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b900768f0 0x7f6b90074d40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.527+0000 7f6b97000640 1 -- 172.21.15.1:0/3317669564 >> 172.21.15.1:0/3317669564 conn(0x7f6b900fdd10 msgr2=0x7f6b90100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.527+0000 7f6b97000640 1 -- 172.21.15.1:0/3317669564 shutdown_connections 2024-04-26T21:26:40.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.527+0000 7f6b97000640 1 -- 172.21.15.1:0/3317669564 wait complete. 2024-04-26T21:26:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.529+0000 7f6b97000640 1 Processor -- start 2024-04-26T21:26:40.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.529+0000 7f6b97000640 1 -- start start 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.529+0000 7f6b97000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b90075310 0x7f6b9019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.529+0000 7f6b97000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b900768f0 0x7f6b9019aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.529+0000 7f6b97000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b9010a650 0x7f6b901a2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.529+0000 7f6b97000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6b901104d0 con 0x7f6b9010a650 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.529+0000 7f6b97000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6b90110350 con 0x7f6b900768f0 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.529+0000 7f6b97000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b90110650 con 0x7f6b90075310 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b90075310 0x7f6b9019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b90075310 0x7f6b9019aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:34092/0 (socket says 172.21.15.1:34092) 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95c00640 1 -- 172.21.15.1:0/3384561885 learned_addr learned my addr 172.21.15.1:0/3384561885 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b96600640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b9010a650 0x7f6b901a2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95200640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b900768f0 0x7f6b9019aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95c00640 1 -- 172.21.15.1:0/3384561885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b900768f0 msgr2=0x7f6b9019aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95c00640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b900768f0 0x7f6b9019aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:40.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95c00640 1 -- 172.21.15.1:0/3384561885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b9010a650 msgr2=0x7f6b901a2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:40.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95c00640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b9010a650 0x7f6b901a2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:40.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95c00640 1 -- 172.21.15.1:0/3384561885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b901a26d0 con 0x7f6b90075310 2024-04-26T21:26:40.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95200640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b900768f0 0x7f6b9019aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:40.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.530+0000 7f6b95c00640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b90075310 0x7f6b9019aab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6b80005940 tx=0x7f6b800042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:40.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.531+0000 7f6b8e000640 1 -- 172.21.15.1:0/3384561885 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8003f030 con 0x7f6b90075310 2024-04-26T21:26:40.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.531+0000 7f6b8e000640 1 -- 172.21.15.1:0/3384561885 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b80003cb0 con 0x7f6b90075310 2024-04-26T21:26:40.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.531+0000 7f6b97000640 1 -- 172.21.15.1:0/3384561885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b901a2960 con 0x7f6b90075310 2024-04-26T21:26:40.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.531+0000 7f6b8e000640 1 -- 172.21.15.1:0/3384561885 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b80035070 con 0x7f6b90075310 2024-04-26T21:26:40.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.531+0000 7f6b97000640 1 -- 172.21.15.1:0/3384561885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b901a2ea0 con 0x7f6b90075310 2024-04-26T21:26:40.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.532+0000 7f6b97000640 1 -- 172.21.15.1:0/3384561885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b90109410 con 0x7f6b90075310 2024-04-26T21:26:40.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.534+0000 7f6b8e000640 1 -- 172.21.15.1:0/3384561885 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6b80042030 con 0x7f6b90075310 2024-04-26T21:26:40.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.535+0000 7f6b8e000640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6b740a3cd0 0x7f6b740a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:40.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.536+0000 7f6b8e000640 1 -- 172.21.15.1:0/3384561885 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f6b800f0350 con 0x7f6b90075310 2024-04-26T21:26:40.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.536+0000 7f6b95200640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6b740a3cd0 0x7f6b740a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:40.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.536+0000 7f6b95200640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6b740a3cd0 0x7f6b740a6190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6b9019bfc0 tx=0x7f6b7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.539+0000 7f6b8e000640 1 -- 172.21.15.1:0/3384561885 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6b80003e60 con 0x7f6b90075310 2024-04-26T21:26:40.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:40.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:40.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:40.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:40.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:40.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:40.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:40.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: Deploying daemon mds.h on smithi038 2024-04-26T21:26:40.624 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.186:6836/3582845013,v1:172.21.15.186:6837/3582845013] up:boot 2024-04-26T21:26:40.624 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: fsmap 7 up:standby 2024-04-26T21:26:40.624 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:40 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:26:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.788+0000 7f6b97000640 1 -- 172.21.15.1:0/3384561885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b9002dad0 con 0x7f6b90075310 2024-04-26T21:26:40.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.789+0000 7f6b8e000640 1 -- 172.21.15.1:0/3384561885 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6343 (secure 0 0 0) 0x7f6b800bcae0 con 0x7f6b90075310 2024-04-26T21:26:40.790 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:40.790 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-26T21:26:40.790 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 8 2024-04-26T21:26:40.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 -- 172.21.15.1:0/3384561885 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6b740a3cd0 msgr2=0x7f6b740a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:40.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6b740a3cd0 0x7f6b740a6190 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6b9019bfc0 tx=0x7f6b7c005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:40.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 -- 172.21.15.1:0/3384561885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b90075310 msgr2=0x7f6b9019aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:40.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b90075310 0x7f6b9019aab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6b80005940 tx=0x7f6b800042f0 comp rx=0 tx=0).stop 2024-04-26T21:26:40.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 -- 172.21.15.1:0/3384561885 shutdown_connections 2024-04-26T21:26:40.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6b740a3cd0 0x7f6b740a6190 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:40.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6b9010a650 0x7f6b901a2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:40.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b900768f0 0x7f6b9019aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:40.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 --2- 172.21.15.1:0/3384561885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b90075310 0x7f6b9019aab0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:40.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 -- 172.21.15.1:0/3384561885 >> 172.21.15.1:0/3384561885 conn(0x7f6b900fdd10 msgr2=0x7f6b901065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:40.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 -- 172.21.15.1:0/3384561885 shutdown_connections 2024-04-26T21:26:40.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:40.792+0000 7f6b97000640 1 -- 172.21.15.1:0/3384561885 wait complete. 2024-04-26T21:26:40.809 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:40.809 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:40.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:40.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:40.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:40.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:40.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:40.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: Deploying daemon mds.h on smithi038 2024-04-26T21:26:40.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.186:6836/3582845013,v1:172.21.15.186:6837/3582845013] up:boot 2024-04-26T21:26:40.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: fsmap 7 up:standby 2024-04-26T21:26:40.810 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:40 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: Deploying daemon mds.h on smithi038 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.186:6836/3582845013,v1:172.21.15.186:6837/3582845013] up:boot 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: fsmap 7 up:standby 2024-04-26T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:40 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:26:41.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:41 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3384561885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:41.788 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:41 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3384561885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:41.788 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:26:41 smithi038 systemd[1]: Starting Ceph mds.h for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:26:41.788 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:26:41 smithi038 podman[62477]: 2024-04-26 21:26:41.690393669 +0000 UTC m=+0.018384096 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:41.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:41 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3384561885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:42.051 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:26:41 smithi038 podman[62477]: 2024-04-26 21:26:41.796791726 +0000 UTC m=+0.124782148 container create eaecce66b2abff9392c0b3d1575c4b5724f6d4848d633a9547e191475723d70f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-26T21:26:42.051 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:26:42 smithi038 podman[62477]: 2024-04-26 21:26:42.024979986 +0000 UTC m=+0.352970401 container init eaecce66b2abff9392c0b3d1575c4b5724f6d4848d633a9547e191475723d70f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:26:42.052 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:26:42 smithi038 podman[62477]: 2024-04-26 21:26:42.027596361 +0000 UTC m=+0.355586778 container start eaecce66b2abff9392c0b3d1575c4b5724f6d4848d633a9547e191475723d70f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=) 2024-04-26T21:26:42.327 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:26:42 smithi038 ceph-mds[62496]: -- 172.21.15.38:0/1092365270 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5564d5f19a40 con 0x5564d5eed000 2024-04-26T21:26:42.327 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:26:42 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h[62492]: starting mds.h at 2024-04-26T21:26:42.327 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:26:42 smithi038 bash[62477]: eaecce66b2abff9392c0b3d1575c4b5724f6d4848d633a9547e191475723d70f 2024-04-26T21:26:42.327 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:26:42 smithi038 systemd[1]: Started Ceph mds.h for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:42.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:42 smithi186 ceph-mon[32991]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:42.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:42.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:42 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:42.813 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:42.822 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:42 smithi001 ceph-mon[28054]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:42.822 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:42.822 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:42.822 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:42.822 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:42.823 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:42.823 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:42 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:42 smithi038 ceph-mon[32724]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:42 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.035+0000 7f5d7da00640 1 -- 172.21.15.1:0/2207002212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d78101940 msgr2=0x7f5d78101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.035+0000 7f5d7da00640 1 --2- 172.21.15.1:0/2207002212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d78101940 0x7f5d78101da0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5d60009900 tx=0x7f5d6002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.035+0000 7f5d7da00640 1 -- 172.21.15.1:0/2207002212 shutdown_connections 2024-04-26T21:26:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.035+0000 7f5d7da00640 1 --2- 172.21.15.1:0/2207002212 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d781022e0 0x7f5d78102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.035+0000 7f5d7da00640 1 --2- 172.21.15.1:0/2207002212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d78101940 0x7f5d78101da0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.035+0000 7f5d7da00640 1 --2- 172.21.15.1:0/2207002212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d78100fe0 0x7f5d78101400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.035+0000 7f5d7da00640 1 -- 172.21.15.1:0/2207002212 >> 172.21.15.1:0/2207002212 conn(0x7f5d780fcc00 msgr2=0x7f5d780ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:43.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.035+0000 7f5d7da00640 1 -- 172.21.15.1:0/2207002212 shutdown_connections 2024-04-26T21:26:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.035+0000 7f5d7da00640 1 -- 172.21.15.1:0/2207002212 wait complete. 2024-04-26T21:26:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.036+0000 7f5d7da00640 1 Processor -- start 2024-04-26T21:26:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.036+0000 7f5d7da00640 1 -- start start 2024-04-26T21:26:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d7da00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d78100fe0 0x7f5d7819aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d7da00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d78101940 0x7f5d7819afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d7da00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d781022e0 0x7f5d781a1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d7da00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78110460 con 0x7f5d781022e0 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d7da00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5d781102e0 con 0x7f5d78100fe0 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d7da00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5d781105e0 con 0x7f5d78101940 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d77400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d78101940 0x7f5d7819afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d77e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d78100fe0 0x7f5d7819aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d77400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d78101940 0x7f5d7819afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:34110/0 (socket says 172.21.15.1:34110) 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d77400640 1 -- 172.21.15.1:0/320686301 learned_addr learned my addr 172.21.15.1:0/320686301 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d77e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d78100fe0 0x7f5d7819aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51678/0 (socket says 172.21.15.1:51678) 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d7d000640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d781022e0 0x7f5d781a1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.037+0000 7f5d77400640 1 -- 172.21.15.1:0/320686301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d78100fe0 msgr2=0x7f5d7819aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.038+0000 7f5d77400640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d78100fe0 0x7f5d7819aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.038+0000 7f5d77400640 1 -- 172.21.15.1:0/320686301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d781022e0 msgr2=0x7f5d781a1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.038+0000 7f5d77400640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d781022e0 0x7f5d781a1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.038+0000 7f5d77400640 1 -- 172.21.15.1:0/320686301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d781a2640 con 0x7f5d78101940 2024-04-26T21:26:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.038+0000 7f5d77400640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d78101940 0x7f5d7819afb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d60005940 tx=0x7f5d600042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:43.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.038+0000 7f5d74c00640 1 -- 172.21.15.1:0/320686301 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6003f030 con 0x7f5d78101940 2024-04-26T21:26:43.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.038+0000 7f5d74c00640 1 -- 172.21.15.1:0/320686301 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d60003cb0 con 0x7f5d78101940 2024-04-26T21:26:43.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.038+0000 7f5d74c00640 1 -- 172.21.15.1:0/320686301 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60035070 con 0x7f5d78101940 2024-04-26T21:26:43.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.038+0000 7f5d7da00640 1 -- 172.21.15.1:0/320686301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d781a28d0 con 0x7f5d78101940 2024-04-26T21:26:43.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.039+0000 7f5d7da00640 1 -- 172.21.15.1:0/320686301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d781a2d70 con 0x7f5d78101940 2024-04-26T21:26:43.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.040+0000 7f5d7da00640 1 -- 172.21.15.1:0/320686301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d7810bd30 con 0x7f5d78101940 2024-04-26T21:26:43.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.041+0000 7f5d74c00640 1 -- 172.21.15.1:0/320686301 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f5d60042030 con 0x7f5d78101940 2024-04-26T21:26:43.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.043+0000 7f5d74c00640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d5c0a3c50 0x7f5d5c0a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:43.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.043+0000 7f5d74c00640 1 -- 172.21.15.1:0/320686301 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f5d600f03c0 con 0x7f5d78101940 2024-04-26T21:26:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.043+0000 7f5d77e00640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d5c0a3c50 0x7f5d5c0a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.044+0000 7f5d77e00640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d5c0a3c50 0x7f5d5c0a6110 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5d68005fd0 tx=0x7f5d68009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:43.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.047+0000 7f5d74c00640 1 -- 172.21.15.1:0/320686301 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5d60003e60 con 0x7f5d78101940 2024-04-26T21:26:43.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.295+0000 7f5d7da00640 1 -- 172.21.15.1:0/320686301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d7802dcf0 con 0x7f5d78101940 2024-04-26T21:26:43.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.296+0000 7f5d74c00640 1 -- 172.21.15.1:0/320686301 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7168 (secure 0 0 0) 0x7f5d600dd0e0 con 0x7f5d78101940 2024-04-26T21:26:43.297 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:43.297 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T21:26:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-04-26T21:26:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 -- 172.21.15.1:0/320686301 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d5c0a3c50 msgr2=0x7f5d5c0a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d5c0a3c50 0x7f5d5c0a6110 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5d68005fd0 tx=0x7f5d68009500 comp rx=0 tx=0).stop 2024-04-26T21:26:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 -- 172.21.15.1:0/320686301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d78101940 msgr2=0x7f5d7819afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d78101940 0x7f5d7819afb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5d60005940 tx=0x7f5d600042f0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 -- 172.21.15.1:0/320686301 shutdown_connections 2024-04-26T21:26:43.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d5c0a3c50 0x7f5d5c0a6110 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d78101940 0x7f5d7819afb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5d781022e0 0x7f5d781a1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 --2- 172.21.15.1:0/320686301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5d78100fe0 0x7f5d7819aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:43.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 -- 172.21.15.1:0/320686301 >> 172.21.15.1:0/320686301 conn(0x7f5d780fcc00 msgr2=0x7f5d780fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:43.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 -- 172.21.15.1:0/320686301 shutdown_connections 2024-04-26T21:26:43.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:43.299+0000 7f5d7da00640 1 -- 172.21.15.1:0/320686301 wait complete. 2024-04-26T21:26:43.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:43 smithi001 ceph-mon[28054]: Deploying daemon mds.i on smithi186 2024-04-26T21:26:43.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:43 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] up:boot 2024-04-26T21:26:43.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:43 smithi001 ceph-mon[28054]: fsmap 8 up:standby 2024-04-26T21:26:43.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:43 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:26:43.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:43 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/320686301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:43.632 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:43 smithi186 ceph-mon[32991]: Deploying daemon mds.i on smithi186 2024-04-26T21:26:43.632 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:43 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] up:boot 2024-04-26T21:26:43.632 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:43 smithi186 ceph-mon[32991]: fsmap 8 up:standby 2024-04-26T21:26:43.632 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:43 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:26:43.632 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:43 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/320686301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:43 smithi038 ceph-mon[32724]: Deploying daemon mds.i on smithi186 2024-04-26T21:26:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:43 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] up:boot 2024-04-26T21:26:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:43 smithi038 ceph-mon[32724]: fsmap 8 up:standby 2024-04-26T21:26:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:43 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:26:43.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:43 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/320686301' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:44.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:44 smithi186 ceph-mon[32991]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:44.705 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:26:44 smithi186 systemd[1]: Starting Ceph mds.i for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:26:44.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:44 smithi001 ceph-mon[28054]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:44.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:44 smithi038 ceph-mon[32724]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:45.064 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:26:44 smithi186 podman[63857]: 2024-04-26 21:26:44.693614575 +0000 UTC m=+0.017466410 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:45.064 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:26:44 smithi186 podman[63857]: 2024-04-26 21:26:44.809824707 +0000 UTC m=+0.133676542 container create b230804d290a2ae26a1640ad057d80c27f7f903232c960f00cd4e500282471b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-26T21:26:45.064 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:26:45 smithi186 podman[63857]: 2024-04-26 21:26:45.037432264 +0000 UTC m=+0.361284099 container init b230804d290a2ae26a1640ad057d80c27f7f903232c960f00cd4e500282471b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-26T21:26:45.064 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:26:45 smithi186 podman[63857]: 2024-04-26 21:26:45.040040324 +0000 UTC m=+0.363892162 container start b230804d290a2ae26a1640ad057d80c27f7f903232c960f00cd4e500282471b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T21:26:45.064 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:26:45 smithi186 ceph-mds[63876]: -- 172.21.15.186:0/3252100183 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56413ceaba40 con 0x56413ce7e800 2024-04-26T21:26:45.321 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:45.454 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:26:45 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: starting mds.i at 2024-04-26T21:26:45.455 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:26:45 smithi186 bash[63857]: b230804d290a2ae26a1640ad057d80c27f7f903232c960f00cd4e500282471b8 2024-04-26T21:26:45.455 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:26:45 smithi186 systemd[1]: Started Ceph mds.i for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.521+0000 7f2847e00640 1 -- 172.21.15.1:0/1797198847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f284810a630 msgr2=0x7f284810cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.521+0000 7f2847e00640 1 --2- 172.21.15.1:0/1797198847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f284810a630 0x7f284810cad0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2840009ca0 tx=0x7f284002a950 comp rx=0 tx=0).stop 2024-04-26T21:26:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.521+0000 7f2847e00640 1 -- 172.21.15.1:0/1797198847 shutdown_connections 2024-04-26T21:26:45.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.521+0000 7f2847e00640 1 --2- 172.21.15.1:0/1797198847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2848075330 0x7f2848075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.521+0000 7f2847e00640 1 --2- 172.21.15.1:0/1797198847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f284810a630 0x7f284810cad0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.522+0000 7f2847e00640 1 --2- 172.21.15.1:0/1797198847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2848076910 0x7f2848074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.522+0000 7f2847e00640 1 -- 172.21.15.1:0/1797198847 >> 172.21.15.1:0/1797198847 conn(0x7f28480fdd10 msgr2=0x7f2848100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.522+0000 7f2847e00640 1 -- 172.21.15.1:0/1797198847 shutdown_connections 2024-04-26T21:26:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.522+0000 7f2847e00640 1 -- 172.21.15.1:0/1797198847 wait complete. 2024-04-26T21:26:45.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.523+0000 7f2847e00640 1 Processor -- start 2024-04-26T21:26:45.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.523+0000 7f2847e00640 1 -- start start 2024-04-26T21:26:45.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2847e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2848075330 0x7f284819aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:45.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2847e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2848076910 0x7f284819af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:45.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2847e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f284810a630 0x7f28481a1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:45.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2847e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2848110520 con 0x7f284810a630 2024-04-26T21:26:45.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2847e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f28481103a0 con 0x7f2848076910 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2847e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28481106a0 con 0x7f2848075330 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2846000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2848076910 0x7f284819af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2846000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2848076910 0x7f284819af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51700/0 (socket says 172.21.15.1:51700) 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2846000640 1 -- 172.21.15.1:0/144214534 learned_addr learned my addr 172.21.15.1:0/144214534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2846a00640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2848075330 0x7f284819aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.524+0000 7f2847400640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f284810a630 0x7f28481a1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f2846a00640 1 -- 172.21.15.1:0/144214534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2848076910 msgr2=0x7f284819af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f2846a00640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2848076910 0x7f284819af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f2846a00640 1 -- 172.21.15.1:0/144214534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f284810a630 msgr2=0x7f28481a1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f2846a00640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f284810a630 0x7f28481a1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f2846a00640 1 -- 172.21.15.1:0/144214534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28481a2660 con 0x7f2848075330 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f2846000640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2848076910 0x7f284819af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f2846a00640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2848075330 0x7f284819aa40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f283800e9f0 tx=0x7f283800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:45.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f283f400640 1 -- 172.21.15.1:0/144214534 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283800ccc0 con 0x7f2848075330 2024-04-26T21:26:45.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f283f400640 1 -- 172.21.15.1:0/144214534 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28380108e0 con 0x7f2848075330 2024-04-26T21:26:45.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f2847e00640 1 -- 172.21.15.1:0/144214534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28481a2950 con 0x7f2848075330 2024-04-26T21:26:45.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f283f400640 1 -- 172.21.15.1:0/144214534 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2838018880 con 0x7f2848075330 2024-04-26T21:26:45.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.525+0000 7f2847e00640 1 -- 172.21.15.1:0/144214534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28481a2ea0 con 0x7f2848075330 2024-04-26T21:26:45.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.527+0000 7f2847e00640 1 -- 172.21.15.1:0/144214534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28481093f0 con 0x7f2848075330 2024-04-26T21:26:45.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.528+0000 7f283f400640 1 -- 172.21.15.1:0/144214534 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f2838010430 con 0x7f2848075330 2024-04-26T21:26:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.530+0000 7f283f400640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f282c0a3c50 0x7f282c0a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.530+0000 7f283f400640 1 -- 172.21.15.1:0/144214534 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f28380d02f0 con 0x7f2848075330 2024-04-26T21:26:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.530+0000 7f2846000640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f282c0a3c50 0x7f282c0a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:45.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.531+0000 7f2846000640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f282c0a3c50 0x7f282c0a6110 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f284819bf50 tx=0x7f2834005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:45.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.534+0000 7f283f400640 1 -- 172.21.15.1:0/144214534 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f283809ca80 con 0x7f2848075330 2024-04-26T21:26:45.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.780+0000 7f2847e00640 1 -- 172.21.15.1:0/144214534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f284802dad0 con 0x7f2848075330 2024-04-26T21:26:45.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.781+0000 7f283f400640 1 -- 172.21.15.1:0/144214534 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7998 (secure 0 0 0) 0x7f28380a0d80 con 0x7f2848075330 2024-04-26T21:26:45.782 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:45.783 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T21:26:45.783 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 10 2024-04-26T21:26:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.784+0000 7f2847e00640 1 -- 172.21.15.1:0/144214534 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f282c0a3c50 msgr2=0x7f282c0a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.784+0000 7f2847e00640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f282c0a3c50 0x7f282c0a6110 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f284819bf50 tx=0x7f2834005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.784+0000 7f2847e00640 1 -- 172.21.15.1:0/144214534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2848075330 msgr2=0x7f284819aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:45.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.784+0000 7f2847e00640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2848075330 0x7f284819aa40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f283800e9f0 tx=0x7f283800eeb0 comp rx=0 tx=0).stop 2024-04-26T21:26:45.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.784+0000 7f2847e00640 1 -- 172.21.15.1:0/144214534 shutdown_connections 2024-04-26T21:26:45.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.784+0000 7f2847e00640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f282c0a3c50 0x7f282c0a6110 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:45.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.784+0000 7f2847e00640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f284810a630 0x7f28481a1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:45.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.784+0000 7f2847e00640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2848076910 0x7f284819af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:45.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.784+0000 7f2847e00640 1 --2- 172.21.15.1:0/144214534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2848075330 0x7f284819aa40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:45.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.785+0000 7f2847e00640 1 -- 172.21.15.1:0/144214534 >> 172.21.15.1:0/144214534 conn(0x7f28480fdd10 msgr2=0x7f28481065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:45.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.785+0000 7f2847e00640 1 -- 172.21.15.1:0/144214534 shutdown_connections 2024-04-26T21:26:45.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:45.785+0000 7f2847e00640 1 -- 172.21.15.1:0/144214534 wait complete. 2024-04-26T21:26:46.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:46.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:46.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:46.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:46.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:46.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:46.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: Deploying daemon mds.g on smithi001 2024-04-26T21:26:46.454 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:46.455 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.186:6838/2477017203,v1:172.21.15.186:6839/2477017203] up:boot 2024-04-26T21:26:46.455 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: fsmap 9 up:standby 2024-04-26T21:26:46.455 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:26:46.455 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:46 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/144214534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:46.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:46.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:46.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:46.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:46.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:46.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:46.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: Deploying daemon mds.g on smithi001 2024-04-26T21:26:46.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:46.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.186:6838/2477017203,v1:172.21.15.186:6839/2477017203] up:boot 2024-04-26T21:26:46.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: fsmap 9 up:standby 2024-04-26T21:26:46.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:26:46.456 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:46 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/144214534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: Deploying daemon mds.g on smithi001 2024-04-26T21:26:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:46.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.186:6838/2477017203,v1:172.21.15.186:6839/2477017203] up:boot 2024-04-26T21:26:46.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: fsmap 9 up:standby 2024-04-26T21:26:46.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:26:46.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:46 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/144214534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:47.804 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.170+0000 7fbd4c800640 1 -- 172.21.15.1:0/2395092666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd44078ed0 msgr2=0x7fbd44079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:48.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.170+0000 7fbd4c800640 1 --2- 172.21.15.1:0/2395092666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd44078ed0 0x7fbd44079770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009be0 tx=0x7fbd4002a6d0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.173+0000 7fbd4c800640 1 -- 172.21.15.1:0/2395092666 shutdown_connections 2024-04-26T21:26:48.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.173+0000 7fbd4c800640 1 --2- 172.21.15.1:0/2395092666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd44078ed0 0x7fbd44079770 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.173+0000 7fbd4c800640 1 --2- 172.21.15.1:0/2395092666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44078530 0x7fbd44078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.173+0000 7fbd4c800640 1 --2- 172.21.15.1:0/2395092666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd440772f0 0x7fbd44077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.174+0000 7fbd4c800640 1 -- 172.21.15.1:0/2395092666 >> 172.21.15.1:0/2395092666 conn(0x7fbd441000e0 msgr2=0x7fbd44102520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.176+0000 7fbd4c800640 1 -- 172.21.15.1:0/2395092666 shutdown_connections 2024-04-26T21:26:48.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.176+0000 7fbd4c800640 1 -- 172.21.15.1:0/2395092666 wait complete. 2024-04-26T21:26:48.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.178+0000 7fbd4c800640 1 Processor -- start 2024-04-26T21:26:48.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.178+0000 7fbd4c800640 1 -- start start 2024-04-26T21:26:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.179+0000 7fbd4c800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd440772f0 0x7fbd4410f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.179+0000 7fbd4c800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44078530 0x7fbd4410fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.179+0000 7fbd4c800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44078ed0 0x7fbd441af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.179+0000 7fbd4c800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbd44114ab0 con 0x7fbd44078ed0 2024-04-26T21:26:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.179+0000 7fbd4c800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbd44114930 con 0x7fbd440772f0 2024-04-26T21:26:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.179+0000 7fbd4c800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbd44114c30 con 0x7fbd44078530 2024-04-26T21:26:48.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.180+0000 7fbd49e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd440772f0 0x7fbd4410f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.180+0000 7fbd49400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44078530 0x7fbd4410fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.180+0000 7fbd49400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44078530 0x7fbd4410fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:39038/0 (socket says 172.21.15.1:39038) 2024-04-26T21:26:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.180+0000 7fbd49400640 1 -- 172.21.15.1:0/2294973817 learned_addr learned my addr 172.21.15.1:0/2294973817 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.180+0000 7fbd4a800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44078ed0 0x7fbd441af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.180+0000 7fbd49e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd440772f0 0x7fbd4410f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34098/0 (socket says 172.21.15.1:34098) 2024-04-26T21:26:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.181+0000 7fbd49e00640 1 -- 172.21.15.1:0/2294973817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44078530 msgr2=0x7fbd4410fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.181+0000 7fbd49e00640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44078530 0x7fbd4410fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.181+0000 7fbd49e00640 1 -- 172.21.15.1:0/2294973817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44078ed0 msgr2=0x7fbd441af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.181+0000 7fbd49e00640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44078ed0 0x7fbd441af180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.181+0000 7fbd49e00640 1 -- 172.21.15.1:0/2294973817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd441af6c0 con 0x7fbd440772f0 2024-04-26T21:26:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.181+0000 7fbd4a800640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44078ed0 0x7fbd441af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.181+0000 7fbd49e00640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd440772f0 0x7fbd4410f590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbd38005fd0 tx=0x7fbd3800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.182+0000 7fbd3ea00640 1 -- 172.21.15.1:0/2294973817 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38009c30 con 0x7fbd440772f0 2024-04-26T21:26:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.182+0000 7fbd3ea00640 1 -- 172.21.15.1:0/2294973817 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd38009dd0 con 0x7fbd440772f0 2024-04-26T21:26:48.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.182+0000 7fbd3ea00640 1 -- 172.21.15.1:0/2294973817 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38003e10 con 0x7fbd440772f0 2024-04-26T21:26:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.182+0000 7fbd4c800640 1 -- 172.21.15.1:0/2294973817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd441af890 con 0x7fbd440772f0 2024-04-26T21:26:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.182+0000 7fbd4c800640 1 -- 172.21.15.1:0/2294973817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd441afd30 con 0x7fbd440772f0 2024-04-26T21:26:48.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.184+0000 7fbd4c800640 1 -- 172.21.15.1:0/2294973817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd44108cd0 con 0x7fbd440772f0 2024-04-26T21:26:48.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.188+0000 7fbd3ea00640 1 -- 172.21.15.1:0/2294973817 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fbd38010460 con 0x7fbd440772f0 2024-04-26T21:26:48.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.191+0000 7fbd3ea00640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbd2c0a3d00 0x7fbd2c0a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.191+0000 7fbd3ea00640 1 -- 172.21.15.1:0/2294973817 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fbd380d01f0 con 0x7fbd440772f0 2024-04-26T21:26:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.191+0000 7fbd49400640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbd2c0a3d00 0x7fbd2c0a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:48.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.192+0000 7fbd49400640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbd2c0a3d00 0x7fbd2c0a61c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbd44110aa0 tx=0x7fbd34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:48.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.194+0000 7fbd3ea00640 1 -- 172.21.15.1:0/2294973817 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fbd38017020 con 0x7fbd440772f0 2024-04-26T21:26:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.451+0000 7fbd4c800640 1 -- 172.21.15.1:0/2294973817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd4402dad0 con 0x7fbd440772f0 2024-04-26T21:26:48.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.452+0000 7fbd3ea00640 1 -- 172.21.15.1:0/2294973817 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7998 (secure 0 0 0) 0x7fbd3809ca00 con 0x7fbd440772f0 2024-04-26T21:26:48.453 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:48.453 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-26T21:26:48.453 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 10 2024-04-26T21:26:48.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.454+0000 7fbd4c800640 1 -- 172.21.15.1:0/2294973817 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbd2c0a3d00 msgr2=0x7fbd2c0a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:48.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.454+0000 7fbd4c800640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbd2c0a3d00 0x7fbd2c0a61c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbd44110aa0 tx=0x7fbd34009260 comp rx=0 tx=0).stop 2024-04-26T21:26:48.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.454+0000 7fbd4c800640 1 -- 172.21.15.1:0/2294973817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd440772f0 msgr2=0x7fbd4410f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:48.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.454+0000 7fbd4c800640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd440772f0 0x7fbd4410f590 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbd38005fd0 tx=0x7fbd3800dde0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.455+0000 7fbd4c800640 1 -- 172.21.15.1:0/2294973817 shutdown_connections 2024-04-26T21:26:48.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.455+0000 7fbd4c800640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbd2c0a3d00 0x7fbd2c0a61c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.455+0000 7fbd4c800640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd44078530 0x7fbd4410fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.455+0000 7fbd4c800640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd44078ed0 0x7fbd441af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.455+0000 7fbd4c800640 1 --2- 172.21.15.1:0/2294973817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd440772f0 0x7fbd4410f590 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:48.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.455+0000 7fbd4c800640 1 -- 172.21.15.1:0/2294973817 >> 172.21.15.1:0/2294973817 conn(0x7fbd441000e0 msgr2=0x7fbd44103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:48.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.455+0000 7fbd4c800640 1 -- 172.21.15.1:0/2294973817 shutdown_connections 2024-04-26T21:26:48.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:48.455+0000 7fbd4c800640 1 -- 172.21.15.1:0/2294973817 wait complete. 2024-04-26T21:26:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:48 smithi186 ceph-mon[32991]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:48.795 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:48 smithi001 ceph-mon[28054]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:48 smithi038 ceph-mon[32724]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:49.069 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:26:48 smithi001 podman[66860]: 2024-04-26 21:26:48.839352827 +0000 UTC m=+0.176135520 container create fa524eb6c02ac82e8ec307a3068e6e4ff62b65de4c0f256699afd6ea4649f0ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-26T21:26:49.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:49 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2294973817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:49.568 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:26:49 smithi001 podman[66860]: 2024-04-26 21:26:49.135601779 +0000 UTC m=+0.472384479 container init fa524eb6c02ac82e8ec307a3068e6e4ff62b65de4c0f256699afd6ea4649f0ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T21:26:49.568 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:26:49 smithi001 podman[66860]: 2024-04-26 21:26:49.14901563 +0000 UTC m=+0.485798333 container start fa524eb6c02ac82e8ec307a3068e6e4ff62b65de4c0f256699afd6ea4649f0ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:26:49.568 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:26:49 smithi001 ceph-mds[66879]: -- 172.21.15.1:0/2184187642 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d2a0281a40 con 0x55d2a0254800 2024-04-26T21:26:49.568 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:26:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g[66875]: starting mds.g at 2024-04-26T21:26:49.568 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:26:49 smithi001 bash[66860]: fa524eb6c02ac82e8ec307a3068e6e4ff62b65de4c0f256699afd6ea4649f0ee 2024-04-26T21:26:49.569 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:26:49 smithi001 systemd[1]: Started Ceph mds.g for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:49.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:49 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2294973817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:49.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:49 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2294973817' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:50.476 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:50.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:50.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] up:boot 2024-04-26T21:26:50.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: fsmap 10 up:standby 2024-04-26T21:26:50.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:26:50.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:50.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:50.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:50.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:50.649 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:50.650 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:50.650 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:50 smithi001 ceph-mon[28054]: Deploying daemon mds.a on smithi001 2024-04-26T21:26:50.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.700+0000 7f0195c00640 1 -- 172.21.15.1:0/643999602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0190108650 msgr2=0x7f019010aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.700+0000 7f0195c00640 1 --2- 172.21.15.1:0/643999602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0190108650 0x7f019010aaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0188009ca0 tx=0x7f018802a8c0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.701+0000 7f0195c00640 1 -- 172.21.15.1:0/643999602 shutdown_connections 2024-04-26T21:26:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.701+0000 7f0195c00640 1 --2- 172.21.15.1:0/643999602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0190108650 0x7f019010aaa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.701+0000 7f0195c00640 1 --2- 172.21.15.1:0/643999602 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0190105d20 0x7f0190108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.701+0000 7f0195c00640 1 --2- 172.21.15.1:0/643999602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01901033f0 0x7f01901057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.701+0000 7f0195c00640 1 -- 172.21.15.1:0/643999602 >> 172.21.15.1:0/643999602 conn(0x7f01900fcde0 msgr2=0x7f01900ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.701+0000 7f0195c00640 1 -- 172.21.15.1:0/643999602 shutdown_connections 2024-04-26T21:26:50.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.701+0000 7f0195c00640 1 -- 172.21.15.1:0/643999602 wait complete. 2024-04-26T21:26:50.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f0195c00640 1 Processor -- start 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f0195c00640 1 -- start start 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f0195c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01901033f0 0x7f019019ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f0195c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0190105d20 0x7f019019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f0195c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0190108650 0x7f01901a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f0195c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f019010e4c0 con 0x7f01901033f0 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f0195c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f019010e340 con 0x7f0190108650 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f0195c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f019010e640 con 0x7f0190105d20 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f018ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01901033f0 0x7f019019ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f018ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01901033f0 0x7f019019ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45504/0 (socket says 172.21.15.1:45504) 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.702+0000 7f018ea00640 1 -- 172.21.15.1:0/491883776 learned_addr learned my addr 172.21.15.1:0/491883776 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.703+0000 7f018e000640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0190105d20 0x7f019019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.703+0000 7f018f400640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0190108650 0x7f01901a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.703+0000 7f018ea00640 1 -- 172.21.15.1:0/491883776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0190105d20 msgr2=0x7f019019b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.703+0000 7f018ea00640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0190105d20 0x7f019019b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.703+0000 7f018ea00640 1 -- 172.21.15.1:0/491883776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0190108650 msgr2=0x7f01901a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.703+0000 7f018ea00640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0190108650 0x7f01901a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.703+0000 7f018ea00640 1 -- 172.21.15.1:0/491883776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01901a2990 con 0x7f01901033f0 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.703+0000 7f018e000640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0190105d20 0x7f019019b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.703+0000 7f018ea00640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01901033f0 0x7f019019ad70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f018000e9f0 tx=0x7f018000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.703+0000 7f0187400640 1 -- 172.21.15.1:0/491883776 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018000cd80 con 0x7f01901033f0 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.704+0000 7f0195c00640 1 -- 172.21.15.1:0/491883776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01901a2c80 con 0x7f01901033f0 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.704+0000 7f0187400640 1 -- 172.21.15.1:0/491883776 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01800108e0 con 0x7f01901033f0 2024-04-26T21:26:50.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.704+0000 7f0187400640 1 -- 172.21.15.1:0/491883776 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0180018980 con 0x7f01901033f0 2024-04-26T21:26:50.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.704+0000 7f0195c00640 1 -- 172.21.15.1:0/491883776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01901a31d0 con 0x7f01901033f0 2024-04-26T21:26:50.707 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:50.707 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] up:boot 2024-04-26T21:26:50.707 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: fsmap 10 up:standby 2024-04-26T21:26:50.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:26:50.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:50.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:50.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:50.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:50.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:50.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:50.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:50 smithi186 ceph-mon[32991]: Deploying daemon mds.a on smithi001 2024-04-26T21:26:50.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.705+0000 7f0195c00640 1 -- 172.21.15.1:0/491883776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f019010a140 con 0x7f01901033f0 2024-04-26T21:26:50.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.707+0000 7f0187400640 1 -- 172.21.15.1:0/491883776 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f0180010430 con 0x7f01901033f0 2024-04-26T21:26:50.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.709+0000 7f0187400640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01740a3c30 0x7f01740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:50.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.709+0000 7f0187400640 1 -- 172.21.15.1:0/491883776 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f01800d07f0 con 0x7f01901033f0 2024-04-26T21:26:50.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.709+0000 7f018e000640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01740a3c30 0x7f01740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:50.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.710+0000 7f018e000640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01740a3c30 0x7f01740a60f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f019019c280 tx=0x7f017c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:50.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.711+0000 7f0187400640 1 -- 172.21.15.1:0/491883776 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f018009d5d0 con 0x7f01901033f0 2024-04-26T21:26:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] up:boot 2024-04-26T21:26:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: fsmap 10 up:standby 2024-04-26T21:26:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:26:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:50.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:50.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:50.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:50.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:50 smithi038 ceph-mon[32724]: Deploying daemon mds.a on smithi001 2024-04-26T21:26:50.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.961+0000 7f0195c00640 1 -- 172.21.15.1:0/491883776 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f019002dd10 con 0x7f01901033f0 2024-04-26T21:26:50.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.962+0000 7f0187400640 1 -- 172.21.15.1:0/491883776 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8818 (secure 0 0 0) 0x7f018009ced0 con 0x7f01901033f0 2024-04-26T21:26:50.962 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:50.963 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24572,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T21:26:50.963 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 11 2024-04-26T21:26:50.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.964+0000 7f0195c00640 1 -- 172.21.15.1:0/491883776 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01740a3c30 msgr2=0x7f01740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:50.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.964+0000 7f0195c00640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01740a3c30 0x7f01740a60f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f019019c280 tx=0x7f017c005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:50.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.964+0000 7f0195c00640 1 -- 172.21.15.1:0/491883776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01901033f0 msgr2=0x7f019019ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:50.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.964+0000 7f0195c00640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01901033f0 0x7f019019ad70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f018000e9f0 tx=0x7f018000eeb0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.965+0000 7f0195c00640 1 -- 172.21.15.1:0/491883776 shutdown_connections 2024-04-26T21:26:50.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.965+0000 7f0195c00640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01740a3c30 0x7f01740a60f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.965+0000 7f0195c00640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0190105d20 0x7f019019b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.965+0000 7f0195c00640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0190108650 0x7f01901a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.965+0000 7f0195c00640 1 --2- 172.21.15.1:0/491883776 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01901033f0 0x7f019019ad70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:50.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.965+0000 7f0195c00640 1 -- 172.21.15.1:0/491883776 >> 172.21.15.1:0/491883776 conn(0x7f01900fcde0 msgr2=0x7f0190102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:50.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.965+0000 7f0195c00640 1 -- 172.21.15.1:0/491883776 shutdown_connections 2024-04-26T21:26:50.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:50.965+0000 7f0195c00640 1 -- 172.21.15.1:0/491883776 wait complete. 2024-04-26T21:26:51.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:51 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/491883776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:51.760 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:51 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/491883776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:51.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:51 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/491883776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:52.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:52 smithi001 ceph-mon[28054]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:52.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:52 smithi186 ceph-mon[32991]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:52.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:52 smithi038 ceph-mon[32724]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:52.969 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:26:52 smithi001 systemd[1]: Starting Ceph mds.a for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:26:52.984 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:53.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.215+0000 7fa2aa400640 1 -- 172.21.15.1:0/3500418286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a41045b0 msgr2=0x7fa2a410ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:53.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.215+0000 7fa2aa400640 1 --2- 172.21.15.1:0/3500418286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a41045b0 0x7fa2a410ae40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009ca0 tx=0x7fa29c02a8c0 comp rx=0 tx=0).stop 2024-04-26T21:26:53.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.216+0000 7fa2aa400640 1 -- 172.21.15.1:0/3500418286 shutdown_connections 2024-04-26T21:26:53.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.216+0000 7fa2aa400640 1 --2- 172.21.15.1:0/3500418286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a41045b0 0x7fa2a410ae40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:53.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.216+0000 7fa2aa400640 1 --2- 172.21.15.1:0/3500418286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2a4103bf0 0x7fa2a4104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:53.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.216+0000 7fa2aa400640 1 --2- 172.21.15.1:0/3500418286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2a4100c20 0x7fa2a4101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:53.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.216+0000 7fa2aa400640 1 -- 172.21.15.1:0/3500418286 >> 172.21.15.1:0/3500418286 conn(0x7fa2a40fc530 msgr2=0x7fa2a40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:53.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.216+0000 7fa2aa400640 1 -- 172.21.15.1:0/3500418286 shutdown_connections 2024-04-26T21:26:53.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.216+0000 7fa2aa400640 1 -- 172.21.15.1:0/3500418286 wait complete. 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.217+0000 7fa2aa400640 1 Processor -- start 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.217+0000 7fa2aa400640 1 -- start start 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2aa400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2a4100c20 0x7fa2a419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2aa400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2a4103bf0 0x7fa2a419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2aa400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a41045b0 0x7fa2a41a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2aa400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a410e4a0 con 0x7fa2a4103bf0 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2aa400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a410e320 con 0x7fa2a41045b0 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2aa400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a410e620 con 0x7fa2a4100c20 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2a3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2a4100c20 0x7fa2a419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa29be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2a4103bf0 0x7fa2a419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2a3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2a4100c20 0x7fa2a419ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:39078/0 (socket says 172.21.15.1:39078) 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2a3400640 1 -- 172.21.15.1:0/1692205700 learned_addr learned my addr 172.21.15.1:0/1692205700 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2a3e00640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a41045b0 0x7fa2a41a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:53.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2a3400640 1 -- 172.21.15.1:0/1692205700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a41045b0 msgr2=0x7fa2a41a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2a3400640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a41045b0 0x7fa2a41a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2a3400640 1 -- 172.21.15.1:0/1692205700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2a4103bf0 msgr2=0x7fa2a419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2a3400640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2a4103bf0 0x7fa2a419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.218+0000 7fa2a3400640 1 -- 172.21.15.1:0/1692205700 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a41a2990 con 0x7fa2a4100c20 2024-04-26T21:26:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.219+0000 7fa2a3400640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2a4100c20 0x7fa2a419ad70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa29400e9c0 tx=0x7fa29400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.219+0000 7fa2a0c00640 1 -- 172.21.15.1:0/1692205700 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29400cd80 con 0x7fa2a4100c20 2024-04-26T21:26:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.219+0000 7fa2aa400640 1 -- 172.21.15.1:0/1692205700 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2a41a2c80 con 0x7fa2a4100c20 2024-04-26T21:26:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.219+0000 7fa2a0c00640 1 -- 172.21.15.1:0/1692205700 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2940035f0 con 0x7fa2a4100c20 2024-04-26T21:26:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.219+0000 7fa2a0c00640 1 -- 172.21.15.1:0/1692205700 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa294010690 con 0x7fa2a4100c20 2024-04-26T21:26:53.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.219+0000 7fa2aa400640 1 -- 172.21.15.1:0/1692205700 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2a41a3180 con 0x7fa2a4100c20 2024-04-26T21:26:53.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.220+0000 7fa2aa400640 1 -- 172.21.15.1:0/1692205700 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2a41a33b0 con 0x7fa2a4100c20 2024-04-26T21:26:53.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.222+0000 7fa2a0c00640 1 -- 172.21.15.1:0/1692205700 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fa294004000 con 0x7fa2a4100c20 2024-04-26T21:26:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.224+0000 7fa2a0c00640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa2880a3bb0 0x7fa2880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.224+0000 7fa2a0c00640 1 -- 172.21.15.1:0/1692205700 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fa2940d04f0 con 0x7fa2a4100c20 2024-04-26T21:26:53.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.224+0000 7fa29be00640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa2880a3bb0 0x7fa2880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:53.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.225+0000 7fa29be00640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa2880a3bb0 0x7fa2880a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa2a419c280 tx=0x7fa28c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:53.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.228+0000 7fa2a0c00640 1 -- 172.21.15.1:0/1692205700 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa29409d2d0 con 0x7fa2a4100c20 2024-04-26T21:26:53.249 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:26:52 smithi001 podman[67203]: 2024-04-26 21:26:52.874112465 +0000 UTC m=+0.037003222 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:53.249 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:26:53 smithi001 podman[67203]: 2024-04-26 21:26:53.020705067 +0000 UTC m=+0.183595861 container create 57763fb7fce3c4f51e016af19f083e4037ff0ca12d73f99f67171a91ee491c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:26:53.249 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:26:53 smithi001 podman[67203]: 2024-04-26 21:26:53.249003522 +0000 UTC m=+0.411894303 container init 57763fb7fce3c4f51e016af19f083e4037ff0ca12d73f99f67171a91ee491c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-26T21:26:53.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.493+0000 7fa2aa400640 1 -- 172.21.15.1:0/1692205700 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2a41a3620 con 0x7fa2a4100c20 2024-04-26T21:26:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.494+0000 7fa2a0c00640 1 -- 172.21.15.1:0/1692205700 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9641 (secure 0 0 0) 0x7fa29409cbd0 con 0x7fa2a4100c20 2024-04-26T21:26:53.495 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:53.495 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24572,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T21:26:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 12 2024-04-26T21:26:53.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 -- 172.21.15.1:0/1692205700 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa2880a3bb0 msgr2=0x7fa2880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:53.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa2880a3bb0 0x7fa2880a6070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa2a419c280 tx=0x7fa28c005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:53.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 -- 172.21.15.1:0/1692205700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2a4100c20 msgr2=0x7fa2a419ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:53.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2a4100c20 0x7fa2a419ad70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa29400e9c0 tx=0x7fa29400ee80 comp rx=0 tx=0).stop 2024-04-26T21:26:53.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 -- 172.21.15.1:0/1692205700 shutdown_connections 2024-04-26T21:26:53.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa2880a3bb0 0x7fa2880a6070 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:53.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2a41045b0 0x7fa2a41a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:53.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2a4103bf0 0x7fa2a419b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:53.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 --2- 172.21.15.1:0/1692205700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2a4100c20 0x7fa2a419ad70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:53.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 -- 172.21.15.1:0/1692205700 >> 172.21.15.1:0/1692205700 conn(0x7fa2a40fc530 msgr2=0x7fa2a41019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:53.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 -- 172.21.15.1:0/1692205700 shutdown_connections 2024-04-26T21:26:53.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:53.497+0000 7fa2aa400640 1 -- 172.21.15.1:0/1692205700 wait complete. 2024-04-26T21:26:53.514 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:26:53 smithi001 podman[67203]: 2024-04-26 21:26:53.258358956 +0000 UTC m=+0.421249719 container start 57763fb7fce3c4f51e016af19f083e4037ff0ca12d73f99f67171a91ee491c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-26T21:26:53.514 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:26:53 smithi001 ceph-mds[67261]: -- 172.21.15.1:0/864418375 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cdafcc3a40 con 0x55cdafc96c00 2024-04-26T21:26:53.514 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:26:53 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-a[67238]: starting mds.a at 2024-04-26T21:26:53.514 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:26:53 smithi001 bash[67203]: 57763fb7fce3c4f51e016af19f083e4037ff0ca12d73f99f67171a91ee491c7e 2024-04-26T21:26:53.514 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:26:53 smithi001 systemd[1]: Started Ceph mds.a for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:54.463 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.1:6840/2045981022,v1:172.21.15.1:6841/2045981022] up:boot 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: fsmap 11 up:standby 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1692205700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:54.464 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:54 smithi038 ceph-mon[32724]: Deploying daemon mds.b on smithi038 2024-04-26T21:26:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.1:6840/2045981022,v1:172.21.15.1:6841/2045981022] up:boot 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: fsmap 11 up:standby 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1692205700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:54.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:54 smithi186 ceph-mon[32991]: Deploying daemon mds.b on smithi038 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.1:6840/2045981022,v1:172.21.15.1:6841/2045981022] up:boot 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: fsmap 11 up:standby 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1692205700' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:26:54.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:54 smithi001 ceph-mon[28054]: Deploying daemon mds.b on smithi038 2024-04-26T21:26:55.518 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.747+0000 7f4faee00640 1 -- 172.21.15.1:0/4047540257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa80768f0 msgr2=0x7f4fa8074d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.747+0000 7f4faee00640 1 --2- 172.21.15.1:0/4047540257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa80768f0 0x7f4fa8074d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f98009900 tx=0x7f4f9802a490 comp rx=0 tx=0).stop 2024-04-26T21:26:55.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.747+0000 7f4faee00640 1 -- 172.21.15.1:0/4047540257 shutdown_connections 2024-04-26T21:26:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.747+0000 7f4faee00640 1 --2- 172.21.15.1:0/4047540257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa8075310 0x7f4fa8075770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.747+0000 7f4faee00640 1 --2- 172.21.15.1:0/4047540257 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa810a820 0x7f4fa810ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.747+0000 7f4faee00640 1 --2- 172.21.15.1:0/4047540257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa80768f0 0x7f4fa8074d40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.747+0000 7f4faee00640 1 -- 172.21.15.1:0/4047540257 >> 172.21.15.1:0/4047540257 conn(0x7f4fa80fdf00 msgr2=0x7f4fa8100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.747+0000 7f4faee00640 1 -- 172.21.15.1:0/4047540257 shutdown_connections 2024-04-26T21:26:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.747+0000 7f4faee00640 1 -- 172.21.15.1:0/4047540257 wait complete. 2024-04-26T21:26:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.749+0000 7f4faee00640 1 Processor -- start 2024-04-26T21:26:55.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.749+0000 7f4faee00640 1 -- start start 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.749+0000 7f4faee00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8075310 0x7f4fa819acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.749+0000 7f4faee00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa80768f0 0x7f4fa819b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.749+0000 7f4faee00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa810a820 0x7f4fa81a2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.749+0000 7f4faee00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa81106d0 con 0x7f4fa80768f0 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.749+0000 7f4faee00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa8110550 con 0x7f4fa8075310 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.749+0000 7f4faee00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa8110850 con 0x7f4fa810a820 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4fa7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8075310 0x7f4fa819acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4fa7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8075310 0x7f4fa819acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34184/0 (socket says 172.21.15.1:34184) 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4fa7e00640 1 -- 172.21.15.1:0/3551449964 learned_addr learned my addr 172.21.15.1:0/3551449964 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4f9f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa80768f0 0x7f4fa819b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4face00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa810a820 0x7f4fa81a2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:55.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4fa7e00640 1 -- 172.21.15.1:0/3551449964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa810a820 msgr2=0x7f4fa81a2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4fa7e00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa810a820 0x7f4fa81a2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4fa7e00640 1 -- 172.21.15.1:0/3551449964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa80768f0 msgr2=0x7f4fa819b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4fa7e00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa80768f0 0x7f4fa819b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4fa7e00640 1 -- 172.21.15.1:0/3551449964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa81a2900 con 0x7f4fa8075310 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4face00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa810a820 0x7f4fa81a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4f9f400640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa80768f0 0x7f4fa819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.750+0000 7f4fa7e00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8075310 0x7f4fa819acf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f98005ab0 tx=0x7f4f98003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.751+0000 7f4fa5600640 1 -- 172.21.15.1:0/3551449964 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9803f070 con 0x7f4fa8075310 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.751+0000 7f4faee00640 1 -- 172.21.15.1:0/3551449964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa81a2b90 con 0x7f4fa8075310 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.751+0000 7f4fa5600640 1 -- 172.21.15.1:0/3551449964 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f9803b460 con 0x7f4fa8075310 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.751+0000 7f4fa5600640 1 -- 172.21.15.1:0/3551449964 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f98035030 con 0x7f4fa8075310 2024-04-26T21:26:55.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.751+0000 7f4faee00640 1 -- 172.21.15.1:0/3551449964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa81a30d0 con 0x7f4fa8075310 2024-04-26T21:26:55.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.752+0000 7f4faee00640 1 -- 172.21.15.1:0/3551449964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fa81095e0 con 0x7f4fa8075310 2024-04-26T21:26:55.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.755+0000 7f4fa5600640 1 -- 172.21.15.1:0/3551449964 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4f98042070 con 0x7f4fa8075310 2024-04-26T21:26:55.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.756+0000 7f4fa5600640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4f8c0a3c30 0x7f4f8c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:55.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.756+0000 7f4fa5600640 1 -- 172.21.15.1:0/3551449964 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f4f980f08f0 con 0x7f4fa8075310 2024-04-26T21:26:55.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.756+0000 7f4f9f400640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4f8c0a3c30 0x7f4f8c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:55.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.757+0000 7f4f9f400640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4f8c0a3c30 0x7f4f8c0a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000afb0 tx=0x7f4f9000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:55.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:55.762+0000 7f4fa5600640 1 -- 172.21.15.1:0/3551449964 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4f980bd6d0 con 0x7f4fa8075310 2024-04-26T21:26:55.827 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:26:55 smithi038 systemd[1]: Starting Ceph mds.b for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:26:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.008+0000 7f4faee00640 1 -- 172.21.15.1:0/3551449964 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fa802dad0 con 0x7f4fa8075310 2024-04-26T21:26:56.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.009+0000 7f4fa5600640 1 -- 172.21.15.1:0/3551449964 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9641 (secure 0 0 0) 0x7f4f980bcfd0 con 0x7f4fa8075310 2024-04-26T21:26:56.010 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:56.010 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24572,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-26T21:26:56.010 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 12 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 -- 172.21.15.1:0/3551449964 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4f8c0a3c30 msgr2=0x7f4f8c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4f8c0a3c30 0x7f4f8c0a60f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000afb0 tx=0x7f4f9000a2d0 comp rx=0 tx=0).stop 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 -- 172.21.15.1:0/3551449964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8075310 msgr2=0x7f4fa819acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8075310 0x7f4fa819acf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f98005ab0 tx=0x7f4f98003cd0 comp rx=0 tx=0).stop 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 -- 172.21.15.1:0/3551449964 shutdown_connections 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4f8c0a3c30 0x7f4f8c0a60f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fa810a820 0x7f4fa81a2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa80768f0 0x7f4fa819b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 --2- 172.21.15.1:0/3551449964 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fa8075310 0x7f4fa819acf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 -- 172.21.15.1:0/3551449964 >> 172.21.15.1:0/3551449964 conn(0x7f4fa80fdf00 msgr2=0x7f4fa81067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 -- 172.21.15.1:0/3551449964 shutdown_connections 2024-04-26T21:26:56.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:56.012+0000 7f4faee00640 1 -- 172.21.15.1:0/3551449964 wait complete. 2024-04-26T21:26:56.232 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:26:55 smithi038 podman[62803]: 2024-04-26 21:26:55.853543532 +0000 UTC m=+0.018566026 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:26:56.232 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:26:55 smithi038 podman[62803]: 2024-04-26 21:26:55.996090097 +0000 UTC m=+0.161112605 container create 1ac8426148a506e6d13dd2447cafbdbdf57b35d83a6b3b42ad9903aa8ec8a02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-26T21:26:56.232 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:26:56 smithi038 podman[62803]: 2024-04-26 21:26:56.232268288 +0000 UTC m=+0.397290759 container init 1ac8426148a506e6d13dd2447cafbdbdf57b35d83a6b3b42ad9903aa8ec8a02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T21:26:56.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:56 smithi038 ceph-mon[32724]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:56.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:56 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3551449964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:56.577 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:26:56 smithi038 podman[62803]: 2024-04-26 21:26:56.235103735 +0000 UTC m=+0.400126211 container start 1ac8426148a506e6d13dd2447cafbdbdf57b35d83a6b3b42ad9903aa8ec8a02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T21:26:56.577 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:26:56 smithi038 ceph-mds[62823]: -- 172.21.15.38:0/1549863374 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a5dab9fa40 con 0x55a5dab73000 2024-04-26T21:26:56.577 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:26:56 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b[62819]: starting mds.b at 2024-04-26T21:26:56.577 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:26:56 smithi038 bash[62803]: 1ac8426148a506e6d13dd2447cafbdbdf57b35d83a6b3b42ad9903aa8ec8a02f 2024-04-26T21:26:56.578 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:26:56 smithi038 systemd[1]: Started Ceph mds.b for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:26:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:56 smithi186 ceph-mon[32991]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:56 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3551449964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:56.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:56 smithi001 ceph-mon[28054]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:56.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:56 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3551449964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:57.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:57 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] up:boot 2024-04-26T21:26:57.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:57 smithi186 ceph-mon[32991]: fsmap 12 up:standby 2024-04-26T21:26:57.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:26:57.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.659 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:57.781 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.781 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:57 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] up:boot 2024-04-26T21:26:57.781 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:57 smithi038 ceph-mon[32724]: fsmap 12 up:standby 2024-04-26T21:26:57.781 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:26:57.781 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.781 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.781 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.781 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.781 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:57.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:57 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] up:boot 2024-04-26T21:26:57.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:57 smithi001 ceph-mon[28054]: fsmap 12 up:standby 2024-04-26T21:26:57.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:26:57.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:57.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:26:58.035 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:26:58.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.262+0000 7fdb8c400640 1 -- 172.21.15.1:0/4068429629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb84105d70 msgr2=0x7fdb841061f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.262+0000 7fdb8c400640 1 --2- 172.21.15.1:0/4068429629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb84105d70 0x7fdb841061f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009900 tx=0x7fdb7002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.263+0000 7fdb8c400640 1 -- 172.21.15.1:0/4068429629 shutdown_connections 2024-04-26T21:26:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.263+0000 7fdb8c400640 1 --2- 172.21.15.1:0/4068429629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb84106730 0x7fdb8410cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.263+0000 7fdb8c400640 1 --2- 172.21.15.1:0/4068429629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb84105d70 0x7fdb841061f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.263+0000 7fdb8c400640 1 --2- 172.21.15.1:0/4068429629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb84104b80 0x7fdb84104fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.263+0000 7fdb8c400640 1 -- 172.21.15.1:0/4068429629 >> 172.21.15.1:0/4068429629 conn(0x7fdb841000f0 msgr2=0x7fdb84102530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.263+0000 7fdb8c400640 1 -- 172.21.15.1:0/4068429629 shutdown_connections 2024-04-26T21:26:58.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.263+0000 7fdb8c400640 1 -- 172.21.15.1:0/4068429629 wait complete. 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.264+0000 7fdb8c400640 1 Processor -- start 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.264+0000 7fdb8c400640 1 -- start start 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.264+0000 7fdb8c400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb84104b80 0x7fdb8419ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.264+0000 7fdb8c400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb84105d70 0x7fdb8419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.264+0000 7fdb8c400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb84106730 0x7fdb841a2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.264+0000 7fdb8c400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841106e0 con 0x7fdb84104b80 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.264+0000 7fdb8c400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84110560 con 0x7fdb84105d70 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.264+0000 7fdb8c400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84110860 con 0x7fdb84106730 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb89000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb84105d70 0x7fdb8419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb8a400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb84106730 0x7fdb841a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:58.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb89a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb84104b80 0x7fdb8419ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb8a400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb84106730 0x7fdb841a2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37772/0 (socket says 172.21.15.1:37772) 2024-04-26T21:26:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb8a400640 1 -- 172.21.15.1:0/2822404837 learned_addr learned my addr 172.21.15.1:0/2822404837 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb89000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb84105d70 0x7fdb8419b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42790/0 (socket says 172.21.15.1:42790) 2024-04-26T21:26:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb8a400640 1 -- 172.21.15.1:0/2822404837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb84105d70 msgr2=0x7fdb8419b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb8a400640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb84105d70 0x7fdb8419b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb8a400640 1 -- 172.21.15.1:0/2822404837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb84104b80 msgr2=0x7fdb8419ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb8a400640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb84104b80 0x7fdb8419ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb8a400640 1 -- 172.21.15.1:0/2822404837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb841a2970 con 0x7fdb84106730 2024-04-26T21:26:58.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.265+0000 7fdb89a00640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb84104b80 0x7fdb8419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.266+0000 7fdb8a400640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb84106730 0x7fdb841a2300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00eb10 tx=0x7fdb7c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.266+0000 7fdb82000640 1 -- 172.21.15.1:0/2822404837 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c004b20 con 0x7fdb84106730 2024-04-26T21:26:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.266+0000 7fdb82000640 1 -- 172.21.15.1:0/2822404837 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb7c004cc0 con 0x7fdb84106730 2024-04-26T21:26:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.266+0000 7fdb8c400640 1 -- 172.21.15.1:0/2822404837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb841a2c60 con 0x7fdb84106730 2024-04-26T21:26:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.266+0000 7fdb82000640 1 -- 172.21.15.1:0/2822404837 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c010670 con 0x7fdb84106730 2024-04-26T21:26:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.266+0000 7fdb8c400640 1 -- 172.21.15.1:0/2822404837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb841a3180 con 0x7fdb84106730 2024-04-26T21:26:58.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.267+0000 7fdb8c400640 1 -- 172.21.15.1:0/2822404837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb8410bea0 con 0x7fdb84106730 2024-04-26T21:26:58.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.269+0000 7fdb82000640 1 -- 172.21.15.1:0/2822404837 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fdb7c005020 con 0x7fdb84106730 2024-04-26T21:26:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.270+0000 7fdb82000640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb6c0a3c30 0x7fdb6c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.271+0000 7fdb82000640 1 -- 172.21.15.1:0/2822404837 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fdb7c0d04e0 con 0x7fdb84106730 2024-04-26T21:26:58.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.271+0000 7fdb89a00640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb6c0a3c30 0x7fdb6c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:58.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.271+0000 7fdb89a00640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb6c0a3c30 0x7fdb6c0a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdb84105bd0 tx=0x7fdb78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:58.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.274+0000 7fdb82000640 1 -- 172.21.15.1:0/2822404837 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fdb7c09d2c0 con 0x7fdb84106730 2024-04-26T21:26:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.524+0000 7fdb8c400640 1 -- 172.21.15.1:0/2822404837 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb8402dad0 con 0x7fdb84106730 2024-04-26T21:26:58.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.525+0000 7fdb82000640 1 -- 172.21.15.1:0/2822404837 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10467 (secure 0 0 0) 0x7fdb7c09cbc0 con 0x7fdb84106730 2024-04-26T21:26:58.526 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:58.526 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24572,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24602,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-26T21:26:58.526 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 13 2024-04-26T21:26:58.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.529+0000 7fdb8c400640 1 -- 172.21.15.1:0/2822404837 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb6c0a3c30 msgr2=0x7fdb6c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:58.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.529+0000 7fdb8c400640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb6c0a3c30 0x7fdb6c0a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdb84105bd0 tx=0x7fdb78005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:58.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.529+0000 7fdb8c400640 1 -- 172.21.15.1:0/2822404837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb84106730 msgr2=0x7fdb841a2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:58.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.529+0000 7fdb8c400640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb84106730 0x7fdb841a2300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00eb10 tx=0x7fdb7c00c2d0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.530+0000 7fdb8c400640 1 -- 172.21.15.1:0/2822404837 shutdown_connections 2024-04-26T21:26:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.530+0000 7fdb8c400640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fdb6c0a3c30 0x7fdb6c0a60f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.530+0000 7fdb8c400640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb84106730 0x7fdb841a2300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.530+0000 7fdb8c400640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb84105d70 0x7fdb8419b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.530+0000 7fdb8c400640 1 --2- 172.21.15.1:0/2822404837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdb84104b80 0x7fdb8419ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.530+0000 7fdb8c400640 1 -- 172.21.15.1:0/2822404837 >> 172.21.15.1:0/2822404837 conn(0x7fdb841000f0 msgr2=0x7fdb84109070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.530+0000 7fdb8c400640 1 -- 172.21.15.1:0/2822404837 shutdown_connections 2024-04-26T21:26:58.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.530+0000 7fdb8c400640 1 -- 172.21.15.1:0/2822404837 wait complete. 2024-04-26T21:26:58.553 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:26:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.775+0000 7f4d59c00640 1 -- 172.21.15.1:0/1935602407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d54105b90 msgr2=0x7f4d54106010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.775+0000 7f4d59c00640 1 --2- 172.21.15.1:0/1935602407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d54105b90 0x7f4d54106010 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4d40009900 tx=0x7f4d4002a410 comp rx=0 tx=0).stop 2024-04-26T21:26:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.775+0000 7f4d59c00640 1 -- 172.21.15.1:0/1935602407 shutdown_connections 2024-04-26T21:26:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.775+0000 7f4d59c00640 1 --2- 172.21.15.1:0/1935602407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d54106550 0x7f4d5410cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.775+0000 7f4d59c00640 1 --2- 172.21.15.1:0/1935602407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d54105b90 0x7f4d54106010 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.775+0000 7f4d59c00640 1 --2- 172.21.15.1:0/1935602407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d541049a0 0x7f4d54104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.775+0000 7f4d59c00640 1 -- 172.21.15.1:0/1935602407 >> 172.21.15.1:0/1935602407 conn(0x7f4d540fff10 msgr2=0x7f4d54102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.775+0000 7f4d59c00640 1 -- 172.21.15.1:0/1935602407 shutdown_connections 2024-04-26T21:26:58.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.775+0000 7f4d59c00640 1 -- 172.21.15.1:0/1935602407 wait complete. 2024-04-26T21:26:58.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.776+0000 7f4d59c00640 1 Processor -- start 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.776+0000 7f4d59c00640 1 -- start start 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.776+0000 7f4d59c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d541049a0 0x7f4d5419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d59c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d54105b90 0x7f4d5419afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d59c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d54106550 0x7f4d541a2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d59c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4d54110520 con 0x7f4d54105b90 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d59c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541103a0 con 0x7f4d541049a0 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d59c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541106a0 con 0x7f4d54106550 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d53e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d541049a0 0x7f4d5419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d53e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d541049a0 0x7f4d5419aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42798/0 (socket says 172.21.15.1:42798) 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d53e00640 1 -- 172.21.15.1:0/2251930827 learned_addr learned my addr 172.21.15.1:0/2251930827 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d59200640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d54106550 0x7f4d541a2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d53400640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d54105b90 0x7f4d5419afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d53e00640 1 -- 172.21.15.1:0/2251930827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d54106550 msgr2=0x7f4d541a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d53e00640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d54106550 0x7f4d541a2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d53e00640 1 -- 172.21.15.1:0/2251930827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d54105b90 msgr2=0x7f4d5419afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:58.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d53e00640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d54105b90 0x7f4d5419afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:58.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.777+0000 7f4d53e00640 1 -- 172.21.15.1:0/2251930827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d541a26c0 con 0x7f4d541049a0 2024-04-26T21:26:58.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.778+0000 7f4d53e00640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d541049a0 0x7f4d5419aaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400e9c0 tx=0x7f4d4400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:58.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.778+0000 7f4d50c00640 1 -- 172.21.15.1:0/2251930827 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4400cd80 con 0x7f4d541049a0 2024-04-26T21:26:58.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.778+0000 7f4d50c00640 1 -- 172.21.15.1:0/2251930827 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d440035f0 con 0x7f4d541049a0 2024-04-26T21:26:58.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.778+0000 7f4d50c00640 1 -- 172.21.15.1:0/2251930827 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44010690 con 0x7f4d541049a0 2024-04-26T21:26:58.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.778+0000 7f4d59c00640 1 -- 172.21.15.1:0/2251930827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d541a29b0 con 0x7f4d541049a0 2024-04-26T21:26:58.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.778+0000 7f4d59c00640 1 -- 172.21.15.1:0/2251930827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d541a2e80 con 0x7f4d541049a0 2024-04-26T21:26:58.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.779+0000 7f4d59c00640 1 -- 172.21.15.1:0/2251930827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d5410bc60 con 0x7f4d541049a0 2024-04-26T21:26:58.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.781+0000 7f4d50c00640 1 -- 172.21.15.1:0/2251930827 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f4d440037a0 con 0x7f4d541049a0 2024-04-26T21:26:58.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.783+0000 7f4d50c00640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4d380a3cd0 0x7f4d380a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:58.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.783+0000 7f4d50c00640 1 -- 172.21.15.1:0/2251930827 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f4d440d02e0 con 0x7f4d541049a0 2024-04-26T21:26:58.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.783+0000 7f4d53400640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4d380a3cd0 0x7f4d380a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:58.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.784+0000 7f4d53400640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4d380a3cd0 0x7f4d380a6190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4d40031500 tx=0x7f4d40005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:58.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:58.787+0000 7f4d50c00640 1 -- 172.21.15.1:0/2251930827 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4d4409d0c0 con 0x7f4d541049a0 2024-04-26T21:26:58.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:58 smithi001 ceph-mon[28054]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:58.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:58 smithi038 ceph-mon[32724]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:58.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:58 smithi186 ceph-mon[32991]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:26:59.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.004+0000 7f4d59c00640 1 -- 172.21.15.1:0/2251930827 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d540635d0 con 0x7f4d541049a0 2024-04-26T21:26:59.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.005+0000 7f4d50c00640 1 -- 172.21.15.1:0/2251930827 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18300 (secure 0 0 0) 0x7f4d4409c9c0 con 0x7f4d541049a0 2024-04-26T21:26:59.006 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:59.006 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":83,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:26:04.333876+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 -- 172.21.15.1:0/2251930827 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4d380a3cd0 msgr2=0x7f4d380a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4d380a3cd0 0x7f4d380a6190 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4d40031500 tx=0x7f4d40005a20 comp rx=0 tx=0).stop 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 -- 172.21.15.1:0/2251930827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d541049a0 msgr2=0x7f4d5419aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d541049a0 0x7f4d5419aaa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400e9c0 tx=0x7f4d4400ee80 comp rx=0 tx=0).stop 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 -- 172.21.15.1:0/2251930827 shutdown_connections 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4d380a3cd0 0x7f4d380a6190 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d54106550 0x7f4d541a2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4d54105b90 0x7f4d5419afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 --2- 172.21.15.1:0/2251930827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d541049a0 0x7f4d5419aaa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 -- 172.21.15.1:0/2251930827 >> 172.21.15.1:0/2251930827 conn(0x7f4d540fff10 msgr2=0x7f4d54108e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 -- 172.21.15.1:0/2251930827 shutdown_connections 2024-04-26T21:26:59.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.008+0000 7f4d59c00640 1 -- 172.21.15.1:0/2251930827 wait complete. 2024-04-26T21:26:59.029 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T21:26:59.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:26:59.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.222+0000 7f8ddf400640 1 -- 172.21.15.1:0/4249188369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dd8105d20 msgr2=0x7f8dd8108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.222+0000 7f8ddf400640 1 --2- 172.21.15.1:0/4249188369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dd8105d20 0x7f8dd8108110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4009900 tx=0x7f8dc402a410 comp rx=0 tx=0).stop 2024-04-26T21:26:59.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.222+0000 7f8ddf400640 1 -- 172.21.15.1:0/4249188369 shutdown_connections 2024-04-26T21:26:59.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.222+0000 7f8ddf400640 1 --2- 172.21.15.1:0/4249188369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd8108650 0x7f8dd810aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.222+0000 7f8ddf400640 1 --2- 172.21.15.1:0/4249188369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dd8105d20 0x7f8dd8108110 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.222+0000 7f8ddf400640 1 --2- 172.21.15.1:0/4249188369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dd81033f0 0x7f8dd81057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.222+0000 7f8ddf400640 1 -- 172.21.15.1:0/4249188369 >> 172.21.15.1:0/4249188369 conn(0x7f8dd80fce00 msgr2=0x7f8dd80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:59.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.222+0000 7f8ddf400640 1 -- 172.21.15.1:0/4249188369 shutdown_connections 2024-04-26T21:26:59.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.222+0000 7f8ddf400640 1 -- 172.21.15.1:0/4249188369 wait complete. 2024-04-26T21:26:59.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.223+0000 7f8ddf400640 1 Processor -- start 2024-04-26T21:26:59.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.223+0000 7f8ddf400640 1 -- start start 2024-04-26T21:26:59.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddf400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dd81033f0 0x7f8dd819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:59.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddf400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd8105d20 0x7f8dd819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:59.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddf400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dd8108650 0x7f8dd81a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:59.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddf400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd810e4e0 con 0x7f8dd8105d20 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddf400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd810e360 con 0x7f8dd8108650 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddf400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd810e660 con 0x7f8dd81033f0 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddd400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dd8108650 0x7f8dd81a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dd81033f0 0x7f8dd819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8dd7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd8105d20 0x7f8dd819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddd400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dd8108650 0x7f8dd81a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42824/0 (socket says 172.21.15.1:42824) 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddd400640 1 -- 172.21.15.1:0/1247237383 learned_addr learned my addr 172.21.15.1:0/1247237383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddd400640 1 -- 172.21.15.1:0/1247237383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dd81033f0 msgr2=0x7f8dd819adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddd400640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dd81033f0 0x7f8dd819adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddd400640 1 -- 172.21.15.1:0/1247237383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd8105d20 msgr2=0x7f8dd819b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddd400640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd8105d20 0x7f8dd819b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddd400640 1 -- 172.21.15.1:0/1247237383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd81a2a10 con 0x7f8dd8108650 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.224+0000 7f8ddca00640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dd81033f0 0x7f8dd819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:26:59.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.225+0000 7f8ddd400640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dd8108650 0x7f8dd81a23a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8dd000eb10 tx=0x7f8dd000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:59.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.225+0000 7f8dd5600640 1 -- 172.21.15.1:0/1247237383 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0004b20 con 0x7f8dd8108650 2024-04-26T21:26:59.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.225+0000 7f8dd5600640 1 -- 172.21.15.1:0/1247237383 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dd0004cc0 con 0x7f8dd8108650 2024-04-26T21:26:59.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.225+0000 7f8dd5600640 1 -- 172.21.15.1:0/1247237383 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0010670 con 0x7f8dd8108650 2024-04-26T21:26:59.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.225+0000 7f8ddf400640 1 -- 172.21.15.1:0/1247237383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd81a2d00 con 0x7f8dd8108650 2024-04-26T21:26:59.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.225+0000 7f8ddf400640 1 -- 172.21.15.1:0/1247237383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd81a31d0 con 0x7f8dd8108650 2024-04-26T21:26:59.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.227+0000 7f8ddf400640 1 -- 172.21.15.1:0/1247237383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd810a1a0 con 0x7f8dd8108650 2024-04-26T21:26:59.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.228+0000 7f8dd5600640 1 -- 172.21.15.1:0/1247237383 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f8dd0005020 con 0x7f8dd8108650 2024-04-26T21:26:59.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.230+0000 7f8dd5600640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8dbc0a3c30 0x7f8dbc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:59.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.230+0000 7f8dd5600640 1 -- 172.21.15.1:0/1247237383 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f8dd00d04e0 con 0x7f8dd8108650 2024-04-26T21:26:59.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.230+0000 7f8ddca00640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8dbc0a3c30 0x7f8dbc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:59.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.231+0000 7f8ddca00640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8dbc0a3c30 0x7f8dbc0a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009fc0 tx=0x7f8dc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:59.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.234+0000 7f8dd5600640 1 -- 172.21.15.1:0/1247237383 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8dd009d2c0 con 0x7f8dd8108650 2024-04-26T21:26:59.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.448+0000 7f8ddf400640 1 -- 172.21.15.1:0/1247237383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8dd8063530 con 0x7f8dd8108650 2024-04-26T21:26:59.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.448+0000 7f8dd5600640 1 -- 172.21.15.1:0/1247237383 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f8dd009cbc0 con 0x7f8dd8108650 2024-04-26T21:26:59.449 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 -- 172.21.15.1:0/1247237383 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8dbc0a3c30 msgr2=0x7f8dbc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8dbc0a3c30 0x7f8dbc0a60f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009fc0 tx=0x7f8dc8005f90 comp rx=0 tx=0).stop 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 -- 172.21.15.1:0/1247237383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dd8108650 msgr2=0x7f8dd81a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dd8108650 0x7f8dd81a23a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8dd000eb10 tx=0x7f8dd000c2d0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 -- 172.21.15.1:0/1247237383 shutdown_connections 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8dbc0a3c30 0x7f8dbc0a60f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dd8108650 0x7f8dd81a23a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dd8105d20 0x7f8dd819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 --2- 172.21.15.1:0/1247237383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dd81033f0 0x7f8dd819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 -- 172.21.15.1:0/1247237383 >> 172.21.15.1:0/1247237383 conn(0x7f8dd80fce00 msgr2=0x7f8dd8102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 -- 172.21.15.1:0/1247237383 shutdown_connections 2024-04-26T21:26:59.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.451+0000 7f8ddf400640 1 -- 172.21.15.1:0/1247237383 wait complete. 2024-04-26T21:26:59.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-26T21:26:59.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.667+0000 7f432be00640 1 -- 172.21.15.1:0/3840602855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f432c103a70 msgr2=0x7f432c103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.667+0000 7f432be00640 1 --2- 172.21.15.1:0/3840602855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f432c103a70 0x7f432c103ef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4314009900 tx=0x7f431402a3f0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.667+0000 7f432be00640 1 -- 172.21.15.1:0/3840602855 shutdown_connections 2024-04-26T21:26:59.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.667+0000 7f432be00640 1 --2- 172.21.15.1:0/3840602855 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f432c104430 0x7f432c10acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.667+0000 7f432be00640 1 --2- 172.21.15.1:0/3840602855 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f432c103a70 0x7f432c103ef0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.667+0000 7f432be00640 1 --2- 172.21.15.1:0/3840602855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c102880 0x7f432c102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.667+0000 7f432be00640 1 -- 172.21.15.1:0/3840602855 >> 172.21.15.1:0/3840602855 conn(0x7f432c0fddf0 msgr2=0x7f432c100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:59.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.667+0000 7f432be00640 1 -- 172.21.15.1:0/3840602855 shutdown_connections 2024-04-26T21:26:59.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.667+0000 7f432be00640 1 -- 172.21.15.1:0/3840602855 wait complete. 2024-04-26T21:26:59.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f432be00640 1 Processor -- start 2024-04-26T21:26:59.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f432be00640 1 -- start start 2024-04-26T21:26:59.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f432be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f432c102880 0x7f432c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:59.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f432be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c103a70 0x7f432c199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f432be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f432c104430 0x7f432c1a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f432be00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f432c10e400 con 0x7f432c104430 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f432be00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f432c10e280 con 0x7f432c102880 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f432be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f432c10e580 con 0x7f432c103a70 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f4323e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c103a70 0x7f432c199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f432aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f432c102880 0x7f432c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.670+0000 7f432aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f432c102880 0x7f432c198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42834/0 (socket says 172.21.15.1:42834) 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.669+0000 7f4323e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c103a70 0x7f432c199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37804/0 (socket says 172.21.15.1:37804) 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.670+0000 7f432aa00640 1 -- 172.21.15.1:0/1306047185 learned_addr learned my addr 172.21.15.1:0/1306047185 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.670+0000 7f432b400640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f432c104430 0x7f432c1a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.670+0000 7f432aa00640 1 -- 172.21.15.1:0/1306047185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c103a70 msgr2=0x7f432c199010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.670+0000 7f432aa00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c103a70 0x7f432c199010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.670+0000 7f432aa00640 1 -- 172.21.15.1:0/1306047185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f432c104430 msgr2=0x7f432c1a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.670+0000 7f432aa00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f432c104430 0x7f432c1a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.670+0000 7f432aa00640 1 -- 172.21.15.1:0/1306047185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432c1a06f0 con 0x7f432c102880 2024-04-26T21:26:59.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.670+0000 7f432aa00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f432c102880 0x7f432c198ad0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f431c00e9c0 tx=0x7f431c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:59.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.673+0000 7f4323400640 1 -- 172.21.15.1:0/1306047185 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431c00ccc0 con 0x7f432c102880 2024-04-26T21:26:59.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.673+0000 7f4323400640 1 -- 172.21.15.1:0/1306047185 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f431c0035f0 con 0x7f432c102880 2024-04-26T21:26:59.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.673+0000 7f4323400640 1 -- 172.21.15.1:0/1306047185 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f431c00ce60 con 0x7f432c102880 2024-04-26T21:26:59.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.673+0000 7f432be00640 1 -- 172.21.15.1:0/1306047185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f432c1a09e0 con 0x7f432c102880 2024-04-26T21:26:59.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.674+0000 7f432be00640 1 -- 172.21.15.1:0/1306047185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f432c1a0eb0 con 0x7f432c102880 2024-04-26T21:26:59.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.675+0000 7f432be00640 1 -- 172.21.15.1:0/1306047185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f432c109cc0 con 0x7f432c102880 2024-04-26T21:26:59.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.677+0000 7f4323400640 1 -- 172.21.15.1:0/1306047185 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f431c004000 con 0x7f432c102880 2024-04-26T21:26:59.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.679+0000 7f4323400640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43100a3c20 0x7f43100a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:26:59.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.679+0000 7f4323400640 1 -- 172.21.15.1:0/1306047185 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f431c0d0450 con 0x7f432c102880 2024-04-26T21:26:59.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.679+0000 7f4323e00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43100a3c20 0x7f43100a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:26:59.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.680+0000 7f4323e00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43100a3c20 0x7f43100a60e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4314031500 tx=0x7f4314005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:26:59.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.682+0000 7f4323400640 1 -- 172.21.15.1:0/1306047185 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f431c09d230 con 0x7f432c102880 2024-04-26T21:26:59.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:59 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2822404837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:59.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:59 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:59 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:59 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:59 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:59 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2251930827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:26:59.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:26:59 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1247237383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:26:59.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:59 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2822404837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:59.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:59 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:59 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:59 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:59 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:59 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2251930827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:26:59.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:26:59 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1247237383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:26:59.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.902+0000 7f432be00640 1 -- 172.21.15.1:0/1306047185 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f432c02dd30 con 0x7f432c102880 2024-04-26T21:26:59.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.903+0000 7f4323400640 1 -- 172.21.15.1:0/1306047185 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) ==== 84+0+65 (secure 0 0 0) 0x7f431c09cb30 con 0x7f432c102880 2024-04-26T21:26:59.904 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:26:59.904 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-04-26T21:26:59.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:26:59.904 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-04-26T21:26:59.904 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-04-26T21:26:59.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:26:59.904 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-04-26T21:26:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.906+0000 7f432be00640 1 -- 172.21.15.1:0/1306047185 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43100a3c20 msgr2=0x7f43100a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.906+0000 7f432be00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43100a3c20 0x7f43100a60e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4314031500 tx=0x7f4314005a20 comp rx=0 tx=0).stop 2024-04-26T21:26:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.906+0000 7f432be00640 1 -- 172.21.15.1:0/1306047185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f432c102880 msgr2=0x7f432c198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:26:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.906+0000 7f432be00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f432c102880 0x7f432c198ad0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f431c00e9c0 tx=0x7f431c00edc0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.906+0000 7f432be00640 1 -- 172.21.15.1:0/1306047185 shutdown_connections 2024-04-26T21:26:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.906+0000 7f432be00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43100a3c20 0x7f43100a60e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.906+0000 7f432be00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f432c103a70 0x7f432c199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.906+0000 7f432be00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f432c104430 0x7f432c1a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.906+0000 7f432be00640 1 --2- 172.21.15.1:0/1306047185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f432c102880 0x7f432c198ad0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:26:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.906+0000 7f432be00640 1 -- 172.21.15.1:0/1306047185 >> 172.21.15.1:0/1306047185 conn(0x7f432c0fddf0 msgr2=0x7f432c106e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:26:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.907+0000 7f432be00640 1 -- 172.21.15.1:0/1306047185 shutdown_connections 2024-04-26T21:26:59.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:26:59.907+0000 7f432be00640 1 -- 172.21.15.1:0/1306047185 wait complete. 2024-04-26T21:26:59.928 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-26T21:26:59.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-26T21:26:59.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:59 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2822404837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:26:59.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:59 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:59 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:59 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:59 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:26:59.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:59 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2251930827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:26:59.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:26:59 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1247237383' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:27:00.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.143+0000 7ff108e00640 1 -- 172.21.15.1:0/591697896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff104105d20 msgr2=0x7ff104108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:00.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.143+0000 7ff108e00640 1 --2- 172.21.15.1:0/591697896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff104105d20 0x7ff104108110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009900 tx=0x7ff0f002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:00.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.143+0000 7ff108e00640 1 -- 172.21.15.1:0/591697896 shutdown_connections 2024-04-26T21:27:00.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.143+0000 7ff108e00640 1 --2- 172.21.15.1:0/591697896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff104105d20 0x7ff104108110 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:00.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.143+0000 7ff108e00640 1 --2- 172.21.15.1:0/591697896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104108650 0x7ff10410aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:00.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.143+0000 7ff108e00640 1 --2- 172.21.15.1:0/591697896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1041033f0 0x7ff1041057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:00.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.143+0000 7ff108e00640 1 -- 172.21.15.1:0/591697896 >> 172.21.15.1:0/591697896 conn(0x7ff1040fce00 msgr2=0x7ff1040ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:00.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.143+0000 7ff108e00640 1 -- 172.21.15.1:0/591697896 shutdown_connections 2024-04-26T21:27:00.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.143+0000 7ff108e00640 1 -- 172.21.15.1:0/591697896 wait complete. 2024-04-26T21:27:00.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.144+0000 7ff108e00640 1 Processor -- start 2024-04-26T21:27:00.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.144+0000 7ff108e00640 1 -- start start 2024-04-26T21:27:00.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.144+0000 7ff108e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1041033f0 0x7ff10419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:00.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.144+0000 7ff108e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff104105d20 0x7ff10419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:00.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.144+0000 7ff108e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104108650 0x7ff1041a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:00.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.144+0000 7ff108e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff10410e4e0 con 0x7ff104108650 2024-04-26T21:27:00.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.144+0000 7ff108e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff10410e360 con 0x7ff1041033f0 2024-04-26T21:27:00.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.144+0000 7ff108e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff10410e660 con 0x7ff104105d20 2024-04-26T21:27:00.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff101e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1041033f0 0x7ff10419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff101e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1041033f0 0x7ff10419ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42856/0 (socket says 172.21.15.1:42856) 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff101e00640 1 -- 172.21.15.1:0/1751426098 learned_addr learned my addr 172.21.15.1:0/1751426098 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff101400640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff104105d20 0x7ff10419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff102800640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104108650 0x7ff1041a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff101e00640 1 -- 172.21.15.1:0/1751426098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff104105d20 msgr2=0x7ff10419b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff101e00640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff104105d20 0x7ff10419b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff101e00640 1 -- 172.21.15.1:0/1751426098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104108650 msgr2=0x7ff1041a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff101e00640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104108650 0x7ff1041a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff101e00640 1 -- 172.21.15.1:0/1751426098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1041a2a20 con 0x7ff1041033f0 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.145+0000 7ff101e00640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1041033f0 0x7ff10419ae00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400ea60 tx=0x7ff0f400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.146+0000 7ff0faa00640 1 -- 172.21.15.1:0/1751426098 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f400cde0 con 0x7ff1041033f0 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.146+0000 7ff0faa00640 1 -- 172.21.15.1:0/1751426098 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0f40035f0 con 0x7ff1041033f0 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.146+0000 7ff0faa00640 1 -- 172.21.15.1:0/1751426098 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4010690 con 0x7ff1041033f0 2024-04-26T21:27:00.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.146+0000 7ff108e00640 1 -- 172.21.15.1:0/1751426098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1041a2d10 con 0x7ff1041033f0 2024-04-26T21:27:00.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.146+0000 7ff108e00640 1 -- 172.21.15.1:0/1751426098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1041a3180 con 0x7ff1041033f0 2024-04-26T21:27:00.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.147+0000 7ff108e00640 1 -- 172.21.15.1:0/1751426098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff10410a1a0 con 0x7ff1041033f0 2024-04-26T21:27:00.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.149+0000 7ff0faa00640 1 -- 172.21.15.1:0/1751426098 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7ff0f40037a0 con 0x7ff1041033f0 2024-04-26T21:27:00.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.150+0000 7ff0faa00640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0e80a3c30 0x7ff0e80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:00.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.150+0000 7ff0faa00640 1 -- 172.21.15.1:0/1751426098 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff0f40d02f0 con 0x7ff1041033f0 2024-04-26T21:27:00.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.151+0000 7ff101400640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0e80a3c30 0x7ff0e80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:00.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.151+0000 7ff101400640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0e80a3c30 0x7ff0e80a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0005ab0 tx=0x7ff0f0005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:00.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.154+0000 7ff0faa00640 1 -- 172.21.15.1:0/1751426098 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff0f409d0d0 con 0x7ff1041033f0 2024-04-26T21:27:00.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:00.366+0000 7ff108e00640 1 -- 172.21.15.1:0/1751426098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff104062d70 con 0x7ff1041033f0 2024-04-26T21:27:00.951 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:27:00.951 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:00.951 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:00.951 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:00.952 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:00.952 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:00.952 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1306047185' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T21:27:00.952 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:00.952 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:00.952 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1751426098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T21:27:00.952 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:00 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1306047185' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1751426098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T21:27:00.955 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:00 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T21:27:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:27:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1306047185' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-26T21:27:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:27:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:01.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1751426098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T21:27:01.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:00 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-26T21:27:01.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.275+0000 7ff0faa00640 1 -- 172.21.15.1:0/1751426098 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) ==== 138+0+0 (secure 0 0 0) 0x7ff0f409c9d0 con 0x7ff1041033f0 2024-04-26T21:27:01.279 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-04-26T21:27:01.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.282+0000 7ff108e00640 1 -- 172.21.15.1:0/1751426098 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0e80a3c30 msgr2=0x7ff0e80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:01.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.282+0000 7ff108e00640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0e80a3c30 0x7ff0e80a60f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0005ab0 tx=0x7ff0f0005a20 comp rx=0 tx=0).stop 2024-04-26T21:27:01.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.282+0000 7ff108e00640 1 -- 172.21.15.1:0/1751426098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1041033f0 msgr2=0x7ff10419ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.282+0000 7ff108e00640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1041033f0 0x7ff10419ae00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400ea60 tx=0x7ff0f400ef20 comp rx=0 tx=0).stop 2024-04-26T21:27:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.283+0000 7ff108e00640 1 -- 172.21.15.1:0/1751426098 shutdown_connections 2024-04-26T21:27:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.283+0000 7ff108e00640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff0e80a3c30 0x7ff0e80a60f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.283+0000 7ff108e00640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff104105d20 0x7ff10419b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.283+0000 7ff108e00640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff104108650 0x7ff1041a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.283+0000 7ff108e00640 1 --2- 172.21.15.1:0/1751426098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1041033f0 0x7ff10419ae00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.283+0000 7ff108e00640 1 -- 172.21.15.1:0/1751426098 >> 172.21.15.1:0/1751426098 conn(0x7ff1040fce00 msgr2=0x7ff104102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.284+0000 7ff108e00640 1 -- 172.21.15.1:0/1751426098 shutdown_connections 2024-04-26T21:27:01.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.284+0000 7ff108e00640 1 -- 172.21.15.1:0/1751426098 wait complete. 2024-04-26T21:27:01.307 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-26T21:27:01.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.509+0000 7f5416600640 1 -- 172.21.15.1:0/430560134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5410105d80 msgr2=0x7f5410106200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:01.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.509+0000 7f5416600640 1 --2- 172.21.15.1:0/430560134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5410105d80 0x7f5410106200 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009900 tx=0x7f53fc02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:01.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.509+0000 7f5416600640 1 -- 172.21.15.1:0/430560134 shutdown_connections 2024-04-26T21:27:01.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.509+0000 7f5416600640 1 --2- 172.21.15.1:0/430560134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5410105d80 0x7f5410106200 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:01.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.509+0000 7f5416600640 1 --2- 172.21.15.1:0/430560134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5410106740 0x7f541010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:01.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.509+0000 7f5416600640 1 --2- 172.21.15.1:0/430560134 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5410104b90 0x7f5410104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:01.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.509+0000 7f5416600640 1 -- 172.21.15.1:0/430560134 >> 172.21.15.1:0/430560134 conn(0x7f5410100120 msgr2=0x7f5410102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:01.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.509+0000 7f5416600640 1 -- 172.21.15.1:0/430560134 shutdown_connections 2024-04-26T21:27:01.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.509+0000 7f5416600640 1 -- 172.21.15.1:0/430560134 wait complete. 2024-04-26T21:27:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.510+0000 7f5416600640 1 Processor -- start 2024-04-26T21:27:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.510+0000 7f5416600640 1 -- start start 2024-04-26T21:27:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.510+0000 7f5416600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5410104b90 0x7f541019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f5416600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5410105d80 0x7f541019b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f5416600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5410106740 0x7f54101a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f5416600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f54101106f0 con 0x7f5410104b90 2024-04-26T21:27:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f5416600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5410110570 con 0x7f5410106740 2024-04-26T21:27:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f5416600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5410110870 con 0x7f5410105d80 2024-04-26T21:27:01.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5410104b90 0x7f541019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5410105d80 0x7f541019b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5410106740 0x7f54101a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5410105d80 0x7f541019b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37844/0 (socket says 172.21.15.1:37844) 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5410104b90 0x7f541019add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49912/0 (socket says 172.21.15.1:49912) 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540ea00640 1 -- 172.21.15.1:0/422908627 learned_addr learned my addr 172.21.15.1:0/422908627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540ea00640 1 -- 172.21.15.1:0/422908627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5410106740 msgr2=0x7f54101a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540ea00640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5410106740 0x7f54101a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540ea00640 1 -- 172.21.15.1:0/422908627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5410104b90 msgr2=0x7f541019add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540ea00640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5410104b90 0x7f541019add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.511+0000 7f540ea00640 1 -- 172.21.15.1:0/422908627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54101a29f0 con 0x7f5410105d80 2024-04-26T21:27:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.512+0000 7f540ea00640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5410105d80 0x7f541019b310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53fc005ab0 tx=0x7f53fc003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.512+0000 7f5407e00640 1 -- 172.21.15.1:0/422908627 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc03f070 con 0x7f5410105d80 2024-04-26T21:27:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.512+0000 7f5407e00640 1 -- 172.21.15.1:0/422908627 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53fc031c50 con 0x7f5410105d80 2024-04-26T21:27:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.512+0000 7f5416600640 1 -- 172.21.15.1:0/422908627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54101a2c80 con 0x7f5410105d80 2024-04-26T21:27:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.512+0000 7f5407e00640 1 -- 172.21.15.1:0/422908627 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc035030 con 0x7f5410105d80 2024-04-26T21:27:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.512+0000 7f5416600640 1 -- 172.21.15.1:0/422908627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54101a31c0 con 0x7f5410105d80 2024-04-26T21:27:01.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.513+0000 7f5416600640 1 -- 172.21.15.1:0/422908627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f541010bed0 con 0x7f5410105d80 2024-04-26T21:27:01.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.516+0000 7f5407e00640 1 -- 172.21.15.1:0/422908627 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f53fc042070 con 0x7f5410105d80 2024-04-26T21:27:01.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.517+0000 7f5407e00640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f53f40a3c30 0x7f53f40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.517+0000 7f5407e00640 1 -- 172.21.15.1:0/422908627 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7f53fc0f0e80 con 0x7f5410105d80 2024-04-26T21:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.517+0000 7f540f400640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f53f40a3c30 0x7f53f40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:01.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.518+0000 7f540f400640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f53f40a3c30 0x7f53f40a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5400005fd0 tx=0x7f540000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:01.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.520+0000 7f5407e00640 1 -- 172.21.15.1:0/422908627 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f53fc0bd4a0 con 0x7f5410105d80 2024-04-26T21:27:01.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:01.729+0000 7f5416600640 1 -- 172.21.15.1:0/422908627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f5410063530 con 0x7f5410105d80 2024-04-26T21:27:02.290 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-04-26T21:27:02.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.287+0000 7f5407e00640 1 -- 172.21.15.1:0/422908627 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v85) ==== 171+0+0 (secure 0 0 0) 0x7f53fc0c17a0 con 0x7f5410105d80 2024-04-26T21:27:02.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.298+0000 7f5416600640 1 -- 172.21.15.1:0/422908627 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f53f40a3c30 msgr2=0x7f53f40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:02.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.298+0000 7f5416600640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f53f40a3c30 0x7f53f40a60f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5400005fd0 tx=0x7f540000b410 comp rx=0 tx=0).stop 2024-04-26T21:27:02.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.298+0000 7f5416600640 1 -- 172.21.15.1:0/422908627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5410105d80 msgr2=0x7f541019b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:02.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.298+0000 7f5416600640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5410105d80 0x7f541019b310 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f53fc005ab0 tx=0x7f53fc003b90 comp rx=0 tx=0).stop 2024-04-26T21:27:02.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.300+0000 7f5416600640 1 -- 172.21.15.1:0/422908627 shutdown_connections 2024-04-26T21:27:02.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.300+0000 7f5416600640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f53f40a3c30 0x7f53f40a60f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:02.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.300+0000 7f5416600640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5410105d80 0x7f541019b310 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:02.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.300+0000 7f5416600640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5410106740 0x7f54101a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:02.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.300+0000 7f5416600640 1 --2- 172.21.15.1:0/422908627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5410104b90 0x7f541019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:02.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.300+0000 7f5416600640 1 -- 172.21.15.1:0/422908627 >> 172.21.15.1:0/422908627 conn(0x7f5410100120 msgr2=0x7f54101090a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:02.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.301+0000 7f5416600640 1 -- 172.21.15.1:0/422908627 shutdown_connections 2024-04-26T21:27:02.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.302+0000 7f5416600640 1 -- 172.21.15.1:0/422908627 wait complete. 2024-04-26T21:27:02.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-26T21:27:02.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.519+0000 7fe7d2a00640 1 -- 172.21.15.1:0/3938174376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7cc113030 msgr2=0x7fe7cc1154c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:02.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.519+0000 7fe7d2a00640 1 --2- 172.21.15.1:0/3938174376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7cc113030 0x7fe7cc1154c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4009ca0 tx=0x7fe7c402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:27:02.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.519+0000 7fe7d2a00640 1 -- 172.21.15.1:0/3938174376 shutdown_connections 2024-04-26T21:27:02.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.519+0000 7fe7d2a00640 1 --2- 172.21.15.1:0/3938174376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7cc06c980 0x7fe7cc06ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:02.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.519+0000 7fe7d2a00640 1 --2- 172.21.15.1:0/3938174376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7cc113030 0x7fe7cc1154c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:02.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.519+0000 7fe7d2a00640 1 --2- 172.21.15.1:0/3938174376 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7cc06df60 0x7fe7cc06c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:02.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.519+0000 7fe7d2a00640 1 -- 172.21.15.1:0/3938174376 >> 172.21.15.1:0/3938174376 conn(0x7fe7cc071910 msgr2=0x7fe7cc073d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:02.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.519+0000 7fe7d2a00640 1 -- 172.21.15.1:0/3938174376 shutdown_connections 2024-04-26T21:27:02.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.519+0000 7fe7d2a00640 1 -- 172.21.15.1:0/3938174376 wait complete. 2024-04-26T21:27:02.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.521+0000 7fe7d2a00640 1 Processor -- start 2024-04-26T21:27:02.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.521+0000 7fe7d2a00640 1 -- start start 2024-04-26T21:27:02.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.522+0000 7fe7d2a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7cc06c980 0x7fe7cc1a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.522+0000 7fe7d2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7cc06df60 0x7fe7cc1a3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.522+0000 7fe7d2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7cc113030 0x7fe7cc1aab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.522+0000 7fe7d2a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc118ef0 con 0x7fe7cc06c980 2024-04-26T21:27:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.522+0000 7fe7d2a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc118d70 con 0x7fe7cc113030 2024-04-26T21:27:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.522+0000 7fe7d2a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc119070 con 0x7fe7cc06df60 2024-04-26T21:27:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.522+0000 7fe7c3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7cc06df60 0x7fe7cc1a3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.522+0000 7fe7c3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7cc06df60 0x7fe7cc1a3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37858/0 (socket says 172.21.15.1:37858) 2024-04-26T21:27:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.522+0000 7fe7c3e00640 1 -- 172.21.15.1:0/4088001630 learned_addr learned my addr 172.21.15.1:0/4088001630 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:02.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.522+0000 7fe7d0a00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7cc113030 0x7fe7cc1aab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7cbe00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7cc06c980 0x7fe7cc1a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7c3e00640 1 -- 172.21.15.1:0/4088001630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7cc113030 msgr2=0x7fe7cc1aab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7c3e00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7cc113030 0x7fe7cc1aab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7c3e00640 1 -- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7cc06c980 msgr2=0x7fe7cc1a35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7c3e00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7cc06c980 0x7fe7cc1a35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7c3e00640 1 -- 172.21.15.1:0/4088001630 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc1ab1c0 con 0x7fe7cc06df60 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7cbe00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7cc06c980 0x7fe7cc1a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7c3e00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7cc06df60 0x7fe7cc1a3ae0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400e9f0 tx=0x7fe7b400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7c9600640 1 -- 172.21.15.1:0/4088001630 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b400cd80 con 0x7fe7cc06df60 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7c9600640 1 -- 172.21.15.1:0/4088001630 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe7b40054d0 con 0x7fe7cc06df60 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7d2a00640 1 -- 172.21.15.1:0/4088001630 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7cc1ab4b0 con 0x7fe7cc06df60 2024-04-26T21:27:02.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.523+0000 7fe7c9600640 1 -- 172.21.15.1:0/4088001630 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4010690 con 0x7fe7cc06df60 2024-04-26T21:27:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.524+0000 7fe7d2a00640 1 -- 172.21.15.1:0/4088001630 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7cc1ab9b0 con 0x7fe7cc06df60 2024-04-26T21:27:02.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.525+0000 7fe7d2a00640 1 -- 172.21.15.1:0/4088001630 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7cc111d60 con 0x7fe7cc06df60 2024-04-26T21:27:02.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.527+0000 7fe7c9600640 1 -- 172.21.15.1:0/4088001630 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fe7b4004e90 con 0x7fe7cc06df60 2024-04-26T21:27:02.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.528+0000 7fe7c9600640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe7b00a3bb0 0x7fe7b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:02.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.528+0000 7fe7c9600640 1 -- 172.21.15.1:0/4088001630 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7fe7b40d0790 con 0x7fe7cc06df60 2024-04-26T21:27:02.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.528+0000 7fe7cbe00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe7b00a3bb0 0x7fe7b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:02.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.529+0000 7fe7cbe00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe7b00a3bb0 0x7fe7b00a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009b50 tx=0x7fe7bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:02.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.532+0000 7fe7c9600640 1 -- 172.21.15.1:0/4088001630 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe7b409d280 con 0x7fe7cc06df60 2024-04-26T21:27:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:02 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T21:27:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:02 smithi001 ceph-mon[28054]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T21:27:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:02 smithi001 ceph-mon[28054]: pgmap v317: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:27:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:02 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/422908627' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T21:27:02.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:02 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:02 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T21:27:02.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:02 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:02 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:02 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:02 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T21:27:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:02 smithi038 ceph-mon[32724]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T21:27:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:02 smithi038 ceph-mon[32724]: pgmap v317: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:27:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:02 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/422908627' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T21:27:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:02 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:02 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T21:27:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:02 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:02 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:02 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:02 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-26T21:27:02.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:02 smithi186 ceph-mon[32991]: osdmap e84: 12 total, 12 up, 12 in 2024-04-26T21:27:02.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:02 smithi186 ceph-mon[32991]: pgmap v317: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:27:02.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:02 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/422908627' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T21:27:02.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:02 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:02 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-26T21:27:02.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:02 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:02 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:02 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:02.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:02.792+0000 7fe7d2a00640 1 -- 172.21.15.1:0/4088001630 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe7cc02dad0 con 0x7fe7cc06df60 2024-04-26T21:27:03.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.295+0000 7fe7c9600640 1 -- 172.21.15.1:0/4088001630 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fe7b409cb80 con 0x7fe7cc06df60 2024-04-26T21:27:03.297 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 2 and data pool 3 2024-04-26T21:27:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.299+0000 7fe7d2a00640 1 -- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe7b00a3bb0 msgr2=0x7fe7b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.299+0000 7fe7d2a00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe7b00a3bb0 0x7fe7b00a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009b50 tx=0x7fe7bc005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:03.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.300+0000 7fe7d2a00640 1 -- 172.21.15.1:0/4088001630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7cc06df60 msgr2=0x7fe7cc1a3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:03.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.300+0000 7fe7d2a00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7cc06df60 0x7fe7cc1a3ae0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400e9f0 tx=0x7fe7b400edf0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.300+0000 7fe7d2a00640 1 -- 172.21.15.1:0/4088001630 shutdown_connections 2024-04-26T21:27:03.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.300+0000 7fe7d2a00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe7b00a3bb0 0x7fe7b00a6070 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc009b50 tx=0x7fe7bc005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:03.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.300+0000 7fe7d2a00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7cc06df60 0x7fe7cc1a3ae0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400e9f0 tx=0x7fe7b400edf0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.300+0000 7fe7d2a00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7cc113030 0x7fe7cc1aab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.300+0000 7fe7d2a00640 1 --2- 172.21.15.1:0/4088001630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7cc06c980 0x7fe7cc1a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.300+0000 7fe7d2a00640 1 -- 172.21.15.1:0/4088001630 >> 172.21.15.1:0/4088001630 conn(0x7fe7cc071910 msgr2=0x7fe7cc10ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:03.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.300+0000 7fe7d2a00640 1 -- 172.21.15.1:0/4088001630 shutdown_connections 2024-04-26T21:27:03.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.302+0000 7fe7d2a00640 1 -- 172.21.15.1:0/4088001630 wait complete. 2024-04-26T21:27:03.330 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-26T21:27:03.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-26T21:27:03.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.541+0000 7f430da00640 1 -- 172.21.15.1:0/435463076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f430810a640 msgr2=0x7f430810cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:03.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.541+0000 7f430da00640 1 --2- 172.21.15.1:0/435463076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f430810a640 0x7f430810cad0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4300009ca0 tx=0x7f430002a8c0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.541+0000 7f430da00640 1 -- 172.21.15.1:0/435463076 shutdown_connections 2024-04-26T21:27:03.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.541+0000 7f430da00640 1 --2- 172.21.15.1:0/435463076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4308075390 0x7f4308075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.541+0000 7f430da00640 1 --2- 172.21.15.1:0/435463076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f430810a640 0x7f430810cad0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.541+0000 7f430da00640 1 --2- 172.21.15.1:0/435463076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4308076970 0x7f4308074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.541+0000 7f430da00640 1 -- 172.21.15.1:0/435463076 >> 172.21.15.1:0/435463076 conn(0x7f43080fdcf0 msgr2=0x7f4308100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:03.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.541+0000 7f430da00640 1 -- 172.21.15.1:0/435463076 shutdown_connections 2024-04-26T21:27:03.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.541+0000 7f430da00640 1 -- 172.21.15.1:0/435463076 wait complete. 2024-04-26T21:27:03.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.542+0000 7f430da00640 1 Processor -- start 2024-04-26T21:27:03.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.543+0000 7f430da00640 1 -- start start 2024-04-26T21:27:03.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.543+0000 7f430da00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4308075390 0x7f430819abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:03.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.543+0000 7f430da00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4308076970 0x7f430819b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:03.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.543+0000 7f430da00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f430810a640 0x7f43081a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:03.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.543+0000 7f430da00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f43081104e0 con 0x7f4308076970 2024-04-26T21:27:03.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.543+0000 7f430da00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4308110360 con 0x7f430810a640 2024-04-26T21:27:03.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.543+0000 7f430da00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4308110660 con 0x7f4308075390 2024-04-26T21:27:03.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.543+0000 7f4307e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4308075390 0x7f430819abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:03.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.543+0000 7f4307e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4308075390 0x7f430819abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37872/0 (socket says 172.21.15.1:37872) 2024-04-26T21:27:03.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.543+0000 7f4307e00640 1 -- 172.21.15.1:0/2243202024 learned_addr learned my addr 172.21.15.1:0/2243202024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:03.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.544+0000 7f4307400640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4308076970 0x7f430819b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:03.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.544+0000 7f430d000640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f430810a640 0x7f43081a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.544+0000 7f4307e00640 1 -- 172.21.15.1:0/2243202024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f430810a640 msgr2=0x7f43081a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.544+0000 7f4307e00640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f430810a640 0x7f43081a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.544+0000 7f4307e00640 1 -- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4308076970 msgr2=0x7f430819b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.544+0000 7f4307e00640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4308076970 0x7f430819b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.544+0000 7f4307e00640 1 -- 172.21.15.1:0/2243202024 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43081a2780 con 0x7f4308075390 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.544+0000 7f4307400640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4308076970 0x7f430819b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.544+0000 7f4307e00640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4308075390 0x7f430819abb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f42f800e9f0 tx=0x7f42f800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.544+0000 7f4304c00640 1 -- 172.21.15.1:0/2243202024 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f800cd80 con 0x7f4308075390 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.545+0000 7f430da00640 1 -- 172.21.15.1:0/2243202024 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43081a2a70 con 0x7f4308075390 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.545+0000 7f4304c00640 1 -- 172.21.15.1:0/2243202024 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42f80108e0 con 0x7f4308075390 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.545+0000 7f4304c00640 1 -- 172.21.15.1:0/2243202024 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8018980 con 0x7f4308075390 2024-04-26T21:27:03.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.545+0000 7f430da00640 1 -- 172.21.15.1:0/2243202024 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43081a2fc0 con 0x7f4308075390 2024-04-26T21:27:03.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.546+0000 7f430da00640 1 -- 172.21.15.1:0/2243202024 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4308109370 con 0x7f4308075390 2024-04-26T21:27:03.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.548+0000 7f4304c00640 1 -- 172.21.15.1:0/2243202024 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f42f8010430 con 0x7f4308075390 2024-04-26T21:27:03.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.549+0000 7f4304c00640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42ec0a3bb0 0x7f42ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:03.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.549+0000 7f4304c00640 1 -- 172.21.15.1:0/2243202024 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f42f80d0a60 con 0x7f4308075390 2024-04-26T21:27:03.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.550+0000 7f4307400640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42ec0a3bb0 0x7f42ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:03.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.550+0000 7f4307400640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42ec0a3bb0 0x7f42ec0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f430819c070 tx=0x7f42f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:03.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.552+0000 7f4304c00640 1 -- 172.21.15.1:0/2243202024 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f42f809d4f0 con 0x7f4308075390 2024-04-26T21:27:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T21:27:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T21:27:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T21:27:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4088001630' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T21:27:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T21:27:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T21:27:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T21:27:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:03.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:03.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4088001630' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T21:27:03.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:03.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:03.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:03.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:03.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: osdmap e85: 12 total, 12 up, 12 in 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4088001630' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: osdmap e86: 12 total, 12 up, 12 in 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:27:03.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:27:03.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.785+0000 7f430da00640 1 -- 172.21.15.1:0/2243202024 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f430802dbf0 con 0x7f4308075390 2024-04-26T21:27:03.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.836+0000 7f4304c00640 1 -- 172.21.15.1:0/2243202024 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v87) ==== 157+0+0 (secure 0 0 0) 0x7f42f809cdf0 con 0x7f4308075390 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.841+0000 7f430da00640 1 -- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42ec0a3bb0 msgr2=0x7f42ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.841+0000 7f430da00640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42ec0a3bb0 0x7f42ec0a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f430819c070 tx=0x7f42f0005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.841+0000 7f430da00640 1 -- 172.21.15.1:0/2243202024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4308075390 msgr2=0x7f430819abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.841+0000 7f430da00640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4308075390 0x7f430819abb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f42f800e9f0 tx=0x7f42f800eeb0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.841+0000 7f430da00640 1 -- 172.21.15.1:0/2243202024 shutdown_connections 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.841+0000 7f430da00640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42ec0a3bb0 0x7f42ec0a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.841+0000 7f430da00640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f430810a640 0x7f43081a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.841+0000 7f430da00640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4308076970 0x7f430819b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.842+0000 7f430da00640 1 --2- 172.21.15.1:0/2243202024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4308075390 0x7f430819abb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.842+0000 7f430da00640 1 -- 172.21.15.1:0/2243202024 >> 172.21.15.1:0/2243202024 conn(0x7f43080fdcf0 msgr2=0x7f4308106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.842+0000 7f430da00640 1 -- 172.21.15.1:0/2243202024 shutdown_connections 2024-04-26T21:27:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:03.842+0000 7f430da00640 1 -- 172.21.15.1:0/2243202024 wait complete. 2024-04-26T21:27:03.863 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-26T21:27:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.090+0000 7f1fe8a00640 1 -- 172.21.15.1:0/3247706864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe4078ed0 msgr2=0x7f1fe4079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.090+0000 7f1fe8a00640 1 --2- 172.21.15.1:0/3247706864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe4078ed0 0x7f1fe4079770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc009be0 tx=0x7f1fdc02a6c0 comp rx=0 tx=0).stop 2024-04-26T21:27:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.090+0000 7f1fe8a00640 1 -- 172.21.15.1:0/3247706864 shutdown_connections 2024-04-26T21:27:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.090+0000 7f1fe8a00640 1 --2- 172.21.15.1:0/3247706864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe4078ed0 0x7f1fe4079770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.090+0000 7f1fe8a00640 1 --2- 172.21.15.1:0/3247706864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe4078530 0x7f1fe4078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.090+0000 7f1fe8a00640 1 --2- 172.21.15.1:0/3247706864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fe40772f0 0x7f1fe4077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.090+0000 7f1fe8a00640 1 -- 172.21.15.1:0/3247706864 >> 172.21.15.1:0/3247706864 conn(0x7f1fe4100000 msgr2=0x7f1fe4102440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.090+0000 7f1fe8a00640 1 -- 172.21.15.1:0/3247706864 shutdown_connections 2024-04-26T21:27:04.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.090+0000 7f1fe8a00640 1 -- 172.21.15.1:0/3247706864 wait complete. 2024-04-26T21:27:04.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.092+0000 7f1fe8a00640 1 Processor -- start 2024-04-26T21:27:04.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.092+0000 7f1fe8a00640 1 -- start start 2024-04-26T21:27:04.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.092+0000 7f1fe8a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe40772f0 0x7f1fe419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:04.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.092+0000 7f1fe8a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe4078530 0x7f1fe419f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:04.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.092+0000 7f1fe8a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fe4078ed0 0x7f1fe41a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:04.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.092+0000 7f1fe8a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe41149d0 con 0x7f1fe40772f0 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.092+0000 7f1fe8a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe4114850 con 0x7f1fe4078ed0 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.092+0000 7f1fe8a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe4114b50 con 0x7f1fe4078530 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fe2800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe40772f0 0x7f1fe419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fdbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe4078530 0x7f1fe419f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fe3200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fe4078ed0 0x7f1fe41a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fe2800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe40772f0 0x7f1fe419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49970/0 (socket says 172.21.15.1:49970) 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fe2800640 1 -- 172.21.15.1:0/693418998 learned_addr learned my addr 172.21.15.1:0/693418998 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fe3200640 1 -- 172.21.15.1:0/693418998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe4078530 msgr2=0x7f1fe419f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fe3200640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe4078530 0x7f1fe419f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fe3200640 1 -- 172.21.15.1:0/693418998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe40772f0 msgr2=0x7f1fe419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fe3200640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe40772f0 0x7f1fe419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:04.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fe3200640 1 -- 172.21.15.1:0/693418998 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe41a6c20 con 0x7f1fe4078ed0 2024-04-26T21:27:04.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.093+0000 7f1fdbe00640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe4078530 0x7f1fe419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:04.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.094+0000 7f1fe3200640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fe4078ed0 0x7f1fe41a65b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc034040 tx=0x7f1fdc030d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:04.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.094+0000 7f1fdb400640 1 -- 172.21.15.1:0/693418998 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc041020 con 0x7f1fe4078ed0 2024-04-26T21:27:04.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.094+0000 7f1fdb400640 1 -- 172.21.15.1:0/693418998 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1fdc0318f0 con 0x7f1fe4078ed0 2024-04-26T21:27:04.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.094+0000 7f1fe8a00640 1 -- 172.21.15.1:0/693418998 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe41a6eb0 con 0x7f1fe4078ed0 2024-04-26T21:27:04.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.094+0000 7f1fdb400640 1 -- 172.21.15.1:0/693418998 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc036040 con 0x7f1fe4078ed0 2024-04-26T21:27:04.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.094+0000 7f1fe8a00640 1 -- 172.21.15.1:0/693418998 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe41a7310 con 0x7f1fe4078ed0 2024-04-26T21:27:04.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.096+0000 7f1fe8a00640 1 -- 172.21.15.1:0/693418998 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe41115a0 con 0x7f1fe4078ed0 2024-04-26T21:27:04.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.098+0000 7f1fdb400640 1 -- 172.21.15.1:0/693418998 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f1fdc044020 con 0x7f1fe4078ed0 2024-04-26T21:27:04.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.100+0000 7f1fdb400640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fc80a3ba0 0x7f1fc80a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:04.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.100+0000 7f1fdb400640 1 -- 172.21.15.1:0/693418998 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8277+0+0 (secure 0 0 0) 0x7f1fdc0f0fb0 con 0x7f1fe4078ed0 2024-04-26T21:27:04.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.100+0000 7f1fe2800640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fc80a3ba0 0x7f1fc80a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:04.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.101+0000 7f1fe2800640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fc80a3ba0 0x7f1fc80a6060 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4005c80 tx=0x7f1fd4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:04.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.104+0000 7f1fdb400640 1 -- 172.21.15.1:0/693418998 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1fdc04b020 con 0x7f1fe4078ed0 2024-04-26T21:27:04.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:04.315+0000 7f1fe8a00640 1 -- 172.21.15.1:0/693418998 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1fe402dad0 con 0x7f1fe4078ed0 2024-04-26T21:27:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:04 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T21:27:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:04 smithi001 ceph-mon[28054]: pgmap v320: 97 pgs: 17 creating+peering, 8 active+clean, 72 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:27:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:04 smithi001 ceph-mon[28054]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T21:27:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:04 smithi001 ceph-mon[28054]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T21:27:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:04 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] up:active 2024-04-26T21:27:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:04 smithi001 ceph-mon[28054]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:04 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2243202024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T21:27:04.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:04 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T21:27:04.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:04 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T21:27:04.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:04 smithi001 ceph-mon[28054]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:04 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:04 smithi038 ceph-mon[32724]: pgmap v320: 97 pgs: 17 creating+peering, 8 active+clean, 72 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:04 smithi038 ceph-mon[32724]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:04 smithi038 ceph-mon[32724]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:04 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] up:active 2024-04-26T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:04 smithi038 ceph-mon[32724]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:04.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:04 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2243202024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T21:27:04.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:04 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T21:27:04.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:04 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T21:27:04.578 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:04 smithi038 ceph-mon[32724]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T21:27:04.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:04 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-26T21:27:04.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:04 smithi186 ceph-mon[32991]: pgmap v320: 97 pgs: 17 creating+peering, 8 active+clean, 72 unknown; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:27:04.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:04 smithi186 ceph-mon[32991]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-26T21:27:04.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:04 smithi186 ceph-mon[32991]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-26T21:27:04.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:04 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] up:active 2024-04-26T21:27:04.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:04 smithi186 ceph-mon[32991]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:04.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:04 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2243202024' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T21:27:04.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:04 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T21:27:04.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:04 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-26T21:27:04.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:04 smithi186 ceph-mon[32991]: osdmap e87: 12 total, 12 up, 12 in 2024-04-26T21:27:05.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:05 smithi001 ceph-mon[28054]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T21:27:05.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:05 smithi001 ceph-mon[28054]: Cluster is now healthy 2024-04-26T21:27:05.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:05 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/693418998' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:27:05.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:05 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:05 smithi038 ceph-mon[32724]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:05 smithi038 ceph-mon[32724]: Cluster is now healthy 2024-04-26T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:05 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/693418998' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:05 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:27:05.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:05 smithi186 ceph-mon[32991]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T21:27:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:05 smithi186 ceph-mon[32991]: Cluster is now healthy 2024-04-26T21:27:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:05 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/693418998' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:27:05.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:05 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-26T21:27:06.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.333+0000 7f1fdb400640 1 -- 172.21.15.1:0/693418998 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v89) ==== 228+0+0 (secure 0 0 0) 0x7f1fdc0bd900 con 0x7f1fe4078ed0 2024-04-26T21:27:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data_ec' created 2024-04-26T21:27:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.339+0000 7f1fe8a00640 1 -- 172.21.15.1:0/693418998 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fc80a3ba0 msgr2=0x7f1fc80a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.339+0000 7f1fe8a00640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fc80a3ba0 0x7f1fc80a6060 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4005c80 tx=0x7f1fd4009260 comp rx=0 tx=0).stop 2024-04-26T21:27:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.339+0000 7f1fe8a00640 1 -- 172.21.15.1:0/693418998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fe4078ed0 msgr2=0x7f1fe41a65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.339+0000 7f1fe8a00640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fe4078ed0 0x7f1fe41a65b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc034040 tx=0x7f1fdc030d50 comp rx=0 tx=0).stop 2024-04-26T21:27:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.339+0000 7f1fe8a00640 1 -- 172.21.15.1:0/693418998 shutdown_connections 2024-04-26T21:27:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.339+0000 7f1fe8a00640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1fc80a3ba0 0x7f1fc80a6060 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.339+0000 7f1fe8a00640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe4078530 0x7f1fe419f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.339+0000 7f1fe8a00640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fe4078ed0 0x7f1fe41a65b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:06.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.339+0000 7f1fe8a00640 1 --2- 172.21.15.1:0/693418998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1fe40772f0 0x7f1fe419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:06.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.339+0000 7f1fe8a00640 1 -- 172.21.15.1:0/693418998 >> 172.21.15.1:0/693418998 conn(0x7f1fe4100000 msgr2=0x7f1fe41016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:06.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.340+0000 7f1fe8a00640 1 -- 172.21.15.1:0/693418998 shutdown_connections 2024-04-26T21:27:06.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.340+0000 7f1fe8a00640 1 -- 172.21.15.1:0/693418998 wait complete. 2024-04-26T21:27:06.367 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-26T21:27:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.555+0000 7ff322600640 1 -- 172.21.15.1:0/3167278256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff31c108540 msgr2=0x7ff31c10a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.555+0000 7ff322600640 1 --2- 172.21.15.1:0/3167278256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff31c108540 0x7ff31c10a930 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff314009be0 tx=0x7ff31402a6c0 comp rx=0 tx=0).stop 2024-04-26T21:27:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.555+0000 7ff322600640 1 -- 172.21.15.1:0/3167278256 shutdown_connections 2024-04-26T21:27:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.555+0000 7ff322600640 1 --2- 172.21.15.1:0/3167278256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff31c105c10 0x7ff31c108000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.555+0000 7ff322600640 1 --2- 172.21.15.1:0/3167278256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff31c108540 0x7ff31c10a930 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.555+0000 7ff322600640 1 --2- 172.21.15.1:0/3167278256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff31c1032e0 0x7ff31c1056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.555+0000 7ff322600640 1 -- 172.21.15.1:0/3167278256 >> 172.21.15.1:0/3167278256 conn(0x7ff31c0fcc10 msgr2=0x7ff31c0ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:06.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.555+0000 7ff322600640 1 -- 172.21.15.1:0/3167278256 shutdown_connections 2024-04-26T21:27:06.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.555+0000 7ff322600640 1 -- 172.21.15.1:0/3167278256 wait complete. 2024-04-26T21:27:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.556+0000 7ff322600640 1 Processor -- start 2024-04-26T21:27:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.557+0000 7ff322600640 1 -- start start 2024-04-26T21:27:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.557+0000 7ff322600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff31c1032e0 0x7ff31c19ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.557+0000 7ff322600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff31c105c10 0x7ff31c19b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.557+0000 7ff322600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff31c108540 0x7ff31c1a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.557+0000 7ff322600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c10e2e0 con 0x7ff31c108540 2024-04-26T21:27:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.557+0000 7ff322600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c10e160 con 0x7ff31c105c10 2024-04-26T21:27:06.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.557+0000 7ff322600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff31c10e460 con 0x7ff31c1032e0 2024-04-26T21:27:06.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.557+0000 7ff321c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff31c108540 0x7ff31c1a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:06.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.557+0000 7ff321200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff31c1032e0 0x7ff31c19ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:06.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.558+0000 7ff321c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff31c108540 0x7ff31c1a2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49988/0 (socket says 172.21.15.1:49988) 2024-04-26T21:27:06.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.558+0000 7ff321c00640 1 -- 172.21.15.1:0/242650120 learned_addr learned my addr 172.21.15.1:0/242650120 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:06.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.558+0000 7ff31be00640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff31c105c10 0x7ff31c19b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:06.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.558+0000 7ff321200640 1 -- 172.21.15.1:0/242650120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff31c105c10 msgr2=0x7ff31c19b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:06.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.558+0000 7ff321200640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff31c105c10 0x7ff31c19b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:06.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.558+0000 7ff321200640 1 -- 172.21.15.1:0/242650120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff31c108540 msgr2=0x7ff31c1a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:06.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.558+0000 7ff321200640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff31c108540 0x7ff31c1a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:06.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.558+0000 7ff321200640 1 -- 172.21.15.1:0/242650120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff31c1a2830 con 0x7ff31c1032e0 2024-04-26T21:27:06.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.558+0000 7ff321200640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff31c1032e0 0x7ff31c19ac20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff310005fd0 tx=0x7ff31000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:06.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.559+0000 7ff319600640 1 -- 172.21.15.1:0/242650120 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31000cd00 con 0x7ff31c1032e0 2024-04-26T21:27:06.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.559+0000 7ff322600640 1 -- 172.21.15.1:0/242650120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff31c1a2b20 con 0x7ff31c1032e0 2024-04-26T21:27:06.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.559+0000 7ff319600640 1 -- 172.21.15.1:0/242650120 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff3100035f0 con 0x7ff31c1032e0 2024-04-26T21:27:06.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.559+0000 7ff319600640 1 -- 172.21.15.1:0/242650120 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff310010670 con 0x7ff31c1032e0 2024-04-26T21:27:06.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.559+0000 7ff322600640 1 -- 172.21.15.1:0/242650120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff31c1a3070 con 0x7ff31c1032e0 2024-04-26T21:27:06.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.561+0000 7ff322600640 1 -- 172.21.15.1:0/242650120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff31c10a030 con 0x7ff31c1032e0 2024-04-26T21:27:06.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.562+0000 7ff319600640 1 -- 172.21.15.1:0/242650120 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7ff3100037a0 con 0x7ff31c1032e0 2024-04-26T21:27:06.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.563+0000 7ff319600640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3040a3bb0 0x7ff3040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:06.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.564+0000 7ff319600640 1 -- 172.21.15.1:0/242650120 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7ff3100d08d0 con 0x7ff31c1032e0 2024-04-26T21:27:06.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.564+0000 7ff31be00640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3040a3bb0 0x7ff3040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:06.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.565+0000 7ff31be00640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3040a3bb0 0x7ff3040a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff31c06a6d0 tx=0x7ff308005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:06.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.568+0000 7ff319600640 1 -- 172.21.15.1:0/242650120 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff310009510 con 0x7ff31c1032e0 2024-04-26T21:27:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:06 smithi038 ceph-mon[32724]: pgmap v322: 97 pgs: 17 creating+peering, 42 active+clean, 38 unknown; 578 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 763 B/s wr, 2 op/s 2024-04-26T21:27:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:06 smithi038 ceph-mon[32724]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T21:27:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:06 smithi186 ceph-mon[32991]: pgmap v322: 97 pgs: 17 creating+peering, 42 active+clean, 38 unknown; 578 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 763 B/s wr, 2 op/s 2024-04-26T21:27:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:06 smithi186 ceph-mon[32991]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T21:27:06.771 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:06 smithi001 ceph-mon[28054]: pgmap v322: 97 pgs: 17 creating+peering, 42 active+clean, 38 unknown; 578 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 763 B/s wr, 2 op/s 2024-04-26T21:27:06.772 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:06 smithi001 ceph-mon[28054]: osdmap e88: 12 total, 12 up, 12 in 2024-04-26T21:27:06.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:06.796+0000 7ff322600640 1 -- 172.21.15.1:0/242650120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7ff31c02dad0 con 0x7ff31c1032e0 2024-04-26T21:27:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.344+0000 7ff319600640 1 -- 172.21.15.1:0/242650120 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v90) ==== 169+0+0 (secure 0 0 0) 0x7ff31009d030 con 0x7ff31c1032e0 2024-04-26T21:27:07.345 INFO:teuthology.orchestra.run.smithi001.stderr:set pool 4 allow_ec_overwrites to true 2024-04-26T21:27:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.348+0000 7ff322600640 1 -- 172.21.15.1:0/242650120 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3040a3bb0 msgr2=0x7ff3040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.349+0000 7ff322600640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3040a3bb0 0x7ff3040a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff31c06a6d0 tx=0x7ff308005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.349+0000 7ff322600640 1 -- 172.21.15.1:0/242650120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff31c1032e0 msgr2=0x7ff31c19ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:07.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.349+0000 7ff322600640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff31c1032e0 0x7ff31c19ac20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff310005fd0 tx=0x7ff31000ee50 comp rx=0 tx=0).stop 2024-04-26T21:27:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.349+0000 7ff322600640 1 -- 172.21.15.1:0/242650120 shutdown_connections 2024-04-26T21:27:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.349+0000 7ff322600640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3040a3bb0 0x7ff3040a6070 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff31c06a6d0 tx=0x7ff308005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.349+0000 7ff322600640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff31c108540 0x7ff31c1a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.349+0000 7ff322600640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff31c105c10 0x7ff31c19b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.349+0000 7ff322600640 1 --2- 172.21.15.1:0/242650120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff31c1032e0 0x7ff31c19ac20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.349+0000 7ff322600640 1 -- 172.21.15.1:0/242650120 >> 172.21.15.1:0/242650120 conn(0x7ff31c0fcc10 msgr2=0x7ff31c102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:07.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.350+0000 7ff322600640 1 -- 172.21.15.1:0/242650120 shutdown_connections 2024-04-26T21:27:07.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.351+0000 7ff322600640 1 -- 172.21.15.1:0/242650120 wait complete. 2024-04-26T21:27:07.382 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:07 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T21:27:07.382 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:07 smithi001 ceph-mon[28054]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T21:27:07.382 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:07 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/242650120' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T21:27:07.382 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:07 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T21:27:07.385 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-26T21:27:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.561+0000 7f1d25200640 1 -- 172.21.15.1:0/2658706204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d201016a0 msgr2=0x7f1d20101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.561+0000 7f1d25200640 1 --2- 172.21.15.1:0/2658706204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d201016a0 0x7f1d20101ac0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1d10009900 tx=0x7f1d1002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.562+0000 7f1d25200640 1 -- 172.21.15.1:0/2658706204 shutdown_connections 2024-04-26T21:27:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.562+0000 7f1d25200640 1 --2- 172.21.15.1:0/2658706204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d20103250 0x7f1d20109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.562+0000 7f1d25200640 1 --2- 172.21.15.1:0/2658706204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d20102890 0x7f1d20102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.562+0000 7f1d25200640 1 --2- 172.21.15.1:0/2658706204 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d201016a0 0x7f1d20101ac0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.562+0000 7f1d25200640 1 -- 172.21.15.1:0/2658706204 >> 172.21.15.1:0/2658706204 conn(0x7f1d200fcc10 msgr2=0x7f1d200ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.562+0000 7f1d25200640 1 -- 172.21.15.1:0/2658706204 shutdown_connections 2024-04-26T21:27:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.562+0000 7f1d25200640 1 -- 172.21.15.1:0/2658706204 wait complete. 2024-04-26T21:27:07.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d25200640 1 Processor -- start 2024-04-26T21:27:07.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d25200640 1 -- start start 2024-04-26T21:27:07.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d25200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d201016a0 0x7f1d2019acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:07.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d25200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d20102890 0x7f1d2019b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:07.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d25200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d20103250 0x7f1d201a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d25200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2010e350 con 0x7f1d20103250 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d25200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2010e1d0 con 0x7f1d20102890 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d25200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2010e4d0 con 0x7f1d201016a0 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d1fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d20103250 0x7f1d201a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d1f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d201016a0 0x7f1d2019acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d1ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d20102890 0x7f1d2019b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d1fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d20103250 0x7f1d201a21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50008/0 (socket says 172.21.15.1:50008) 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d1f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d201016a0 0x7f1d2019acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37934/0 (socket says 172.21.15.1:37934) 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.563+0000 7f1d1fe00640 1 -- 172.21.15.1:0/680492971 learned_addr learned my addr 172.21.15.1:0/680492971 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.564+0000 7f1d1ea00640 1 -- 172.21.15.1:0/680492971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d201016a0 msgr2=0x7f1d2019acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.564+0000 7f1d1ea00640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d201016a0 0x7f1d2019acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.564+0000 7f1d1ea00640 1 -- 172.21.15.1:0/680492971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d20103250 msgr2=0x7f1d201a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.564+0000 7f1d1ea00640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d20103250 0x7f1d201a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.564+0000 7f1d1ea00640 1 -- 172.21.15.1:0/680492971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d201a2870 con 0x7f1d20102890 2024-04-26T21:27:07.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.564+0000 7f1d1ea00640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d20102890 0x7f1d2019b1f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00e9f0 tx=0x7f1d0c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.564+0000 7f1d17e00640 1 -- 172.21.15.1:0/680492971 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c00cd80 con 0x7f1d20102890 2024-04-26T21:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.565+0000 7f1d17e00640 1 -- 172.21.15.1:0/680492971 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d0c0035f0 con 0x7f1d20102890 2024-04-26T21:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.565+0000 7f1d25200640 1 -- 172.21.15.1:0/680492971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d201a2b60 con 0x7f1d20102890 2024-04-26T21:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.565+0000 7f1d17e00640 1 -- 172.21.15.1:0/680492971 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c010670 con 0x7f1d20102890 2024-04-26T21:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.565+0000 7f1d25200640 1 -- 172.21.15.1:0/680492971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d201a3080 con 0x7f1d20102890 2024-04-26T21:27:07.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.566+0000 7f1d25200640 1 -- 172.21.15.1:0/680492971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d201a32b0 con 0x7f1d20102890 2024-04-26T21:27:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.569+0000 7f1d17e00640 1 -- 172.21.15.1:0/680492971 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f1d0c0037a0 con 0x7f1d20102890 2024-04-26T21:27:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.570+0000 7f1d17e00640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1d040a3bb0 0x7f1d040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.570+0000 7f1d17e00640 1 -- 172.21.15.1:0/680492971 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8770+0+0 (secure 0 0 0) 0x7f1d0c0d0980 con 0x7f1d20102890 2024-04-26T21:27:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.570+0000 7f1d1f400640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1d040a3bb0 0x7f1d040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.571+0000 7f1d1f400640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1d040a3bb0 0x7f1d040a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1d10005ab0 tx=0x7f1d10005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.572+0000 7f1d17e00640 1 -- 172.21.15.1:0/680492971 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1d0c09d0e0 con 0x7f1d20102890 2024-04-26T21:27:07.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:07 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T21:27:07.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:07 smithi186 ceph-mon[32991]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T21:27:07.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:07 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/242650120' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T21:27:07.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:07 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T21:27:07.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:07.813+0000 7f1d25200640 1 -- 172.21.15.1:0/680492971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f1d201a36a0 con 0x7f1d20102890 2024-04-26T21:27:07.827 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:27:07 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b[62819]: 2024-04-26T21:27:07.404+0000 7f2f14698700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-26T21:27:07.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:07 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-26T21:27:07.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:07 smithi038 ceph-mon[32724]: osdmap e89: 12 total, 12 up, 12 in 2024-04-26T21:27:07.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:07 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/242650120' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T21:27:07.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:07 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-26T21:27:08.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.353+0000 7f1d17e00640 1 -- 172.21.15.1:0/680492971 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7f1d0c09c9e0 con 0x7f1d20102890 2024-04-26T21:27:08.356 INFO:teuthology.orchestra.run.smithi001.stderr:added data pool 4 to fsmap 2024-04-26T21:27:08.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.358+0000 7f1d25200640 1 -- 172.21.15.1:0/680492971 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1d040a3bb0 msgr2=0x7f1d040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:08.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.358+0000 7f1d25200640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1d040a3bb0 0x7f1d040a6070 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1d10005ab0 tx=0x7f1d10005a20 comp rx=0 tx=0).stop 2024-04-26T21:27:08.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.358+0000 7f1d25200640 1 -- 172.21.15.1:0/680492971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d20102890 msgr2=0x7f1d2019b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:08.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.358+0000 7f1d25200640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d20102890 0x7f1d2019b1f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00e9f0 tx=0x7f1d0c00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.358+0000 7f1d25200640 1 -- 172.21.15.1:0/680492971 shutdown_connections 2024-04-26T21:27:08.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.358+0000 7f1d25200640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1d040a3bb0 0x7f1d040a6070 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.358+0000 7f1d25200640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d20103250 0x7f1d201a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.358+0000 7f1d25200640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d20102890 0x7f1d2019b1f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.358+0000 7f1d25200640 1 --2- 172.21.15.1:0/680492971 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d201016a0 0x7f1d2019acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.358+0000 7f1d25200640 1 -- 172.21.15.1:0/680492971 >> 172.21.15.1:0/680492971 conn(0x7f1d200fcc10 msgr2=0x7f1d2010a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:08.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.359+0000 7f1d25200640 1 -- 172.21.15.1:0/680492971 shutdown_connections 2024-04-26T21:27:08.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.359+0000 7f1d25200640 1 -- 172.21.15.1:0/680492971 wait complete. 2024-04-26T21:27:08.384 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:27:08.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.578+0000 7f25caa00640 1 -- 172.21.15.1:0/2729945428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25c4109f80 msgr2=0x7f25c410a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.578+0000 7f25caa00640 1 --2- 172.21.15.1:0/2729945428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25c4109f80 0x7f25c410a400 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009900 tx=0x7f25b002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.579+0000 7f25caa00640 1 -- 172.21.15.1:0/2729945428 shutdown_connections 2024-04-26T21:27:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.579+0000 7f25caa00640 1 --2- 172.21.15.1:0/2729945428 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c410a940 0x7f25c41111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.579+0000 7f25caa00640 1 --2- 172.21.15.1:0/2729945428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25c4109f80 0x7f25c410a400 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.579+0000 7f25caa00640 1 --2- 172.21.15.1:0/2729945428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25c4108d90 0x7f25c41091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.579+0000 7f25caa00640 1 -- 172.21.15.1:0/2729945428 >> 172.21.15.1:0/2729945428 conn(0x7f25c4077d30 msgr2=0x7f25c407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.579+0000 7f25caa00640 1 -- 172.21.15.1:0/2729945428 shutdown_connections 2024-04-26T21:27:08.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.579+0000 7f25caa00640 1 -- 172.21.15.1:0/2729945428 wait complete. 2024-04-26T21:27:08.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.580+0000 7f25caa00640 1 Processor -- start 2024-04-26T21:27:08.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.580+0000 7f25caa00640 1 -- start start 2024-04-26T21:27:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25caa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25c4108d90 0x7f25c419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25caa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c4109f80 0x7f25c419f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25caa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25c410a940 0x7f25c41a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25caa00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f25c41148c0 con 0x7f25c4109f80 2024-04-26T21:27:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25caa00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f25c4114740 con 0x7f25c410a940 2024-04-26T21:27:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25caa00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f25c4114a40 con 0x7f25c4108d90 2024-04-26T21:27:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25c9600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25c4108d90 0x7f25c419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25c8c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c4109f80 0x7f25c419f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:08.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25c9600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25c4108d90 0x7f25c419ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44764/0 (socket says 172.21.15.1:44764) 2024-04-26T21:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25c9600640 1 -- 172.21.15.1:0/831375413 learned_addr learned my addr 172.21.15.1:0/831375413 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.581+0000 7f25ca000640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25c410a940 0x7f25c41a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25c9600640 1 -- 172.21.15.1:0/831375413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25c410a940 msgr2=0x7f25c41a64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25c9600640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25c410a940 0x7f25c41a64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25c9600640 1 -- 172.21.15.1:0/831375413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c4109f80 msgr2=0x7f25c419f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25c9600640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c4109f80 0x7f25c419f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25c9600640 1 -- 172.21.15.1:0/831375413 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c41a6b50 con 0x7f25c4108d90 2024-04-26T21:27:08.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25c8c00640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c4109f80 0x7f25c419f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25c9600640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25c4108d90 0x7f25c419ef30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f25b800e9c0 tx=0x7f25b800edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25c2000640 1 -- 172.21.15.1:0/831375413 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b800ccc0 con 0x7f25c4108d90 2024-04-26T21:27:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25c2000640 1 -- 172.21.15.1:0/831375413 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25b80035f0 con 0x7f25c4108d90 2024-04-26T21:27:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25caa00640 1 -- 172.21.15.1:0/831375413 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c41a6e40 con 0x7f25c4108d90 2024-04-26T21:27:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25c2000640 1 -- 172.21.15.1:0/831375413 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b800ce60 con 0x7f25c4108d90 2024-04-26T21:27:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.582+0000 7f25caa00640 1 -- 172.21.15.1:0/831375413 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c41a72a0 con 0x7f25c4108d90 2024-04-26T21:27:08.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.583+0000 7f25caa00640 1 -- 172.21.15.1:0/831375413 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25c41100b0 con 0x7f25c4108d90 2024-04-26T21:27:08.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.586+0000 7f25c2000640 1 -- 172.21.15.1:0/831375413 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f25b8004000 con 0x7f25c4108d90 2024-04-26T21:27:08.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.587+0000 7f25c2000640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25ac0a3cd0 0x7f25ac0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:08.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.587+0000 7f25c2000640 1 -- 172.21.15.1:0/831375413 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f25b80d0c80 con 0x7f25c4108d90 2024-04-26T21:27:08.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.587+0000 7f25c8c00640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25ac0a3cd0 0x7f25ac0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:08.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.588+0000 7f25c8c00640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25ac0a3cd0 0x7f25ac0a6190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f25b0000c00 tx=0x7f25b00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:08.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.589+0000 7f25c2000640 1 -- 172.21.15.1:0/831375413 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f25b809d3c0 con 0x7f25c4108d90 2024-04-26T21:27:08.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:08 smithi186 ceph-mon[32991]: pgmap v325: 129 pgs: 18 creating+peering, 85 active+clean, 26 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 8 op/s 2024-04-26T21:27:08.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:08 smithi186 ceph-mon[32991]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T21:27:08.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:08 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T21:27:08.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:08 smithi186 ceph-mon[32991]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T21:27:08.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:08 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/680492971' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T21:27:08.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:08 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T21:27:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:08 smithi001 ceph-mon[28054]: pgmap v325: 129 pgs: 18 creating+peering, 85 active+clean, 26 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 8 op/s 2024-04-26T21:27:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:08 smithi001 ceph-mon[28054]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T21:27:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:08 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T21:27:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:08 smithi001 ceph-mon[28054]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T21:27:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:08 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/680492971' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T21:27:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:08 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T21:27:08.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:08 smithi038 ceph-mon[32724]: pgmap v325: 129 pgs: 18 creating+peering, 85 active+clean, 26 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 8 op/s 2024-04-26T21:27:08.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:08 smithi038 ceph-mon[32724]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T21:27:08.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:08 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-26T21:27:08.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:08 smithi038 ceph-mon[32724]: osdmap e90: 12 total, 12 up, 12 in 2024-04-26T21:27:08.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:08 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/680492971' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T21:27:08.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:08 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-26T21:27:08.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.839+0000 7f25caa00640 1 -- 172.21.15.1:0/831375413 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25c402dd60 con 0x7f25c4108d90 2024-04-26T21:27:08.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.841+0000 7f25c2000640 1 -- 172.21.15.1:0/831375413 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11785 (secure 0 0 0) 0x7f25b809ccc0 con 0x7f25c4108d90 2024-04-26T21:27:08.842 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:08.843 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24572,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T21:27:02.795246+0000","modified":"2024-04-26T21:27:07.817139+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24602},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24602,"qdb_cluster":[24602]},"id":1}]} 2024-04-26T21:27:08.843 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-04-26T21:27:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 -- 172.21.15.1:0/831375413 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25ac0a3cd0 msgr2=0x7f25ac0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25ac0a3cd0 0x7f25ac0a6190 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f25b0000c00 tx=0x7f25b00314a0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 -- 172.21.15.1:0/831375413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25c4108d90 msgr2=0x7f25c419ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25c4108d90 0x7f25c419ef30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f25b800e9c0 tx=0x7f25b800edc0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 -- 172.21.15.1:0/831375413 shutdown_connections 2024-04-26T21:27:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25ac0a3cd0 0x7f25ac0a6190 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25c410a940 0x7f25c41a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25c4109f80 0x7f25c419f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 --2- 172.21.15.1:0/831375413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25c4108d90 0x7f25c419ef30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:08.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 -- 172.21.15.1:0/831375413 >> 172.21.15.1:0/831375413 conn(0x7f25c4077d30 msgr2=0x7f25c410d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:08.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 -- 172.21.15.1:0/831375413 shutdown_connections 2024-04-26T21:27:08.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:08.844+0000 7f25caa00640 1 -- 172.21.15.1:0/831375413 wait complete. 2024-04-26T21:27:08.865 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:27:09.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.068+0000 7f493a200640 1 -- 172.21.15.1:0/278004689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934109f80 msgr2=0x7f493410a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.068+0000 7f493a200640 1 --2- 172.21.15.1:0/278004689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934109f80 0x7f493410a400 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4924009900 tx=0x7f492402a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.069+0000 7f493a200640 1 -- 172.21.15.1:0/278004689 shutdown_connections 2024-04-26T21:27:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.069+0000 7f493a200640 1 --2- 172.21.15.1:0/278004689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f493410a940 0x7f49341111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.069+0000 7f493a200640 1 --2- 172.21.15.1:0/278004689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4934109f80 0x7f493410a400 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.069+0000 7f493a200640 1 --2- 172.21.15.1:0/278004689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4934108d90 0x7f49341091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.069+0000 7f493a200640 1 -- 172.21.15.1:0/278004689 >> 172.21.15.1:0/278004689 conn(0x7f4934077d30 msgr2=0x7f493407a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.069+0000 7f493a200640 1 -- 172.21.15.1:0/278004689 shutdown_connections 2024-04-26T21:27:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.069+0000 7f493a200640 1 -- 172.21.15.1:0/278004689 wait complete. 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.070+0000 7f493a200640 1 Processor -- start 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.070+0000 7f493a200640 1 -- start start 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.070+0000 7f493a200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4934108d90 0x7f493419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.070+0000 7f493a200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4934109f80 0x7f493419f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.070+0000 7f493a200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493410a940 0x7f49341a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.070+0000 7f493a200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f49341148a0 con 0x7f4934109f80 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.070+0000 7f493a200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4934114720 con 0x7f493410a940 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.070+0000 7f493a200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4934114a20 con 0x7f4934108d90 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4938e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4934108d90 0x7f493419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4938e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4934108d90 0x7f493419eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44786/0 (socket says 172.21.15.1:44786) 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4938e00640 1 -- 172.21.15.1:0/150201626 learned_addr learned my addr 172.21.15.1:0/150201626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4939800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493410a940 0x7f49341a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4933e00640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4934109f80 0x7f493419f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4938e00640 1 -- 172.21.15.1:0/150201626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493410a940 msgr2=0x7f49341a64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4938e00640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493410a940 0x7f49341a64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4938e00640 1 -- 172.21.15.1:0/150201626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4934109f80 msgr2=0x7f493419f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4938e00640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4934109f80 0x7f493419f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4938e00640 1 -- 172.21.15.1:0/150201626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49341a6b10 con 0x7f4934108d90 2024-04-26T21:27:09.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.071+0000 7f4938e00640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4934108d90 0x7f493419eef0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f491c00e9c0 tx=0x7f491c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.072+0000 7f4931600640 1 -- 172.21.15.1:0/150201626 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c00cde0 con 0x7f4934108d90 2024-04-26T21:27:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.072+0000 7f4931600640 1 -- 172.21.15.1:0/150201626 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f491c0108e0 con 0x7f4934108d90 2024-04-26T21:27:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.072+0000 7f493a200640 1 -- 172.21.15.1:0/150201626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49341a6e00 con 0x7f4934108d90 2024-04-26T21:27:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.072+0000 7f4931600640 1 -- 172.21.15.1:0/150201626 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c018960 con 0x7f4934108d90 2024-04-26T21:27:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.072+0000 7f493a200640 1 -- 172.21.15.1:0/150201626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49341a7270 con 0x7f4934108d90 2024-04-26T21:27:09.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.073+0000 7f493a200640 1 -- 172.21.15.1:0/150201626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4934110050 con 0x7f4934108d90 2024-04-26T21:27:09.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.075+0000 7f4931600640 1 -- 172.21.15.1:0/150201626 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f491c01f020 con 0x7f4934108d90 2024-04-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.077+0000 7f4931600640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49180a3cd0 0x7f49180a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.077+0000 7f4931600640 1 -- 172.21.15.1:0/150201626 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f491c0d0870 con 0x7f4934108d90 2024-04-26T21:27:09.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.077+0000 7f4933e00640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49180a3cd0 0x7f49180a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.078+0000 7f4933e00640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49180a3cd0 0x7f49180a6190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4924000c00 tx=0x7f49240314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:09.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.081+0000 7f4931600640 1 -- 172.21.15.1:0/150201626 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f491c054060 con 0x7f4934108d90 2024-04-26T21:27:09.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.288+0000 7f493a200640 1 -- 172.21.15.1:0/150201626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4934063670 con 0x7f4934108d90 2024-04-26T21:27:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.291+0000 7f4931600640 1 -- 172.21.15.1:0/150201626 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23972 (secure 0 0 0) 0x7f491c09cfb0 con 0x7f4934108d90 2024-04-26T21:27:09.293 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:09.293 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:27:07.816919+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:27:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.293+0000 7f493a200640 1 -- 172.21.15.1:0/150201626 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49180a3cd0 msgr2=0x7f49180a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.293+0000 7f493a200640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49180a3cd0 0x7f49180a6190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4924000c00 tx=0x7f49240314a0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.293+0000 7f493a200640 1 -- 172.21.15.1:0/150201626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4934108d90 msgr2=0x7f493419eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.293+0000 7f493a200640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4934108d90 0x7f493419eef0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f491c00e9c0 tx=0x7f491c00ee80 comp rx=0 tx=0).stop 2024-04-26T21:27:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.294+0000 7f493a200640 1 -- 172.21.15.1:0/150201626 shutdown_connections 2024-04-26T21:27:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.294+0000 7f493a200640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49180a3cd0 0x7f49180a6190 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.294+0000 7f493a200640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f493410a940 0x7f49341a64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.294+0000 7f493a200640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4934109f80 0x7f493419f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.294+0000 7f493a200640 1 --2- 172.21.15.1:0/150201626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4934108d90 0x7f493419eef0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.294+0000 7f493a200640 1 -- 172.21.15.1:0/150201626 >> 172.21.15.1:0/150201626 conn(0x7f4934077d30 msgr2=0x7f493410d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.294+0000 7f493a200640 1 -- 172.21.15.1:0/150201626 shutdown_connections 2024-04-26T21:27:09.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.294+0000 7f493a200640 1 -- 172.21.15.1:0/150201626 wait complete. 2024-04-26T21:27:09.316 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:27:09.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.518+0000 7fc291200640 1 -- 172.21.15.1:0/238440959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc28c10a940 msgr2=0x7fc28c1111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.518+0000 7fc291200640 1 --2- 172.21.15.1:0/238440959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc28c10a940 0x7fc28c1111d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc284009be0 tx=0x7fc28402a6d0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.518+0000 7fc291200640 1 -- 172.21.15.1:0/238440959 shutdown_connections 2024-04-26T21:27:09.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.518+0000 7fc291200640 1 --2- 172.21.15.1:0/238440959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc28c10a940 0x7fc28c1111d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.518+0000 7fc291200640 1 --2- 172.21.15.1:0/238440959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc28c109f80 0x7fc28c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.518+0000 7fc291200640 1 --2- 172.21.15.1:0/238440959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28c108d90 0x7fc28c1091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.518+0000 7fc291200640 1 -- 172.21.15.1:0/238440959 >> 172.21.15.1:0/238440959 conn(0x7fc28c077d30 msgr2=0x7fc28c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:09.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.519+0000 7fc291200640 1 -- 172.21.15.1:0/238440959 shutdown_connections 2024-04-26T21:27:09.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.519+0000 7fc291200640 1 -- 172.21.15.1:0/238440959 wait complete. 2024-04-26T21:27:09.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.519+0000 7fc291200640 1 Processor -- start 2024-04-26T21:27:09.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc291200640 1 -- start start 2024-04-26T21:27:09.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc291200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc28c108d90 0x7fc28c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc291200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28c109f80 0x7fc28c19f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc291200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc28c10a940 0x7fc28c1a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc291200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc28c1148c0 con 0x7fc28c108d90 2024-04-26T21:27:09.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc291200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc28c114740 con 0x7fc28c10a940 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc291200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc28c114a40 con 0x7fc28c109f80 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc28b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc28c108d90 0x7fc28c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc28b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc28c108d90 0x7fc28c19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59962/0 (socket says 172.21.15.1:59962) 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc28b400640 1 -- 172.21.15.1:0/249227883 learned_addr learned my addr 172.21.15.1:0/249227883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc28aa00640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28c109f80 0x7fc28c19f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.520+0000 7fc28be00640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc28c10a940 0x7fc28c1a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.521+0000 7fc28b400640 1 -- 172.21.15.1:0/249227883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28c109f80 msgr2=0x7fc28c19f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.521+0000 7fc28b400640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28c109f80 0x7fc28c19f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.521+0000 7fc28b400640 1 -- 172.21.15.1:0/249227883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc28c10a940 msgr2=0x7fc28c1a6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.521+0000 7fc28b400640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc28c10a940 0x7fc28c1a6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.521+0000 7fc28b400640 1 -- 172.21.15.1:0/249227883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28c1a6a90 con 0x7fc28c108d90 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.521+0000 7fc28b400640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc28c108d90 0x7fc28c19ee70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc27400e9f0 tx=0x7fc27400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.521+0000 7fc283e00640 1 -- 172.21.15.1:0/249227883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27400ccc0 con 0x7fc28c108d90 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.521+0000 7fc283e00640 1 -- 172.21.15.1:0/249227883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2740035f0 con 0x7fc28c108d90 2024-04-26T21:27:09.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.521+0000 7fc283e00640 1 -- 172.21.15.1:0/249227883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27400ce60 con 0x7fc28c108d90 2024-04-26T21:27:09.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.522+0000 7fc291200640 1 -- 172.21.15.1:0/249227883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc28c1a6d80 con 0x7fc28c108d90 2024-04-26T21:27:09.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.522+0000 7fc291200640 1 -- 172.21.15.1:0/249227883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc28c1a71c0 con 0x7fc28c108d90 2024-04-26T21:27:09.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.523+0000 7fc291200640 1 -- 172.21.15.1:0/249227883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc28c110050 con 0x7fc28c108d90 2024-04-26T21:27:09.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.525+0000 7fc283e00640 1 -- 172.21.15.1:0/249227883 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc274004000 con 0x7fc28c108d90 2024-04-26T21:27:09.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.526+0000 7fc283e00640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2700a3dc0 0x7fc2700a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.527+0000 7fc283e00640 1 -- 172.21.15.1:0/249227883 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fc2740d08f0 con 0x7fc28c108d90 2024-04-26T21:27:09.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.527+0000 7fc28aa00640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2700a3dc0 0x7fc2700a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.527+0000 7fc28aa00640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2700a3dc0 0x7fc2700a6280 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc28c1a0380 tx=0x7fc27c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:09.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.530+0000 7fc283e00640 1 -- 172.21.15.1:0/249227883 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc27409d0b0 con 0x7fc28c108d90 2024-04-26T21:27:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:09 smithi001 ceph-mon[28054]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T21:27:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:09 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T21:27:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:09 smithi001 ceph-mon[28054]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:09 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/831375413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:27:09.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:09 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/150201626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:09.606 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:09 smithi186 ceph-mon[32991]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T21:27:09.606 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:09 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T21:27:09.606 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:09 smithi186 ceph-mon[32991]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:09.606 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:09 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/831375413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:27:09.606 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:09 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/150201626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:09.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.737+0000 7fc291200640 1 -- 172.21.15.1:0/249227883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc28c0636b0 con 0x7fc28c108d90 2024-04-26T21:27:09.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.740+0000 7fc283e00640 1 -- 172.21.15.1:0/249227883 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23972 (secure 0 0 0) 0x7fc27409c9b0 con 0x7fc28c108d90 2024-04-26T21:27:09.742 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:09.742 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:27:07.816919+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:27:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.742+0000 7fc291200640 1 -- 172.21.15.1:0/249227883 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2700a3dc0 msgr2=0x7fc2700a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.742+0000 7fc291200640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2700a3dc0 0x7fc2700a6280 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc28c1a0380 tx=0x7fc27c00b390 comp rx=0 tx=0).stop 2024-04-26T21:27:09.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.742+0000 7fc291200640 1 -- 172.21.15.1:0/249227883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc28c108d90 msgr2=0x7fc28c19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.742+0000 7fc291200640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc28c108d90 0x7fc28c19ee70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc27400e9f0 tx=0x7fc27400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.743+0000 7fc291200640 1 -- 172.21.15.1:0/249227883 shutdown_connections 2024-04-26T21:27:09.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.743+0000 7fc291200640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc2700a3dc0 0x7fc2700a6280 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.743+0000 7fc291200640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc28c109f80 0x7fc28c19f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.743+0000 7fc291200640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc28c10a940 0x7fc28c1a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.743+0000 7fc291200640 1 --2- 172.21.15.1:0/249227883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc28c108d90 0x7fc28c19ee70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.743+0000 7fc291200640 1 -- 172.21.15.1:0/249227883 >> 172.21.15.1:0/249227883 conn(0x7fc28c077d30 msgr2=0x7fc28c10d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:09.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.743+0000 7fc291200640 1 -- 172.21.15.1:0/249227883 shutdown_connections 2024-04-26T21:27:09.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.743+0000 7fc291200640 1 -- 172.21.15.1:0/249227883 wait complete. 2024-04-26T21:27:09.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:27:09.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:09 smithi038 ceph-mon[32724]: osdmap e91: 12 total, 12 up, 12 in 2024-04-26T21:27:09.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:09 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-26T21:27:09.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:09 smithi038 ceph-mon[32724]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:09.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:09 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/831375413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:27:09.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:09 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/150201626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.990+0000 7faa68a00640 1 -- 172.21.15.1:0/589768468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa64105b20 msgr2=0x7faa64107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.990+0000 7faa68a00640 1 --2- 172.21.15.1:0/589768468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa64105b20 0x7faa64107f10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faa4c009860 tx=0x7faa4c02a340 comp rx=0 tx=0).stop 2024-04-26T21:27:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.990+0000 7faa68a00640 1 -- 172.21.15.1:0/589768468 shutdown_connections 2024-04-26T21:27:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.990+0000 7faa68a00640 1 --2- 172.21.15.1:0/589768468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa64105b20 0x7faa64107f10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.990+0000 7faa68a00640 1 --2- 172.21.15.1:0/589768468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa64108450 0x7faa6410a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.990+0000 7faa68a00640 1 --2- 172.21.15.1:0/589768468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa641031f0 0x7faa641055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.990+0000 7faa68a00640 1 -- 172.21.15.1:0/589768468 >> 172.21.15.1:0/589768468 conn(0x7faa640fcc00 msgr2=0x7faa640ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.990+0000 7faa68a00640 1 -- 172.21.15.1:0/589768468 shutdown_connections 2024-04-26T21:27:09.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.990+0000 7faa68a00640 1 -- 172.21.15.1:0/589768468 wait complete. 2024-04-26T21:27:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.991+0000 7faa68a00640 1 Processor -- start 2024-04-26T21:27:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa68a00640 1 -- start start 2024-04-26T21:27:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa68a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa641031f0 0x7faa64102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa68a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa64105b20 0x7faa641011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa68a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa64108450 0x7faa641016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa68a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faa6410e290 con 0x7faa64105b20 2024-04-26T21:27:09.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa68a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa6410e110 con 0x7faa64108450 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa68a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faa6410e410 con 0x7faa641031f0 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa59e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa64105b20 0x7faa641011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa62800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa641031f0 0x7faa64102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa62800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa641031f0 0x7faa64102b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44834/0 (socket says 172.21.15.1:44834) 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.992+0000 7faa62800640 1 -- 172.21.15.1:0/1562418750 learned_addr learned my addr 172.21.15.1:0/1562418750 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa63200640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa64108450 0x7faa641016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa62800640 1 -- 172.21.15.1:0/1562418750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa64108450 msgr2=0x7faa641016e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa62800640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa64108450 0x7faa641016e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa62800640 1 -- 172.21.15.1:0/1562418750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa64105b20 msgr2=0x7faa641011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa62800640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa64105b20 0x7faa641011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa62800640 1 -- 172.21.15.1:0/1562418750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa64101f70 con 0x7faa641031f0 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa63200640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa64108450 0x7faa641016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa62800640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa641031f0 0x7faa64102b60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faa5400e9c0 tx=0x7faa5400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:09.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa5be00640 1 -- 172.21.15.1:0/1562418750 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5400ccc0 con 0x7faa641031f0 2024-04-26T21:27:09.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa68a00640 1 -- 172.21.15.1:0/1562418750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa641a29d0 con 0x7faa641031f0 2024-04-26T21:27:09.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa5be00640 1 -- 172.21.15.1:0/1562418750 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa540054d0 con 0x7faa641031f0 2024-04-26T21:27:09.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.993+0000 7faa5be00640 1 -- 172.21.15.1:0/1562418750 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54010430 con 0x7faa641031f0 2024-04-26T21:27:09.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.994+0000 7faa68a00640 1 -- 172.21.15.1:0/1562418750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa641a2f40 con 0x7faa641031f0 2024-04-26T21:27:09.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.995+0000 7faa68a00640 1 -- 172.21.15.1:0/1562418750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa640fe6f0 con 0x7faa641031f0 2024-04-26T21:27:09.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.997+0000 7faa5be00640 1 -- 172.21.15.1:0/1562418750 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7faa54004f80 con 0x7faa641031f0 2024-04-26T21:27:10.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.998+0000 7faa5be00640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7faa480a3c50 0x7faa480a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.998+0000 7faa5be00640 1 -- 172.21.15.1:0/1562418750 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7faa540d0bc0 con 0x7faa641031f0 2024-04-26T21:27:10.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.998+0000 7faa59e00640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7faa480a3c50 0x7faa480a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:09.999+0000 7faa59e00640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7faa480a3c50 0x7faa480a6110 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faa4c009830 tx=0x7faa4c030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:10.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.003+0000 7faa5be00640 1 -- 172.21.15.1:0/1562418750 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7faa5409d300 con 0x7faa641031f0 2024-04-26T21:27:10.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.245+0000 7faa68a00640 1 -- 172.21.15.1:0/1562418750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa6402dad0 con 0x7faa641031f0 2024-04-26T21:27:10.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.247+0000 7faa5be00640 1 -- 172.21.15.1:0/1562418750 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11785 (secure 0 0 0) 0x7faa6402dad0 con 0x7faa641031f0 2024-04-26T21:27:10.249 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:10.249 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24515,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24548,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24572,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-26T21:27:02.795246+0000","modified":"2024-04-26T21:27:07.817139+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24602},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24602,"qdb_cluster":[24602]},"id":1}]} 2024-04-26T21:27:10.249 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-04-26T21:27:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.249+0000 7faa68a00640 1 -- 172.21.15.1:0/1562418750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7faa480a3c50 msgr2=0x7faa480a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.249+0000 7faa68a00640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7faa480a3c50 0x7faa480a6110 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faa4c009830 tx=0x7faa4c030450 comp rx=0 tx=0).stop 2024-04-26T21:27:10.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.249+0000 7faa68a00640 1 -- 172.21.15.1:0/1562418750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa641031f0 msgr2=0x7faa64102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.249+0000 7faa68a00640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa641031f0 0x7faa64102b60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faa5400e9c0 tx=0x7faa5400ee80 comp rx=0 tx=0).stop 2024-04-26T21:27:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.250+0000 7faa68a00640 1 -- 172.21.15.1:0/1562418750 shutdown_connections 2024-04-26T21:27:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.250+0000 7faa68a00640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7faa480a3c50 0x7faa480a6110 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.250+0000 7faa68a00640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa64108450 0x7faa641016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.250+0000 7faa68a00640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa64105b20 0x7faa641011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.250+0000 7faa68a00640 1 --2- 172.21.15.1:0/1562418750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa641031f0 0x7faa64102b60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.250+0000 7faa68a00640 1 -- 172.21.15.1:0/1562418750 >> 172.21.15.1:0/1562418750 conn(0x7faa640fcc00 msgr2=0x7faa64103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.250+0000 7faa68a00640 1 -- 172.21.15.1:0/1562418750 shutdown_connections 2024-04-26T21:27:10.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.250+0000 7faa68a00640 1 -- 172.21.15.1:0/1562418750 wait complete. 2024-04-26T21:27:10.271 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:27:10.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.467+0000 7f9432c00640 1 -- 172.21.15.1:0/2481037226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f942c078530 msgr2=0x7f942c078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.467+0000 7f9432c00640 1 --2- 172.21.15.1:0/2481037226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f942c078530 0x7f942c078990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9418009900 tx=0x7f941802a410 comp rx=0 tx=0).stop 2024-04-26T21:27:10.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.468+0000 7f9432c00640 1 -- 172.21.15.1:0/2481037226 shutdown_connections 2024-04-26T21:27:10.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.468+0000 7f9432c00640 1 --2- 172.21.15.1:0/2481037226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942c078ed0 0x7f942c079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.468+0000 7f9432c00640 1 --2- 172.21.15.1:0/2481037226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f942c078530 0x7f942c078990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.468+0000 7f9432c00640 1 --2- 172.21.15.1:0/2481037226 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f942c0772f0 0x7f942c077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.468+0000 7f9432c00640 1 -- 172.21.15.1:0/2481037226 >> 172.21.15.1:0/2481037226 conn(0x7f942c0fff00 msgr2=0x7f942c102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:10.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.468+0000 7f9432c00640 1 -- 172.21.15.1:0/2481037226 shutdown_connections 2024-04-26T21:27:10.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.468+0000 7f9432c00640 1 -- 172.21.15.1:0/2481037226 wait complete. 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.469+0000 7f9432c00640 1 Processor -- start 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.469+0000 7f9432c00640 1 -- start start 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.469+0000 7f9432c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f942c0772f0 0x7f942c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.469+0000 7f9432c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942c078530 0x7f942c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.469+0000 7f9432c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f942c078ed0 0x7f942c1a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.469+0000 7f9432c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f942c114910 con 0x7f942c078ed0 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.469+0000 7f9432c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f942c114790 con 0x7f942c078530 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.469+0000 7f9432c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f942c114a90 con 0x7f942c0772f0 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9432200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f942c078ed0 0x7f942c1a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9432200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f942c078ed0 0x7f942c1a6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59998/0 (socket says 172.21.15.1:59998) 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9432200640 1 -- 172.21.15.1:0/4286311946 learned_addr learned my addr 172.21.15.1:0/4286311946 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9430e00640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942c078530 0x7f942c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9431800640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f942c0772f0 0x7f942c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9432200640 1 -- 172.21.15.1:0/4286311946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f942c0772f0 msgr2=0x7f942c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9432200640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f942c0772f0 0x7f942c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9432200640 1 -- 172.21.15.1:0/4286311946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942c078530 msgr2=0x7f942c19f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9432200640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942c078530 0x7f942c19f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9432200640 1 -- 172.21.15.1:0/4286311946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942c1a6b00 con 0x7f942c078ed0 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9430e00640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942c078530 0x7f942c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.470+0000 7f9432200640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f942c078ed0 0x7f942c1a6490 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f942400eb10 tx=0x7f942400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.471+0000 7f942a000640 1 -- 172.21.15.1:0/4286311946 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424004b20 con 0x7f942c078ed0 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.471+0000 7f942a000640 1 -- 172.21.15.1:0/4286311946 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9424004cc0 con 0x7f942c078ed0 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.471+0000 7f942a000640 1 -- 172.21.15.1:0/4286311946 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424010670 con 0x7f942c078ed0 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.471+0000 7f9432c00640 1 -- 172.21.15.1:0/4286311946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942c1a6df0 con 0x7f942c078ed0 2024-04-26T21:27:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.471+0000 7f9432c00640 1 -- 172.21.15.1:0/4286311946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942c1a7260 con 0x7f942c078ed0 2024-04-26T21:27:10.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.472+0000 7f9432c00640 1 -- 172.21.15.1:0/4286311946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f942c0506f0 con 0x7f942c078ed0 2024-04-26T21:27:10.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.474+0000 7f942a000640 1 -- 172.21.15.1:0/4286311946 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f9424005020 con 0x7f942c078ed0 2024-04-26T21:27:10.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.476+0000 7f942a000640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f94140a3c50 0x7f94140a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.476+0000 7f942a000640 1 -- 172.21.15.1:0/4286311946 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f94240d0c00 con 0x7f942c078ed0 2024-04-26T21:27:10.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.476+0000 7f9431800640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f94140a3c50 0x7f94140a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.477+0000 7f9431800640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f94140a3c50 0x7f94140a6110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9420009bd0 tx=0x7f9420005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:10.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.480+0000 7f942a000640 1 -- 172.21.15.1:0/4286311946 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f942c0506f0 con 0x7f942c078ed0 2024-04-26T21:27:10.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:10 smithi001 ceph-mon[28054]: pgmap v328: 129 pgs: 2 creating+activating, 6 creating+peering, 110 active+clean, 11 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-26T21:27:10.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:10 smithi001 ceph-mon[28054]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T21:27:10.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:10 smithi001 ceph-mon[28054]: Cluster is now healthy 2024-04-26T21:27:10.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:10 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/249227883' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:10.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:10 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1562418750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:27:10.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.687+0000 7f9432c00640 1 -- 172.21.15.1:0/4286311946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f942c0635d0 con 0x7f942c078ed0 2024-04-26T21:27:10.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.689+0000 7f942a000640 1 -- 172.21.15.1:0/4286311946 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23972 (secure 0 0 0) 0x7f942409d340 con 0x7f942c078ed0 2024-04-26T21:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:10.692 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:27:07.816919+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:27:10.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.692+0000 7f9432c00640 1 -- 172.21.15.1:0/4286311946 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f94140a3c50 msgr2=0x7f94140a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.692+0000 7f9432c00640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f94140a3c50 0x7f94140a6110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9420009bd0 tx=0x7f9420005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.692+0000 7f9432c00640 1 -- 172.21.15.1:0/4286311946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f942c078ed0 msgr2=0x7f942c1a6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.692+0000 7f9432c00640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f942c078ed0 0x7f942c1a6490 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f942400eb10 tx=0x7f942400c2d0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.692+0000 7f9432c00640 1 -- 172.21.15.1:0/4286311946 shutdown_connections 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.692+0000 7f9432c00640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f94140a3c50 0x7f94140a6110 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.692+0000 7f9432c00640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f942c078ed0 0x7f942c1a6490 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.692+0000 7f9432c00640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f942c078530 0x7f942c19f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.692+0000 7f9432c00640 1 --2- 172.21.15.1:0/4286311946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f942c0772f0 0x7f942c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.692+0000 7f9432c00640 1 -- 172.21.15.1:0/4286311946 >> 172.21.15.1:0/4286311946 conn(0x7f942c0fff00 msgr2=0x7f942c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.693+0000 7f9432c00640 1 -- 172.21.15.1:0/4286311946 shutdown_connections 2024-04-26T21:27:10.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.693+0000 7f9432c00640 1 -- 172.21.15.1:0/4286311946 wait complete. 2024-04-26T21:27:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:10 smithi186 ceph-mon[32991]: pgmap v328: 129 pgs: 2 creating+activating, 6 creating+peering, 110 active+clean, 11 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-26T21:27:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:10 smithi186 ceph-mon[32991]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T21:27:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:10 smithi186 ceph-mon[32991]: Cluster is now healthy 2024-04-26T21:27:10.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:10 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/249227883' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:10.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:10 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1562418750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:27:10.714 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-26T21:27:02.795246+0000', 'modified': '2024-04-26T21:27:07.817139+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24602}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24602': {'gid': 24602, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.38:6839/1675711617', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 1675711617}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 1675711617}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24602, 'qdb_cluster': [24602]} 2024-04-26T21:27:10.714 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-26T21:27:10.714 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T21:27:10.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:10 smithi038 ceph-mon[32724]: pgmap v328: 129 pgs: 2 creating+activating, 6 creating+peering, 110 active+clean, 11 unknown; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-26T21:27:10.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:10 smithi038 ceph-mon[32724]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T21:27:10.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:10 smithi038 ceph-mon[32724]: Cluster is now healthy 2024-04-26T21:27:10.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:10 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/249227883' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:10.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:10 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1562418750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:27:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.931+0000 7fd8cd600640 1 -- 172.21.15.1:0/2014982766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8c8105b30 msgr2=0x7fd8c8107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.931+0000 7fd8cd600640 1 --2- 172.21.15.1:0/2014982766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8c8105b30 0x7fd8c8107f20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0009900 tx=0x7fd8b002a400 comp rx=0 tx=0).stop 2024-04-26T21:27:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.932+0000 7fd8cd600640 1 -- 172.21.15.1:0/2014982766 shutdown_connections 2024-04-26T21:27:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.932+0000 7fd8cd600640 1 --2- 172.21.15.1:0/2014982766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8c8105b30 0x7fd8c8107f20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.932+0000 7fd8cd600640 1 --2- 172.21.15.1:0/2014982766 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8c8108460 0x7fd8c810a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.932+0000 7fd8cd600640 1 --2- 172.21.15.1:0/2014982766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8103200 0x7fd8c81055f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.932+0000 7fd8cd600640 1 -- 172.21.15.1:0/2014982766 >> 172.21.15.1:0/2014982766 conn(0x7fd8c80fcc10 msgr2=0x7fd8c80ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.932+0000 7fd8cd600640 1 -- 172.21.15.1:0/2014982766 shutdown_connections 2024-04-26T21:27:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.932+0000 7fd8cd600640 1 -- 172.21.15.1:0/2014982766 wait complete. 2024-04-26T21:27:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.933+0000 7fd8cd600640 1 Processor -- start 2024-04-26T21:27:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.933+0000 7fd8cd600640 1 -- start start 2024-04-26T21:27:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8cd600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8c8103200 0x7fd8c819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8cd600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8105b30 0x7fd8c819b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8cd600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8c8108460 0x7fd8c81a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8cd600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c810e2f0 con 0x7fd8c8108460 2024-04-26T21:27:10.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8cd600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c810e170 con 0x7fd8c8105b30 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8cd600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c810e470 con 0x7fd8c8103200 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8c7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8105b30 0x7fd8c819b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8ccc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8c8108460 0x7fd8c81a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8c7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8105b30 0x7fd8c819b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:57026/0 (socket says 172.21.15.1:57026) 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8c7400640 1 -- 172.21.15.1:0/3312267474 learned_addr learned my addr 172.21.15.1:0/3312267474 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8c7e00640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8c8103200 0x7fd8c819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8c7400640 1 -- 172.21.15.1:0/3312267474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8c8103200 msgr2=0x7fd8c819aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.934+0000 7fd8c7400640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8c8103200 0x7fd8c819aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.935+0000 7fd8c7400640 1 -- 172.21.15.1:0/3312267474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8c8108460 msgr2=0x7fd8c81a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.935+0000 7fd8c7400640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8c8108460 0x7fd8c81a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.935+0000 7fd8c7400640 1 -- 172.21.15.1:0/3312267474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c81a27c0 con 0x7fd8c8105b30 2024-04-26T21:27:10.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.935+0000 7fd8ccc00640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8c8108460 0x7fd8c81a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:10.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.935+0000 7fd8c7400640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8105b30 0x7fd8c819b0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0031500 tx=0x7fd8b0003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:10.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.935+0000 7fd8c4c00640 1 -- 172.21.15.1:0/3312267474 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b003f070 con 0x7fd8c8105b30 2024-04-26T21:27:10.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.935+0000 7fd8c4c00640 1 -- 172.21.15.1:0/3312267474 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b0031c50 con 0x7fd8c8105b30 2024-04-26T21:27:10.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.935+0000 7fd8cd600640 1 -- 172.21.15.1:0/3312267474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c81a2a50 con 0x7fd8c8105b30 2024-04-26T21:27:10.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.935+0000 7fd8c4c00640 1 -- 172.21.15.1:0/3312267474 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0035030 con 0x7fd8c8105b30 2024-04-26T21:27:10.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.936+0000 7fd8cd600640 1 -- 172.21.15.1:0/3312267474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c81a2f40 con 0x7fd8c8105b30 2024-04-26T21:27:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.939+0000 7fd8d15fd740 1 Processor -- start 2024-04-26T21:27:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.939+0000 7fd8d15fd740 1 -- start start 2024-04-26T21:27:10.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.939+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc74976e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.939+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 0x55bfc749c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.939+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc749f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.939+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55bfc73fb600 con 0x55bfc749ce20 2024-04-26T21:27:10.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.939+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bfc7398100 con 0x55bfc74972c0 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.939+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bfc73f75c0 con 0x55bfc7497cb0 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8c4c00640 1 -- 172.21.15.1:0/3312267474 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd8b0042070 con 0x7fd8c8105b30 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8ccc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc74976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8ccc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc74976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:57034/0 (socket says 172.21.15.1:57034) 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8ccc00640 1 -- 172.21.15.1:0/2362506968 learned_addr learned my addr 172.21.15.1:0/2362506968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8c7e00640 1 --2- 172.21.15.1:0/2362506968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 0x55bfc749c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8c7400640 1 --2- 172.21.15.1:0/2362506968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc749f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8ccc00640 1 -- 172.21.15.1:0/2362506968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 msgr2=0x55bfc749c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8ccc00640 1 --2- 172.21.15.1:0/2362506968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 0x55bfc749c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8ccc00640 1 -- 172.21.15.1:0/2362506968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 msgr2=0x55bfc749f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8ccc00640 1 --2- 172.21.15.1:0/2362506968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc749f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8ccc00640 1 -- 172.21.15.1:0/2362506968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfc710d2d0 con 0x55bfc74972c0 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8c7400640 1 --2- 172.21.15.1:0/2362506968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc749f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.940+0000 7fd8ccc00640 1 --2- 172.21.15.1:0/2362506968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc74976e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8011570 tx=0x7fd8b8011a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea246fd6d3c6d9cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.941+0000 7fd8bd200640 1 -- 172.21.15.1:0/2362506968 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b801e070 con 0x55bfc74972c0 2024-04-26T21:27:10.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.941+0000 7fd8bd200640 1 -- 172.21.15.1:0/2362506968 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b800fa70 con 0x55bfc74972c0 2024-04-26T21:27:10.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.941+0000 7fd8bd200640 1 -- 172.21.15.1:0/2362506968 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8019b00 con 0x55bfc74972c0 2024-04-26T21:27:10.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.941+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2362506968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 msgr2=0x55bfc74976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.941+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/2362506968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc74976e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8011570 tx=0x7fd8b8011a30 comp rx=0 tx=0).stop 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2362506968 shutdown_connections 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/2362506968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 0x55bfc749c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/2362506968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc749f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/2362506968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc74976e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2362506968 >> 172.21.15.1:0/2362506968 conn(0x55bfc742bd10 msgr2=0x55bfc7492d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2362506968 shutdown_connections 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2362506968 wait complete. 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8c4c00640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd8ac0a3ab0 0x7fd8ac0a5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8c4c00640 1 -- 172.21.15.1:0/3312267474 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd8b00f0d70 con 0x7fd8c8105b30 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8c7e00640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd8ac0a3ab0 0x7fd8ac0a5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8d15fd740 1 -- start start 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.942+0000 7fd8d15fd740 1 Processor -- start 2024-04-26T21:27:10.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc74a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8c7e00640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd8ac0a3ab0 0x7fd8ac0a5f70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55bfc7493bf0 tx=0x7fd8c0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc7497cb0 0x55bfc74a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc749ce20 0x55bfc74a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55bfc749fdd0 con 0x55bfc7497cb0 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bfc7382f70 con 0x55bfc74972c0 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bfc749ff50 con 0x55bfc749ce20 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8d15fd740 10 client.0 _ll_register_callbacks cb 0x55bfc73f1480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8ccc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc749ce20 0x55bfc74a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8c7400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc7497cb0 0x55bfc74a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8ccc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc749ce20 0x55bfc74a8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44896/0 (socket says 172.21.15.1:44896) 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8ccc00640 1 -- 172.21.15.1:0/2502197583 learned_addr learned my addr 172.21.15.1:0/2502197583 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.943+0000 7fd8c7e00640 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc74a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.944+0000 7fd8ccc00640 1 -- 172.21.15.1:0/2502197583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 msgr2=0x55bfc74a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:27:10.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.944+0000 7fd8ccc00640 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc74a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.944+0000 7fd8ccc00640 1 -- 172.21.15.1:0/2502197583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc7497cb0 msgr2=0x55bfc74a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.944+0000 7fd8ccc00640 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc7497cb0 0x55bfc74a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.944+0000 7fd8ccc00640 1 -- 172.21.15.1:0/2502197583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfc71d76b0 con 0x55bfc749ce20 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.944+0000 7fd8ccc00640 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc749ce20 0x55bfc74a8d70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8000c00 tx=0x7fd8b800f870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16253e40f59f9a28 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.944+0000 7fd8b5600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.944+0000 7fd8b5600640 1 -- 172.21.15.1:0/2502197583 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b801e040 con 0x55bfc749ce20 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.944+0000 7fd8b5600640 1 -- 172.21.15.1:0/2502197583 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd8b80198a0 con 0x55bfc749ce20 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.944+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2502197583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b800fa70 con 0x55bfc749ce20 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.945+0000 7fd8b5600640 1 -- 172.21.15.1:0/2502197583 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fd8b8030050 con 0x55bfc749ce20 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.945+0000 7fd8d15fd740 10 client.24700 fetch_fsmap learned FSMap version 16 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.945+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2502197583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x55bfc74abd30 con 0x55bfc749ce20 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.945+0000 7fd8b5600640 1 -- 172.21.15.1:0/2502197583 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b800fa70 con 0x55bfc749ce20 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.945+0000 7fd8b5600640 1 -- 172.21.15.1:0/2502197583 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7fd8b8021070 con 0x55bfc749ce20 2024-04-26T21:27:10.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.945+0000 7fd8d15fd740 10 client.24700 fetch_fsmap finished waiting for FSMap version 16 2024-04-26T21:27:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.945+0000 7fd8d15fd740 10 client.24700 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T21:27:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.945+0000 7fd8d15fd740 20 client.24700 populate_metadata read hostname 'smithi001' 2024-04-26T21:27:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.946+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bfc74ae1b0 0x55bfc74af620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.946+0000 7fd8d15fd740 4 client.24700 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:27:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.946+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2502197583 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfc7409210 con 0x55bfc74ae1b0 2024-04-26T21:27:10.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.946+0000 7fd8c7e00640 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bfc74ae1b0 0x55bfc74af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.946+0000 7fd8c7e00640 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bfc74ae1b0 0x55bfc74af620 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b7da8125f54bcc2 server_cookie=310263efe3d568f2 in_seq=0 out_seq=0 2024-04-26T21:27:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.946+0000 7fd8b5600640 10 client.24700 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:27:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.947+0000 7fd8b5600640 1 -- 172.21.15.1:0/2502197583 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bfc7409210 con 0x55bfc74ae1b0 2024-04-26T21:27:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.947+0000 7fd8b5600640 10 client.24700 handle_command_reply: tid=0 2024-04-26T21:27:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.947+0000 7fd8d15fd740 1 client.24700 shutdown 2024-04-26T21:27:10.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.947+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2502197583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc749ce20 msgr2=0x55bfc74a8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.947+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc749ce20 0x55bfc74a8d70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8000c00 tx=0x7fd8b800f870 comp rx=0 tx=0).stop 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.948+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2502197583 shutdown_connections 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.948+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bfc74ae1b0 0x55bfc74af620 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.948+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc749ce20 0x55bfc74a8d70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.948+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc7497cb0 0x55bfc74a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.948+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/2502197583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc74a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.948+0000 7fd8b5600640 0 client.24700 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.948+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2502197583 >> 172.21.15.1:0/2502197583 conn(0x55bfc742bd10 msgr2=0x55bfc7491f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.948+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2502197583 shutdown_connections 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.948+0000 7fd8d15fd740 1 -- 172.21.15.1:0/2502197583 wait complete. 2024-04-26T21:27:10.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.948+0000 7fd8d15fd740 20 client.24700 trim_cache size 0 max 16384 2024-04-26T21:27:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.968+0000 7fd8d15fd740 1 Processor -- start 2024-04-26T21:27:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.968+0000 7fd8d15fd740 1 -- start start 2024-04-26T21:27:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.968+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc74972c0 0x55bfc7498750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.968+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc7497cb0 0x55bfc74a8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.968+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc74927c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.968+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55bfc74ad960 con 0x55bfc749ce20 2024-04-26T21:27:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.968+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bfc74ae030 con 0x55bfc7497cb0 2024-04-26T21:27:10.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.968+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bfc74ae700 con 0x55bfc74972c0 2024-04-26T21:27:10.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8c7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc74972c0 0x55bfc7498750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8c7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc74972c0 0x55bfc7498750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44902/0 (socket says 172.21.15.1:44902) 2024-04-26T21:27:10.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8c7e00640 1 -- 172.21.15.1:0/916212365 learned_addr learned my addr 172.21.15.1:0/916212365 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:10.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8c7400640 1 --2- 172.21.15.1:0/916212365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc7497cb0 0x55bfc74a8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8ccc00640 1 --2- 172.21.15.1:0/916212365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc74927c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8c7400640 1 -- 172.21.15.1:0/916212365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc74972c0 msgr2=0x55bfc7498750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8c7400640 1 --2- 172.21.15.1:0/916212365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc74972c0 0x55bfc7498750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8c7400640 1 -- 172.21.15.1:0/916212365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 msgr2=0x55bfc74927c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8c7400640 1 --2- 172.21.15.1:0/916212365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc74927c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8c7400640 1 -- 172.21.15.1:0/916212365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfc74a0c00 con 0x55bfc7497cb0 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.969+0000 7fd8ccc00640 1 --2- 172.21.15.1:0/916212365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc74927c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8c7400640 1 --2- 172.21.15.1:0/916212365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc7497cb0 0x55bfc74a8c90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00bcd40 tx=0x7fd8b0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=434cf53055b7bb81 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8b7e00640 1 -- 172.21.15.1:0/916212365 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0049050 con 0x55bfc7497cb0 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8b7e00640 1 -- 172.21.15.1:0/916212365 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b00bd4b0 con 0x55bfc7497cb0 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8b7e00640 1 -- 172.21.15.1:0/916212365 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b00b9420 con 0x55bfc7497cb0 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8d15fd740 1 -- 172.21.15.1:0/916212365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc7497cb0 msgr2=0x55bfc74a8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/916212365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc7497cb0 0x55bfc74a8c90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00bcd40 tx=0x7fd8b0009260 comp rx=0 tx=0).stop 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8d15fd740 1 -- 172.21.15.1:0/916212365 shutdown_connections 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/916212365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc74927c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/916212365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc7497cb0 0x55bfc74a8c90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/916212365 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc74972c0 0x55bfc7498750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.970+0000 7fd8d15fd740 1 -- 172.21.15.1:0/916212365 >> 172.21.15.1:0/916212365 conn(0x55bfc742bd10 msgr2=0x55bfc74ad130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.971+0000 7fd8d15fd740 1 -- 172.21.15.1:0/916212365 shutdown_connections 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.971+0000 7fd8d15fd740 1 -- 172.21.15.1:0/916212365 wait complete. 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.971+0000 7fd8d15fd740 1 -- start start 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.971+0000 7fd8d15fd740 1 Processor -- start 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc749b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 0x55bfc749b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8d15fd740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc7512150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55bfc74af4a0 con 0x55bfc749ce20 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bfc74aedd0 con 0x55bfc74972c0 2024-04-26T21:27:10.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8d15fd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bfc74afb70 con 0x55bfc7497cb0 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8d15fd740 10 client.0 _ll_register_callbacks cb 0x7fd8b801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8c7400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 0x55bfc749b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8ccc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc7512150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8c7400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 0x55bfc749b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44912/0 (socket says 172.21.15.1:44912) 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8c7400640 1 -- 172.21.15.1:0/1553662402 learned_addr learned my addr 172.21.15.1:0/1553662402 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8c7e00640 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc749b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8c7400640 1 -- 172.21.15.1:0/1553662402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 msgr2=0x55bfc749b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8c7400640 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc749b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8c7400640 1 -- 172.21.15.1:0/1553662402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 msgr2=0x55bfc7512150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8c7400640 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc7512150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.972+0000 7fd8c7400640 1 -- 172.21.15.1:0/1553662402 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfc7512760 con 0x55bfc7497cb0 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.973+0000 7fd8c7400640 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 0x55bfc749b870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0005e90 tx=0x7fd8b002a400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22d49355d8d7fe3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:10.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.973+0000 7fd8b5600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.973+0000 7fd8b5600640 1 -- 172.21.15.1:0/1553662402 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b00c5a20 con 0x55bfc7497cb0 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.973+0000 7fd8b5600640 1 -- 172.21.15.1:0/1553662402 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd8b00b9420 con 0x55bfc7497cb0 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.973+0000 7fd8d15fd740 1 -- 172.21.15.1:0/1553662402 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8b00bd4b0 con 0x55bfc7497cb0 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.973+0000 7fd8b5600640 1 -- 172.21.15.1:0/1553662402 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fd8b0049070 con 0x55bfc7497cb0 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.973+0000 7fd8d15fd740 10 client.24706 fetch_fsmap learned FSMap version 16 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.973+0000 7fd8d15fd740 1 -- 172.21.15.1:0/1553662402 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x55bfc7513a60 con 0x55bfc7497cb0 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.973+0000 7fd8b5600640 1 -- 172.21.15.1:0/1553662402 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8b00bd4b0 con 0x55bfc7497cb0 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.974+0000 7fd8b5600640 1 -- 172.21.15.1:0/1553662402 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7fd8b00dd0b0 con 0x55bfc7497cb0 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.974+0000 7fd8d15fd740 10 client.24706 fetch_fsmap finished waiting for FSMap version 16 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.974+0000 7fd8d15fd740 10 client.24706 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.974+0000 7fd8d15fd740 20 client.24706 populate_metadata read hostname 'smithi001' 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.974+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bfc7518420 0x55bfc7517130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.974+0000 7fd8d15fd740 4 client.24706 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.974+0000 7fd8d15fd740 1 -- 172.21.15.1:0/1553662402 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "status"}) -- 0x55bfc7403080 con 0x55bfc7518420 2024-04-26T21:27:10.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.974+0000 7fd8c7e00640 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bfc7518420 0x55bfc7517130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8c7e00640 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bfc7518420 0x55bfc7517130 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68cc1f20bef054a2 server_cookie=5c7bc1d40ff72a64 in_seq=0 out_seq=0 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8b5600640 10 client.24706 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8b5600640 1 -- 172.21.15.1:0/1553662402 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55bfc7403080 con 0x55bfc7518420 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8b5600640 10 client.24706 handle_command_reply: tid=0 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 client.24706 shutdown 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 -- 172.21.15.1:0/1553662402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 msgr2=0x55bfc749b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 0x55bfc749b870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0005e90 tx=0x7fd8b002a400 comp rx=0 tx=0).stop 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 -- 172.21.15.1:0/1553662402 shutdown_connections 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bfc7518420 0x55bfc7517130 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bfc7497cb0 0x55bfc749b870 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8b5600640 0 client.24706 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:27:10.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bfc749ce20 0x55bfc7512150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 --2- 172.21.15.1:0/1553662402 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bfc74972c0 0x55bfc749b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 -- 172.21.15.1:0/1553662402 >> 172.21.15.1:0/1553662402 conn(0x55bfc742bd10 msgr2=0x55bfc74927c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:10.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 -- 172.21.15.1:0/1553662402 shutdown_connections 2024-04-26T21:27:10.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 1 -- 172.21.15.1:0/1553662402 wait complete. 2024-04-26T21:27:10.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.975+0000 7fd8d15fd740 20 client.24706 trim_cache size 0 max 16384 2024-04-26T21:27:10.982 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:27:10.982 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:27:10.982 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-04-26T21:27:10.982 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24602, 2024-04-26T21:27:10.983 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T21:27:10.983 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T21:27:10.983 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T21:27:10.983 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 7.6812192340000003, 2024-04-26T21:27:10.983 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 16, 2024-04-26T21:27:10.983 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 91, 2024-04-26T21:27:10.983 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T21:27:10.983 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 14.71552273 2024-04-26T21:27:10.983 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 -- 172.21.15.1:0/3312267474 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd8ac0a3ab0 msgr2=0x7fd8ac0a5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd8ac0a3ab0 0x7fd8ac0a5f70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55bfc7493bf0 tx=0x7fd8c0009500 comp rx=0 tx=0).stop 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 -- 172.21.15.1:0/3312267474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8105b30 msgr2=0x7fd8c819b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8105b30 0x7fd8c819b0e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0031500 tx=0x7fd8b0003c10 comp rx=0 tx=0).stop 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 -- 172.21.15.1:0/3312267474 shutdown_connections 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd8ac0a3ab0 0x7fd8ac0a5f70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8c8108460 0x7fd8c81a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8105b30 0x7fd8c819b0e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 --2- 172.21.15.1:0/3312267474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8c8103200 0x7fd8c819aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 -- 172.21.15.1:0/3312267474 >> 172.21.15.1:0/3312267474 conn(0x7fd8c80fcc10 msgr2=0x7fd8c8102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 -- 172.21.15.1:0/3312267474 shutdown_connections 2024-04-26T21:27:10.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:10.976+0000 7fd8cd600640 1 -- 172.21.15.1:0/3312267474 wait complete. 2024-04-26T21:27:10.998 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T21:27:10.998 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T21:27:10.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T21:27:11.461 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:11.609 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:11 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4286311946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:11.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:11 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4286311946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:11.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:11 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4286311946' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:12.650 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:12 smithi001 ceph-mon[28054]: pgmap v329: 129 pgs: 2 creating+activating, 6 creating+peering, 121 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 6 op/s 2024-04-26T21:27:12.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:12 smithi186 ceph-mon[32991]: pgmap v329: 129 pgs: 2 creating+activating, 6 creating+peering, 121 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 6 op/s 2024-04-26T21:27:12.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:12 smithi038 ceph-mon[32724]: pgmap v329: 129 pgs: 2 creating+activating, 6 creating+peering, 121 active+clean; 579 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 6 op/s 2024-04-26T21:27:13.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.845+0000 7fa465600700 1 -- 172.21.15.1:0/4045690946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4601145f0 msgr2=0x7fa4601169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.845+0000 7fa465600700 1 --2- 172.21.15.1:0/4045690946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4601145f0 0x7fa4601169e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa458009f90 tx=0x7fa45800b330 comp rx=0 tx=0).stop 2024-04-26T21:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.845+0000 7fa465600700 1 -- 172.21.15.1:0/4045690946 shutdown_connections 2024-04-26T21:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.845+0000 7fa465600700 1 --2- 172.21.15.1:0/4045690946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4601145f0 0x7fa4601169e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.845+0000 7fa465600700 1 --2- 172.21.15.1:0/4045690946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa460075650 0x7fa460075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.845+0000 7fa465600700 1 --2- 172.21.15.1:0/4045690946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa460075f70 0x7fa4601140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.845+0000 7fa465600700 1 -- 172.21.15.1:0/4045690946 >> 172.21.15.1:0/4045690946 conn(0x7fa46007a580 msgr2=0x7fa46007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.845+0000 7fa465600700 1 -- 172.21.15.1:0/4045690946 shutdown_connections 2024-04-26T21:27:13.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.846+0000 7fa465600700 1 -- 172.21.15.1:0/4045690946 wait complete. 2024-04-26T21:27:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.847+0000 7fa465600700 1 Processor -- start 2024-04-26T21:27:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.847+0000 7fa465600700 1 -- start start 2024-04-26T21:27:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.847+0000 7fa465600700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa460075650 0x7fa4601a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:13.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa465600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa460075f70 0x7fa4601a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa465600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4601145f0 0x7fa4601a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa465600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa460119c10 con 0x7fa4601145f0 2024-04-26T21:27:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa465600700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa460119a90 con 0x7fa460075f70 2024-04-26T21:27:13.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa465600700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa460119d90 con 0x7fa460075650 2024-04-26T21:27:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa45ea00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa460075650 0x7fa4601a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa45ea00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa460075650 0x7fa4601a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44924/0 (socket says 172.21.15.1:44924) 2024-04-26T21:27:13.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa45ea00700 1 -- 172.21.15.1:0/3420461576 learned_addr learned my addr 172.21.15.1:0/3420461576 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa45e000700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa460075f70 0x7fa4601a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa45f400700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4601145f0 0x7fa4601a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa45ea00700 1 -- 172.21.15.1:0/3420461576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa460075f70 msgr2=0x7fa4601a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa45ea00700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa460075f70 0x7fa4601a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa45ea00700 1 -- 172.21.15.1:0/3420461576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4601145f0 msgr2=0x7fa4601a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa45ea00700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4601145f0 0x7fa4601a4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.848+0000 7fa45ea00700 1 -- 172.21.15.1:0/3420461576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4601a51b0 con 0x7fa460075650 2024-04-26T21:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.849+0000 7fa45ea00700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa460075650 0x7fa4601a01a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa45000ec10 tx=0x7fa45000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:13.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.849+0000 7fa457400700 1 -- 172.21.15.1:0/3420461576 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45000cc90 con 0x7fa460075650 2024-04-26T21:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.849+0000 7fa457400700 1 -- 172.21.15.1:0/3420461576 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45000ce30 con 0x7fa460075650 2024-04-26T21:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.849+0000 7fa457400700 1 -- 172.21.15.1:0/3420461576 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa450010430 con 0x7fa460075650 2024-04-26T21:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.849+0000 7fa465600700 1 -- 172.21.15.1:0/3420461576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4601a54a0 con 0x7fa460075650 2024-04-26T21:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.849+0000 7fa465600700 1 -- 172.21.15.1:0/3420461576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4601acd20 con 0x7fa460075650 2024-04-26T21:27:13.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.850+0000 7fa465600700 1 -- 172.21.15.1:0/3420461576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa46006a110 con 0x7fa460075650 2024-04-26T21:27:13.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.852+0000 7fa457400700 1 -- 172.21.15.1:0/3420461576 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fa450004000 con 0x7fa460075650 2024-04-26T21:27:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.854+0000 7fa457400700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa4440a3cc0 0x7fa4440a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.854+0000 7fa457400700 1 -- 172.21.15.1:0/3420461576 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fa4500d08a0 con 0x7fa460075650 2024-04-26T21:27:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.854+0000 7fa45e000700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa4440a3cc0 0x7fa4440a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:13.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.855+0000 7fa45e000700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa4440a3cc0 0x7fa4440a6180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009680 tx=0x7fa44c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:13.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:13.858+0000 7fa457400700 1 -- 172.21.15.1:0/3420461576 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa45009cfe0 con 0x7fa460075650 2024-04-26T21:27:14.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.139+0000 7fa465600700 1 -- 172.21.15.1:0/3420461576 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa4601acf20 con 0x7fa460075650 2024-04-26T21:27:14.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.140+0000 7fa457400700 1 -- 172.21.15.1:0/3420461576 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7fa45009c8e0 con 0x7fa460075650 2024-04-26T21:27:14.142 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T21:27:14.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 -- 172.21.15.1:0/3420461576 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa4440a3cc0 msgr2=0x7fa4440a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:14.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa4440a3cc0 0x7fa4440a6180 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009680 tx=0x7fa44c009260 comp rx=0 tx=0).stop 2024-04-26T21:27:14.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 -- 172.21.15.1:0/3420461576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa460075650 msgr2=0x7fa4601a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:14.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa460075650 0x7fa4601a01a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa45000ec10 tx=0x7fa45000ef20 comp rx=0 tx=0).stop 2024-04-26T21:27:14.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 -- 172.21.15.1:0/3420461576 shutdown_connections 2024-04-26T21:27:14.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa4440a3cc0 0x7fa4440a6180 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:14.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa460075650 0x7fa4601a01a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:14.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4601145f0 0x7fa4601a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 --2- 172.21.15.1:0/3420461576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa460075f70 0x7fa4601a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 -- 172.21.15.1:0/3420461576 >> 172.21.15.1:0/3420461576 conn(0x7fa46007a580 msgr2=0x7fa460113420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.143+0000 7fa465600700 1 -- 172.21.15.1:0/3420461576 shutdown_connections 2024-04-26T21:27:14.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:14.144+0000 7fa465600700 1 -- 172.21.15.1:0/3420461576 wait complete. 2024-04-26T21:27:14.542 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:14 smithi001 ceph-mon[28054]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-26T21:27:14.542 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:14 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3420461576' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T21:27:14.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:14 smithi186 ceph-mon[32991]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-26T21:27:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:14 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3420461576' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T21:27:14.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:14 smithi038 ceph-mon[32724]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-26T21:27:14.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:14 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3420461576' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T21:27:15.075 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-26T21:27:15.075 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T21:27:15.075 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-04-26T21:27:15.075 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-26T21:27:15.075 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-26T21:27:15.075 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-26T21:27:15.076 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T21:27:15.076 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T21:27:15.076 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T21:27:15.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:15.076 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-04-26T21:27:15.134 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-04-26T21:27:15.135 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T21:27:15.136 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-04-26T21:27:15.136 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-04-26T21:27:15.136 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T21:27:15.136 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42077sec preferred_lft 42077sec 2024-04-26T21:27:15.136 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-04-26T21:27:15.136 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-04-26T21:27:15.136 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T21:27:15.136 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-04-26T21:27:15.136 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-04-26T21:27:15.137 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T21:27:15.138 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:27:15.138 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T21:27:15.138 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add name ceph-brx type bridge 2024-04-26T21:27:15.138 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr flush dev ceph-brx 2024-04-26T21:27:15.138 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set ceph-brx up 2024-04-26T21:27:15.138 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T21:27:15.138 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T21:27:15.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:27:15.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:27:15.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:15.637 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T21:27:15.704 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-04-26T21:27:15.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:15.707 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-04-26T21:27:15.765 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-04-26T21:27:15.765 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T21:27:15.765 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T21:27:15.765 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T21:27:15.765 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T21:27:15.765 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:27:15.765 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T21:27:15.766 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T21:27:15.766 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T21:27:15.766 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T21:27:15.766 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T21:27:15.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:27:16.506 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:16 smithi001 ceph-mon[28054]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2024-04-26T21:27:16.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:27:16.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:16.520 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-04-26T21:27:16.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:16.576 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-04-26T21:27:16.629 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:16 smithi038 ceph-mon[32724]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2024-04-26T21:27:16.631 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:27:16.631 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T21:27:16.631 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T21:27:16.631 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-26T21:27:16.631 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T21:27:16.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:16 smithi186 ceph-mon[32991]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 895 B/s wr, 0 op/s 2024-04-26T21:27:16.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:27:16.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:27:16.826 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-26T21:27:16.826 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:27:16.826 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T21:27:16.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-26T21:27:16.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T21:27:16.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-26T21:27:16.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-26T21:27:16.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-26T21:27:16.826 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T21:27:16.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:27:17.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:27:17.136 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:27:17.136 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T21:27:17.136 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-04-26T21:27:17.136 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T21:27:17.137 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T21:27:17.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:27:17.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:27:17.352 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-26T21:27:17.353 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T21:27:17.353 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-26T21:27:17.405 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-26T21:27:17.406 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T21:27:17.406 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-26T21:27:17.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-04-26T21:27:17.529 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T21:27:17.577 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-04-26T21:27:17.577 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-04-26T21:27:17.577 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-04-26T21:27:17.577 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-04-26T21:27:17.577 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-04-26T21:27:17.578 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/23ca000bacdb02bc16445ee95eb5cb137898dbcd085e8244ddc13996195a03d0/merged 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/1398c84e4ba30ac8c2f5a53eabc28a032e8b86fbab1fbb8c79f273c6d97fb2cc/merged 2024-04-26T21:27:17.579 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/0 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/e4da9549b7faee507a9bab196ef872bb27da1cc12b6f366d99522149dac92c27/merged 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/dab89c5e262b80a7a157d0db5a6cff446e394c7213aeaafd5e003d35aef830e7/merged 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/975c8050cdda4d6ef3b395cda43ecbe69e8cd4fc81f40ab1eafed37787df3d5d/merged 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/85f51e31017364135e18435448556369129a02f74b3a92923de4427ba15ee917/merged 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/3d2513c65ae2686a0ce831873e070b519171831531b09b82ee215117cb03dbb2/merged 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/b2f3b375ea2f1a175663bd2f288195cb58875fe77ba8b58e20ec147fcbe34a16/merged 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/8832fdcf6eccc6481a5adffc5a712500790cd6a4221c71adc961254a5882ff49/merged 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/e45a3aff48bad92913610423f54d4deb1bb5819d0b29f5f640cddcd6c6c91009/merged 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T21:27:17.580 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T21:27:17.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:17.581 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-04-26T21:27:17.625 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-26T21:27:17.625 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-26T21:27:17.673 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-04-26T21:27:17.747 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-04-26T21:27:17.789 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-04-26T21:27:17.790 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/23ca000bacdb02bc16445ee95eb5cb137898dbcd085e8244ddc13996195a03d0/merged 2024-04-26T21:27:17.791 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/1398c84e4ba30ac8c2f5a53eabc28a032e8b86fbab1fbb8c79f273c6d97fb2cc/merged 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/0 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/e4da9549b7faee507a9bab196ef872bb27da1cc12b6f366d99522149dac92c27/merged 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/dab89c5e262b80a7a157d0db5a6cff446e394c7213aeaafd5e003d35aef830e7/merged 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/975c8050cdda4d6ef3b395cda43ecbe69e8cd4fc81f40ab1eafed37787df3d5d/merged 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/85f51e31017364135e18435448556369129a02f74b3a92923de4427ba15ee917/merged 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/3d2513c65ae2686a0ce831873e070b519171831531b09b82ee215117cb03dbb2/merged 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/b2f3b375ea2f1a175663bd2f288195cb58875fe77ba8b58e20ec147fcbe34a16/merged 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/8832fdcf6eccc6481a5adffc5a712500790cd6a4221c71adc961254a5882ff49/merged 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/containers/storage/overlay/e45a3aff48bad92913610423f54d4deb1bb5819d0b29f5f640cddcd6c6c91009/merged 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T21:27:17.792 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T21:27:17.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:17.793 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-04-26T21:27:17.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:2024-04-26T21:27:17.864+0000 7f82b3462480 -1 init, newargv = 0x5595fd681290 newargc=15 2024-04-26T21:27:17.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:ceph-fuse[69385]: starting ceph client 2024-04-26T21:27:17.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:ceph-fuse[69385]: starting fuse 2024-04-26T21:27:17.882 INFO:teuthology.orchestra.run.smithi001.stdout:114 2024-04-26T21:27:17.883 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-26T21:27:17.883 DEBUG:teuthology.orchestra.run.smithi001:> 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-26T21:27:17.884 DEBUG:teuthology.orchestra.run.smithi001:> 2024-04-26T21:27:17.885 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.admin")) 2024-04-26T21:27:17.885 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-04-26T21:27:18.078 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.admin.69385.asok 2024-04-26T21:27:18.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:27:18.095 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.69385.asok 2024-04-26T21:27:18.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:18.095 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.69385.asok status 2024-04-26T21:27:18.324 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:27:18.324 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-04-26T21:27:18.324 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "bc612ffb9c1d5bc27536bc917a705262a36d4387", 2024-04-26T21:27:18.324 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-3306-gbc612ffb (bc612ffb9c1d5bc27536bc917a705262a36d4387) squid (dev)", 2024-04-26T21:27:18.324 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "admin", 2024-04-26T21:27:18.324 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-04-26T21:27:18.324 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-26T21:27:18.324 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "69385", 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24728, 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 24728 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:0", 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2833590015 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:27:18.325 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:0", 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2833590015 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.24728 172.21.15.1:0/2833590015", 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "172.21.15.1:0/2833590015", 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 16, 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 91, 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-04-26T21:27:18.326 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:27:18.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:18.339 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-04-26T21:27:18.339 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-26T21:27:18.390 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-04-26T21:27:18.390 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-26T21:27:18.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:18.391 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-26T21:27:18.473 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-26T21:27:18.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T21:27:18.593 INFO:teuthology.orchestra.run.smithi001.stdout:# file: . 2024-04-26T21:27:18.593 INFO:teuthology.orchestra.run.smithi001.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-26T21:27:18.593 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:18.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:27:18.603 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:27:18.603 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T21:27:18.651 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:27:18.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:18 smithi186 ceph-mon[32991]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 719 B/s wr, 0 op/s 2024-04-26T21:27:18.764 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:18 smithi001 ceph-mon[28054]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 719 B/s wr, 0 op/s 2024-04-26T21:27:18.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:18 smithi038 ceph-mon[32724]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 719 B/s wr, 0 op/s 2024-04-26T21:27:18.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.907+0000 7f70a0c00640 1 -- 172.21.15.1:0/1153344395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c108d90 msgr2=0x7f709c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:18.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.907+0000 7f70a0c00640 1 --2- 172.21.15.1:0/1153344395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c108d90 0x7f709c1091b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f708c009900 tx=0x7f708c02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:18.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.907+0000 7f70a0c00640 1 -- 172.21.15.1:0/1153344395 shutdown_connections 2024-04-26T21:27:18.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.907+0000 7f70a0c00640 1 --2- 172.21.15.1:0/1153344395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c109f80 0x7f709c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:18.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.907+0000 7f70a0c00640 1 --2- 172.21.15.1:0/1153344395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c10a940 0x7f709c1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:18.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.907+0000 7f70a0c00640 1 --2- 172.21.15.1:0/1153344395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c108d90 0x7f709c1091b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:18.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.907+0000 7f70a0c00640 1 -- 172.21.15.1:0/1153344395 >> 172.21.15.1:0/1153344395 conn(0x7f709c077d30 msgr2=0x7f709c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:18.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.907+0000 7f70a0c00640 1 -- 172.21.15.1:0/1153344395 shutdown_connections 2024-04-26T21:27:18.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.907+0000 7f70a0c00640 1 -- 172.21.15.1:0/1153344395 wait complete. 2024-04-26T21:27:18.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f70a0c00640 1 Processor -- start 2024-04-26T21:27:18.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f70a0c00640 1 -- start start 2024-04-26T21:27:18.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f70a0c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c108d90 0x7f709c19eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f70a0c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c109f80 0x7f709c19f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f70a0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c10a940 0x7f709c1a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f70a0c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f709c1148b0 con 0x7f709c109f80 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f70a0c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f709c114730 con 0x7f709c108d90 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f70a0c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f709c114a30 con 0x7f709c10a940 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f7093e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c109f80 0x7f709c19f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f709be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c10a940 0x7f709c1a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.909+0000 7f7093e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c109f80 0x7f709c19f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39000/0 (socket says 172.21.15.1:39000) 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.910+0000 7f7093e00640 1 -- 172.21.15.1:0/1925376747 learned_addr learned my addr 172.21.15.1:0/1925376747 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.910+0000 7f709b400640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c108d90 0x7f709c19eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.910+0000 7f709be00640 1 -- 172.21.15.1:0/1925376747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c108d90 msgr2=0x7f709c19eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:18.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.910+0000 7f709be00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c108d90 0x7f709c19eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:18.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.910+0000 7f709be00640 1 -- 172.21.15.1:0/1925376747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c109f80 msgr2=0x7f709c19f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:18.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.910+0000 7f709be00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c109f80 0x7f709c19f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:18.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.910+0000 7f709be00640 1 -- 172.21.15.1:0/1925376747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709c1a6b10 con 0x7f709c10a940 2024-04-26T21:27:18.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.910+0000 7f7093e00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c109f80 0x7f709c19f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:18.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.910+0000 7f709be00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c10a940 0x7f709c1a64a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7094009fd0 tx=0x7f709400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:18.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.911+0000 7f7098c00640 1 -- 172.21.15.1:0/1925376747 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709400ce20 con 0x7f709c10a940 2024-04-26T21:27:18.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.911+0000 7f70a0c00640 1 -- 172.21.15.1:0/1925376747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709c1a6e00 con 0x7f709c10a940 2024-04-26T21:27:18.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.911+0000 7f7098c00640 1 -- 172.21.15.1:0/1925376747 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7094005020 con 0x7f709c10a940 2024-04-26T21:27:18.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.911+0000 7f7098c00640 1 -- 172.21.15.1:0/1925376747 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094010690 con 0x7f709c10a940 2024-04-26T21:27:18.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.911+0000 7f70a0c00640 1 -- 172.21.15.1:0/1925376747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709c1a72c0 con 0x7f709c10a940 2024-04-26T21:27:18.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.912+0000 7f70a0c00640 1 -- 172.21.15.1:0/1925376747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f709c1100b0 con 0x7f709c10a940 2024-04-26T21:27:18.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.914+0000 7f7098c00640 1 -- 172.21.15.1:0/1925376747 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f70940054d0 con 0x7f709c10a940 2024-04-26T21:27:18.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.916+0000 7f7098c00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70800a3c50 0x7f70800a6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:18.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.916+0000 7f7098c00640 1 -- 172.21.15.1:0/1925376747 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f70940d0cf0 con 0x7f709c10a940 2024-04-26T21:27:18.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.916+0000 7f709b400640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70800a3c50 0x7f70800a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:18.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.917+0000 7f709b400640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70800a3c50 0x7f70800a6110 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f708c009230 tx=0x7f708c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:18.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:18.918+0000 7f7098c00640 1 -- 172.21.15.1:0/1925376747 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f709409d430 con 0x7f709c10a940 2024-04-26T21:27:19.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.127+0000 7f70a0c00640 1 -- 172.21.15.1:0/1925376747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f709c02dc50 con 0x7f709c10a940 2024-04-26T21:27:19.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.129+0000 7f7098c00640 1 -- 172.21.15.1:0/1925376747 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23972 (secure 0 0 0) 0x7f709409cd30 con 0x7f709c10a940 2024-04-26T21:27:19.131 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:19.131 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:27:07.816919+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:27:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.132+0000 7f70a0c00640 1 -- 172.21.15.1:0/1925376747 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70800a3c50 msgr2=0x7f70800a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.132+0000 7f70a0c00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70800a3c50 0x7f70800a6110 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f708c009230 tx=0x7f708c0314a0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.132+0000 7f70a0c00640 1 -- 172.21.15.1:0/1925376747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c10a940 msgr2=0x7f709c1a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.132+0000 7f70a0c00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c10a940 0x7f709c1a64a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7094009fd0 tx=0x7f709400ef10 comp rx=0 tx=0).stop 2024-04-26T21:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.132+0000 7f70a0c00640 1 -- 172.21.15.1:0/1925376747 shutdown_connections 2024-04-26T21:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.132+0000 7f70a0c00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70800a3c50 0x7f70800a6110 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.132+0000 7f70a0c00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c10a940 0x7f709c1a64a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.132+0000 7f70a0c00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c109f80 0x7f709c19f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.132+0000 7f70a0c00640 1 --2- 172.21.15.1:0/1925376747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c108d90 0x7f709c19eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.132+0000 7f70a0c00640 1 -- 172.21.15.1:0/1925376747 >> 172.21.15.1:0/1925376747 conn(0x7f709c077d30 msgr2=0x7f709c10d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.133+0000 7f70a0c00640 1 -- 172.21.15.1:0/1925376747 shutdown_connections 2024-04-26T21:27:19.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.133+0000 7f70a0c00640 1 -- 172.21.15.1:0/1925376747 wait complete. 2024-04-26T21:27:19.154 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:27:00.381106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:27:01.743898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T21:27:05.341216+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T21:27:19.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:27:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.361+0000 7fe3d5000640 1 -- 172.21.15.1:0/1205559782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3d010a960 msgr2=0x7fe3d01111f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.361+0000 7fe3d5000640 1 --2- 172.21.15.1:0/1205559782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3d010a960 0x7fe3d01111f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009ca0 tx=0x7fe3c802a8c0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.361+0000 7fe3d5000640 1 -- 172.21.15.1:0/1205559782 shutdown_connections 2024-04-26T21:27:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.361+0000 7fe3d5000640 1 --2- 172.21.15.1:0/1205559782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d0109fa0 0x7fe3d010a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.361+0000 7fe3d5000640 1 --2- 172.21.15.1:0/1205559782 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3d010a960 0x7fe3d01111f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.361+0000 7fe3d5000640 1 --2- 172.21.15.1:0/1205559782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3d0108db0 0x7fe3d01091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.361+0000 7fe3d5000640 1 -- 172.21.15.1:0/1205559782 >> 172.21.15.1:0/1205559782 conn(0x7fe3d0077d70 msgr2=0x7fe3d007a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.361+0000 7fe3d5000640 1 -- 172.21.15.1:0/1205559782 shutdown_connections 2024-04-26T21:27:19.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.361+0000 7fe3d5000640 1 -- 172.21.15.1:0/1205559782 wait complete. 2024-04-26T21:27:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.362+0000 7fe3d5000640 1 Processor -- start 2024-04-26T21:27:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.362+0000 7fe3d5000640 1 -- start start 2024-04-26T21:27:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3d5000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d0108db0 0x7fe3d019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3d5000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3d0109fa0 0x7fe3d019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3d5000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3d010a960 0x7fe3d01a6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:19.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3d5000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d0114910 con 0x7fe3d010a960 2024-04-26T21:27:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3d5000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d0114790 con 0x7fe3d0109fa0 2024-04-26T21:27:19.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3d5000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe3d0114a90 con 0x7fe3d0108db0 2024-04-26T21:27:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3cea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3d0109fa0 0x7fe3d019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3cf400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d0108db0 0x7fe3d019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3cfe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3d010a960 0x7fe3d01a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3cea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3d0109fa0 0x7fe3d019f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:41882/0 (socket says 172.21.15.1:41882) 2024-04-26T21:27:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3cea00640 1 -- 172.21.15.1:0/4133926445 learned_addr learned my addr 172.21.15.1:0/4133926445 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.363+0000 7fe3cf400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d0108db0 0x7fe3d019efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46572/0 (socket says 172.21.15.1:46572) 2024-04-26T21:27:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.364+0000 7fe3cea00640 1 -- 172.21.15.1:0/4133926445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d0108db0 msgr2=0x7fe3d019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.364+0000 7fe3cea00640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d0108db0 0x7fe3d019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.364+0000 7fe3cea00640 1 -- 172.21.15.1:0/4133926445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3d010a960 msgr2=0x7fe3d01a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.364+0000 7fe3cea00640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3d010a960 0x7fe3d01a6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.364+0000 7fe3cea00640 1 -- 172.21.15.1:0/4133926445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3d01a6bb0 con 0x7fe3d0109fa0 2024-04-26T21:27:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.364+0000 7fe3cf400640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d0108db0 0x7fe3d019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.364+0000 7fe3cea00640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3d0109fa0 0x7fe3d019f520 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00e9f0 tx=0x7fe3bc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.364+0000 7fe3c7e00640 1 -- 172.21.15.1:0/4133926445 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc00cd80 con 0x7fe3d0109fa0 2024-04-26T21:27:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.364+0000 7fe3c7e00640 1 -- 172.21.15.1:0/4133926445 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3bc0108e0 con 0x7fe3d0109fa0 2024-04-26T21:27:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.364+0000 7fe3d5000640 1 -- 172.21.15.1:0/4133926445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3d01a6ea0 con 0x7fe3d0109fa0 2024-04-26T21:27:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.365+0000 7fe3c7e00640 1 -- 172.21.15.1:0/4133926445 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc018980 con 0x7fe3d0109fa0 2024-04-26T21:27:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.365+0000 7fe3d5000640 1 -- 172.21.15.1:0/4133926445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3d01a73b0 con 0x7fe3d0109fa0 2024-04-26T21:27:19.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.366+0000 7fe3d5000640 1 -- 172.21.15.1:0/4133926445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3d01100d0 con 0x7fe3d0109fa0 2024-04-26T21:27:19.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.371+0000 7fe3c7e00640 1 -- 172.21.15.1:0/4133926445 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fe3bc010430 con 0x7fe3d0109fa0 2024-04-26T21:27:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.373+0000 7fe3c7e00640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe3b40a3c30 0x7fe3b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.373+0000 7fe3c7e00640 1 -- 172.21.15.1:0/4133926445 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fe3bc0d0d30 con 0x7fe3d0109fa0 2024-04-26T21:27:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.373+0000 7fe3cf400640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe3b40a3c30 0x7fe3b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.374+0000 7fe3c7e00640 1 -- 172.21.15.1:0/4133926445 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe3bc09ce20 con 0x7fe3d0109fa0 2024-04-26T21:27:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.374+0000 7fe3cf400640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe3b40a3c30 0x7fe3b40a60f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0109e00 tx=0x7fe3c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:19.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:19 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1925376747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:19.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.577+0000 7fe3d5000640 1 -- 172.21.15.1:0/4133926445 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe3d002dcc0 con 0x7fe3d0109fa0 2024-04-26T21:27:19.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.578+0000 7fe3c7e00640 1 -- 172.21.15.1:0/4133926445 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7fe3bc0a1120 con 0x7fe3d0109fa0 2024-04-26T21:27:19.580 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:27:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 -- 172.21.15.1:0/4133926445 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe3b40a3c30 msgr2=0x7fe3b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe3b40a3c30 0x7fe3b40a60f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0109e00 tx=0x7fe3c0005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 -- 172.21.15.1:0/4133926445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3d0109fa0 msgr2=0x7fe3d019f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3d0109fa0 0x7fe3d019f520 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00e9f0 tx=0x7fe3bc00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 -- 172.21.15.1:0/4133926445 shutdown_connections 2024-04-26T21:27:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe3b40a3c30 0x7fe3b40a60f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3d010a960 0x7fe3d01a6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3d0109fa0 0x7fe3d019f520 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 --2- 172.21.15.1:0/4133926445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3d0108db0 0x7fe3d019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 -- 172.21.15.1:0/4133926445 >> 172.21.15.1:0/4133926445 conn(0x7fe3d0077d70 msgr2=0x7fe3d010d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 -- 172.21.15.1:0/4133926445 shutdown_connections 2024-04-26T21:27:19.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.581+0000 7fe3d5000640 1 -- 172.21.15.1:0/4133926445 wait complete. 2024-04-26T21:27:19.601 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:27:19.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:19 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1925376747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:19.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.815+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2788095423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ac4104370 msgr2=0x7f8ac410ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:19.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.815+0000 7f8ac8c00640 1 --2- 172.21.15.1:0/2788095423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ac4104370 0x7f8ac410ac00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009be0 tx=0x7f8abc02a6c0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.815+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2788095423 shutdown_connections 2024-04-26T21:27:19.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.815+0000 7f8ac8c00640 1 --2- 172.21.15.1:0/2788095423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ac4104370 0x7f8ac410ac00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.815+0000 7f8ac8c00640 1 --2- 172.21.15.1:0/2788095423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ac41039b0 0x7f8ac4103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.815+0000 7f8ac8c00640 1 --2- 172.21.15.1:0/2788095423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ac41027c0 0x7f8ac4102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.815+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2788095423 >> 172.21.15.1:0/2788095423 conn(0x7f8ac40fdd30 msgr2=0x7f8ac4100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.815+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2788095423 shutdown_connections 2024-04-26T21:27:19.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.815+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2788095423 wait complete. 2024-04-26T21:27:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.816+0000 7f8ac8c00640 1 Processor -- start 2024-04-26T21:27:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.816+0000 7f8ac8c00640 1 -- start start 2024-04-26T21:27:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ac41027c0 0x7f8ac419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ac41039b0 0x7f8ac419b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ac4104370 0x7f8ac41a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac8c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac410e2d0 con 0x7f8ac41027c0 2024-04-26T21:27:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac8c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac410e150 con 0x7f8ac4104370 2024-04-26T21:27:19.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac8c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac410e450 con 0x7f8ac41039b0 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ac41039b0 0x7f8ac419b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ac4104370 0x7f8ac41a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ac41039b0 0x7f8ac419b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46594/0 (socket says 172.21.15.1:46594) 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac3400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ac41027c0 0x7f8ac419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.817+0000 7f8ac2a00640 1 -- 172.21.15.1:0/2054325697 learned_addr learned my addr 172.21.15.1:0/2054325697 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8ac2a00640 1 -- 172.21.15.1:0/2054325697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ac4104370 msgr2=0x7f8ac41a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8ac2a00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ac4104370 0x7f8ac41a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8ac2a00640 1 -- 172.21.15.1:0/2054325697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ac41027c0 msgr2=0x7f8ac419ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8ac2a00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ac41027c0 0x7f8ac419ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8ac2a00640 1 -- 172.21.15.1:0/2054325697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac41a27b0 con 0x7f8ac41039b0 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8ac3e00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ac4104370 0x7f8ac41a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8ac2a00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ac41039b0 0x7f8ac419b0d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00e9c0 tx=0x7f8aac00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:19.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8abbe00640 1 -- 172.21.15.1:0/2054325697 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac00cde0 con 0x7f8ac41039b0 2024-04-26T21:27:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8abbe00640 1 -- 172.21.15.1:0/2054325697 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8aac0035f0 con 0x7f8ac41039b0 2024-04-26T21:27:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2054325697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ac41a2aa0 con 0x7f8ac41039b0 2024-04-26T21:27:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8abbe00640 1 -- 172.21.15.1:0/2054325697 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac010690 con 0x7f8ac41039b0 2024-04-26T21:27:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.818+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2054325697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac41a2ff0 con 0x7f8ac41039b0 2024-04-26T21:27:19.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.820+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2054325697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ac41a3280 con 0x7f8ac41039b0 2024-04-26T21:27:19.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.822+0000 7f8abbe00640 1 -- 172.21.15.1:0/2054325697 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f8aac004000 con 0x7f8ac41039b0 2024-04-26T21:27:19.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.823+0000 7f8abbe00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8aa80a3bb0 0x7f8aa80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.823+0000 7f8abbe00640 1 -- 172.21.15.1:0/2054325697 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f8aac0d0990 con 0x7f8ac41039b0 2024-04-26T21:27:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.823+0000 7f8ac3400640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8aa80a3bb0 0x7f8aa80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:19.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.824+0000 7f8ac3400640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8aa80a3bb0 0x7f8aa80a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40095f0 tx=0x7f8ab4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:19.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:19 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1925376747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:19.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:19.827+0000 7f8abbe00640 1 -- 172.21.15.1:0/2054325697 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8aac009510 con 0x7f8ac41039b0 2024-04-26T21:27:20.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.033+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2054325697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8ac41a3450 con 0x7f8ac41039b0 2024-04-26T21:27:20.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.034+0000 7f8abbe00640 1 -- 172.21.15.1:0/2054325697 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f8aac09d0d0 con 0x7f8ac41039b0 2024-04-26T21:27:20.036 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:27:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2054325697 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8aa80a3bb0 msgr2=0x7f8aa80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8aa80a3bb0 0x7f8aa80a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40095f0 tx=0x7f8ab4005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2054325697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ac41039b0 msgr2=0x7f8ac419b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ac41039b0 0x7f8ac419b0d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00e9c0 tx=0x7f8aac00ee80 comp rx=0 tx=0).stop 2024-04-26T21:27:20.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2054325697 shutdown_connections 2024-04-26T21:27:20.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8aa80a3bb0 0x7f8aa80a6070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ac41039b0 0x7f8ac419b0d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ac4104370 0x7f8ac41a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 --2- 172.21.15.1:0/2054325697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ac41027c0 0x7f8ac419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2054325697 >> 172.21.15.1:0/2054325697 conn(0x7f8ac40fdd30 msgr2=0x7f8ac4109180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:20.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2054325697 shutdown_connections 2024-04-26T21:27:20.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.037+0000 7f8ac8c00640 1 -- 172.21.15.1:0/2054325697 wait complete. 2024-04-26T21:27:20.059 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:27:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.270+0000 7f3741800640 1 -- 172.21.15.1:0/3938116389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f373c10ab60 msgr2=0x7f373c1113f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.270+0000 7f3741800640 1 --2- 172.21.15.1:0/3938116389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f373c10ab60 0x7f373c1113f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3734009ca0 tx=0x7f373402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.270+0000 7f3741800640 1 -- 172.21.15.1:0/3938116389 shutdown_connections 2024-04-26T21:27:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.270+0000 7f3741800640 1 --2- 172.21.15.1:0/3938116389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c10a1a0 0x7f373c10a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.270+0000 7f3741800640 1 --2- 172.21.15.1:0/3938116389 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f373c10ab60 0x7f373c1113f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.270+0000 7f3741800640 1 --2- 172.21.15.1:0/3938116389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f373c108fb0 0x7f373c1093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.270+0000 7f3741800640 1 -- 172.21.15.1:0/3938116389 >> 172.21.15.1:0/3938116389 conn(0x7f373c077f80 msgr2=0x7f373c07a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.270+0000 7f3741800640 1 -- 172.21.15.1:0/3938116389 shutdown_connections 2024-04-26T21:27:20.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.270+0000 7f3741800640 1 -- 172.21.15.1:0/3938116389 wait complete. 2024-04-26T21:27:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.271+0000 7f3741800640 1 Processor -- start 2024-04-26T21:27:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.271+0000 7f3741800640 1 -- start start 2024-04-26T21:27:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.271+0000 7f3741800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c108fb0 0x7f373c19f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.271+0000 7f3741800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f373c10a1a0 0x7f373c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.271+0000 7f3741800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f373c10ab60 0x7f373c1a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.271+0000 7f3741800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f373c114af0 con 0x7f373c10a1a0 2024-04-26T21:27:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.271+0000 7f3741800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f373c114970 con 0x7f373c10ab60 2024-04-26T21:27:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.271+0000 7f3741800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f373c114c70 con 0x7f373c108fb0 2024-04-26T21:27:20.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f373c10ab60 0x7f373c1a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c108fb0 0x7f373c19f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f373c10ab60 0x7f373c1a6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:41934/0 (socket says 172.21.15.1:41934) 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c108fb0 0x7f373c19f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46616/0 (socket says 172.21.15.1:46616) 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373b400640 1 -- 172.21.15.1:0/4047194804 learned_addr learned my addr 172.21.15.1:0/4047194804 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373a000640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f373c10a1a0 0x7f373c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373aa00640 1 -- 172.21.15.1:0/4047194804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f373c10ab60 msgr2=0x7f373c1a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373aa00640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f373c10ab60 0x7f373c1a6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373aa00640 1 -- 172.21.15.1:0/4047194804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f373c10a1a0 msgr2=0x7f373c19f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373aa00640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f373c10a1a0 0x7f373c19f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.272+0000 7f373aa00640 1 -- 172.21.15.1:0/4047194804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373c1a6d70 con 0x7f373c108fb0 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.273+0000 7f373aa00640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c108fb0 0x7f373c19f150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f372c00d6a0 tx=0x7f372c00db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.273+0000 7f3733400640 1 -- 172.21.15.1:0/4047194804 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c017070 con 0x7f373c108fb0 2024-04-26T21:27:20.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.273+0000 7f3741800640 1 -- 172.21.15.1:0/4047194804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373c1a7060 con 0x7f373c108fb0 2024-04-26T21:27:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.273+0000 7f3733400640 1 -- 172.21.15.1:0/4047194804 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f372c009cd0 con 0x7f373c108fb0 2024-04-26T21:27:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.273+0000 7f3733400640 1 -- 172.21.15.1:0/4047194804 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c003b60 con 0x7f373c108fb0 2024-04-26T21:27:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.273+0000 7f3741800640 1 -- 172.21.15.1:0/4047194804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f373c1a7600 con 0x7f373c108fb0 2024-04-26T21:27:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.274+0000 7f3741800640 1 -- 172.21.15.1:0/4047194804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f373c1102d0 con 0x7f373c108fb0 2024-04-26T21:27:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.276+0000 7f3733400640 1 -- 172.21.15.1:0/4047194804 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f372c0042b0 con 0x7f373c108fb0 2024-04-26T21:27:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.278+0000 7f3733400640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f37200a3c30 0x7f37200a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.278+0000 7f3733400640 1 -- 172.21.15.1:0/4047194804 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f372c0cff50 con 0x7f373c108fb0 2024-04-26T21:27:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.279+0000 7f373a000640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f37200a3c30 0x7f37200a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.279+0000 7f373a000640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f37200a3c30 0x7f37200a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f373c1a0660 tx=0x7f372800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:20.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.282+0000 7f3733400640 1 -- 172.21.15.1:0/4047194804 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f372c09c040 con 0x7f373c108fb0 2024-04-26T21:27:20.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.491+0000 7f3741800640 1 -- 172.21.15.1:0/4047194804 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f373c063530 con 0x7f373c108fb0 2024-04-26T21:27:20.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.492+0000 7f3733400640 1 -- 172.21.15.1:0/4047194804 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f372c0a0340 con 0x7f373c108fb0 2024-04-26T21:27:20.493 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T21:27:20.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.495+0000 7f3741800640 1 -- 172.21.15.1:0/4047194804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f37200a3c30 msgr2=0x7f37200a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.495+0000 7f3741800640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f37200a3c30 0x7f37200a60f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f373c1a0660 tx=0x7f372800b390 comp rx=0 tx=0).stop 2024-04-26T21:27:20.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.496+0000 7f3741800640 1 -- 172.21.15.1:0/4047194804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c108fb0 msgr2=0x7f373c19f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.496+0000 7f3741800640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c108fb0 0x7f373c19f150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f372c00d6a0 tx=0x7f372c00db60 comp rx=0 tx=0).stop 2024-04-26T21:27:20.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.496+0000 7f3741800640 1 -- 172.21.15.1:0/4047194804 shutdown_connections 2024-04-26T21:27:20.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.496+0000 7f3741800640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f37200a3c30 0x7f37200a60f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.496+0000 7f3741800640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f373c10ab60 0x7f373c1a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.496+0000 7f3741800640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f373c10a1a0 0x7f373c19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.496+0000 7f3741800640 1 --2- 172.21.15.1:0/4047194804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c108fb0 0x7f373c19f150 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.496+0000 7f3741800640 1 -- 172.21.15.1:0/4047194804 >> 172.21.15.1:0/4047194804 conn(0x7f373c077f80 msgr2=0x7f373c10d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:20.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.496+0000 7f3741800640 1 -- 172.21.15.1:0/4047194804 shutdown_connections 2024-04-26T21:27:20.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.496+0000 7f3741800640 1 -- 172.21.15.1:0/4047194804 wait complete. 2024-04-26T21:27:20.515 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T21:27:20.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:20 smithi186 ceph-mon[32991]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 373 B/s wr, 0 op/s 2024-04-26T21:27:20.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:20 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4133926445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:27:20.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:20 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2054325697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:27:20.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.727+0000 7fd6c1e00640 1 -- 172.21.15.1:0/2656200763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6bc1027c0 msgr2=0x7fd6bc102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.727+0000 7fd6c1e00640 1 --2- 172.21.15.1:0/2656200763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6bc1027c0 0x7fd6bc102be0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac009900 tx=0x7fd6ac02a400 comp rx=0 tx=0).stop 2024-04-26T21:27:20.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.727+0000 7fd6c1e00640 1 -- 172.21.15.1:0/2656200763 shutdown_connections 2024-04-26T21:27:20.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.727+0000 7fd6c1e00640 1 --2- 172.21.15.1:0/2656200763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6bc1039b0 0x7fd6bc103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.727+0000 7fd6c1e00640 1 --2- 172.21.15.1:0/2656200763 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc104370 0x7fd6bc10ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.727+0000 7fd6c1e00640 1 --2- 172.21.15.1:0/2656200763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6bc1027c0 0x7fd6bc102be0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.727+0000 7fd6c1e00640 1 -- 172.21.15.1:0/2656200763 >> 172.21.15.1:0/2656200763 conn(0x7fd6bc0fdd30 msgr2=0x7fd6bc100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:20.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.727+0000 7fd6c1e00640 1 -- 172.21.15.1:0/2656200763 shutdown_connections 2024-04-26T21:27:20.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.727+0000 7fd6c1e00640 1 -- 172.21.15.1:0/2656200763 wait complete. 2024-04-26T21:27:20.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.729+0000 7fd6c1e00640 1 Processor -- start 2024-04-26T21:27:20.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.729+0000 7fd6c1e00640 1 -- start start 2024-04-26T21:27:20.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.729+0000 7fd6c1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6bc1027c0 0x7fd6bc19ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:20.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.729+0000 7fd6c1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc1039b0 0x7fd6bc19b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:20.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.729+0000 7fd6c1e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6bc104370 0x7fd6bc1a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:20.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.729+0000 7fd6c1e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc10e310 con 0x7fd6bc104370 2024-04-26T21:27:20.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.729+0000 7fd6c1e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc10e190 con 0x7fd6bc1039b0 2024-04-26T21:27:20.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.729+0000 7fd6c1e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc10e490 con 0x7fd6bc1027c0 2024-04-26T21:27:20.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6c0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6bc1027c0 0x7fd6bc19ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:20.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6c1400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6bc104370 0x7fd6bc1a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:20.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6c0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6bc1027c0 0x7fd6bc19ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46632/0 (socket says 172.21.15.1:46632) 2024-04-26T21:27:20.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc1039b0 0x7fd6bc19b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:20.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6c0a00640 1 -- 172.21.15.1:0/1686321995 learned_addr learned my addr 172.21.15.1:0/1686321995 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:20.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6c1400640 1 -- 172.21.15.1:0/1686321995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6bc1027c0 msgr2=0x7fd6bc19ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6c1400640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6bc1027c0 0x7fd6bc19ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6c1400640 1 -- 172.21.15.1:0/1686321995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc1039b0 msgr2=0x7fd6bc19b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6c1400640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc1039b0 0x7fd6bc19b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6c1400640 1 -- 172.21.15.1:0/1686321995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6bc1a27b0 con 0x7fd6bc104370 2024-04-26T21:27:20.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6c0a00640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6bc1027c0 0x7fd6bc19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:20.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.730+0000 7fd6bbe00640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc1039b0 0x7fd6bc19b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:20.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.731+0000 7fd6c1400640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6bc104370 0x7fd6bc1a2140 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400c970 tx=0x7fd6b400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:20.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.731+0000 7fd6b9600640 1 -- 172.21.15.1:0/1686321995 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4007bb0 con 0x7fd6bc104370 2024-04-26T21:27:20.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.731+0000 7fd6b9600640 1 -- 172.21.15.1:0/1686321995 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6b4007d50 con 0x7fd6bc104370 2024-04-26T21:27:20.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.731+0000 7fd6b9600640 1 -- 172.21.15.1:0/1686321995 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4018660 con 0x7fd6bc104370 2024-04-26T21:27:20.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.731+0000 7fd6c1e00640 1 -- 172.21.15.1:0/1686321995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6bc1a2aa0 con 0x7fd6bc104370 2024-04-26T21:27:20.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.731+0000 7fd6c1e00640 1 -- 172.21.15.1:0/1686321995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6bc1a2fa0 con 0x7fd6bc104370 2024-04-26T21:27:20.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.732+0000 7fd6c1e00640 1 -- 172.21.15.1:0/1686321995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6bc1a3230 con 0x7fd6bc104370 2024-04-26T21:27:20.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.735+0000 7fd6b9600640 1 -- 172.21.15.1:0/1686321995 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd6b400f450 con 0x7fd6bc104370 2024-04-26T21:27:20.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.736+0000 7fd6b9600640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd69c0a3bb0 0x7fd69c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:20.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.736+0000 7fd6b9600640 1 -- 172.21.15.1:0/1686321995 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd6b40d05c0 con 0x7fd6bc104370 2024-04-26T21:27:20.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.736+0000 7fd6c0a00640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd69c0a3bb0 0x7fd69c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:20.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.737+0000 7fd6c0a00640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd69c0a3bb0 0x7fd69c0a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac000c00 tx=0x7fd6ac005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:20.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.739+0000 7fd6b9600640 1 -- 172.21.15.1:0/1686321995 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd6b4014030 con 0x7fd6bc104370 2024-04-26T21:27:20.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:20 smithi001 ceph-mon[28054]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 373 B/s wr, 0 op/s 2024-04-26T21:27:20.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:20 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4133926445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:27:20.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:20 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2054325697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:27:20.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:20 smithi038 ceph-mon[32724]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 373 B/s wr, 0 op/s 2024-04-26T21:27:20.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:20 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4133926445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:27:20.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:20 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2054325697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:27:20.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.944+0000 7fd6c1e00640 1 -- 172.21.15.1:0/1686321995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd6bc1a3400 con 0x7fd6bc104370 2024-04-26T21:27:20.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.945+0000 7fd6b9600640 1 -- 172.21.15.1:0/1686321995 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fd6b409cd00 con 0x7fd6bc104370 2024-04-26T21:27:20.947 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 -- 172.21.15.1:0/1686321995 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd69c0a3bb0 msgr2=0x7fd69c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd69c0a3bb0 0x7fd69c0a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac000c00 tx=0x7fd6ac005a20 comp rx=0 tx=0).stop 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 -- 172.21.15.1:0/1686321995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6bc104370 msgr2=0x7fd6bc1a2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6bc104370 0x7fd6bc1a2140 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400c970 tx=0x7fd6b400ce30 comp rx=0 tx=0).stop 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 -- 172.21.15.1:0/1686321995 shutdown_connections 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd69c0a3bb0 0x7fd69c0a6070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6bc104370 0x7fd6bc1a2140 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc1039b0 0x7fd6bc19b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 --2- 172.21.15.1:0/1686321995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6bc1027c0 0x7fd6bc19ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 -- 172.21.15.1:0/1686321995 >> 172.21.15.1:0/1686321995 conn(0x7fd6bc0fdd30 msgr2=0x7fd6bc074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:20.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 -- 172.21.15.1:0/1686321995 shutdown_connections 2024-04-26T21:27:20.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:20.949+0000 7fd6c1e00640 1 -- 172.21.15.1:0/1686321995 wait complete. 2024-04-26T21:27:20.969 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T21:27:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.178+0000 7f40ce600640 1 -- 172.21.15.1:0/3226842181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c81049d0 msgr2=0x7f40c8104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.178+0000 7f40ce600640 1 --2- 172.21.15.1:0/3226842181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c81049d0 0x7f40c8104df0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009900 tx=0x7f40bc02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.178+0000 7f40ce600640 1 -- 172.21.15.1:0/3226842181 shutdown_connections 2024-04-26T21:27:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.178+0000 7f40ce600640 1 --2- 172.21.15.1:0/3226842181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c8105bc0 0x7f40c8106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.178+0000 7f40ce600640 1 --2- 172.21.15.1:0/3226842181 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c8106580 0x7f40c810ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.178+0000 7f40ce600640 1 --2- 172.21.15.1:0/3226842181 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c81049d0 0x7f40c8104df0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.178+0000 7f40ce600640 1 -- 172.21.15.1:0/3226842181 >> 172.21.15.1:0/3226842181 conn(0x7f40c80fff40 msgr2=0x7f40c8102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:21.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.178+0000 7f40ce600640 1 -- 172.21.15.1:0/3226842181 shutdown_connections 2024-04-26T21:27:21.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.178+0000 7f40ce600640 1 -- 172.21.15.1:0/3226842181 wait complete. 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.179+0000 7f40ce600640 1 Processor -- start 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.179+0000 7f40ce600640 1 -- start start 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40ce600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c81049d0 0x7f40c819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40ce600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c8105bc0 0x7f40c819b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40ce600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8106580 0x7f40c81a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40ce600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f40c8110520 con 0x7f40c8106580 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40ce600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81103a0 con 0x7f40c8105bc0 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40ce600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f40c81106a0 con 0x7f40c81049d0 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40cdc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8106580 0x7f40c81a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40cd200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c81049d0 0x7f40c819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40cdc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8106580 0x7f40c81a2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39086/0 (socket says 172.21.15.1:39086) 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40cdc00640 1 -- 172.21.15.1:0/3393481067 learned_addr learned my addr 172.21.15.1:0/3393481067 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40c7e00640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c8105bc0 0x7f40c819b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40cdc00640 1 -- 172.21.15.1:0/3393481067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c81049d0 msgr2=0x7f40c819aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:21.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40cdc00640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c81049d0 0x7f40c819aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40cdc00640 1 -- 172.21.15.1:0/3393481067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c8105bc0 msgr2=0x7f40c819b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40cdc00640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c8105bc0 0x7f40c819b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.180+0000 7f40cdc00640 1 -- 172.21.15.1:0/3393481067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c81a27c0 con 0x7f40c8106580 2024-04-26T21:27:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.181+0000 7f40cdc00640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8106580 0x7f40c81a2150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f40c000eb10 tx=0x7f40c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.181+0000 7f40c5600640 1 -- 172.21.15.1:0/3393481067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0004b20 con 0x7f40c8106580 2024-04-26T21:27:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.181+0000 7f40c5600640 1 -- 172.21.15.1:0/3393481067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f40c0004cc0 con 0x7f40c8106580 2024-04-26T21:27:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.181+0000 7f40ce600640 1 -- 172.21.15.1:0/3393481067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c81a2ab0 con 0x7f40c8106580 2024-04-26T21:27:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.181+0000 7f40c5600640 1 -- 172.21.15.1:0/3393481067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0010670 con 0x7f40c8106580 2024-04-26T21:27:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.181+0000 7f40ce600640 1 -- 172.21.15.1:0/3393481067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c81a2fb0 con 0x7f40c8106580 2024-04-26T21:27:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.183+0000 7f40ce600640 1 -- 172.21.15.1:0/3393481067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c810bd70 con 0x7f40c8106580 2024-04-26T21:27:21.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.185+0000 7f40c5600640 1 -- 172.21.15.1:0/3393481067 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f40c0005020 con 0x7f40c8106580 2024-04-26T21:27:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.186+0000 7f40c5600640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f40b00a3ba0 0x7f40b00a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.186+0000 7f40c5600640 1 -- 172.21.15.1:0/3393481067 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f40c00d0c50 con 0x7f40c8106580 2024-04-26T21:27:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.187+0000 7f40cd200640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f40b00a3ba0 0x7f40b00a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:21.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.187+0000 7f40cd200640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f40b00a3ba0 0x7f40b00a6060 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f40bc005ab0 tx=0x7f40bc005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:21.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.191+0000 7f40c5600640 1 -- 172.21.15.1:0/3393481067 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f40c009d390 con 0x7f40c8106580 2024-04-26T21:27:21.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.421+0000 7f40ce600640 1 -- 172.21.15.1:0/3393481067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f40c802dad0 con 0x7f40c8106580 2024-04-26T21:27:21.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.422+0000 7f40c5600640 1 -- 172.21.15.1:0/3393481067 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+578 (secure 0 0 0) 0x7f40c009cc90 con 0x7f40c8106580 2024-04-26T21:27:21.424 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3822624839 2024-04-27T21:18:55.151699+0000 2024-04-26T21:27:21.424 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/3159681858 2024-04-27T21:18:55.151699+0000 2024-04-26T21:27:21.424 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/3159681858 2024-04-27T21:18:55.151699+0000 2024-04-26T21:27:21.424 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/304032214 2024-04-27T21:18:20.022040+0000 2024-04-26T21:27:21.424 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/678258621 2024-04-27T21:18:20.022040+0000 2024-04-26T21:27:21.425 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2647613577 2024-04-27T21:18:55.151699+0000 2024-04-26T21:27:21.425 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/278183393 2024-04-27T21:18:20.022040+0000 2024-04-26T21:27:21.425 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/304032214 2024-04-27T21:18:20.022040+0000 2024-04-26T21:27:21.425 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1051804891 2024-04-27T21:18:20.022040+0000 2024-04-26T21:27:21.425 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3004159799 2024-04-27T21:18:55.151699+0000 2024-04-26T21:27:21.425 INFO:teuthology.orchestra.run.smithi001.stderr:listed 10 entries 2024-04-26T21:27:21.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 -- 172.21.15.1:0/3393481067 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f40b00a3ba0 msgr2=0x7f40b00a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f40b00a3ba0 0x7f40b00a6060 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f40bc005ab0 tx=0x7f40bc005a20 comp rx=0 tx=0).stop 2024-04-26T21:27:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 -- 172.21.15.1:0/3393481067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8106580 msgr2=0x7f40c81a2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8106580 0x7f40c81a2150 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f40c000eb10 tx=0x7f40c000c2d0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 -- 172.21.15.1:0/3393481067 shutdown_connections 2024-04-26T21:27:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f40b00a3ba0 0x7f40b00a6060 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40c8106580 0x7f40c81a2150 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f40c8105bc0 0x7f40c819b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 --2- 172.21.15.1:0/3393481067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c81049d0 0x7f40c819aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 -- 172.21.15.1:0/3393481067 >> 172.21.15.1:0/3393481067 conn(0x7f40c80fff40 msgr2=0x7f40c8108f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:21.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 -- 172.21.15.1:0/3393481067 shutdown_connections 2024-04-26T21:27:21.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.425+0000 7f40ce600640 1 -- 172.21.15.1:0/3393481067 wait complete. 2024-04-26T21:27:21.447 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:27:21.447 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T21:27:21.501 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:27:21.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:21 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4047194804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:27:21.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:21 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1686321995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:27:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.715+0000 7fda29200640 1 -- 172.21.15.1:0/942409939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda24105d60 msgr2=0x7fda241061e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:21.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.715+0000 7fda29200640 1 --2- 172.21.15.1:0/942409939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda24105d60 0x7fda241061e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fda10009900 tx=0x7fda1002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.716+0000 7fda29200640 1 -- 172.21.15.1:0/942409939 shutdown_connections 2024-04-26T21:27:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.716+0000 7fda29200640 1 --2- 172.21.15.1:0/942409939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda24106720 0x7fda2410cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.716+0000 7fda29200640 1 --2- 172.21.15.1:0/942409939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda24105d60 0x7fda241061e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.716+0000 7fda29200640 1 --2- 172.21.15.1:0/942409939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda24104b70 0x7fda24104f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.716+0000 7fda29200640 1 -- 172.21.15.1:0/942409939 >> 172.21.15.1:0/942409939 conn(0x7fda241000e0 msgr2=0x7fda24102520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.716+0000 7fda29200640 1 -- 172.21.15.1:0/942409939 shutdown_connections 2024-04-26T21:27:21.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.716+0000 7fda29200640 1 -- 172.21.15.1:0/942409939 wait complete. 2024-04-26T21:27:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.717+0000 7fda29200640 1 Processor -- start 2024-04-26T21:27:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.717+0000 7fda29200640 1 -- start start 2024-04-26T21:27:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.717+0000 7fda29200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda24104b70 0x7fda2419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.717+0000 7fda29200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda24105d60 0x7fda2419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.717+0000 7fda29200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda24106720 0x7fda241a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.717+0000 7fda29200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fda241106d0 con 0x7fda24106720 2024-04-26T21:27:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.717+0000 7fda29200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fda24110550 con 0x7fda24104b70 2024-04-26T21:27:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.717+0000 7fda29200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda24110850 con 0x7fda24105d60 2024-04-26T21:27:21.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda21e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda24104b70 0x7fda2419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda22800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda24106720 0x7fda241a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda21400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda24105d60 0x7fda2419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda21e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda24104b70 0x7fda2419ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:41976/0 (socket says 172.21.15.1:41976) 2024-04-26T21:27:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda21e00640 1 -- 172.21.15.1:0/662840926 learned_addr learned my addr 172.21.15.1:0/662840926 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda21400640 1 -- 172.21.15.1:0/662840926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda24104b70 msgr2=0x7fda2419ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda21400640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda24104b70 0x7fda2419ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda21400640 1 -- 172.21.15.1:0/662840926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda24106720 msgr2=0x7fda241a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda21400640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda24106720 0x7fda241a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda21400640 1 -- 172.21.15.1:0/662840926 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda241a2990 con 0x7fda24105d60 2024-04-26T21:27:21.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.718+0000 7fda21e00640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda24104b70 0x7fda2419ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.719+0000 7fda21400640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda24105d60 0x7fda2419b2b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fda1002a9e0 tx=0x7fda10003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.719+0000 7fda1aa00640 1 -- 172.21.15.1:0/662840926 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1003f070 con 0x7fda24105d60 2024-04-26T21:27:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.719+0000 7fda1aa00640 1 -- 172.21.15.1:0/662840926 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda10031c50 con 0x7fda24105d60 2024-04-26T21:27:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.719+0000 7fda29200640 1 -- 172.21.15.1:0/662840926 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda241a2c20 con 0x7fda24105d60 2024-04-26T21:27:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.719+0000 7fda1aa00640 1 -- 172.21.15.1:0/662840926 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda10035030 con 0x7fda24105d60 2024-04-26T21:27:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.719+0000 7fda29200640 1 -- 172.21.15.1:0/662840926 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda241a30e0 con 0x7fda24105d60 2024-04-26T21:27:21.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.720+0000 7fda29200640 1 -- 172.21.15.1:0/662840926 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda2410be90 con 0x7fda24105d60 2024-04-26T21:27:21.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.723+0000 7fda1aa00640 1 -- 172.21.15.1:0/662840926 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fda10042070 con 0x7fda24105d60 2024-04-26T21:27:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.724+0000 7fda1aa00640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fda080a3bb0 0x7fda080a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.724+0000 7fda1aa00640 1 -- 172.21.15.1:0/662840926 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fda100f15e0 con 0x7fda24105d60 2024-04-26T21:27:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.724+0000 7fda21e00640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fda080a3bb0 0x7fda080a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:21.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.725+0000 7fda21e00640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fda080a3bb0 0x7fda080a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda140095f0 tx=0x7fda14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:21.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.726+0000 7fda1aa00640 1 -- 172.21.15.1:0/662840926 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fda100bd6d0 con 0x7fda24105d60 2024-04-26T21:27:21.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:21 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4047194804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:27:21.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:21 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1686321995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:27:21.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:21 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4047194804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:27:21.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:21 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1686321995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:27:21.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.935+0000 7fda29200640 1 -- 172.21.15.1:0/662840926 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda2402dad0 con 0x7fda24105d60 2024-04-26T21:27:21.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.938+0000 7fda1aa00640 1 -- 172.21.15.1:0/662840926 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23972 (secure 0 0 0) 0x7fda100c19d0 con 0x7fda24105d60 2024-04-26T21:27:21.939 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:21.939 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:27:07.816919+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 -- 172.21.15.1:0/662840926 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fda080a3bb0 msgr2=0x7fda080a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fda080a3bb0 0x7fda080a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fda140095f0 tx=0x7fda14005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 -- 172.21.15.1:0/662840926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda24105d60 msgr2=0x7fda2419b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda24105d60 0x7fda2419b2b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fda1002a9e0 tx=0x7fda10003c10 comp rx=0 tx=0).stop 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 -- 172.21.15.1:0/662840926 shutdown_connections 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fda080a3bb0 0x7fda080a6070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda24105d60 0x7fda2419b2b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda24106720 0x7fda241a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 --2- 172.21.15.1:0/662840926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda24104b70 0x7fda2419ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 -- 172.21.15.1:0/662840926 >> 172.21.15.1:0/662840926 conn(0x7fda241000e0 msgr2=0x7fda24109060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 -- 172.21.15.1:0/662840926 shutdown_connections 2024-04-26T21:27:21.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:21.940+0000 7fda29200640 1 -- 172.21.15.1:0/662840926 wait complete. 2024-04-26T21:27:21.961 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:27:00.381106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:27:01.743898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T21:27:05.341216+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T21:27:21.962 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:27:22.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.181+0000 7f046fe00640 1 -- 172.21.15.1:0/104156157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0470108db0 msgr2=0x7f04701091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:22.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.181+0000 7f046fe00640 1 --2- 172.21.15.1:0/104156157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0470108db0 0x7f04701091d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0460009900 tx=0x7f046002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.181+0000 7f046fe00640 1 -- 172.21.15.1:0/104156157 shutdown_connections 2024-04-26T21:27:22.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.181+0000 7f046fe00640 1 --2- 172.21.15.1:0/104156157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0470109fa0 0x7f047010a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.181+0000 7f046fe00640 1 --2- 172.21.15.1:0/104156157 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f047010a960 0x7f04701111f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.181+0000 7f046fe00640 1 --2- 172.21.15.1:0/104156157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0470108db0 0x7f04701091d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.181+0000 7f046fe00640 1 -- 172.21.15.1:0/104156157 >> 172.21.15.1:0/104156157 conn(0x7f0470077d70 msgr2=0x7f047007a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:22.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.181+0000 7f046fe00640 1 -- 172.21.15.1:0/104156157 shutdown_connections 2024-04-26T21:27:22.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.181+0000 7f046fe00640 1 -- 172.21.15.1:0/104156157 wait complete. 2024-04-26T21:27:22.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.183+0000 7f046fe00640 1 Processor -- start 2024-04-26T21:27:22.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.183+0000 7f046fe00640 1 -- start start 2024-04-26T21:27:22.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.183+0000 7f046fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0470108db0 0x7f047019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:22.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.183+0000 7f046fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0470109fa0 0x7f047019f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:22.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.183+0000 7f046fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f047010a960 0x7f04701a65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:22.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.183+0000 7f046fe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0470114900 con 0x7f0470109fa0 2024-04-26T21:27:22.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046fe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0470114780 con 0x7f047010a960 2024-04-26T21:27:22.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0470114a80 con 0x7f0470108db0 2024-04-26T21:27:22.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f047010a960 0x7f04701a65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:22.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f0467e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0470109fa0 0x7f047019f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:22.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0470108db0 0x7f047019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:22.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f047010a960 0x7f04701a65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:41986/0 (socket says 172.21.15.1:41986) 2024-04-26T21:27:22.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046f400640 1 -- 172.21.15.1:0/75053551 learned_addr learned my addr 172.21.15.1:0/75053551 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:22.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046ea00640 1 -- 172.21.15.1:0/75053551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f047010a960 msgr2=0x7f04701a65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:22.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046ea00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f047010a960 0x7f04701a65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046ea00640 1 -- 172.21.15.1:0/75053551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0470109fa0 msgr2=0x7f047019f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:22.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046ea00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0470109fa0 0x7f047019f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046ea00640 1 -- 172.21.15.1:0/75053551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04701a6c60 con 0x7f0470108db0 2024-04-26T21:27:22.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.184+0000 7f046f400640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f047010a960 0x7f04701a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:22.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.185+0000 7f046ea00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0470108db0 0x7f047019f090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f046002a910 tx=0x7f0460003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:22.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.185+0000 7f0467400640 1 -- 172.21.15.1:0/75053551 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046003f070 con 0x7f0470108db0 2024-04-26T21:27:22.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.185+0000 7f046fe00640 1 -- 172.21.15.1:0/75053551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04701a6ef0 con 0x7f0470108db0 2024-04-26T21:27:22.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.185+0000 7f0467400640 1 -- 172.21.15.1:0/75053551 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0460031c50 con 0x7f0470108db0 2024-04-26T21:27:22.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.185+0000 7f0467400640 1 -- 172.21.15.1:0/75053551 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460035030 con 0x7f0470108db0 2024-04-26T21:27:22.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.185+0000 7f046fe00640 1 -- 172.21.15.1:0/75053551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04701a7350 con 0x7f0470108db0 2024-04-26T21:27:22.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.186+0000 7f046fe00640 1 -- 172.21.15.1:0/75053551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04701101f0 con 0x7f0470108db0 2024-04-26T21:27:22.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.188+0000 7f0467400640 1 -- 172.21.15.1:0/75053551 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f0460042070 con 0x7f0470108db0 2024-04-26T21:27:22.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.190+0000 7f0467400640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04540a3c30 0x7f04540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:22.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.190+0000 7f0467400640 1 -- 172.21.15.1:0/75053551 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f04600f13b0 con 0x7f0470108db0 2024-04-26T21:27:22.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.190+0000 7f0467e00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04540a3c30 0x7f04540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:22.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.191+0000 7f0467e00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04540a3c30 0x7f04540a60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f04701a0550 tx=0x7f0458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:22.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.192+0000 7f0467400640 1 -- 172.21.15.1:0/75053551 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f04600bd4a0 con 0x7f0470108db0 2024-04-26T21:27:22.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.398+0000 7f046fe00640 1 -- 172.21.15.1:0/75053551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f047002dc60 con 0x7f0470108db0 2024-04-26T21:27:22.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.400+0000 7f0467400640 1 -- 172.21.15.1:0/75053551 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f04600c17a0 con 0x7f0470108db0 2024-04-26T21:27:22.402 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 -- 172.21.15.1:0/75053551 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04540a3c30 msgr2=0x7f04540a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04540a3c30 0x7f04540a60f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f04701a0550 tx=0x7f0458005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 -- 172.21.15.1:0/75053551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0470108db0 msgr2=0x7f047019f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0470108db0 0x7f047019f090 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f046002a910 tx=0x7f0460003c10 comp rx=0 tx=0).stop 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 -- 172.21.15.1:0/75053551 shutdown_connections 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04540a3c30 0x7f04540a60f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f047010a960 0x7f04701a65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0470109fa0 0x7f047019f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 --2- 172.21.15.1:0/75053551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0470108db0 0x7f047019f090 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 -- 172.21.15.1:0/75053551 >> 172.21.15.1:0/75053551 conn(0x7f0470077d70 msgr2=0x7f047010d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 -- 172.21.15.1:0/75053551 shutdown_connections 2024-04-26T21:27:22.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.403+0000 7f046fe00640 1 -- 172.21.15.1:0/75053551 wait complete. 2024-04-26T21:27:22.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:27:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.635+0000 7f6222600640 1 -- 172.21.15.1:0/2956815070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c10a180 msgr2=0x7f621c10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.635+0000 7f6222600640 1 --2- 172.21.15.1:0/2956815070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c10a180 0x7f621c10a600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6204009900 tx=0x7f620402a410 comp rx=0 tx=0).stop 2024-04-26T21:27:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.635+0000 7f6222600640 1 -- 172.21.15.1:0/2956815070 shutdown_connections 2024-04-26T21:27:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.635+0000 7f6222600640 1 --2- 172.21.15.1:0/2956815070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f621c10ab40 0x7f621c1113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.635+0000 7f6222600640 1 --2- 172.21.15.1:0/2956815070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c10a180 0x7f621c10a600 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.635+0000 7f6222600640 1 --2- 172.21.15.1:0/2956815070 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c108f90 0x7f621c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.635+0000 7f6222600640 1 -- 172.21.15.1:0/2956815070 >> 172.21.15.1:0/2956815070 conn(0x7f621c077f40 msgr2=0x7f621c07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:22.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.635+0000 7f6222600640 1 -- 172.21.15.1:0/2956815070 shutdown_connections 2024-04-26T21:27:22.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.635+0000 7f6222600640 1 -- 172.21.15.1:0/2956815070 wait complete. 2024-04-26T21:27:22.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.636+0000 7f6222600640 1 Processor -- start 2024-04-26T21:27:22.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.636+0000 7f6222600640 1 -- start start 2024-04-26T21:27:22.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f6222600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f621c108f90 0x7f621c19f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:22.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f6222600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c10a180 0x7f621c19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f6222600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c10ab40 0x7f621c1a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f6222600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f621c114ad0 con 0x7f621c10a180 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f6222600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f621c114950 con 0x7f621c10ab40 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f6222600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f621c114c50 con 0x7f621c108f90 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f621b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f621c108f90 0x7f621c19f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f621aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c10a180 0x7f621c19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f621b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f621c108f90 0x7f621c19f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46720/0 (socket says 172.21.15.1:46720) 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f621b400640 1 -- 172.21.15.1:0/798141858 learned_addr learned my addr 172.21.15.1:0/798141858 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f621be00640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c10ab40 0x7f621c1a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.637+0000 7f621b400640 1 -- 172.21.15.1:0/798141858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c10ab40 msgr2=0x7f621c1a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.638+0000 7f621b400640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c10ab40 0x7f621c1a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.638+0000 7f621b400640 1 -- 172.21.15.1:0/798141858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c10a180 msgr2=0x7f621c19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.638+0000 7f621b400640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c10a180 0x7f621c19f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.638+0000 7f621b400640 1 -- 172.21.15.1:0/798141858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621c1a6df0 con 0x7f621c108f90 2024-04-26T21:27:22.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.638+0000 7f621b400640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f621c108f90 0x7f621c19f1d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f620c00ebc0 tx=0x7f620c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:22.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.638+0000 7f6213e00640 1 -- 172.21.15.1:0/798141858 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c003b70 con 0x7f621c108f90 2024-04-26T21:27:22.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.638+0000 7f6213e00640 1 -- 172.21.15.1:0/798141858 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f620c003d10 con 0x7f621c108f90 2024-04-26T21:27:22.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.638+0000 7f6213e00640 1 -- 172.21.15.1:0/798141858 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c019800 con 0x7f621c108f90 2024-04-26T21:27:22.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.638+0000 7f6222600640 1 -- 172.21.15.1:0/798141858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f621c1a70e0 con 0x7f621c108f90 2024-04-26T21:27:22.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.639+0000 7f6222600640 1 -- 172.21.15.1:0/798141858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f621c1a75b0 con 0x7f621c108f90 2024-04-26T21:27:22.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.640+0000 7f6222600640 1 -- 172.21.15.1:0/798141858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f621c110320 con 0x7f621c108f90 2024-04-26T21:27:22.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.642+0000 7f6213e00640 1 -- 172.21.15.1:0/798141858 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f620c0199a0 con 0x7f621c108f90 2024-04-26T21:27:22.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.643+0000 7f6213e00640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f62000a3c30 0x7f62000a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:22.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.643+0000 7f6213e00640 1 -- 172.21.15.1:0/798141858 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f620c0d0f50 con 0x7f621c108f90 2024-04-26T21:27:22.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.643+0000 7f621aa00640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f62000a3c30 0x7f62000a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:22.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.644+0000 7f621aa00640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f62000a3c30 0x7f62000a60f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6204000c00 tx=0x7f62040314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:22.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.647+0000 7f6213e00640 1 -- 172.21.15.1:0/798141858 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f620c09d690 con 0x7f621c108f90 2024-04-26T21:27:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:22 smithi186 ceph-mon[32991]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:22 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3393481067' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:27:22.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:22 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/662840926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:22.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:22 smithi001 ceph-mon[28054]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:22.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:22 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3393481067' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:27:22.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:22 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/662840926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:22.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:22 smithi038 ceph-mon[32724]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:22.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:22 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3393481067' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:27:22.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:22 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/662840926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:22.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.853+0000 7f6222600640 1 -- 172.21.15.1:0/798141858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f621c0635d0 con 0x7f621c108f90 2024-04-26T21:27:22.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.854+0000 7f6213e00640 1 -- 172.21.15.1:0/798141858 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f620c09cf90 con 0x7f621c108f90 2024-04-26T21:27:22.856 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:27:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 -- 172.21.15.1:0/798141858 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f62000a3c30 msgr2=0x7f62000a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f62000a3c30 0x7f62000a60f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6204000c00 tx=0x7f62040314a0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 -- 172.21.15.1:0/798141858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f621c108f90 msgr2=0x7f621c19f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f621c108f90 0x7f621c19f1d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f620c00ebc0 tx=0x7f620c00c6a0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 -- 172.21.15.1:0/798141858 shutdown_connections 2024-04-26T21:27:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f62000a3c30 0x7f62000a60f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f621c10ab40 0x7f621c1a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f621c10a180 0x7f621c19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 --2- 172.21.15.1:0/798141858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f621c108f90 0x7f621c19f1d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 -- 172.21.15.1:0/798141858 >> 172.21.15.1:0/798141858 conn(0x7f621c077f40 msgr2=0x7f621c10d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.857+0000 7f6222600640 1 -- 172.21.15.1:0/798141858 shutdown_connections 2024-04-26T21:27:22.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:22.858+0000 7f6222600640 1 -- 172.21.15.1:0/798141858 wait complete. 2024-04-26T21:27:22.878 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:27:23.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.099+0000 7f6febe00640 1 -- 172.21.15.1:0/1369730018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fec104970 msgr2=0x7f6fec104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:23.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.099+0000 7f6febe00640 1 --2- 172.21.15.1:0/1369730018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fec104970 0x7f6fec104d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc009900 tx=0x7f6fdc02a410 comp rx=0 tx=0).stop 2024-04-26T21:27:23.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.099+0000 7f6febe00640 1 -- 172.21.15.1:0/1369730018 shutdown_connections 2024-04-26T21:27:23.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.099+0000 7f6febe00640 1 --2- 172.21.15.1:0/1369730018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fec105b60 0x7f6fec105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.099+0000 7f6febe00640 1 --2- 172.21.15.1:0/1369730018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fec106520 0x7f6fec10cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.099+0000 7f6febe00640 1 --2- 172.21.15.1:0/1369730018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fec104970 0x7f6fec104d90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.099+0000 7f6febe00640 1 -- 172.21.15.1:0/1369730018 >> 172.21.15.1:0/1369730018 conn(0x7f6fec0fff00 msgr2=0x7f6fec102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:23.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.099+0000 7f6febe00640 1 -- 172.21.15.1:0/1369730018 shutdown_connections 2024-04-26T21:27:23.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.099+0000 7f6febe00640 1 -- 172.21.15.1:0/1369730018 wait complete. 2024-04-26T21:27:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.100+0000 7f6febe00640 1 Processor -- start 2024-04-26T21:27:23.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6febe00640 1 -- start start 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6febe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fec104970 0x7f6fec19ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6febe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fec105b60 0x7f6fec19b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6febe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fec106520 0x7f6fec1a2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6febe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec110490 con 0x7f6fec106520 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6febe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec110310 con 0x7f6fec104970 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6febe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec110610 con 0x7f6fec105b60 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6feaa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fec104970 0x7f6fec19ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6feb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fec106520 0x7f6fec1a2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6feaa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fec104970 0x7f6fec19ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42032/0 (socket says 172.21.15.1:42032) 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6feaa00640 1 -- 172.21.15.1:0/2131644146 learned_addr learned my addr 172.21.15.1:0/2131644146 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.101+0000 7f6fea000640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fec105b60 0x7f6fec19b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.102+0000 7f6feaa00640 1 -- 172.21.15.1:0/2131644146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fec105b60 msgr2=0x7f6fec19b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.102+0000 7f6feaa00640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fec105b60 0x7f6fec19b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.102+0000 7f6feaa00640 1 -- 172.21.15.1:0/2131644146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fec106520 msgr2=0x7f6fec1a2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.102+0000 7f6feaa00640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fec106520 0x7f6fec1a2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.102+0000 7f6feaa00640 1 -- 172.21.15.1:0/2131644146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fec1a27f0 con 0x7f6fec104970 2024-04-26T21:27:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.102+0000 7f6feaa00640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fec104970 0x7f6fec19ac20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc006010 tx=0x7f6fdc0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.103+0000 7f6fe3400640 1 -- 172.21.15.1:0/2131644146 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc03f030 con 0x7f6fec104970 2024-04-26T21:27:23.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.103+0000 7f6fe3400640 1 -- 172.21.15.1:0/2131644146 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fdc031910 con 0x7f6fec104970 2024-04-26T21:27:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.103+0000 7f6febe00640 1 -- 172.21.15.1:0/2131644146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fec1a2a80 con 0x7f6fec104970 2024-04-26T21:27:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.103+0000 7f6fe3400640 1 -- 172.21.15.1:0/2131644146 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc035070 con 0x7f6fec104970 2024-04-26T21:27:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.103+0000 7f6febe00640 1 -- 172.21.15.1:0/2131644146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fec1a2fc0 con 0x7f6fec104970 2024-04-26T21:27:23.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.104+0000 7f6febe00640 1 -- 172.21.15.1:0/2131644146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fec10be00 con 0x7f6fec104970 2024-04-26T21:27:23.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.106+0000 7f6fe3400640 1 -- 172.21.15.1:0/2131644146 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6fdc042030 con 0x7f6fec104970 2024-04-26T21:27:23.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.107+0000 7f6fe3400640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6fd00a3c20 0x7f6fd00a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:23.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.107+0000 7f6fe3400640 1 -- 172.21.15.1:0/2131644146 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f6fdc0f0f00 con 0x7f6fec104970 2024-04-26T21:27:23.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.108+0000 7f6fea000640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6fd00a3c20 0x7f6fd00a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:23.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.108+0000 7f6fea000640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6fd00a3c20 0x7f6fd00a60e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6fec19c0e0 tx=0x7f6fd8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:23.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.110+0000 7f6fe3400640 1 -- 172.21.15.1:0/2131644146 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6fdc0bcff0 con 0x7f6fec104970 2024-04-26T21:27:23.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.316+0000 7f6febe00640 1 -- 172.21.15.1:0/2131644146 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6fec02dc60 con 0x7f6fec104970 2024-04-26T21:27:23.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.317+0000 7f6fe3400640 1 -- 172.21.15.1:0/2131644146 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f6fdc0c12f0 con 0x7f6fec104970 2024-04-26T21:27:23.319 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T21:27:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 -- 172.21.15.1:0/2131644146 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6fd00a3c20 msgr2=0x7f6fd00a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6fd00a3c20 0x7f6fd00a60e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6fec19c0e0 tx=0x7f6fd8005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 -- 172.21.15.1:0/2131644146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fec104970 msgr2=0x7f6fec19ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fec104970 0x7f6fec19ac20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc006010 tx=0x7f6fdc0042f0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 -- 172.21.15.1:0/2131644146 shutdown_connections 2024-04-26T21:27:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f6fd00a3c20 0x7f6fd00a60e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fec105b60 0x7f6fec19b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fec106520 0x7f6fec1a2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 --2- 172.21.15.1:0/2131644146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6fec104970 0x7f6fec19ac20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 -- 172.21.15.1:0/2131644146 >> 172.21.15.1:0/2131644146 conn(0x7f6fec0fff00 msgr2=0x7f6fec108fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 -- 172.21.15.1:0/2131644146 shutdown_connections 2024-04-26T21:27:23.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.320+0000 7f6febe00640 1 -- 172.21.15.1:0/2131644146 wait complete. 2024-04-26T21:27:23.342 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T21:27:23.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.558+0000 7f5935400640 1 -- 172.21.15.1:0/1521008136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5930078530 msgr2=0x7f5930078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:23.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.558+0000 7f5935400640 1 --2- 172.21.15.1:0/1521008136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5930078530 0x7f5930078990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5918009860 tx=0x7f591802a340 comp rx=0 tx=0).stop 2024-04-26T21:27:23.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.558+0000 7f5935400640 1 -- 172.21.15.1:0/1521008136 shutdown_connections 2024-04-26T21:27:23.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.558+0000 7f5935400640 1 --2- 172.21.15.1:0/1521008136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5930078530 0x7f5930078990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.558+0000 7f5935400640 1 --2- 172.21.15.1:0/1521008136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5930078ed0 0x7f5930079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.558+0000 7f5935400640 1 --2- 172.21.15.1:0/1521008136 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59300772f0 0x7f5930077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.558+0000 7f5935400640 1 -- 172.21.15.1:0/1521008136 >> 172.21.15.1:0/1521008136 conn(0x7f59300fff40 msgr2=0x7f5930102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:23.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.558+0000 7f5935400640 1 -- 172.21.15.1:0/1521008136 shutdown_connections 2024-04-26T21:27:23.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.558+0000 7f5935400640 1 -- 172.21.15.1:0/1521008136 wait complete. 2024-04-26T21:27:23.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.560+0000 7f5935400640 1 Processor -- start 2024-04-26T21:27:23.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.560+0000 7f5935400640 1 -- start start 2024-04-26T21:27:23.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.560+0000 7f5935400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59300772f0 0x7f593019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:23.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.560+0000 7f5935400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5930078530 0x7f593019f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:23.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.560+0000 7f5935400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5930078ed0 0x7f59301a65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:23.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.560+0000 7f5935400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f59301148b0 con 0x7f5930078530 2024-04-26T21:27:23.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.560+0000 7f5935400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5930114730 con 0x7f5930078ed0 2024-04-26T21:27:23.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.560+0000 7f5935400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5930114a30 con 0x7f59300772f0 2024-04-26T21:27:23.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f592f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5930078530 0x7f593019f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:23.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f592fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59300772f0 0x7f593019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:23.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f592fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59300772f0 0x7f593019f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46758/0 (socket says 172.21.15.1:46758) 2024-04-26T21:27:23.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f592fe00640 1 -- 172.21.15.1:0/4087193606 learned_addr learned my addr 172.21.15.1:0/4087193606 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:23.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f5934a00640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5930078ed0 0x7f59301a65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f592f400640 1 -- 172.21.15.1:0/4087193606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59300772f0 msgr2=0x7f593019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f592f400640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59300772f0 0x7f593019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f592f400640 1 -- 172.21.15.1:0/4087193606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5930078ed0 msgr2=0x7f59301a65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f592f400640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5930078ed0 0x7f59301a65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f592f400640 1 -- 172.21.15.1:0/4087193606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59301a6c30 con 0x7f5930078530 2024-04-26T21:27:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.561+0000 7f592fe00640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59300772f0 0x7f593019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.562+0000 7f592f400640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5930078530 0x7f593019f550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f591802a860 tx=0x7f5918032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.562+0000 7f592cc00640 1 -- 172.21.15.1:0/4087193606 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918040020 con 0x7f5930078530 2024-04-26T21:27:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.562+0000 7f592cc00640 1 -- 172.21.15.1:0/4087193606 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5918030d50 con 0x7f5930078530 2024-04-26T21:27:23.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.562+0000 7f592cc00640 1 -- 172.21.15.1:0/4087193606 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5918037040 con 0x7f5930078530 2024-04-26T21:27:23.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.562+0000 7f5935400640 1 -- 172.21.15.1:0/4087193606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59301a6ec0 con 0x7f5930078530 2024-04-26T21:27:23.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.562+0000 7f5935400640 1 -- 172.21.15.1:0/4087193606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59301a7330 con 0x7f5930078530 2024-04-26T21:27:23.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.564+0000 7f5935400640 1 -- 172.21.15.1:0/4087193606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5930111320 con 0x7f5930078530 2024-04-26T21:27:23.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.566+0000 7f592cc00640 1 -- 172.21.15.1:0/4087193606 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f5918043020 con 0x7f5930078530 2024-04-26T21:27:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.568+0000 7f592cc00640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f59140a3c30 0x7f59140a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.569+0000 7f592cc00640 1 -- 172.21.15.1:0/4087193606 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f59180f02b0 con 0x7f5930078530 2024-04-26T21:27:23.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.569+0000 7f592fe00640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f59140a3c30 0x7f59140a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:23.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.570+0000 7f592fe00640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f59140a3c30 0x7f59140a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f592000e8b0 tx=0x7f59200095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:23.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.573+0000 7f592cc00640 1 -- 172.21.15.1:0/4087193606 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f591804a020 con 0x7f5930078530 2024-04-26T21:27:23.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:23 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/75053551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:27:23.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:23 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/798141858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:27:23.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:23 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2131644146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:27:23.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.782+0000 7f5935400640 1 -- 172.21.15.1:0/4087193606 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5930063530 con 0x7f5930078530 2024-04-26T21:27:23.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.783+0000 7f592cc00640 1 -- 172.21.15.1:0/4087193606 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f59180bc9f0 con 0x7f5930078530 2024-04-26T21:27:23.785 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:27:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.786+0000 7f5935400640 1 -- 172.21.15.1:0/4087193606 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f59140a3c30 msgr2=0x7f59140a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:23.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.786+0000 7f5935400640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f59140a3c30 0x7f59140a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f592000e8b0 tx=0x7f59200095c0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.786+0000 7f5935400640 1 -- 172.21.15.1:0/4087193606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5930078530 msgr2=0x7f593019f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:23.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.786+0000 7f5935400640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5930078530 0x7f593019f550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f591802a860 tx=0x7f5918032d50 comp rx=0 tx=0).stop 2024-04-26T21:27:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.786+0000 7f5935400640 1 -- 172.21.15.1:0/4087193606 shutdown_connections 2024-04-26T21:27:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.786+0000 7f5935400640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f59140a3c30 0x7f59140a60f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.786+0000 7f5935400640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5930078ed0 0x7f59301a65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.786+0000 7f5935400640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5930078530 0x7f593019f550 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.786+0000 7f5935400640 1 --2- 172.21.15.1:0/4087193606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59300772f0 0x7f593019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.786+0000 7f5935400640 1 -- 172.21.15.1:0/4087193606 >> 172.21.15.1:0/4087193606 conn(0x7f59300fff40 msgr2=0x7f5930101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.787+0000 7f5935400640 1 -- 172.21.15.1:0/4087193606 shutdown_connections 2024-04-26T21:27:23.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:23.787+0000 7f5935400640 1 -- 172.21.15.1:0/4087193606 wait complete. 2024-04-26T21:27:23.804 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:23 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/75053551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:27:23.804 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:23 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/798141858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:27:23.804 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:23 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2131644146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:27:23.807 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T21:27:23.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:23 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/75053551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:27:23.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:23 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/798141858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:27:23.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:23 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2131644146' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:27:24.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.015+0000 7fd1c7e00640 1 -- 172.21.15.1:0/3472588301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c8100ff0 msgr2=0x7fd1c8103420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:24.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.015+0000 7fd1c7e00640 1 --2- 172.21.15.1:0/3472588301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c8100ff0 0x7fd1c8103420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009900 tx=0x7fd1b802a410 comp rx=0 tx=0).stop 2024-04-26T21:27:24.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.015+0000 7fd1c7e00640 1 -- 172.21.15.1:0/3472588301 shutdown_connections 2024-04-26T21:27:24.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.015+0000 7fd1c7e00640 1 --2- 172.21.15.1:0/3472588301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c8103960 0x7fd1c8105dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.015+0000 7fd1c7e00640 1 --2- 172.21.15.1:0/3472588301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1c8106310 0x7fd1c8106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.015+0000 7fd1c7e00640 1 --2- 172.21.15.1:0/3472588301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c8100ff0 0x7fd1c8103420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.015+0000 7fd1c7e00640 1 -- 172.21.15.1:0/3472588301 >> 172.21.15.1:0/3472588301 conn(0x7fd1c80fcc10 msgr2=0x7fd1c80ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:24.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.015+0000 7fd1c7e00640 1 -- 172.21.15.1:0/3472588301 shutdown_connections 2024-04-26T21:27:24.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.015+0000 7fd1c7e00640 1 -- 172.21.15.1:0/3472588301 wait complete. 2024-04-26T21:27:24.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.016+0000 7fd1c7e00640 1 Processor -- start 2024-04-26T21:27:24.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c7e00640 1 -- start start 2024-04-26T21:27:24.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c8100ff0 0x7fd1c819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:24.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1c8103960 0x7fd1c819f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:24.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c8106310 0x7fd1c81a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:24.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c7e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c81148d0 con 0x7fd1c8106310 2024-04-26T21:27:24.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c7e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c8114750 con 0x7fd1c8103960 2024-04-26T21:27:24.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c7e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c8114a50 con 0x7fd1c8100ff0 2024-04-26T21:27:24.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c7400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c8106310 0x7fd1c81a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1be000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1c8103960 0x7fd1c819f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c8100ff0 0x7fd1c819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c7400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c8106310 0x7fd1c81a6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39204/0 (socket says 172.21.15.1:39204) 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.017+0000 7fd1c7400640 1 -- 172.21.15.1:0/1061235091 learned_addr learned my addr 172.21.15.1:0/1061235091 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1c6a00640 1 -- 172.21.15.1:0/1061235091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1c8103960 msgr2=0x7fd1c819f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1c6a00640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1c8103960 0x7fd1c819f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1c6a00640 1 -- 172.21.15.1:0/1061235091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c8106310 msgr2=0x7fd1c81a6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1c6a00640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c8106310 0x7fd1c81a6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1c6a00640 1 -- 172.21.15.1:0/1061235091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c81a6b80 con 0x7fd1c8100ff0 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1c6a00640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c8100ff0 0x7fd1c819efb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd1b802aa00 tx=0x7fd1b8003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1bfe00640 1 -- 172.21.15.1:0/1061235091 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b803f070 con 0x7fd1c8100ff0 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1c7e00640 1 -- 172.21.15.1:0/1061235091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c81a6e10 con 0x7fd1c8100ff0 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1bfe00640 1 -- 172.21.15.1:0/1061235091 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1b8031c50 con 0x7fd1c8100ff0 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1bfe00640 1 -- 172.21.15.1:0/1061235091 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8035030 con 0x7fd1c8100ff0 2024-04-26T21:27:24.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.018+0000 7fd1c7e00640 1 -- 172.21.15.1:0/1061235091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c81a7270 con 0x7fd1c8100ff0 2024-04-26T21:27:24.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.020+0000 7fd1c7e00640 1 -- 172.21.15.1:0/1061235091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c80fe700 con 0x7fd1c8100ff0 2024-04-26T21:27:24.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.022+0000 7fd1bfe00640 1 -- 172.21.15.1:0/1061235091 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fd1b8042070 con 0x7fd1c8100ff0 2024-04-26T21:27:24.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.023+0000 7fd1bfe00640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1ac0a3c20 0x7fd1ac0a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:24.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.023+0000 7fd1bfe00640 1 -- 172.21.15.1:0/1061235091 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd1b80f0c90 con 0x7fd1c8100ff0 2024-04-26T21:27:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.023+0000 7fd1be000640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1ac0a3c20 0x7fd1ac0a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:24.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.024+0000 7fd1be000640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1ac0a3c20 0x7fd1ac0a60e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00095f0 tx=0x7fd1b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:24.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.028+0000 7fd1bfe00640 1 -- 172.21.15.1:0/1061235091 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd1b80bcd80 con 0x7fd1c8100ff0 2024-04-26T21:27:24.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.250+0000 7fd1c7e00640 1 -- 172.21.15.1:0/1061235091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd1c8063630 con 0x7fd1c8100ff0 2024-04-26T21:27:24.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.251+0000 7fd1bfe00640 1 -- 172.21.15.1:0/1061235091 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+578 (secure 0 0 0) 0x7fd1b80c1080 con 0x7fd1c8100ff0 2024-04-26T21:27:24.253 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3822624839 2024-04-27T21:18:55.151699+0000 2024-04-26T21:27:24.253 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/3159681858 2024-04-27T21:18:55.151699+0000 2024-04-26T21:27:24.253 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/3159681858 2024-04-27T21:18:55.151699+0000 2024-04-26T21:27:24.253 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/304032214 2024-04-27T21:18:20.022040+0000 2024-04-26T21:27:24.254 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/678258621 2024-04-27T21:18:20.022040+0000 2024-04-26T21:27:24.254 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2647613577 2024-04-27T21:18:55.151699+0000 2024-04-26T21:27:24.254 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/278183393 2024-04-27T21:18:20.022040+0000 2024-04-26T21:27:24.254 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/304032214 2024-04-27T21:18:20.022040+0000 2024-04-26T21:27:24.254 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1051804891 2024-04-27T21:18:20.022040+0000 2024-04-26T21:27:24.254 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3004159799 2024-04-27T21:18:55.151699+0000 2024-04-26T21:27:24.254 INFO:teuthology.orchestra.run.smithi001.stderr:listed 10 entries 2024-04-26T21:27:24.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 -- 172.21.15.1:0/1061235091 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1ac0a3c20 msgr2=0x7fd1ac0a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:24.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1ac0a3c20 0x7fd1ac0a60e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00095f0 tx=0x7fd1b0005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:24.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 -- 172.21.15.1:0/1061235091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c8100ff0 msgr2=0x7fd1c819efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:24.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c8100ff0 0x7fd1c819efb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd1b802aa00 tx=0x7fd1b8003c10 comp rx=0 tx=0).stop 2024-04-26T21:27:24.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 -- 172.21.15.1:0/1061235091 shutdown_connections 2024-04-26T21:27:24.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd1ac0a3c20 0x7fd1ac0a60e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c8106310 0x7fd1c81a6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1c8103960 0x7fd1c819f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 --2- 172.21.15.1:0/1061235091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1c8100ff0 0x7fd1c819efb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 -- 172.21.15.1:0/1061235091 >> 172.21.15.1:0/1061235091 conn(0x7fd1c80fcc10 msgr2=0x7fd1c81046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:24.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 -- 172.21.15.1:0/1061235091 shutdown_connections 2024-04-26T21:27:24.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.254+0000 7fd1c7e00640 1 -- 172.21.15.1:0/1061235091 wait complete. 2024-04-26T21:27:24.276 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-04-26T21:27:24.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:24.276 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-26T21:27:24.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi001.stderr:ceph-fuse[69385]: fuse finished with error 0 and tester_r 0 2024-04-26T21:27:24.364 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T21:27:24.457 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:27:24.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:27:24.458 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-26T21:27:24.526 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:27:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:24 smithi001 ceph-mon[28054]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:24 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4087193606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:27:24.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:24 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1061235091' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:27:24.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:24 smithi186 ceph-mon[32991]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:24.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:24 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4087193606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:27:24.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:24 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1061235091' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:27:24.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.742+0000 7f85b5a00640 1 -- 172.21.15.1:0/3270059475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85b0078530 msgr2=0x7f85b0078990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.742+0000 7f85b5a00640 1 --2- 172.21.15.1:0/3270059475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85b0078530 0x7f85b0078990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8598009860 tx=0x7f859802a340 comp rx=0 tx=0).stop 2024-04-26T21:27:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.743+0000 7f85b5a00640 1 -- 172.21.15.1:0/3270059475 shutdown_connections 2024-04-26T21:27:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.743+0000 7f85b5a00640 1 --2- 172.21.15.1:0/3270059475 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85b0078ed0 0x7f85b0079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.743+0000 7f85b5a00640 1 --2- 172.21.15.1:0/3270059475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85b0078530 0x7f85b0078990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.743+0000 7f85b5a00640 1 --2- 172.21.15.1:0/3270059475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b00772f0 0x7f85b0077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.743+0000 7f85b5a00640 1 -- 172.21.15.1:0/3270059475 >> 172.21.15.1:0/3270059475 conn(0x7f85b01000e0 msgr2=0x7f85b0102520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.743+0000 7f85b5a00640 1 -- 172.21.15.1:0/3270059475 shutdown_connections 2024-04-26T21:27:24.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.743+0000 7f85b5a00640 1 -- 172.21.15.1:0/3270059475 wait complete. 2024-04-26T21:27:24.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.744+0000 7f85b5a00640 1 Processor -- start 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.744+0000 7f85b5a00640 1 -- start start 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.744+0000 7f85b5a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85b00772f0 0x7f85b019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.744+0000 7f85b5a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85b0078530 0x7f85b019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.744+0000 7f85b5a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b0078ed0 0x7f85b01a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.744+0000 7f85b5a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f85b0114a70 con 0x7f85b0078530 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.744+0000 7f85b5a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f85b01148f0 con 0x7f85b00772f0 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.744+0000 7f85b5a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f85b0114bf0 con 0x7f85b0078ed0 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.744+0000 7f85aea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85b00772f0 0x7f85b019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85ae000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85b0078530 0x7f85b019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85aea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85b00772f0 0x7f85b019f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42096/0 (socket says 172.21.15.1:42096) 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85aea00640 1 -- 172.21.15.1:0/3884365909 learned_addr learned my addr 172.21.15.1:0/3884365909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:24.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85af400640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b0078ed0 0x7f85b01a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85aea00640 1 -- 172.21.15.1:0/3884365909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b0078ed0 msgr2=0x7f85b01a6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85aea00640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b0078ed0 0x7f85b01a6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85aea00640 1 -- 172.21.15.1:0/3884365909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85b0078530 msgr2=0x7f85b019f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85aea00640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85b0078530 0x7f85b019f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85aea00640 1 -- 172.21.15.1:0/3884365909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b01a6dc0 con 0x7f85b00772f0 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85ae000640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85b0078530 0x7f85b019f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85af400640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b0078ed0 0x7f85b01a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.745+0000 7f85aea00640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85b00772f0 0x7f85b019f1a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85a000d950 tx=0x7f85a000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.746+0000 7f85a7400640 1 -- 172.21.15.1:0/3884365909 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0017070 con 0x7f85b00772f0 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.746+0000 7f85a7400640 1 -- 172.21.15.1:0/3884365909 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85a0009c30 con 0x7f85b00772f0 2024-04-26T21:27:24.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.746+0000 7f85a7400640 1 -- 172.21.15.1:0/3884365909 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a000f620 con 0x7f85b00772f0 2024-04-26T21:27:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.746+0000 7f85b5a00640 1 -- 172.21.15.1:0/3884365909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85b01a70b0 con 0x7f85b00772f0 2024-04-26T21:27:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.746+0000 7f85b5a00640 1 -- 172.21.15.1:0/3884365909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85b01a75d0 con 0x7f85b00772f0 2024-04-26T21:27:24.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.747+0000 7f85b5a00640 1 -- 172.21.15.1:0/3884365909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85b01114e0 con 0x7f85b00772f0 2024-04-26T21:27:24.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.749+0000 7f85a7400640 1 -- 172.21.15.1:0/3884365909 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f85a0005030 con 0x7f85b00772f0 2024-04-26T21:27:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.751+0000 7f85a7400640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85940a3bb0 0x7f85940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.751+0000 7f85a7400640 1 -- 172.21.15.1:0/3884365909 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f85a00d0250 con 0x7f85b00772f0 2024-04-26T21:27:24.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.751+0000 7f85ae000640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85940a3bb0 0x7f85940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:24.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.752+0000 7f85ae000640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85940a3bb0 0x7f85940a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f85980304b0 tx=0x7f8598005780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:24.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.753+0000 7f85a7400640 1 -- 172.21.15.1:0/3884365909 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f85a0009de0 con 0x7f85b00772f0 2024-04-26T21:27:24.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:24 smithi038 ceph-mon[32724]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:24.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:24 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4087193606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:27:24.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:24 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1061235091' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:27:24.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.958+0000 7f85b5a00640 1 -- 172.21.15.1:0/3884365909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85b002dad0 con 0x7f85b00772f0 2024-04-26T21:27:24.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.962+0000 7f85a7400640 1 -- 172.21.15.1:0/3884365909 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23972 (secure 0 0 0) 0x7f85a009c340 con 0x7f85b00772f0 2024-04-26T21:27:24.963 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:24.963 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:27:07.816919+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 -- 172.21.15.1:0/3884365909 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85940a3bb0 msgr2=0x7f85940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85940a3bb0 0x7f85940a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f85980304b0 tx=0x7f8598005780 comp rx=0 tx=0).stop 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 -- 172.21.15.1:0/3884365909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85b00772f0 msgr2=0x7f85b019f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85b00772f0 0x7f85b019f1a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85a000d950 tx=0x7f85a000de10 comp rx=0 tx=0).stop 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 -- 172.21.15.1:0/3884365909 shutdown_connections 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85940a3bb0 0x7f85940a6070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b0078ed0 0x7f85b01a6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85b0078530 0x7f85b019f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 --2- 172.21.15.1:0/3884365909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85b00772f0 0x7f85b019f1a0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 -- 172.21.15.1:0/3884365909 >> 172.21.15.1:0/3884365909 conn(0x7f85b01000e0 msgr2=0x7f85b0101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:24.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 -- 172.21.15.1:0/3884365909 shutdown_connections 2024-04-26T21:27:24.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:24.964+0000 7f85b5a00640 1 -- 172.21.15.1:0/3884365909 wait complete. 2024-04-26T21:27:24.986 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:27:25.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.239+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1843854215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf94078ed0 msgr2=0x7fcf94079770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:25.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.239+0000 7fcf9ae00640 1 --2- 172.21.15.1:0/1843854215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf94078ed0 0x7fcf94079770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c009ca0 tx=0x7fcf8c02a8b0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.239+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1843854215 shutdown_connections 2024-04-26T21:27:25.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.239+0000 7fcf9ae00640 1 --2- 172.21.15.1:0/1843854215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf94078ed0 0x7fcf94079770 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.239+0000 7fcf9ae00640 1 --2- 172.21.15.1:0/1843854215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf94078530 0x7fcf94078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.239+0000 7fcf9ae00640 1 --2- 172.21.15.1:0/1843854215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf940772f0 0x7fcf94077710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.239+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1843854215 >> 172.21.15.1:0/1843854215 conn(0x7fcf940fff00 msgr2=0x7fcf94102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:25.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.239+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1843854215 shutdown_connections 2024-04-26T21:27:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.239+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1843854215 wait complete. 2024-04-26T21:27:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf9ae00640 1 Processor -- start 2024-04-26T21:27:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf9ae00640 1 -- start start 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf9ae00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf940772f0 0x7fcf9419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf9ae00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf94078530 0x7fcf9419f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf9ae00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf94078ed0 0x7fcf941a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf9ae00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcf94114910 con 0x7fcf94078530 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf9ae00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcf94114790 con 0x7fcf940772f0 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf9ae00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcf94114a90 con 0x7fcf94078ed0 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf99000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf94078530 0x7fcf9419f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf99a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf940772f0 0x7fcf9419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf99a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf940772f0 0x7fcf9419eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42112/0 (socket says 172.21.15.1:42112) 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf99000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf94078530 0x7fcf9419f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39230/0 (socket says 172.21.15.1:39230) 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf99a00640 1 -- 172.21.15.1:0/1443351980 learned_addr learned my addr 172.21.15.1:0/1443351980 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf9a400640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf94078ed0 0x7fcf941a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf99a00640 1 -- 172.21.15.1:0/1443351980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf94078ed0 msgr2=0x7fcf941a63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf99a00640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf94078ed0 0x7fcf941a63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf99a00640 1 -- 172.21.15.1:0/1443351980 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf94078530 msgr2=0x7fcf9419f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:25.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf99a00640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf94078530 0x7fcf9419f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf99a00640 1 -- 172.21.15.1:0/1443351980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf941a6a40 con 0x7fcf940772f0 2024-04-26T21:27:25.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.241+0000 7fcf9a400640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf94078ed0 0x7fcf941a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:27:25.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.242+0000 7fcf99a00640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf940772f0 0x7fcf9419eee0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800e9f0 tx=0x7fcf8800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:25.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.242+0000 7fcf92000640 1 -- 172.21.15.1:0/1443351980 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8800cd80 con 0x7fcf940772f0 2024-04-26T21:27:25.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.242+0000 7fcf92000640 1 -- 172.21.15.1:0/1443351980 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf880054d0 con 0x7fcf940772f0 2024-04-26T21:27:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.242+0000 7fcf92000640 1 -- 172.21.15.1:0/1443351980 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf88010690 con 0x7fcf940772f0 2024-04-26T21:27:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.242+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1443351980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf941a6d30 con 0x7fcf940772f0 2024-04-26T21:27:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.242+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1443351980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf941a71d0 con 0x7fcf940772f0 2024-04-26T21:27:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.243+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1443351980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf94111300 con 0x7fcf940772f0 2024-04-26T21:27:25.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.246+0000 7fcf92000640 1 -- 172.21.15.1:0/1443351980 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fcf88004e90 con 0x7fcf940772f0 2024-04-26T21:27:25.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.248+0000 7fcf92000640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcf7c0a3cd0 0x7fcf7c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.248+0000 7fcf92000640 1 -- 172.21.15.1:0/1443351980 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fcf880d0ae0 con 0x7fcf940772f0 2024-04-26T21:27:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.248+0000 7fcf99000640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcf7c0a3cd0 0x7fcf7c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.248+0000 7fcf99000640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcf7c0a3cd0 0x7fcf7c0a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcf941a0330 tx=0x7fcf80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:25.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.252+0000 7fcf92000640 1 -- 172.21.15.1:0/1443351980 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcf8809d220 con 0x7fcf940772f0 2024-04-26T21:27:25.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.464+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1443351980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf9402dd30 con 0x7fcf940772f0 2024-04-26T21:27:25.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.467+0000 7fcf92000640 1 -- 172.21.15.1:0/1443351980 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23972 (secure 0 0 0) 0x7fcf8809cb20 con 0x7fcf940772f0 2024-04-26T21:27:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:25.468 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:27:07.816919+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:27:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1443351980 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcf7c0a3cd0 msgr2=0x7fcf7c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcf7c0a3cd0 0x7fcf7c0a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcf941a0330 tx=0x7fcf80005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1443351980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf940772f0 msgr2=0x7fcf9419eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf940772f0 0x7fcf9419eee0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcf8800e9f0 tx=0x7fcf8800edf0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1443351980 shutdown_connections 2024-04-26T21:27:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcf7c0a3cd0 0x7fcf7c0a6190 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf94078ed0 0x7fcf941a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf94078530 0x7fcf9419f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 --2- 172.21.15.1:0/1443351980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf940772f0 0x7fcf9419eee0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1443351980 >> 172.21.15.1:0/1443351980 conn(0x7fcf940fff00 msgr2=0x7fcf94101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:25.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1443351980 shutdown_connections 2024-04-26T21:27:25.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.469+0000 7fcf9ae00640 1 -- 172.21.15.1:0/1443351980 wait complete. 2024-04-26T21:27:25.492 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-26T21:27:25.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.701+0000 7f9c5de00640 1 -- 172.21.15.1:0/1148014188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58105ba0 msgr2=0x7f9c58106020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:25.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.701+0000 7f9c5de00640 1 --2- 172.21.15.1:0/1148014188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58105ba0 0x7f9c58106020 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009900 tx=0x7f9c4402a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.702+0000 7f9c5de00640 1 -- 172.21.15.1:0/1148014188 shutdown_connections 2024-04-26T21:27:25.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.702+0000 7f9c5de00640 1 --2- 172.21.15.1:0/1148014188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c58106560 0x7f9c5810cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.702+0000 7f9c5de00640 1 --2- 172.21.15.1:0/1148014188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c58105ba0 0x7f9c58106020 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.702+0000 7f9c5de00640 1 --2- 172.21.15.1:0/1148014188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c581049b0 0x7f9c58104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.702+0000 7f9c5de00640 1 -- 172.21.15.1:0/1148014188 >> 172.21.15.1:0/1148014188 conn(0x7f9c580fff40 msgr2=0x7f9c58102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:25.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.702+0000 7f9c5de00640 1 -- 172.21.15.1:0/1148014188 shutdown_connections 2024-04-26T21:27:25.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.702+0000 7f9c5de00640 1 -- 172.21.15.1:0/1148014188 wait complete. 2024-04-26T21:27:25.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:25 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3884365909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.703+0000 7f9c5de00640 1 Processor -- start 2024-04-26T21:27:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.703+0000 7f9c5de00640 1 -- start start 2024-04-26T21:27:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c581049b0 0x7f9c5819abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c58105ba0 0x7f9c5819b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c58106560 0x7f9c581a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5de00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9c581104c0 con 0x7f9c581049b0 2024-04-26T21:27:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5de00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9c58110340 con 0x7f9c58105ba0 2024-04-26T21:27:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5de00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9c58110640 con 0x7f9c58106560 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5ca00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c581049b0 0x7f9c5819abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5ca00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c581049b0 0x7f9c5819abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39244/0 (socket says 172.21.15.1:39244) 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5ca00640 1 -- 172.21.15.1:0/1926290276 learned_addr learned my addr 172.21.15.1:0/1926290276 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c57e00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c58105ba0 0x7f9c5819b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5d400640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c58106560 0x7f9c581a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5ca00640 1 -- 172.21.15.1:0/1926290276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c58106560 msgr2=0x7f9c581a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5ca00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c58106560 0x7f9c581a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.704+0000 7f9c5ca00640 1 -- 172.21.15.1:0/1926290276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c58105ba0 msgr2=0x7f9c5819b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.705+0000 7f9c5ca00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c58105ba0 0x7f9c5819b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.705+0000 7f9c5ca00640 1 -- 172.21.15.1:0/1926290276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c581a2810 con 0x7f9c581049b0 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.705+0000 7f9c57e00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c58105ba0 0x7f9c5819b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:27:25.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.705+0000 7f9c5ca00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c581049b0 0x7f9c5819abf0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00ebc0 tx=0x7f9c4c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.705+0000 7f9c55600640 1 -- 172.21.15.1:0/1926290276 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c003b70 con 0x7f9c581049b0 2024-04-26T21:27:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.705+0000 7f9c55600640 1 -- 172.21.15.1:0/1926290276 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9c4c003d10 con 0x7f9c581049b0 2024-04-26T21:27:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.705+0000 7f9c55600640 1 -- 172.21.15.1:0/1926290276 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c019800 con 0x7f9c581049b0 2024-04-26T21:27:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.705+0000 7f9c5de00640 1 -- 172.21.15.1:0/1926290276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c581a2b00 con 0x7f9c581049b0 2024-04-26T21:27:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.706+0000 7f9c5de00640 1 -- 172.21.15.1:0/1926290276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c581a3000 con 0x7f9c581049b0 2024-04-26T21:27:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.707+0000 7f9c5de00640 1 -- 172.21.15.1:0/1926290276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c5810bd50 con 0x7f9c581049b0 2024-04-26T21:27:25.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:25 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3884365909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.709+0000 7f9c55600640 1 -- 172.21.15.1:0/1926290276 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f9c4c0199a0 con 0x7f9c581049b0 2024-04-26T21:27:25.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.711+0000 7f9c55600640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9c400a3c30 0x7f9c400a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:25.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.711+0000 7f9c55600640 1 -- 172.21.15.1:0/1926290276 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f9c4c0d0ff0 con 0x7f9c581049b0 2024-04-26T21:27:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.711+0000 7f9c57e00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9c400a3c30 0x7f9c400a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:25.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.712+0000 7f9c57e00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9c400a3c30 0x7f9c400a60f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9c5819c100 tx=0x7f9c440314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:25.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.716+0000 7f9c55600640 1 -- 172.21.15.1:0/1926290276 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9c4c09d730 con 0x7f9c581049b0 2024-04-26T21:27:25.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:25 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3884365909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:25.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:25.975+0000 7f9c5de00640 1 -- 172.21.15.1:0/1926290276 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9c5802dad0 con 0x7f9c581049b0 2024-04-26T21:27:26.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.415+0000 7f9c55600640 1 -- 172.21.15.1:0/1926290276 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f9c4c09d030 con 0x7f9c581049b0 2024-04-26T21:27:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.419+0000 7f9c5de00640 1 -- 172.21.15.1:0/1926290276 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9c400a3c30 msgr2=0x7f9c400a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.419+0000 7f9c5de00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9c400a3c30 0x7f9c400a60f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9c5819c100 tx=0x7f9c440314a0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.419+0000 7f9c5de00640 1 -- 172.21.15.1:0/1926290276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c581049b0 msgr2=0x7f9c5819abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.419+0000 7f9c5de00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c581049b0 0x7f9c5819abf0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c00ebc0 tx=0x7f9c4c00c6a0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.419+0000 7f9c5de00640 1 -- 172.21.15.1:0/1926290276 shutdown_connections 2024-04-26T21:27:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.419+0000 7f9c5de00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9c400a3c30 0x7f9c400a60f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.419+0000 7f9c5de00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c58106560 0x7f9c581a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.419+0000 7f9c5de00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c58105ba0 0x7f9c5819b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.419+0000 7f9c5de00640 1 --2- 172.21.15.1:0/1926290276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c581049b0 0x7f9c5819abf0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.419+0000 7f9c5de00640 1 -- 172.21.15.1:0/1926290276 >> 172.21.15.1:0/1926290276 conn(0x7f9c580fff40 msgr2=0x7f9c58108f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.420+0000 7f9c5de00640 1 -- 172.21.15.1:0/1926290276 shutdown_connections 2024-04-26T21:27:26.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.420+0000 7f9c5de00640 1 -- 172.21.15.1:0/1926290276 wait complete. 2024-04-26T21:27:26.446 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-26T21:27:26.446 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-26T21:27:26.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.664+0000 7f62ae400640 1 -- 172.21.15.1:0/4095135517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a81027c0 msgr2=0x7f62a8102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:26.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.664+0000 7f62ae400640 1 --2- 172.21.15.1:0/4095135517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a81027c0 0x7f62a8102be0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6298009860 tx=0x7f629802a340 comp rx=0 tx=0).stop 2024-04-26T21:27:26.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.664+0000 7f62ae400640 1 -- 172.21.15.1:0/4095135517 shutdown_connections 2024-04-26T21:27:26.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.664+0000 7f62ae400640 1 --2- 172.21.15.1:0/4095135517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62a8104370 0x7f62a810ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.664+0000 7f62ae400640 1 --2- 172.21.15.1:0/4095135517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62a81039b0 0x7f62a8103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.664+0000 7f62ae400640 1 --2- 172.21.15.1:0/4095135517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a81027c0 0x7f62a8102be0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.664+0000 7f62ae400640 1 -- 172.21.15.1:0/4095135517 >> 172.21.15.1:0/4095135517 conn(0x7f62a80fdd30 msgr2=0x7f62a8100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:26.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.665+0000 7f62ae400640 1 -- 172.21.15.1:0/4095135517 shutdown_connections 2024-04-26T21:27:26.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.665+0000 7f62ae400640 1 -- 172.21.15.1:0/4095135517 wait complete. 2024-04-26T21:27:26.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.666+0000 7f62ae400640 1 Processor -- start 2024-04-26T21:27:26.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.666+0000 7f62ae400640 1 -- start start 2024-04-26T21:27:26.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.666+0000 7f62ae400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62a81027c0 0x7f62a819acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:26.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.666+0000 7f62ae400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62a81039b0 0x7f62a819b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:26.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f62ae400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8104370 0x7f62a81a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:26.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f62ae400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f62a810e2f0 con 0x7f62a81039b0 2024-04-26T21:27:26.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f62ae400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f62a810e170 con 0x7f62a8104370 2024-04-26T21:27:26.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f62ae400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f62a810e470 con 0x7f62a81027c0 2024-04-26T21:27:26.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f629fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62a81039b0 0x7f62a819b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:26.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f629fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62a81039b0 0x7f62a819b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39268/0 (socket says 172.21.15.1:39268) 2024-04-26T21:27:26.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f629fe00640 1 -- 172.21.15.1:0/952302542 learned_addr learned my addr 172.21.15.1:0/952302542 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:26.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f62ad000640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62a81027c0 0x7f62a819acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:26.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f62ada00640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8104370 0x7f62a81a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:26.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f629fe00640 1 -- 172.21.15.1:0/952302542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62a81027c0 msgr2=0x7f62a819acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:26.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f629fe00640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62a81027c0 0x7f62a819acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f629fe00640 1 -- 172.21.15.1:0/952302542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8104370 msgr2=0x7f62a81a21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f629fe00640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8104370 0x7f62a81a21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f629fe00640 1 -- 172.21.15.1:0/952302542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a81a2890 con 0x7f62a81039b0 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f62ad000640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62a81027c0 0x7f62a819acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.667+0000 7f62ada00640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8104370 0x7f62a81a21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.668+0000 7f629fe00640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62a81039b0 0x7f62a819b210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f629400e9c0 tx=0x7f629400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.668+0000 7f62a6000640 1 -- 172.21.15.1:0/952302542 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629400cde0 con 0x7f62a81039b0 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.668+0000 7f62a6000640 1 -- 172.21.15.1:0/952302542 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62940035f0 con 0x7f62a81039b0 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.668+0000 7f62ae400640 1 -- 172.21.15.1:0/952302542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62a81a2b80 con 0x7f62a81039b0 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.668+0000 7f62a6000640 1 -- 172.21.15.1:0/952302542 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294010690 con 0x7f62a81039b0 2024-04-26T21:27:26.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.668+0000 7f62ae400640 1 -- 172.21.15.1:0/952302542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62a81a3120 con 0x7f62a81039b0 2024-04-26T21:27:26.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.670+0000 7f62ae400640 1 -- 172.21.15.1:0/952302542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62a81a3350 con 0x7f62a81039b0 2024-04-26T21:27:26.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.673+0000 7f62a6000640 1 -- 172.21.15.1:0/952302542 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f6294004000 con 0x7f62a81039b0 2024-04-26T21:27:26.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.674+0000 7f62a6000640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f628c0a3c30 0x7f628c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:26.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.674+0000 7f62a6000640 1 -- 172.21.15.1:0/952302542 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f62940d09c0 con 0x7f62a81039b0 2024-04-26T21:27:26.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.674+0000 7f62ad000640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f628c0a3c30 0x7f628c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:26.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.675+0000 7f62ad000640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f628c0a3c30 0x7f628c0a60f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f62a819c150 tx=0x7f6298030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:26.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.679+0000 7f62a6000640 1 -- 172.21.15.1:0/952302542 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f6294009510 con 0x7f62a81039b0 2024-04-26T21:27:26.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:26 smithi186 ceph-mon[32991]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:26.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:26 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1443351980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:26.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:26 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1926290276' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T21:27:26.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:26 smithi001 ceph-mon[28054]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:26.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:26 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1443351980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:26.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:26 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1926290276' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T21:27:26.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:26 smithi038 ceph-mon[32724]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:26.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:26 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1443351980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:26.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:26 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1926290276' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-26T21:27:26.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:26.933+0000 7f62ae400640 1 -- 172.21.15.1:0/952302542 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f62a81a3520 con 0x7f62a81039b0 2024-04-26T21:27:27.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.438+0000 7f62a6000640 1 -- 172.21.15.1:0/952302542 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f629409d100 con 0x7f62a81039b0 2024-04-26T21:27:27.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.442+0000 7f62ae400640 1 -- 172.21.15.1:0/952302542 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f628c0a3c30 msgr2=0x7f628c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:27.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.442+0000 7f62ae400640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f628c0a3c30 0x7f628c0a60f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f62a819c150 tx=0x7f6298030450 comp rx=0 tx=0).stop 2024-04-26T21:27:27.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.442+0000 7f62ae400640 1 -- 172.21.15.1:0/952302542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62a81039b0 msgr2=0x7f62a819b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:27.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.442+0000 7f62ae400640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62a81039b0 0x7f62a819b210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f629400e9c0 tx=0x7f629400ee80 comp rx=0 tx=0).stop 2024-04-26T21:27:27.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.442+0000 7f62ae400640 1 -- 172.21.15.1:0/952302542 shutdown_connections 2024-04-26T21:27:27.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.442+0000 7f62ae400640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f628c0a3c30 0x7f628c0a60f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:27.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.442+0000 7f62ae400640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8104370 0x7f62a81a21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:27.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.442+0000 7f62ae400640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62a81039b0 0x7f62a819b210 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:27.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.442+0000 7f62ae400640 1 --2- 172.21.15.1:0/952302542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62a81027c0 0x7f62a819acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:27.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.442+0000 7f62ae400640 1 -- 172.21.15.1:0/952302542 >> 172.21.15.1:0/952302542 conn(0x7f62a80fdd30 msgr2=0x7f62a8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:27.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.443+0000 7f62ae400640 1 -- 172.21.15.1:0/952302542 shutdown_connections 2024-04-26T21:27:27.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.443+0000 7f62ae400640 1 -- 172.21.15.1:0/952302542 wait complete. 2024-04-26T21:27:27.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-26T21:27:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.664+0000 7f8fffa00640 1 -- 172.21.15.1:0/1817403555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff8106740 msgr2=0x7f8ff810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.664+0000 7f8fffa00640 1 --2- 172.21.15.1:0/1817403555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff8106740 0x7f8ff810cfd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009ca0 tx=0x7f8ff002a8c0 comp rx=0 tx=0).stop 2024-04-26T21:27:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.664+0000 7f8fffa00640 1 -- 172.21.15.1:0/1817403555 shutdown_connections 2024-04-26T21:27:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.664+0000 7f8fffa00640 1 --2- 172.21.15.1:0/1817403555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ff8105d80 0x7f8ff8106200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.664+0000 7f8fffa00640 1 --2- 172.21.15.1:0/1817403555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff8106740 0x7f8ff810cfd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.664+0000 7f8fffa00640 1 --2- 172.21.15.1:0/1817403555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ff8104b90 0x7f8ff8104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:27.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.664+0000 7f8fffa00640 1 -- 172.21.15.1:0/1817403555 >> 172.21.15.1:0/1817403555 conn(0x7f8ff8100120 msgr2=0x7f8ff8102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.664+0000 7f8fffa00640 1 -- 172.21.15.1:0/1817403555 shutdown_connections 2024-04-26T21:27:27.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.664+0000 7f8fffa00640 1 -- 172.21.15.1:0/1817403555 wait complete. 2024-04-26T21:27:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.665+0000 7f8fffa00640 1 Processor -- start 2024-04-26T21:27:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.665+0000 7f8fffa00640 1 -- start start 2024-04-26T21:27:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8fffa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff8104b90 0x7f8ff819ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8fffa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ff8105d80 0x7f8ff819b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8fffa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ff8106740 0x7f8ff81a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8fffa00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff81106f0 con 0x7f8ff8104b90 2024-04-26T21:27:27.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8fffa00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8110570 con 0x7f8ff8106740 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8fffa00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff8110870 con 0x7f8ff8105d80 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8ffd000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff8104b90 0x7f8ff819ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8ff7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ff8105d80 0x7f8ff819b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8ff7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ff8105d80 0x7f8ff819b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46850/0 (socket says 172.21.15.1:46850) 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8ff7e00640 1 -- 172.21.15.1:0/3102977587 learned_addr learned my addr 172.21.15.1:0/3102977587 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.666+0000 7f8ffda00640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ff8106740 0x7f8ff81a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8ff7e00640 1 -- 172.21.15.1:0/3102977587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ff8106740 msgr2=0x7f8ff81a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8ff7e00640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ff8106740 0x7f8ff81a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8ff7e00640 1 -- 172.21.15.1:0/3102977587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff8104b90 msgr2=0x7f8ff819ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8ff7e00640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff8104b90 0x7f8ff819ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8ff7e00640 1 -- 172.21.15.1:0/3102977587 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff81a2980 con 0x7f8ff8105d80 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8ffd000640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff8104b90 0x7f8ff819ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:27.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8ff7e00640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ff8105d80 0x7f8ff819b2a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400d950 tx=0x7f8fe400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8ff5600640 1 -- 172.21.15.1:0/3102977587 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe4017070 con 0x7f8ff8105d80 2024-04-26T21:27:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8ff5600640 1 -- 172.21.15.1:0/3102977587 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fe4009c30 con 0x7f8ff8105d80 2024-04-26T21:27:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8fffa00640 1 -- 172.21.15.1:0/3102977587 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff81a2c70 con 0x7f8ff8105d80 2024-04-26T21:27:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8ff5600640 1 -- 172.21.15.1:0/3102977587 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe400f620 con 0x7f8ff8105d80 2024-04-26T21:27:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.667+0000 7f8fffa00640 1 -- 172.21.15.1:0/3102977587 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff81a31c0 con 0x7f8ff8105d80 2024-04-26T21:27:27.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.669+0000 7f8fffa00640 1 -- 172.21.15.1:0/3102977587 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff810beb0 con 0x7f8ff8105d80 2024-04-26T21:27:27.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.671+0000 7f8ff5600640 1 -- 172.21.15.1:0/3102977587 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f8fe4010460 con 0x7f8ff8105d80 2024-04-26T21:27:27.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.672+0000 7f8ff5600640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8fe00a3bb0 0x7f8fe00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:27.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.672+0000 7f8ff5600640 1 -- 172.21.15.1:0/3102977587 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f8fe40d0190 con 0x7f8ff8105d80 2024-04-26T21:27:27.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.672+0000 7f8ffd000640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8fe00a3bb0 0x7f8fe00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:27.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.673+0000 7f8ffd000640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8fe00a3bb0 0x7f8fe00a6070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8fec005fd0 tx=0x7f8fec00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:27.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.676+0000 7f8ff5600640 1 -- 172.21.15.1:0/3102977587 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8fe4009de0 con 0x7f8ff8105d80 2024-04-26T21:27:27.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:27 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1926290276' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T21:27:27.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:27 smithi186 ceph-mon[32991]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:27.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:27 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/952302542' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T21:27:27.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:27 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1926290276' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T21:27:27.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:27 smithi001 ceph-mon[28054]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:27.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:27 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/952302542' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T21:27:27.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:27 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1926290276' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-26T21:27:27.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:27 smithi038 ceph-mon[32724]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:27.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:27 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/952302542' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-26T21:27:27.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:27.927+0000 7f8fffa00640 1 -- 172.21.15.1:0/3102977587 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f8ff802dad0 con 0x7f8ff8105d80 2024-04-26T21:27:28.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.455+0000 7f8ff5600640 1 -- 172.21.15.1:0/3102977587 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f8fe409c280 con 0x7f8ff8105d80 2024-04-26T21:27:28.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.458+0000 7f8fffa00640 1 -- 172.21.15.1:0/3102977587 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8fe00a3bb0 msgr2=0x7f8fe00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8fe00a3bb0 0x7f8fe00a6070 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8fec005fd0 tx=0x7f8fec00b410 comp rx=0 tx=0).stop 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 -- 172.21.15.1:0/3102977587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ff8105d80 msgr2=0x7f8ff819b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ff8105d80 0x7f8ff819b2a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400d950 tx=0x7f8fe400de10 comp rx=0 tx=0).stop 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 -- 172.21.15.1:0/3102977587 shutdown_connections 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8fe00a3bb0 0x7f8fe00a6070 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ff8105d80 0x7f8ff819b2a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8ff8106740 0x7f8ff81a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 --2- 172.21.15.1:0/3102977587 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ff8104b90 0x7f8ff819ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 -- 172.21.15.1:0/3102977587 >> 172.21.15.1:0/3102977587 conn(0x7f8ff8100120 msgr2=0x7f8ff8109080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 -- 172.21.15.1:0/3102977587 shutdown_connections 2024-04-26T21:27:28.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.459+0000 7f8fffa00640 1 -- 172.21.15.1:0/3102977587 wait complete. 2024-04-26T21:27:28.486 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-26T21:27:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.693+0000 7f06a8a00640 1 -- 172.21.15.1:0/2538908815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06a41027b0 msgr2=0x7f06a4102bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.693+0000 7f06a8a00640 1 --2- 172.21.15.1:0/2538908815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06a41027b0 0x7f06a4102bd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0694009900 tx=0x7f069402a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.693+0000 7f06a8a00640 1 -- 172.21.15.1:0/2538908815 shutdown_connections 2024-04-26T21:27:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.693+0000 7f06a8a00640 1 --2- 172.21.15.1:0/2538908815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06a4104360 0x7f06a410abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.693+0000 7f06a8a00640 1 --2- 172.21.15.1:0/2538908815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a41039a0 0x7f06a4103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.693+0000 7f06a8a00640 1 --2- 172.21.15.1:0/2538908815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06a41027b0 0x7f06a4102bd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.693+0000 7f06a8a00640 1 -- 172.21.15.1:0/2538908815 >> 172.21.15.1:0/2538908815 conn(0x7f06a40fdd40 msgr2=0x7f06a4100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.693+0000 7f06a8a00640 1 -- 172.21.15.1:0/2538908815 shutdown_connections 2024-04-26T21:27:28.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.693+0000 7f06a8a00640 1 -- 172.21.15.1:0/2538908815 wait complete. 2024-04-26T21:27:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.695+0000 7f06a8a00640 1 Processor -- start 2024-04-26T21:27:28.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.695+0000 7f06a8a00640 1 -- start start 2024-04-26T21:27:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.695+0000 7f06a8a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06a41027b0 0x7f06a419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.695+0000 7f06a8a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a41039a0 0x7f06a419b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.695+0000 7f06a8a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06a4104360 0x7f06a41a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.695+0000 7f06a8a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f06a410e2b0 con 0x7f06a41039a0 2024-04-26T21:27:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.695+0000 7f06a8a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f06a410e130 con 0x7f06a41027b0 2024-04-26T21:27:28.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a8a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f06a410e430 con 0x7f06a4104360 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a2800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06a41027b0 0x7f06a419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a2800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06a41027b0 0x7f06a419acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:46682/0 (socket says 172.21.15.1:46682) 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a2800640 1 -- 172.21.15.1:0/3888346672 learned_addr learned my addr 172.21.15.1:0/3888346672 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f0699e00640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a41039a0 0x7f06a419b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a3200640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06a4104360 0x7f06a41a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a2800640 1 -- 172.21.15.1:0/3888346672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06a4104360 msgr2=0x7f06a41a21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a2800640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06a4104360 0x7f06a41a21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a2800640 1 -- 172.21.15.1:0/3888346672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a41039a0 msgr2=0x7f06a419b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a2800640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a41039a0 0x7f06a419b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a2800640 1 -- 172.21.15.1:0/3888346672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a41a2880 con 0x7f06a41027b0 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.696+0000 7f06a2800640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06a41027b0 0x7f06a419acc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0694006010 tx=0x7f06940042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.697+0000 7f069be00640 1 -- 172.21.15.1:0/3888346672 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069403f030 con 0x7f06a41027b0 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.697+0000 7f069be00640 1 -- 172.21.15.1:0/3888346672 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06940318f0 con 0x7f06a41027b0 2024-04-26T21:27:28.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.697+0000 7f06a8a00640 1 -- 172.21.15.1:0/3888346672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a41a2b10 con 0x7f06a41027b0 2024-04-26T21:27:28.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.697+0000 7f069be00640 1 -- 172.21.15.1:0/3888346672 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0694035070 con 0x7f06a41027b0 2024-04-26T21:27:28.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.697+0000 7f06a8a00640 1 -- 172.21.15.1:0/3888346672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a41a3000 con 0x7f06a41027b0 2024-04-26T21:27:28.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.698+0000 7f06a8a00640 1 -- 172.21.15.1:0/3888346672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a41a31d0 con 0x7f06a41027b0 2024-04-26T21:27:28.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.700+0000 7f069be00640 1 -- 172.21.15.1:0/3888346672 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f0694042030 con 0x7f06a41027b0 2024-04-26T21:27:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.701+0000 7f069be00640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f06880a3bb0 0x7f06880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:28.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.701+0000 7f069be00640 1 -- 172.21.15.1:0/3888346672 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f06940f0dd0 con 0x7f06a41027b0 2024-04-26T21:27:28.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.701+0000 7f0699e00640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f06880a3bb0 0x7f06880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:28.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.702+0000 7f0699e00640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f06880a3bb0 0x7f06880a6070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f068c005fd0 tx=0x7f068c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:28.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:28 smithi186 ceph-mon[32991]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:28.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:28 smithi186 ceph-mon[32991]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T21:27:28.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:28 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/952302542' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T21:27:28.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:28 smithi186 ceph-mon[32991]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T21:27:28.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:28 smithi186 ceph-mon[32991]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:28.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:28 smithi186 ceph-mon[32991]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-26T21:27:28.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:28 smithi186 ceph-mon[32991]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T21:27:28.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:28 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3102977587' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T21:27:28.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:28 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T21:27:28.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.706+0000 7f069be00640 1 -- 172.21.15.1:0/3888346672 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f06940bd510 con 0x7f06a41027b0 2024-04-26T21:27:28.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:28 smithi001 ceph-mon[28054]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:28.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:28 smithi001 ceph-mon[28054]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T21:27:28.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:28 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/952302542' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T21:27:28.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:28 smithi001 ceph-mon[28054]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T21:27:28.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:28 smithi001 ceph-mon[28054]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:28.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:28 smithi001 ceph-mon[28054]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-26T21:27:28.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:28 smithi001 ceph-mon[28054]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T21:27:28.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:28 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3102977587' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T21:27:28.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:28 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T21:27:28.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:28 smithi038 ceph-mon[32724]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-26T21:27:28.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:28 smithi038 ceph-mon[32724]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-26T21:27:28.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:28 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/952302542' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-26T21:27:28.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:28 smithi038 ceph-mon[32724]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-26T21:27:28.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:28 smithi038 ceph-mon[32724]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-26T21:27:28.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:28 smithi038 ceph-mon[32724]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-26T21:27:28.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:28 smithi038 ceph-mon[32724]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-26T21:27:28.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:28 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3102977587' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T21:27:28.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:28 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-26T21:27:28.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:28.955+0000 7f06a8a00640 1 -- 172.21.15.1:0/3888346672 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f06a41a38a0 con 0x7f06a41027b0 2024-04-26T21:27:29.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.689+0000 7f069be00640 1 -- 172.21.15.1:0/3888346672 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f06940bce10 con 0x7f06a41027b0 2024-04-26T21:27:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.693+0000 7f06a8a00640 1 -- 172.21.15.1:0/3888346672 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f06880a3bb0 msgr2=0x7f06880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.693+0000 7f06a8a00640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f06880a3bb0 0x7f06880a6070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f068c005fd0 tx=0x7f068c005850 comp rx=0 tx=0).stop 2024-04-26T21:27:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.693+0000 7f06a8a00640 1 -- 172.21.15.1:0/3888346672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06a41027b0 msgr2=0x7f06a419acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.693+0000 7f06a8a00640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06a41027b0 0x7f06a419acc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0694006010 tx=0x7f06940042f0 comp rx=0 tx=0).stop 2024-04-26T21:27:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.693+0000 7f06a8a00640 1 -- 172.21.15.1:0/3888346672 shutdown_connections 2024-04-26T21:27:29.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.693+0000 7f06a8a00640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f06880a3bb0 0x7f06880a6070 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:29.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.693+0000 7f06a8a00640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06a4104360 0x7f06a41a21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:29.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.693+0000 7f06a8a00640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06a41039a0 0x7f06a419b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:29.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.693+0000 7f06a8a00640 1 --2- 172.21.15.1:0/3888346672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06a41027b0 0x7f06a419acc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:29.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.693+0000 7f06a8a00640 1 -- 172.21.15.1:0/3888346672 >> 172.21.15.1:0/3888346672 conn(0x7f06a40fdd40 msgr2=0x7f06a4074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:29.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.694+0000 7f06a8a00640 1 -- 172.21.15.1:0/3888346672 shutdown_connections 2024-04-26T21:27:29.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.694+0000 7f06a8a00640 1 -- 172.21.15.1:0/3888346672 wait complete. 2024-04-26T21:27:29.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T21:27:29.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] up:active 2024-04-26T21:27:29.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T21:27:29.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T21:27:29.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: Cluster is now healthy 2024-04-26T21:27:29.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-26T21:27:29.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:creating} 9 up:standby 2024-04-26T21:27:29.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-26T21:27:29.715 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] up:active 2024-04-26T21:27:29.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:27:29.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3888346672' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T21:27:29.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:29 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T21:27:29.718 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-26T21:27:29.718 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] up:active 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: Cluster is now healthy 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:creating} 9 up:standby 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] up:active 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:27:29.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3888346672' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T21:27:29.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:29 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T21:27:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.938+0000 7fc881800640 1 -- 172.21.15.1:0/2874505963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87c10ab40 msgr2=0x7fc87c1113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.938+0000 7fc881800640 1 --2- 172.21.15.1:0/2874505963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87c10ab40 0x7fc87c1113d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc874009900 tx=0x7fc87402a410 comp rx=0 tx=0).stop 2024-04-26T21:27:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.938+0000 7fc881800640 1 -- 172.21.15.1:0/2874505963 shutdown_connections 2024-04-26T21:27:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.938+0000 7fc881800640 1 --2- 172.21.15.1:0/2874505963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87c10ab40 0x7fc87c1113d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.938+0000 7fc881800640 1 --2- 172.21.15.1:0/2874505963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc87c10a180 0x7fc87c10a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.938+0000 7fc881800640 1 --2- 172.21.15.1:0/2874505963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc87c108f90 0x7fc87c1093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.938+0000 7fc881800640 1 -- 172.21.15.1:0/2874505963 >> 172.21.15.1:0/2874505963 conn(0x7fc87c077f40 msgr2=0x7fc87c07a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.938+0000 7fc881800640 1 -- 172.21.15.1:0/2874505963 shutdown_connections 2024-04-26T21:27:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.938+0000 7fc881800640 1 -- 172.21.15.1:0/2874505963 wait complete. 2024-04-26T21:27:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.939+0000 7fc881800640 1 Processor -- start 2024-04-26T21:27:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.939+0000 7fc881800640 1 -- start start 2024-04-26T21:27:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.939+0000 7fc881800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87c108f90 0x7fc87c19f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.939+0000 7fc881800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc87c10a180 0x7fc87c19f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.939+0000 7fc881800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc87c10ab40 0x7fc87c1a6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.939+0000 7fc881800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c114b20 con 0x7fc87c108f90 2024-04-26T21:27:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.939+0000 7fc881800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c1149a0 con 0x7fc87c10a180 2024-04-26T21:27:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.939+0000 7fc881800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c114ca0 con 0x7fc87c10ab40 2024-04-26T21:27:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87c108f90 0x7fc87c19f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc87c10a180 0x7fc87c19f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc87c10a180 0x7fc87c19f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:46696/0 (socket says 172.21.15.1:46696) 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87a000640 1 -- 172.21.15.1:0/2699162307 learned_addr learned my addr 172.21.15.1:0/2699162307 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87c108f90 0x7fc87c19f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41072/0 (socket says 172.21.15.1:41072) 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87b400640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc87c10ab40 0x7fc87c1a6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87a000640 1 -- 172.21.15.1:0/2699162307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc87c10ab40 msgr2=0x7fc87c1a6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87a000640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc87c10ab40 0x7fc87c1a6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87a000640 1 -- 172.21.15.1:0/2699162307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87c108f90 msgr2=0x7fc87c19f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87a000640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87c108f90 0x7fc87c19f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87a000640 1 -- 172.21.15.1:0/2699162307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87c1a6da0 con 0x7fc87c10a180 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87b400640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc87c10ab40 0x7fc87c1a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.940+0000 7fc87a000640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc87c10a180 0x7fc87c19f6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc86800ea60 tx=0x7fc86800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.941+0000 7fc873400640 1 -- 172.21.15.1:0/2699162307 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86800cde0 con 0x7fc87c10a180 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.941+0000 7fc873400640 1 -- 172.21.15.1:0/2699162307 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8680035f0 con 0x7fc87c10a180 2024-04-26T21:27:29.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.941+0000 7fc873400640 1 -- 172.21.15.1:0/2699162307 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc868010690 con 0x7fc87c10a180 2024-04-26T21:27:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.941+0000 7fc881800640 1 -- 172.21.15.1:0/2699162307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87c1a7090 con 0x7fc87c10a180 2024-04-26T21:27:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.941+0000 7fc881800640 1 -- 172.21.15.1:0/2699162307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc87c1a7590 con 0x7fc87c10a180 2024-04-26T21:27:29.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.942+0000 7fc881800640 1 -- 172.21.15.1:0/2699162307 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc87c1102b0 con 0x7fc87c10a180 2024-04-26T21:27:29.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.945+0000 7fc873400640 1 -- 172.21.15.1:0/2699162307 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fc8680037a0 con 0x7fc87c10a180 2024-04-26T21:27:29.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.946+0000 7fc873400640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc8600a3c30 0x7fc8600a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:29.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.946+0000 7fc873400640 1 -- 172.21.15.1:0/2699162307 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fc8680d0960 con 0x7fc87c10a180 2024-04-26T21:27:29.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.946+0000 7fc87aa00640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc8600a3c30 0x7fc8600a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:29.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.947+0000 7fc87aa00640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc8600a3c30 0x7fc8600a60f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc87c109fe0 tx=0x7fc86c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:29.950+0000 7fc873400640 1 -- 172.21.15.1:0/2699162307 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc86809d0a0 con 0x7fc87c10a180 2024-04-26T21:27:29.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-26T21:27:29.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] up:active 2024-04-26T21:27:29.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-26T21:27:29.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-26T21:27:29.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: Cluster is now healthy 2024-04-26T21:27:29.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-26T21:27:29.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:creating} 9 up:standby 2024-04-26T21:27:29.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-04-26T21:27:29.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] up:active 2024-04-26T21:27:29.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:27:29.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3888346672' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T21:27:29.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:29 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-26T21:27:30.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.179+0000 7fc881800640 1 -- 172.21.15.1:0/2699162307 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc87c062d70 con 0x7fc8600a3c30 2024-04-26T21:27:30.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.204+0000 7fc873400640 1 -- 172.21.15.1:0/2699162307 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc87c062d70 con 0x7fc8600a3c30 2024-04-26T21:27:30.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 -- 172.21.15.1:0/2699162307 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc8600a3c30 msgr2=0x7fc8600a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc8600a3c30 0x7fc8600a60f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc87c109fe0 tx=0x7fc86c009500 comp rx=0 tx=0).stop 2024-04-26T21:27:30.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 -- 172.21.15.1:0/2699162307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc87c10a180 msgr2=0x7fc87c19f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc87c10a180 0x7fc87c19f6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc86800ea60 tx=0x7fc86800ef20 comp rx=0 tx=0).stop 2024-04-26T21:27:30.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 -- 172.21.15.1:0/2699162307 shutdown_connections 2024-04-26T21:27:30.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc8600a3c30 0x7fc8600a60f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc87c10ab40 0x7fc87c1a6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc87c10a180 0x7fc87c19f6c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 --2- 172.21.15.1:0/2699162307 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc87c108f90 0x7fc87c19f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 -- 172.21.15.1:0/2699162307 >> 172.21.15.1:0/2699162307 conn(0x7fc87c077f40 msgr2=0x7fc87c10d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:30.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 -- 172.21.15.1:0/2699162307 shutdown_connections 2024-04-26T21:27:30.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.206+0000 7fc881800640 1 -- 172.21.15.1:0/2699162307 wait complete. 2024-04-26T21:27:30.224 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-26T21:27:30.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.424+0000 7f2e3ee00640 1 -- 172.21.15.1:0/4021176396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e38103220 msgr2=0x7f2e38105610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.424+0000 7f2e3ee00640 1 --2- 172.21.15.1:0/4021176396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e38103220 0x7f2e38105610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2e28009900 tx=0x7f2e2802a410 comp rx=0 tx=0).stop 2024-04-26T21:27:30.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.424+0000 7f2e3ee00640 1 -- 172.21.15.1:0/4021176396 shutdown_connections 2024-04-26T21:27:30.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.424+0000 7f2e3ee00640 1 --2- 172.21.15.1:0/4021176396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e38108480 0x7f2e3810a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.424+0000 7f2e3ee00640 1 --2- 172.21.15.1:0/4021176396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38105b50 0x7f2e38107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.424+0000 7f2e3ee00640 1 --2- 172.21.15.1:0/4021176396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e38103220 0x7f2e38105610 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.424+0000 7f2e3ee00640 1 -- 172.21.15.1:0/4021176396 >> 172.21.15.1:0/4021176396 conn(0x7f2e380fcc10 msgr2=0x7f2e380ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:30.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.424+0000 7f2e3ee00640 1 -- 172.21.15.1:0/4021176396 shutdown_connections 2024-04-26T21:27:30.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.424+0000 7f2e3ee00640 1 -- 172.21.15.1:0/4021176396 wait complete. 2024-04-26T21:27:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.425+0000 7f2e3ee00640 1 Processor -- start 2024-04-26T21:27:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.425+0000 7f2e3ee00640 1 -- start start 2024-04-26T21:27:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3ee00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38103220 0x7f2e38102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3ee00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e38105b50 0x7f2e381011d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3ee00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e38108480 0x7f2e38101710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:30.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3ee00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3810e340 con 0x7f2e38103220 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3ee00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3810e1c0 con 0x7f2e38108480 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3ee00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3810e4c0 con 0x7f2e38105b50 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3e400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e38108480 0x7f2e38101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3da00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38103220 0x7f2e38102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3e400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e38108480 0x7f2e38101710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:46724/0 (socket says 172.21.15.1:46724) 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3e400640 1 -- 172.21.15.1:0/2224420594 learned_addr learned my addr 172.21.15.1:0/2224420594 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.426+0000 7f2e3d000640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e38105b50 0x7f2e381011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e3e400640 1 -- 172.21.15.1:0/2224420594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e38105b50 msgr2=0x7f2e381011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e3e400640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e38105b50 0x7f2e381011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e3e400640 1 -- 172.21.15.1:0/2224420594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38103220 msgr2=0x7f2e38102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e3e400640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38103220 0x7f2e38102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e3e400640 1 -- 172.21.15.1:0/2224420594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e38101fa0 con 0x7f2e38108480 2024-04-26T21:27:30.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e3e400640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e38108480 0x7f2e38101710 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000eaa0 tx=0x7f2e3000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:30.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e36000640 1 -- 172.21.15.1:0/2224420594 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3000cdb0 con 0x7f2e38108480 2024-04-26T21:27:30.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e36000640 1 -- 172.21.15.1:0/2224420594 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e30005020 con 0x7f2e38108480 2024-04-26T21:27:30.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e36000640 1 -- 172.21.15.1:0/2224420594 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e300116d0 con 0x7f2e38108480 2024-04-26T21:27:30.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e3ee00640 1 -- 172.21.15.1:0/2224420594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e381a2b30 con 0x7f2e38108480 2024-04-26T21:27:30.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.427+0000 7f2e3ee00640 1 -- 172.21.15.1:0/2224420594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e381a3030 con 0x7f2e38108480 2024-04-26T21:27:30.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.428+0000 7f2e3ee00640 1 -- 172.21.15.1:0/2224420594 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e380fe700 con 0x7f2e38108480 2024-04-26T21:27:30.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.430+0000 7f2e36000640 1 -- 172.21.15.1:0/2224420594 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f2e300051d0 con 0x7f2e38108480 2024-04-26T21:27:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.431+0000 7f2e36000640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2e1c0a3bb0 0x7f2e1c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.432+0000 7f2e36000640 1 -- 172.21.15.1:0/2224420594 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f2e300d0740 con 0x7f2e38108480 2024-04-26T21:27:30.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.432+0000 7f2e3da00640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2e1c0a3bb0 0x7f2e1c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.432+0000 7f2e3da00640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2e1c0a3bb0 0x7f2e1c0a6070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2e28031500 tx=0x7f2e28005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.436+0000 7f2e36000640 1 -- 172.21.15.1:0/2224420594 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2e30016030 con 0x7f2e38108480 2024-04-26T21:27:30.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:27:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:27:30.187+0000 7fa16b200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:27:30.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:27:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:27:30.187+0000 7fa16b200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:27:30.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:27:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:27:30.187+0000 7fa16b200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:27:30.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:27:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:27:30.187+0000 7fa16b200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:27:30.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:27:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:27:30.187+0000 7fa16b200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:27:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.674+0000 7f2e3ee00640 1 -- 172.21.15.1:0/2224420594 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f2e38062e00 con 0x7f2e1c0a3bb0 2024-04-26T21:27:30.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.716+0000 7f2e36000640 1 -- 172.21.15.1:0/2224420594 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2e38062e00 con 0x7f2e1c0a3bb0 2024-04-26T21:27:30.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.719+0000 7f2e3ee00640 1 -- 172.21.15.1:0/2224420594 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2e1c0a3bb0 msgr2=0x7f2e1c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.719+0000 7f2e3ee00640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2e1c0a3bb0 0x7f2e1c0a6070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2e28031500 tx=0x7f2e28005a20 comp rx=0 tx=0).stop 2024-04-26T21:27:30.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.719+0000 7f2e3ee00640 1 -- 172.21.15.1:0/2224420594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e38108480 msgr2=0x7f2e38101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.719+0000 7f2e3ee00640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e38108480 0x7f2e38101710 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000eaa0 tx=0x7f2e3000ef60 comp rx=0 tx=0).stop 2024-04-26T21:27:30.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.719+0000 7f2e3ee00640 1 -- 172.21.15.1:0/2224420594 shutdown_connections 2024-04-26T21:27:30.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.719+0000 7f2e3ee00640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2e1c0a3bb0 0x7f2e1c0a6070 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.719+0000 7f2e3ee00640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e38105b50 0x7f2e381011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.719+0000 7f2e3ee00640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e38108480 0x7f2e38101710 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.719+0000 7f2e3ee00640 1 --2- 172.21.15.1:0/2224420594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e38103220 0x7f2e38102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.719+0000 7f2e3ee00640 1 -- 172.21.15.1:0/2224420594 >> 172.21.15.1:0/2224420594 conn(0x7f2e380fcc10 msgr2=0x7f2e38103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:30.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.720+0000 7f2e3ee00640 1 -- 172.21.15.1:0/2224420594 shutdown_connections 2024-04-26T21:27:30.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.720+0000 7f2e3ee00640 1 -- 172.21.15.1:0/2224420594 wait complete. 2024-04-26T21:27:30.740 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-26T21:27:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.953+0000 7f0a38e00640 1 -- 172.21.15.1:0/3071790306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a341049b0 msgr2=0x7f0a34104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.953+0000 7f0a38e00640 1 --2- 172.21.15.1:0/3071790306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a341049b0 0x7f0a34104dd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009900 tx=0x7f0a2402a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.953+0000 7f0a38e00640 1 -- 172.21.15.1:0/3071790306 shutdown_connections 2024-04-26T21:27:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.953+0000 7f0a38e00640 1 --2- 172.21.15.1:0/3071790306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a34106560 0x7f0a3410cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.953+0000 7f0a38e00640 1 --2- 172.21.15.1:0/3071790306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a34105ba0 0x7f0a34106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.953+0000 7f0a38e00640 1 --2- 172.21.15.1:0/3071790306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a341049b0 0x7f0a34104dd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.953+0000 7f0a38e00640 1 -- 172.21.15.1:0/3071790306 >> 172.21.15.1:0/3071790306 conn(0x7f0a340fff40 msgr2=0x7f0a34102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.953+0000 7f0a38e00640 1 -- 172.21.15.1:0/3071790306 shutdown_connections 2024-04-26T21:27:30.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.953+0000 7f0a38e00640 1 -- 172.21.15.1:0/3071790306 wait complete. 2024-04-26T21:27:30.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:30 smithi186 ceph-mon[32991]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-26T21:27:30.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:30 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T21:27:30.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:30 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:27:30.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.954+0000 7f0a38e00640 1 Processor -- start 2024-04-26T21:27:30.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.954+0000 7f0a38e00640 1 -- start start 2024-04-26T21:27:30.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a38e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a341049b0 0x7f0a3419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:30.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a38e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a34105ba0 0x7f0a3419b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a38e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a34106560 0x7f0a341a21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a38e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0a341104b0 con 0x7f0a34106560 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a38e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0a34110330 con 0x7f0a34105ba0 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a38e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0a34110630 con 0x7f0a341049b0 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a33400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a341049b0 0x7f0a3419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a32a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a34105ba0 0x7f0a3419b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a33400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a341049b0 0x7f0a3419ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:40872/0 (socket says 172.21.15.1:40872) 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a33400640 1 -- 172.21.15.1:0/3369511725 learned_addr learned my addr 172.21.15.1:0/3369511725 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.955+0000 7f0a33e00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a34106560 0x7f0a341a21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a32a00640 1 -- 172.21.15.1:0/3369511725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a341049b0 msgr2=0x7f0a3419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a32a00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a341049b0 0x7f0a3419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a32a00640 1 -- 172.21.15.1:0/3369511725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a34106560 msgr2=0x7f0a341a21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a32a00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a34106560 0x7f0a341a21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:30.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a32a00640 1 -- 172.21.15.1:0/3369511725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a341a2840 con 0x7f0a34105ba0 2024-04-26T21:27:30.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a32a00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a34105ba0 0x7f0a3419b170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000e9f0 tx=0x7f0a2000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:30.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a2be00640 1 -- 172.21.15.1:0/3369511725 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2000ccc0 con 0x7f0a34105ba0 2024-04-26T21:27:30.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a2be00640 1 -- 172.21.15.1:0/3369511725 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a200108e0 con 0x7f0a34105ba0 2024-04-26T21:27:30.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a2be00640 1 -- 172.21.15.1:0/3369511725 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20018880 con 0x7f0a34105ba0 2024-04-26T21:27:30.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a38e00640 1 -- 172.21.15.1:0/3369511725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a341a2b30 con 0x7f0a34105ba0 2024-04-26T21:27:30.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.956+0000 7f0a38e00640 1 -- 172.21.15.1:0/3369511725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a341a30d0 con 0x7f0a34105ba0 2024-04-26T21:27:30.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.957+0000 7f0a38e00640 1 -- 172.21.15.1:0/3369511725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a3410be40 con 0x7f0a34105ba0 2024-04-26T21:27:30.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.960+0000 7f0a2be00640 1 -- 172.21.15.1:0/3369511725 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7f0a20010430 con 0x7f0a34105ba0 2024-04-26T21:27:30.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.961+0000 7f0a2be00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0a180a3c30 0x7f0a180a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:30.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.961+0000 7f0a2be00640 1 -- 172.21.15.1:0/3369511725 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f0a200d09a0 con 0x7f0a34105ba0 2024-04-26T21:27:30.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.962+0000 7f0a33400640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0a180a3c30 0x7f0a180a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:30.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.963+0000 7f0a33400640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0a180a3c30 0x7f0a180a60f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0a24000c00 tx=0x7f0a240314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:30.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:30.966+0000 7f0a2be00640 1 -- 172.21.15.1:0/3369511725 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0a2009ca90 con 0x7f0a34105ba0 2024-04-26T21:27:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:30 smithi001 ceph-mon[28054]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-26T21:27:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:30 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T21:27:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:30 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:27:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:30 smithi038 ceph-mon[32724]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-04-26T21:27:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:30 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-26T21:27:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:30 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:27:31.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.195+0000 7f0a38e00640 1 -- 172.21.15.1:0/3369511725 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f0a34062e00 con 0x7f0a180a3c30 2024-04-26T21:27:31.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.205+0000 7f0a2be00640 1 -- 172.21.15.1:0/3369511725 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0a34062e00 con 0x7f0a180a3c30 2024-04-26T21:27:31.205 INFO:teuthology.orchestra.run.smithi001.stdout:/volumes/qa/sv_0/b60fb08b-ccfe-44f2-9991-49cd2d815af4 2024-04-26T21:27:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.207+0000 7f0a38e00640 1 -- 172.21.15.1:0/3369511725 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0a180a3c30 msgr2=0x7f0a180a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.207+0000 7f0a38e00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0a180a3c30 0x7f0a180a60f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0a24000c00 tx=0x7f0a240314a0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.207+0000 7f0a38e00640 1 -- 172.21.15.1:0/3369511725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a34105ba0 msgr2=0x7f0a3419b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.207+0000 7f0a38e00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a34105ba0 0x7f0a3419b170 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000e9f0 tx=0x7f0a2000eeb0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.208+0000 7f0a38e00640 1 -- 172.21.15.1:0/3369511725 shutdown_connections 2024-04-26T21:27:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.208+0000 7f0a38e00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0a180a3c30 0x7f0a180a60f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.208+0000 7f0a38e00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a34106560 0x7f0a341a21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.208+0000 7f0a38e00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a34105ba0 0x7f0a3419b170 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.208+0000 7f0a38e00640 1 --2- 172.21.15.1:0/3369511725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a341049b0 0x7f0a3419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.208+0000 7f0a38e00640 1 -- 172.21.15.1:0/3369511725 >> 172.21.15.1:0/3369511725 conn(0x7f0a340fff40 msgr2=0x7f0a34109010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.208+0000 7f0a38e00640 1 -- 172.21.15.1:0/3369511725 shutdown_connections 2024-04-26T21:27:31.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.208+0000 7f0a38e00640 1 -- 172.21.15.1:0/3369511725 wait complete. 2024-04-26T21:27:31.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-26T21:27:31.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.453+0000 7fa185400640 1 -- 172.21.15.1:0/4050856683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1800772f0 msgr2=0x7fa180077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:31.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.453+0000 7fa185400640 1 --2- 172.21.15.1:0/4050856683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1800772f0 0x7fa180077710 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa170009900 tx=0x7fa17002a400 comp rx=0 tx=0).stop 2024-04-26T21:27:31.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.453+0000 7fa185400640 1 -- 172.21.15.1:0/4050856683 shutdown_connections 2024-04-26T21:27:31.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.453+0000 7fa185400640 1 --2- 172.21.15.1:0/4050856683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa180078ed0 0x7fa180079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.453+0000 7fa185400640 1 --2- 172.21.15.1:0/4050856683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa180078530 0x7fa180078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.453+0000 7fa185400640 1 --2- 172.21.15.1:0/4050856683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa1800772f0 0x7fa180077710 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.453+0000 7fa185400640 1 -- 172.21.15.1:0/4050856683 >> 172.21.15.1:0/4050856683 conn(0x7fa1800fff40 msgr2=0x7fa180102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:31.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.453+0000 7fa185400640 1 -- 172.21.15.1:0/4050856683 shutdown_connections 2024-04-26T21:27:31.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.453+0000 7fa185400640 1 -- 172.21.15.1:0/4050856683 wait complete. 2024-04-26T21:27:31.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa185400640 1 Processor -- start 2024-04-26T21:27:31.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa185400640 1 -- start start 2024-04-26T21:27:31.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa185400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1800772f0 0x7fa18019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:31.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa185400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa180078530 0x7fa18019f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:31.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa185400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa180078ed0 0x7fa1801a65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:31.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa185400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa1801148f0 con 0x7fa180078ed0 2024-04-26T21:27:31.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa185400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa180114770 con 0x7fa180078530 2024-04-26T21:27:31.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa185400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa180114a70 con 0x7fa1800772f0 2024-04-26T21:27:31.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa184a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa180078ed0 0x7fa1801a65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa17f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa180078530 0x7fa18019f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa17fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1800772f0 0x7fa18019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa17f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa180078530 0x7fa18019f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:46758/0 (socket says 172.21.15.1:46758) 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa17f400640 1 -- 172.21.15.1:0/1744744647 learned_addr learned my addr 172.21.15.1:0/1744744647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.455+0000 7fa17fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1800772f0 0x7fa18019f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:40896/0 (socket says 172.21.15.1:40896) 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.456+0000 7fa17fe00640 1 -- 172.21.15.1:0/1744744647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa180078530 msgr2=0x7fa18019f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.456+0000 7fa17fe00640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa180078530 0x7fa18019f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.456+0000 7fa17fe00640 1 -- 172.21.15.1:0/1744744647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa180078ed0 msgr2=0x7fa1801a65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.456+0000 7fa17fe00640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa180078ed0 0x7fa1801a65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.456+0000 7fa17fe00640 1 -- 172.21.15.1:0/1744744647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1801a6c20 con 0x7fa1800772f0 2024-04-26T21:27:31.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.456+0000 7fa17fe00640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1800772f0 0x7fa18019f000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa170006010 tx=0x7fa1700042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.457+0000 7fa17cc00640 1 -- 172.21.15.1:0/1744744647 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17003f030 con 0x7fa1800772f0 2024-04-26T21:27:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.457+0000 7fa17cc00640 1 -- 172.21.15.1:0/1744744647 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa170031910 con 0x7fa1800772f0 2024-04-26T21:27:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.457+0000 7fa185400640 1 -- 172.21.15.1:0/1744744647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1801a6eb0 con 0x7fa1800772f0 2024-04-26T21:27:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.457+0000 7fa17cc00640 1 -- 172.21.15.1:0/1744744647 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa170035070 con 0x7fa1800772f0 2024-04-26T21:27:31.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.457+0000 7fa185400640 1 -- 172.21.15.1:0/1744744647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1801a72c0 con 0x7fa1800772f0 2024-04-26T21:27:31.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.458+0000 7fa185400640 1 -- 172.21.15.1:0/1744744647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa180111320 con 0x7fa1800772f0 2024-04-26T21:27:31.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.460+0000 7fa17cc00640 1 -- 172.21.15.1:0/1744744647 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 138645+0+0 (secure 0 0 0) 0x7fa170042030 con 0x7fa1800772f0 2024-04-26T21:27:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.461+0000 7fa17cc00640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa1640a3bb0 0x7fa1640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.462+0000 7fa17cc00640 1 -- 172.21.15.1:0/1744744647 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fa1700f0bc0 con 0x7fa1800772f0 2024-04-26T21:27:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.462+0000 7fa17f400640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa1640a3bb0 0x7fa1640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:31.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.462+0000 7fa17f400640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa1640a3bb0 0x7fa1640a6070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa1801a0510 tx=0x7fa16c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:31.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.466+0000 7fa17cc00640 1 -- 172.21.15.1:0/1744744647 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa1700bccb0 con 0x7fa1800772f0 2024-04-26T21:27:31.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.710+0000 7fa185400640 1 -- 172.21.15.1:0/1744744647 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa180062e00 con 0x7fa1640a3bb0 2024-04-26T21:27:31.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.716+0000 7fa17cc00640 1 -- 172.21.15.1:0/1744744647 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fa1700b9320 con 0x7fa1800772f0 2024-04-26T21:27:31.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.753+0000 7fa17cc00640 1 -- 172.21.15.1:0/1744744647 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa180062e00 con 0x7fa1640a3bb0 2024-04-26T21:27:31.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.756+0000 7fa185400640 1 -- 172.21.15.1:0/1744744647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa1640a3bb0 msgr2=0x7fa1640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:31.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.756+0000 7fa185400640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa1640a3bb0 0x7fa1640a6070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa1801a0510 tx=0x7fa16c005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:31.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.756+0000 7fa185400640 1 -- 172.21.15.1:0/1744744647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1800772f0 msgr2=0x7fa18019f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:31.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.756+0000 7fa185400640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1800772f0 0x7fa18019f000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa170006010 tx=0x7fa1700042f0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.756+0000 7fa185400640 1 -- 172.21.15.1:0/1744744647 shutdown_connections 2024-04-26T21:27:31.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.756+0000 7fa185400640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa1640a3bb0 0x7fa1640a6070 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.756+0000 7fa185400640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa180078ed0 0x7fa1801a65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.756+0000 7fa185400640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa180078530 0x7fa18019f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.756+0000 7fa185400640 1 --2- 172.21.15.1:0/1744744647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1800772f0 0x7fa18019f000 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.756+0000 7fa185400640 1 -- 172.21.15.1:0/1744744647 >> 172.21.15.1:0/1744744647 conn(0x7fa1800fff40 msgr2=0x7fa180101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:31.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.757+0000 7fa185400640 1 -- 172.21.15.1:0/1744744647 shutdown_connections 2024-04-26T21:27:31.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.757+0000 7fa185400640 1 -- 172.21.15.1:0/1744744647 wait complete. 2024-04-26T21:27:31.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-26T21:27:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.985+0000 7f3687e00640 1 -- 172.21.15.1:0/2739861833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3688105b50 msgr2=0x7f3688107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.985+0000 7f3687e00640 1 --2- 172.21.15.1:0/2739861833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3688105b50 0x7f3688107f40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3678009ca0 tx=0x7f367802a8c0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.985+0000 7f3687e00640 1 -- 172.21.15.1:0/2739861833 shutdown_connections 2024-04-26T21:27:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.985+0000 7f3687e00640 1 --2- 172.21.15.1:0/2739861833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3688108480 0x7f368810a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.985+0000 7f3687e00640 1 --2- 172.21.15.1:0/2739861833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3688105b50 0x7f3688107f40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.985+0000 7f3687e00640 1 --2- 172.21.15.1:0/2739861833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3688103220 0x7f3688105610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.985+0000 7f3687e00640 1 -- 172.21.15.1:0/2739861833 >> 172.21.15.1:0/2739861833 conn(0x7f36880fcc10 msgr2=0x7f36880ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.985+0000 7f3687e00640 1 -- 172.21.15.1:0/2739861833 shutdown_connections 2024-04-26T21:27:31.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.985+0000 7f3687e00640 1 -- 172.21.15.1:0/2739861833 wait complete. 2024-04-26T21:27:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.986+0000 7f3687e00640 1 Processor -- start 2024-04-26T21:27:31.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.986+0000 7f3687e00640 1 -- start start 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3687e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3688103220 0x7f3688198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3687e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3688105b50 0x7f3688198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3687e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3688108480 0x7f368819ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3687e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f368810e330 con 0x7f3688103220 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3687e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f368810e1b0 con 0x7f3688105b50 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3687e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f368810e4b0 con 0x7f3688108480 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3686a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3688103220 0x7f3688198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3687400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3688108480 0x7f368819ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3686a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3688103220 0x7f3688198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41164/0 (socket says 172.21.15.1:41164) 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3686a00640 1 -- 172.21.15.1:0/1679720430 learned_addr learned my addr 172.21.15.1:0/1679720430 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:31.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.987+0000 7f3686000640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3688105b50 0x7f3688198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.988+0000 7f3687400640 1 -- 172.21.15.1:0/1679720430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3688105b50 msgr2=0x7f3688198f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.988+0000 7f3687400640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3688105b50 0x7f3688198f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.988+0000 7f3687400640 1 -- 172.21.15.1:0/1679720430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3688103220 msgr2=0x7f3688198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.988+0000 7f3687400640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3688103220 0x7f3688198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.988+0000 7f3687400640 1 -- 172.21.15.1:0/1679720430 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36881a0630 con 0x7f3688108480 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.988+0000 7f3687400640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3688108480 0x7f368819ffc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f368000e970 tx=0x7f368000ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.988+0000 7f367f400640 1 -- 172.21.15.1:0/1679720430 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368000ccc0 con 0x7f3688108480 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.988+0000 7f367f400640 1 -- 172.21.15.1:0/1679720430 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3680005020 con 0x7f3688108480 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.988+0000 7f367f400640 1 -- 172.21.15.1:0/1679720430 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3680004a00 con 0x7f3688108480 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.988+0000 7f3687e00640 1 -- 172.21.15.1:0/1679720430 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36881a0920 con 0x7f3688108480 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.989+0000 7f3687e00640 1 -- 172.21.15.1:0/1679720430 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36881a0e40 con 0x7f3688108480 2024-04-26T21:27:31.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.990+0000 7f3687e00640 1 -- 172.21.15.1:0/1679720430 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36880fe700 con 0x7f3688108480 2024-04-26T21:27:31.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.991+0000 7f367f400640 1 -- 172.21.15.1:0/1679720430 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f3680004ba0 con 0x7f3688108480 2024-04-26T21:27:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.993+0000 7f367f400640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f366c0a3cc0 0x7f366c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.993+0000 7f367f400640 1 -- 172.21.15.1:0/1679720430 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f36800d06d0 con 0x7f3688108480 2024-04-26T21:27:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.993+0000 7f3686a00640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f366c0a3cc0 0x7f366c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.994+0000 7f3686a00640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f366c0a3cc0 0x7f366c0a6180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3670009cd0 tx=0x7f3670005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:31.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:31.997+0000 7f367f400640 1 -- 172.21.15.1:0/1679720430 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f368009ccb0 con 0x7f3688108480 2024-04-26T21:27:32.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:31 smithi001 ceph-mon[28054]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:32.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:31 smithi001 ceph-mon[28054]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:32.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:31 smithi038 ceph-mon[32724]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:32.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:31 smithi038 ceph-mon[32724]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:31 smithi186 ceph-mon[32991]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:31 smithi186 ceph-mon[32991]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:32.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.237+0000 7f3687e00640 1 -- 172.21.15.1:0/1679720430 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f3688062d70 con 0x7f366c0a3cc0 2024-04-26T21:27:32.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.245+0000 7f367f400640 1 -- 172.21.15.1:0/1679720430 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3688062d70 con 0x7f366c0a3cc0 2024-04-26T21:27:32.245 INFO:teuthology.orchestra.run.smithi001.stdout:/volumes/qa/sv_1/298b6b07-2bc1-45c5-bc65-b7e2df0eafa2 2024-04-26T21:27:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 -- 172.21.15.1:0/1679720430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f366c0a3cc0 msgr2=0x7f366c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f366c0a3cc0 0x7f366c0a6180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3670009cd0 tx=0x7f3670005f90 comp rx=0 tx=0).stop 2024-04-26T21:27:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 -- 172.21.15.1:0/1679720430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3688108480 msgr2=0x7f368819ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3688108480 0x7f368819ffc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f368000e970 tx=0x7f368000ee30 comp rx=0 tx=0).stop 2024-04-26T21:27:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 -- 172.21.15.1:0/1679720430 shutdown_connections 2024-04-26T21:27:32.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f366c0a3cc0 0x7f366c0a6180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3688108480 0x7f368819ffc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3688105b50 0x7f3688198f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 --2- 172.21.15.1:0/1679720430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3688103220 0x7f3688198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 -- 172.21.15.1:0/1679720430 >> 172.21.15.1:0/1679720430 conn(0x7f36880fcc10 msgr2=0x7f3688103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:32.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.247+0000 7f3687e00640 1 -- 172.21.15.1:0/1679720430 shutdown_connections 2024-04-26T21:27:32.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.248+0000 7f3687e00640 1 -- 172.21.15.1:0/1679720430 wait complete. 2024-04-26T21:27:32.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:27:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.479+0000 7f41d5a00640 1 -- 172.21.15.1:0/2240151038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41d0104300 msgr2=0x7f41d010ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:32.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.479+0000 7f41d5a00640 1 --2- 172.21.15.1:0/2240151038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41d0104300 0x7f41d010ab90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009860 tx=0x7f41c802a340 comp rx=0 tx=0).stop 2024-04-26T21:27:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.479+0000 7f41d5a00640 1 -- 172.21.15.1:0/2240151038 shutdown_connections 2024-04-26T21:27:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.479+0000 7f41d5a00640 1 --2- 172.21.15.1:0/2240151038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41d0104300 0x7f41d010ab90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.479+0000 7f41d5a00640 1 --2- 172.21.15.1:0/2240151038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41d0103940 0x7f41d0103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.479+0000 7f41d5a00640 1 --2- 172.21.15.1:0/2240151038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41d0102750 0x7f41d0102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.479+0000 7f41d5a00640 1 -- 172.21.15.1:0/2240151038 >> 172.21.15.1:0/2240151038 conn(0x7f41d00fdcc0 msgr2=0x7f41d0100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.480+0000 7f41d5a00640 1 -- 172.21.15.1:0/2240151038 shutdown_connections 2024-04-26T21:27:32.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.480+0000 7f41d5a00640 1 -- 172.21.15.1:0/2240151038 wait complete. 2024-04-26T21:27:32.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41d5a00640 1 Processor -- start 2024-04-26T21:27:32.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41d5a00640 1 -- start start 2024-04-26T21:27:32.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41d5a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41d0102750 0x7f41d019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:32.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41d5a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41d0103940 0x7f41d019afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:32.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41d5a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41d0104300 0x7f41d01a2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:32.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41d5a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f41d010e260 con 0x7f41d0103940 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41d5a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f41d010e0e0 con 0x7f41d0102750 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41d5a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41d010e3e0 con 0x7f41d0104300 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41cf400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41d0103940 0x7f41d019afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41cf400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41d0103940 0x7f41d019afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41178/0 (socket says 172.21.15.1:41178) 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41cf400640 1 -- 172.21.15.1:0/2695207121 learned_addr learned my addr 172.21.15.1:0/2695207121 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.481+0000 7f41cfe00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41d0102750 0x7f41d019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41d5000640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41d0104300 0x7f41d01a2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41cf400640 1 -- 172.21.15.1:0/2695207121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41d0104300 msgr2=0x7f41d01a2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41cf400640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41d0104300 0x7f41d01a2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41cf400640 1 -- 172.21.15.1:0/2695207121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41d0102750 msgr2=0x7f41d019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41cf400640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41d0102750 0x7f41d019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41cf400640 1 -- 172.21.15.1:0/2695207121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d01a2680 con 0x7f41d0103940 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41cfe00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41d0102750 0x7f41d019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:32.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41cf400640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41d0103940 0x7f41d019afa0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f41b800e9c0 tx=0x7f41b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41ccc00640 1 -- 172.21.15.1:0/2695207121 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b800cde0 con 0x7f41d0103940 2024-04-26T21:27:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41ccc00640 1 -- 172.21.15.1:0/2695207121 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41b80035f0 con 0x7f41d0103940 2024-04-26T21:27:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.482+0000 7f41d5a00640 1 -- 172.21.15.1:0/2695207121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41d01a2970 con 0x7f41d0103940 2024-04-26T21:27:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.483+0000 7f41ccc00640 1 -- 172.21.15.1:0/2695207121 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b8010690 con 0x7f41d0103940 2024-04-26T21:27:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.483+0000 7f41d5a00640 1 -- 172.21.15.1:0/2695207121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41d01a2ec0 con 0x7f41d0103940 2024-04-26T21:27:32.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.484+0000 7f41d5a00640 1 -- 172.21.15.1:0/2695207121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41d01a3150 con 0x7f41d0103940 2024-04-26T21:27:32.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.486+0000 7f41ccc00640 1 -- 172.21.15.1:0/2695207121 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f41b8004000 con 0x7f41d0103940 2024-04-26T21:27:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.487+0000 7f41ccc00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f41b40a3cf0 0x7f41b40a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.487+0000 7f41ccc00640 1 -- 172.21.15.1:0/2695207121 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f41b80d0b20 con 0x7f41d0103940 2024-04-26T21:27:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.487+0000 7f41cfe00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f41b40a3cf0 0x7f41b40a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:32.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.488+0000 7f41cfe00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f41b40a3cf0 0x7f41b40a61b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f41d01037a0 tx=0x7f41c0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:32.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.494+0000 7f41ccc00640 1 -- 172.21.15.1:0/2695207121 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f41b8009510 con 0x7f41d0103940 2024-04-26T21:27:32.708 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:27:32 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b[62819]: 2024-04-26T21:27:32.407+0000 7f2f14698700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-26T21:27:32.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.746+0000 7f41d5a00640 1 -- 172.21.15.1:0/2695207121 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41d01a3320 con 0x7f41d0103940 2024-04-26T21:27:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.747+0000 7f41ccc00640 1 -- 172.21.15.1:0/2695207121 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11857 (secure 0 0 0) 0x7f41b809d100 con 0x7f41d0103940 2024-04-26T21:27:32.748 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:32.748 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:27:02.795246+0000","modified":"2024-04-26T21:27:28.957703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24572,"mds_2":24548},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24548":{"gid":24548,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24548,24572,24602]},"id":1}]} 2024-04-26T21:27:32.748 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 23 2024-04-26T21:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 -- 172.21.15.1:0/2695207121 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f41b40a3cf0 msgr2=0x7f41b40a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f41b40a3cf0 0x7f41b40a61b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f41d01037a0 tx=0x7f41c0009500 comp rx=0 tx=0).stop 2024-04-26T21:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 -- 172.21.15.1:0/2695207121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41d0103940 msgr2=0x7f41d019afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41d0103940 0x7f41d019afa0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f41b800e9c0 tx=0x7f41b800ee80 comp rx=0 tx=0).stop 2024-04-26T21:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 -- 172.21.15.1:0/2695207121 shutdown_connections 2024-04-26T21:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f41b40a3cf0 0x7f41b40a61b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41d0104300 0x7f41d01a2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41d0103940 0x7f41d019afa0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 --2- 172.21.15.1:0/2695207121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41d0102750 0x7f41d019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 -- 172.21.15.1:0/2695207121 >> 172.21.15.1:0/2695207121 conn(0x7f41d00fdcc0 msgr2=0x7f41d0074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 -- 172.21.15.1:0/2695207121 shutdown_connections 2024-04-26T21:27:32.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.750+0000 7f41d5a00640 1 -- 172.21.15.1:0/2695207121 wait complete. 2024-04-26T21:27:32.771 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:27:32.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.974+0000 7f1c1f200640 1 -- 172.21.15.1:0/2623964455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c20108450 msgr2=0x7f1c2010a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:32.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.974+0000 7f1c1f200640 1 --2- 172.21.15.1:0/2623964455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c20108450 0x7f1c2010a8a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1c18009900 tx=0x7f1c1802a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.974+0000 7f1c1f200640 1 -- 172.21.15.1:0/2623964455 shutdown_connections 2024-04-26T21:27:32.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.974+0000 7f1c1f200640 1 --2- 172.21.15.1:0/2623964455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c20105b20 0x7f1c20107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.974+0000 7f1c1f200640 1 --2- 172.21.15.1:0/2623964455 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c20108450 0x7f1c2010a8a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.974+0000 7f1c1f200640 1 --2- 172.21.15.1:0/2623964455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c201031f0 0x7f1c201055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.974+0000 7f1c1f200640 1 -- 172.21.15.1:0/2623964455 >> 172.21.15.1:0/2623964455 conn(0x7f1c200fcc20 msgr2=0x7f1c200ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:32.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.974+0000 7f1c1f200640 1 -- 172.21.15.1:0/2623964455 shutdown_connections 2024-04-26T21:27:32.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.974+0000 7f1c1f200640 1 -- 172.21.15.1:0/2623964455 wait complete. 2024-04-26T21:27:32.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.975+0000 7f1c1f200640 1 Processor -- start 2024-04-26T21:27:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.976+0000 7f1c1f200640 1 -- start start 2024-04-26T21:27:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.976+0000 7f1c1f200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c201031f0 0x7f1c20102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.976+0000 7f1c1f200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c20105b20 0x7f1c201011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.976+0000 7f1c1f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c20108450 0x7f1c201016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.976+0000 7f1c1f200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2010e2f0 con 0x7f1c201031f0 2024-04-26T21:27:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.976+0000 7f1c1f200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2010e170 con 0x7f1c20105b20 2024-04-26T21:27:32.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.976+0000 7f1c1f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1c2010e470 con 0x7f1c20108450 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c20108450 0x7f1c201016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c20108450 0x7f1c201016e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:40954/0 (socket says 172.21.15.1:40954) 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1e800640 1 -- 172.21.15.1:0/390356270 learned_addr learned my addr 172.21.15.1:0/390356270 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1de00640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c201031f0 0x7f1c20102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1d400640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c20105b20 0x7f1c201011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1e800640 1 -- 172.21.15.1:0/390356270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c20105b20 msgr2=0x7f1c201011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1e800640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c20105b20 0x7f1c201011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1e800640 1 -- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c201031f0 msgr2=0x7f1c20102b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1e800640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c201031f0 0x7f1c20102b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1e800640 1 -- 172.21.15.1:0/390356270 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c20101f70 con 0x7f1c20108450 2024-04-26T21:27:32.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1de00640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c201031f0 0x7f1c20102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:32.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.977+0000 7f1c1e800640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c20108450 0x7f1c201016e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1c18005ab0 tx=0x7f1c18003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:32.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.978+0000 7f1c16a00640 1 -- 172.21.15.1:0/390356270 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1803f070 con 0x7f1c20108450 2024-04-26T21:27:32.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.978+0000 7f1c1f200640 1 -- 172.21.15.1:0/390356270 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c201a2900 con 0x7f1c20108450 2024-04-26T21:27:32.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.978+0000 7f1c16a00640 1 -- 172.21.15.1:0/390356270 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c18031c50 con 0x7f1c20108450 2024-04-26T21:27:32.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.978+0000 7f1c16a00640 1 -- 172.21.15.1:0/390356270 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c18035030 con 0x7f1c20108450 2024-04-26T21:27:32.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.978+0000 7f1c1f200640 1 -- 172.21.15.1:0/390356270 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c201a2e50 con 0x7f1c20108450 2024-04-26T21:27:32.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.979+0000 7f1c1f200640 1 -- 172.21.15.1:0/390356270 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c200fe710 con 0x7f1c20108450 2024-04-26T21:27:32.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.981+0000 7f1c16a00640 1 -- 172.21.15.1:0/390356270 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f1c18042070 con 0x7f1c20108450 2024-04-26T21:27:32.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.982+0000 7f1c16a00640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1c040a3cf0 0x7f1c040a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:32.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.982+0000 7f1c16a00640 1 -- 172.21.15.1:0/390356270 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f1c180f1620 con 0x7f1c20108450 2024-04-26T21:27:32.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.983+0000 7f1c1de00640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1c040a3cf0 0x7f1c040a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:32.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.983+0000 7f1c1de00640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1c040a3cf0 0x7f1c040a61b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c10005fd0 tx=0x7f1c10009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:32.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:32.985+0000 7f1c16a00640 1 -- 172.21.15.1:0/390356270 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1c180bd610 con 0x7f1c20108450 2024-04-26T21:27:33.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:32 smithi001 ceph-mon[28054]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:33.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:32 smithi001 ceph-mon[28054]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-26T21:27:33.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:32 smithi001 ceph-mon[28054]: from='client.24856 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:33.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:32 smithi001 ceph-mon[28054]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T21:27:33.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:32 smithi038 ceph-mon[32724]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:33.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:32 smithi038 ceph-mon[32724]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-26T21:27:33.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:32 smithi038 ceph-mon[32724]: from='client.24856 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:33.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:32 smithi038 ceph-mon[32724]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T21:27:33.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.194+0000 7f1c1f200640 1 -- 172.21.15.1:0/390356270 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c2002dad0 con 0x7f1c20108450 2024-04-26T21:27:33.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.196+0000 7f1c16a00640 1 -- 172.21.15.1:0/390356270 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23972 (secure 0 0 0) 0x7f1c2002dad0 con 0x7f1c20108450 2024-04-26T21:27:33.197 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:33.197 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:27:07.816919+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:27:33.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.199+0000 7f1c1f200640 1 -- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1c040a3cf0 msgr2=0x7f1c040a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:33.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.199+0000 7f1c1f200640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1c040a3cf0 0x7f1c040a61b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c10005fd0 tx=0x7f1c10009500 comp rx=0 tx=0).stop 2024-04-26T21:27:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.199+0000 7f1c1f200640 1 -- 172.21.15.1:0/390356270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c20108450 msgr2=0x7f1c201016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.199+0000 7f1c1f200640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c20108450 0x7f1c201016e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1c18005ab0 tx=0x7f1c18003b90 comp rx=0 tx=0).stop 2024-04-26T21:27:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.199+0000 7f1c1f200640 1 -- 172.21.15.1:0/390356270 shutdown_connections 2024-04-26T21:27:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.199+0000 7f1c1f200640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1c040a3cf0 0x7f1c040a61b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.199+0000 7f1c1f200640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c20108450 0x7f1c201016e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.199+0000 7f1c1f200640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c20105b20 0x7f1c201011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.199+0000 7f1c1f200640 1 --2- 172.21.15.1:0/390356270 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c201031f0 0x7f1c20102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.199+0000 7f1c1f200640 1 -- 172.21.15.1:0/390356270 >> 172.21.15.1:0/390356270 conn(0x7f1c200fcc20 msgr2=0x7f1c20103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.200+0000 7f1c1f200640 1 -- 172.21.15.1:0/390356270 shutdown_connections 2024-04-26T21:27:33.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.200+0000 7f1c1f200640 1 -- 172.21.15.1:0/390356270 wait complete. 2024-04-26T21:27:33.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:32 smithi186 ceph-mon[32991]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:33.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:32 smithi186 ceph-mon[32991]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-26T21:27:33.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:32 smithi186 ceph-mon[32991]: from='client.24856 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:33.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:32 smithi186 ceph-mon[32991]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-26T21:27:33.222 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-26T21:27:33.222 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T21:27:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.436+0000 7f157f000640 1 -- 172.21.15.1:0/4249691727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15780772f0 msgr2=0x7f1578077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.436+0000 7f157f000640 1 --2- 172.21.15.1:0/4249691727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15780772f0 0x7f1578077710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f156c009900 tx=0x7f156c02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.436+0000 7f157f000640 1 -- 172.21.15.1:0/4249691727 shutdown_connections 2024-04-26T21:27:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.436+0000 7f157f000640 1 --2- 172.21.15.1:0/4249691727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1578078530 0x7f1578078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.436+0000 7f157f000640 1 --2- 172.21.15.1:0/4249691727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1578078ed0 0x7f1578079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.436+0000 7f157f000640 1 --2- 172.21.15.1:0/4249691727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15780772f0 0x7f1578077710 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.436+0000 7f157f000640 1 -- 172.21.15.1:0/4249691727 >> 172.21.15.1:0/4249691727 conn(0x7f15780fff00 msgr2=0x7f1578102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.436+0000 7f157f000640 1 -- 172.21.15.1:0/4249691727 shutdown_connections 2024-04-26T21:27:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.436+0000 7f157f000640 1 -- 172.21.15.1:0/4249691727 wait complete. 2024-04-26T21:27:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.438+0000 7f157f000640 1 Processor -- start 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.438+0000 7f157f000640 1 -- start start 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.438+0000 7f157f000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15780772f0 0x7f157819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.438+0000 7f157f000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1578078530 0x7f157819f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.438+0000 7f157f000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1578078ed0 0x7f15781a64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.438+0000 7f157f000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1578114910 con 0x7f1578078530 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.438+0000 7f157f000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1578114790 con 0x7f1578078ed0 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.438+0000 7f157f000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1578114a90 con 0x7f15780772f0 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157d200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1578078530 0x7f157819f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157dc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15780772f0 0x7f157819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157e600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1578078ed0 0x7f15781a64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157d200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1578078530 0x7f157819f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41218/0 (socket says 172.21.15.1:41218) 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157dc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15780772f0 0x7f157819ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:40970/0 (socket says 172.21.15.1:40970) 2024-04-26T21:27:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157d200640 1 -- 172.21.15.1:0/735698314 learned_addr learned my addr 172.21.15.1:0/735698314 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157e600640 1 -- 172.21.15.1:0/735698314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15780772f0 msgr2=0x7f157819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157e600640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15780772f0 0x7f157819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157e600640 1 -- 172.21.15.1:0/735698314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1578078530 msgr2=0x7f157819f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157e600640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1578078530 0x7f157819f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157e600640 1 -- 172.21.15.1:0/735698314 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15781a6b30 con 0x7f1578078ed0 2024-04-26T21:27:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157d200640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1578078530 0x7f157819f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.439+0000 7f157e600640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1578078ed0 0x7f15781a64c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f157000da10 tx=0x7f157000ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.440+0000 7f1576000640 1 -- 172.21.15.1:0/735698314 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157000bd70 con 0x7f1578078ed0 2024-04-26T21:27:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.440+0000 7f157f000640 1 -- 172.21.15.1:0/735698314 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15781a6e20 con 0x7f1578078ed0 2024-04-26T21:27:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.440+0000 7f157f000640 1 -- 172.21.15.1:0/735698314 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15781a72e0 con 0x7f1578078ed0 2024-04-26T21:27:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.440+0000 7f1576000640 1 -- 172.21.15.1:0/735698314 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1570003e20 con 0x7f1578078ed0 2024-04-26T21:27:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.440+0000 7f1576000640 1 -- 172.21.15.1:0/735698314 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15700106c0 con 0x7f1578078ed0 2024-04-26T21:27:33.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.441+0000 7f157f000640 1 -- 172.21.15.1:0/735698314 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1578111300 con 0x7f1578078ed0 2024-04-26T21:27:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.443+0000 7f1576000640 1 -- 172.21.15.1:0/735698314 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f1570004230 con 0x7f1578078ed0 2024-04-26T21:27:33.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.445+0000 7f1576000640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f15600a3d70 0x7f15600a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:33.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.445+0000 7f1576000640 1 -- 172.21.15.1:0/735698314 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f15700d0860 con 0x7f1578078ed0 2024-04-26T21:27:33.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.445+0000 7f157dc00640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f15600a3d70 0x7f15600a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.446+0000 7f157dc00640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f15600a3d70 0x7f15600a6230 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f156c031500 tx=0x7f156c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:33.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.449+0000 7f1576000640 1 -- 172.21.15.1:0/735698314 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1570021070 con 0x7f1578078ed0 2024-04-26T21:27:33.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.654+0000 7f157f000640 1 -- 172.21.15.1:0/735698314 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1578062d70 con 0x7f15600a3d70 2024-04-26T21:27:33.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.691+0000 7f1576000640 1 -- 172.21.15.1:0/735698314 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340383 (secure 0 0 0) 0x7f1578062d70 con 0x7f15600a3d70 2024-04-26T21:27:33.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:33.702 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T21:27:33.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 -- 172.21.15.1:0/735698314 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f15600a3d70 msgr2=0x7f15600a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:33.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f15600a3d70 0x7f15600a6230 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f156c031500 tx=0x7f156c005a20 comp rx=0 tx=0).stop 2024-04-26T21:27:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 -- 172.21.15.1:0/735698314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1578078ed0 msgr2=0x7f15781a64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1578078ed0 0x7f15781a64c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f157000da10 tx=0x7f157000ded0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 -- 172.21.15.1:0/735698314 shutdown_connections 2024-04-26T21:27:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f15600a3d70 0x7f15600a6230 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1578078ed0 0x7f15781a64c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1578078530 0x7f157819f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 --2- 172.21.15.1:0/735698314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15780772f0 0x7f157819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 -- 172.21.15.1:0/735698314 >> 172.21.15.1:0/735698314 conn(0x7f15780fff00 msgr2=0x7f1578101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.702+0000 7f157f000640 1 -- 172.21.15.1:0/735698314 shutdown_connections 2024-04-26T21:27:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.703+0000 7f157f000640 1 -- 172.21.15.1:0/735698314 wait complete. 2024-04-26T21:27:33.725 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-04-26T21:27:33.310518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597937,"num_objects":63,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":167,"num_read_kb":144,"num_write":313,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379612,"kb_used_data":13820,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562212,"statfs":{"total":1151940427776,"available":1151551705088,"internally_reserved":0,"allocated":14151680,"data_stored":7708652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3070,"num_objects":39,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":84,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006175"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399037+0000","last_change":"2024-04-26T21:27:02.854724+0000","last_active":"2024-04-26T21:27:08.399037+0000","last_peered":"2024-04-26T21:27:08.399037+0000","last_clean":"2024-04-26T21:27:08.399037+0000","last_became_active":"2024-04-26T21:27:02.854125+0000","last_became_peered":"2024-04-26T21:27:02.854125+0000","last_unstale":"2024-04-26T21:27:08.399037+0000","last_undegraded":"2024-04-26T21:27:08.399037+0000","last_fullsized":"2024-04-26T21:27:08.399037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:16.500220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.339983+0000","last_change":"2024-04-26T21:27:02.852846+0000","last_active":"2024-04-26T21:27:08.339983+0000","last_peered":"2024-04-26T21:27:08.339983+0000","last_clean":"2024-04-26T21:27:08.339983+0000","last_became_active":"2024-04-26T21:27:02.851243+0000","last_became_peered":"2024-04-26T21:27:02.851243+0000","last_unstale":"2024-04-26T21:27:08.339983+0000","last_undegraded":"2024-04-26T21:27:08.339983+0000","last_fullsized":"2024-04-26T21:27:08.339983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:09:11.240955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.339547+0000","last_change":"2024-04-26T21:27:02.855856+0000","last_active":"2024-04-26T21:27:08.339547+0000","last_peered":"2024-04-26T21:27:08.339547+0000","last_clean":"2024-04-26T21:27:08.339547+0000","last_became_active":"2024-04-26T21:27:02.855446+0000","last_became_peered":"2024-04-26T21:27:02.855446+0000","last_unstale":"2024-04-26T21:27:08.339547+0000","last_undegraded":"2024-04-26T21:27:08.339547+0000","last_fullsized":"2024-04-26T21:27:08.339547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:35.183449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341803+0000","last_change":"2024-04-26T21:27:02.860949+0000","last_active":"2024-04-26T21:27:08.341803+0000","last_peered":"2024-04-26T21:27:08.341803+0000","last_clean":"2024-04-26T21:27:08.341803+0000","last_became_active":"2024-04-26T21:27:02.860567+0000","last_became_peered":"2024-04-26T21:27:02.860567+0000","last_unstale":"2024-04-26T21:27:08.341803+0000","last_undegraded":"2024-04-26T21:27:08.341803+0000","last_fullsized":"2024-04-26T21:27:08.341803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:09.851885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397220+0000","last_change":"2024-04-26T21:27:02.876019+0000","last_active":"2024-04-26T21:27:08.397220+0000","last_peered":"2024-04-26T21:27:08.397220+0000","last_clean":"2024-04-26T21:27:08.397220+0000","last_became_active":"2024-04-26T21:27:02.875652+0000","last_became_peered":"2024-04-26T21:27:02.875652+0000","last_unstale":"2024-04-26T21:27:08.397220+0000","last_undegraded":"2024-04-26T21:27:08.397220+0000","last_fullsized":"2024-04-26T21:27:08.397220+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:35.017787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397751+0000","last_change":"2024-04-26T21:27:02.860229+0000","last_active":"2024-04-26T21:27:08.397751+0000","last_peered":"2024-04-26T21:27:08.397751+0000","last_clean":"2024-04-26T21:27:08.397751+0000","last_became_active":"2024-04-26T21:27:02.859863+0000","last_became_peered":"2024-04-26T21:27:02.859863+0000","last_unstale":"2024-04-26T21:27:08.397751+0000","last_undegraded":"2024-04-26T21:27:08.397751+0000","last_fullsized":"2024-04-26T21:27:08.397751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:47.337073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398287+0000","last_change":"2024-04-26T21:27:02.863253+0000","last_active":"2024-04-26T21:27:08.398287+0000","last_peered":"2024-04-26T21:27:08.398287+0000","last_clean":"2024-04-26T21:27:08.398287+0000","last_became_active":"2024-04-26T21:27:02.861623+0000","last_became_peered":"2024-04-26T21:27:02.861623+0000","last_unstale":"2024-04-26T21:27:08.398287+0000","last_undegraded":"2024-04-26T21:27:08.398287+0000","last_fullsized":"2024-04-26T21:27:08.398287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:57.642504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399585+0000","last_change":"2024-04-26T21:27:02.860822+0000","last_active":"2024-04-26T21:27:08.399585+0000","last_peered":"2024-04-26T21:27:08.399585+0000","last_clean":"2024-04-26T21:27:08.399585+0000","last_became_active":"2024-04-26T21:27:02.860482+0000","last_became_peered":"2024-04-26T21:27:02.860482+0000","last_unstale":"2024-04-26T21:27:08.399585+0000","last_undegraded":"2024-04-26T21:27:08.399585+0000","last_fullsized":"2024-04-26T21:27:08.399585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:59.224624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398297+0000","last_change":"2024-04-26T21:27:02.854782+0000","last_active":"2024-04-26T21:27:08.398297+0000","last_peered":"2024-04-26T21:27:08.398297+0000","last_clean":"2024-04-26T21:27:08.398297+0000","last_became_active":"2024-04-26T21:27:02.854423+0000","last_became_peered":"2024-04-26T21:27:02.854423+0000","last_unstale":"2024-04-26T21:27:08.398297+0000","last_undegraded":"2024-04-26T21:27:08.398297+0000","last_fullsized":"2024-04-26T21:27:08.398297+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:21.559775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399195+0000","last_change":"2024-04-26T21:27:02.860545+0000","last_active":"2024-04-26T21:27:08.399195+0000","last_peered":"2024-04-26T21:27:08.399195+0000","last_clean":"2024-04-26T21:27:08.399195+0000","last_became_active":"2024-04-26T21:27:02.858577+0000","last_became_peered":"2024-04-26T21:27:02.858577+0000","last_unstale":"2024-04-26T21:27:08.399195+0000","last_undegraded":"2024-04-26T21:27:08.399195+0000","last_fullsized":"2024-04-26T21:27:08.399195+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:43.390592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399281+0000","last_change":"2024-04-26T21:27:02.858685+0000","last_active":"2024-04-26T21:27:08.399281+0000","last_peered":"2024-04-26T21:27:08.399281+0000","last_clean":"2024-04-26T21:27:08.399281+0000","last_became_active":"2024-04-26T21:27:02.857054+0000","last_became_peered":"2024-04-26T21:27:02.857054+0000","last_unstale":"2024-04-26T21:27:08.399281+0000","last_undegraded":"2024-04-26T21:27:08.399281+0000","last_fullsized":"2024-04-26T21:27:08.399281+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:29.406132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341751+0000","last_change":"2024-04-26T21:27:02.852457+0000","last_active":"2024-04-26T21:27:08.341751+0000","last_peered":"2024-04-26T21:27:08.341751+0000","last_clean":"2024-04-26T21:27:08.341751+0000","last_became_active":"2024-04-26T21:27:02.852064+0000","last_became_peered":"2024-04-26T21:27:02.852064+0000","last_unstale":"2024-04-26T21:27:08.341751+0000","last_undegraded":"2024-04-26T21:27:08.341751+0000","last_fullsized":"2024-04-26T21:27:08.341751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:16.798413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399056+0000","last_change":"2024-04-26T21:27:02.872618+0000","last_active":"2024-04-26T21:27:08.399056+0000","last_peered":"2024-04-26T21:27:08.399056+0000","last_clean":"2024-04-26T21:27:08.399056+0000","last_became_active":"2024-04-26T21:27:02.872270+0000","last_became_peered":"2024-04-26T21:27:02.872270+0000","last_unstale":"2024-04-26T21:27:08.399056+0000","last_undegraded":"2024-04-26T21:27:08.399056+0000","last_fullsized":"2024-04-26T21:27:08.399056+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:45.828419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.338795+0000","last_change":"2024-04-26T21:27:02.868590+0000","last_active":"2024-04-26T21:27:08.338795+0000","last_peered":"2024-04-26T21:27:08.338795+0000","last_clean":"2024-04-26T21:27:08.338795+0000","last_became_active":"2024-04-26T21:27:02.866641+0000","last_became_peered":"2024-04-26T21:27:02.866641+0000","last_unstale":"2024-04-26T21:27:08.338795+0000","last_undegraded":"2024-04-26T21:27:08.338795+0000","last_fullsized":"2024-04-26T21:27:08.338795+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:17.313420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.338900+0000","last_change":"2024-04-26T21:27:02.867735+0000","last_active":"2024-04-26T21:27:08.338900+0000","last_peered":"2024-04-26T21:27:08.338900+0000","last_clean":"2024-04-26T21:27:08.338900+0000","last_became_active":"2024-04-26T21:27:02.867217+0000","last_became_peered":"2024-04-26T21:27:02.867217+0000","last_unstale":"2024-04-26T21:27:08.338900+0000","last_undegraded":"2024-04-26T21:27:08.338900+0000","last_fullsized":"2024-04-26T21:27:08.338900+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:04:45.533220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.396965+0000","last_change":"2024-04-26T21:27:02.857541+0000","last_active":"2024-04-26T21:27:08.396965+0000","last_peered":"2024-04-26T21:27:08.396965+0000","last_clean":"2024-04-26T21:27:08.396965+0000","last_became_active":"2024-04-26T21:27:02.855563+0000","last_became_peered":"2024-04-26T21:27:02.855563+0000","last_unstale":"2024-04-26T21:27:08.396965+0000","last_undegraded":"2024-04-26T21:27:08.396965+0000","last_fullsized":"2024-04-26T21:27:08.396965+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:39.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398898+0000","last_change":"2024-04-26T21:27:02.854461+0000","last_active":"2024-04-26T21:27:08.398898+0000","last_peered":"2024-04-26T21:27:08.398898+0000","last_clean":"2024-04-26T21:27:08.398898+0000","last_became_active":"2024-04-26T21:27:02.853691+0000","last_became_peered":"2024-04-26T21:27:02.853691+0000","last_unstale":"2024-04-26T21:27:08.398898+0000","last_undegraded":"2024-04-26T21:27:08.398898+0000","last_fullsized":"2024-04-26T21:27:08.398898+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:09:48.284180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400836+0000","last_change":"2024-04-26T21:27:02.864308+0000","last_active":"2024-04-26T21:27:08.400836+0000","last_peered":"2024-04-26T21:27:08.400836+0000","last_clean":"2024-04-26T21:27:08.400836+0000","last_became_active":"2024-04-26T21:27:02.863928+0000","last_became_peered":"2024-04-26T21:27:02.863928+0000","last_unstale":"2024-04-26T21:27:08.400836+0000","last_undegraded":"2024-04-26T21:27:08.400836+0000","last_fullsized":"2024-04-26T21:27:08.400836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:12.169394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337904+0000","last_change":"2024-04-26T21:27:02.876044+0000","last_active":"2024-04-26T21:27:08.337904+0000","last_peered":"2024-04-26T21:27:08.337904+0000","last_clean":"2024-04-26T21:27:08.337904+0000","last_became_active":"2024-04-26T21:27:02.875681+0000","last_became_peered":"2024-04-26T21:27:02.875681+0000","last_unstale":"2024-04-26T21:27:08.337904+0000","last_undegraded":"2024-04-26T21:27:08.337904+0000","last_fullsized":"2024-04-26T21:27:08.337904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:26.246916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337192+0000","last_change":"2024-04-26T21:27:07.374736+0000","last_active":"2024-04-26T21:27:08.337192+0000","last_peered":"2024-04-26T21:27:08.337192+0000","last_clean":"2024-04-26T21:27:08.337192+0000","last_became_active":"2024-04-26T21:27:07.374116+0000","last_became_peered":"2024-04-26T21:27:07.374116+0000","last_unstale":"2024-04-26T21:27:08.337192+0000","last_undegraded":"2024-04-26T21:27:08.337192+0000","last_fullsized":"2024-04-26T21:27:08.337192+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:11.450926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.478026+0000","last_change":"2024-04-26T21:27:02.302510+0000","last_active":"2024-04-26T21:27:28.478026+0000","last_peered":"2024-04-26T21:27:28.478026+0000","last_clean":"2024-04-26T21:27:28.478026+0000","last_became_active":"2024-04-26T21:27:02.302088+0000","last_became_peered":"2024-04-26T21:27:02.302088+0000","last_unstale":"2024-04-26T21:27:28.478026+0000","last_undegraded":"2024-04-26T21:27:28.478026+0000","last_fullsized":"2024-04-26T21:27:28.478026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:43.007722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401188+0000","last_change":"2024-04-26T21:27:02.861455+0000","last_active":"2024-04-26T21:27:08.401188+0000","last_peered":"2024-04-26T21:27:08.401188+0000","last_clean":"2024-04-26T21:27:08.401188+0000","last_became_active":"2024-04-26T21:27:02.861107+0000","last_became_peered":"2024-04-26T21:27:02.861107+0000","last_unstale":"2024-04-26T21:27:08.401188+0000","last_undegraded":"2024-04-26T21:27:08.401188+0000","last_fullsized":"2024-04-26T21:27:08.401188+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:10:18.382881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341432+0000","last_change":"2024-04-26T21:27:07.377797+0000","last_active":"2024-04-26T21:27:08.341432+0000","last_peered":"2024-04-26T21:27:08.341432+0000","last_clean":"2024-04-26T21:27:08.341432+0000","last_became_active":"2024-04-26T21:27:07.377314+0000","last_became_peered":"2024-04-26T21:27:07.377314+0000","last_unstale":"2024-04-26T21:27:08.341432+0000","last_undegraded":"2024-04-26T21:27:08.341432+0000","last_fullsized":"2024-04-26T21:27:08.341432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:02:34.761034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401344+0000","last_change":"2024-04-26T21:27:02.309488+0000","last_active":"2024-04-26T21:27:08.401344+0000","last_peered":"2024-04-26T21:27:08.401344+0000","last_clean":"2024-04-26T21:27:08.401344+0000","last_became_active":"2024-04-26T21:27:02.308592+0000","last_became_peered":"2024-04-26T21:27:02.308592+0000","last_unstale":"2024-04-26T21:27:08.401344+0000","last_undegraded":"2024-04-26T21:27:08.401344+0000","last_fullsized":"2024-04-26T21:27:08.401344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:36.563847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.340948+0000","last_change":"2024-04-26T21:27:02.863204+0000","last_active":"2024-04-26T21:27:08.340948+0000","last_peered":"2024-04-26T21:27:08.340948+0000","last_clean":"2024-04-26T21:27:08.340948+0000","last_became_active":"2024-04-26T21:27:02.862466+0000","last_became_peered":"2024-04-26T21:27:02.862466+0000","last_unstale":"2024-04-26T21:27:08.340948+0000","last_undegraded":"2024-04-26T21:27:08.340948+0000","last_fullsized":"2024-04-26T21:27:08.340948+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:29.360830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341357+0000","last_change":"2024-04-26T21:27:07.378032+0000","last_active":"2024-04-26T21:27:08.341357+0000","last_peered":"2024-04-26T21:27:08.341357+0000","last_clean":"2024-04-26T21:27:08.341357+0000","last_became_active":"2024-04-26T21:27:07.377255+0000","last_became_peered":"2024-04-26T21:27:07.377255+0000","last_unstale":"2024-04-26T21:27:08.341357+0000","last_undegraded":"2024-04-26T21:27:08.341357+0000","last_fullsized":"2024-04-26T21:27:08.341357+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:18.989977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.472454+0000","last_change":"2024-04-26T21:27:02.294734+0000","last_active":"2024-04-26T21:27:28.472454+0000","last_peered":"2024-04-26T21:27:28.472454+0000","last_clean":"2024-04-26T21:27:28.472454+0000","last_became_active":"2024-04-26T21:27:02.294415+0000","last_became_peered":"2024-04-26T21:27:02.294415+0000","last_unstale":"2024-04-26T21:27:28.472454+0000","last_undegraded":"2024-04-26T21:27:28.472454+0000","last_fullsized":"2024-04-26T21:27:28.472454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:13.017537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397311+0000","last_change":"2024-04-26T21:27:02.876447+0000","last_active":"2024-04-26T21:27:08.397311+0000","last_peered":"2024-04-26T21:27:08.397311+0000","last_clean":"2024-04-26T21:27:08.397311+0000","last_became_active":"2024-04-26T21:27:02.876017+0000","last_became_peered":"2024-04-26T21:27:02.876017+0000","last_unstale":"2024-04-26T21:27:08.397311+0000","last_undegraded":"2024-04-26T21:27:08.397311+0000","last_fullsized":"2024-04-26T21:27:08.397311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:57:05.453787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341317+0000","last_change":"2024-04-26T21:27:07.373700+0000","last_active":"2024-04-26T21:27:08.341317+0000","last_peered":"2024-04-26T21:27:08.341317+0000","last_clean":"2024-04-26T21:27:08.341317+0000","last_became_active":"2024-04-26T21:27:07.373255+0000","last_became_peered":"2024-04-26T21:27:07.373255+0000","last_unstale":"2024-04-26T21:27:08.341317+0000","last_undegraded":"2024-04-26T21:27:08.341317+0000","last_fullsized":"2024-04-26T21:27:08.341317+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:48.517197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397904+0000","last_change":"2024-04-26T21:27:02.300817+0000","last_active":"2024-04-26T21:27:08.397904+0000","last_peered":"2024-04-26T21:27:08.397904+0000","last_clean":"2024-04-26T21:27:08.397904+0000","last_became_active":"2024-04-26T21:27:02.300493+0000","last_became_peered":"2024-04-26T21:27:02.300493+0000","last_unstale":"2024-04-26T21:27:08.397904+0000","last_undegraded":"2024-04-26T21:27:08.397904+0000","last_fullsized":"2024-04-26T21:27:08.397904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:18.783104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397752+0000","last_change":"2024-04-26T21:27:02.859982+0000","last_active":"2024-04-26T21:27:08.397752+0000","last_peered":"2024-04-26T21:27:08.397752+0000","last_clean":"2024-04-26T21:27:08.397752+0000","last_became_active":"2024-04-26T21:27:02.857204+0000","last_became_peered":"2024-04-26T21:27:02.857204+0000","last_unstale":"2024-04-26T21:27:08.397752+0000","last_undegraded":"2024-04-26T21:27:08.397752+0000","last_fullsized":"2024-04-26T21:27:08.397752+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:22.423956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337711+0000","last_change":"2024-04-26T21:27:07.366039+0000","last_active":"2024-04-26T21:27:08.337711+0000","last_peered":"2024-04-26T21:27:08.337711+0000","last_clean":"2024-04-26T21:27:08.337711+0000","last_became_active":"2024-04-26T21:27:07.365560+0000","last_became_peered":"2024-04-26T21:27:07.365560+0000","last_unstale":"2024-04-26T21:27:08.337711+0000","last_undegraded":"2024-04-26T21:27:08.337711+0000","last_fullsized":"2024-04-26T21:27:08.337711+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:47.833476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:30.689622+0000","last_change":"2024-04-26T21:27:02.306364+0000","last_active":"2024-04-26T21:27:30.689622+0000","last_peered":"2024-04-26T21:27:30.689622+0000","last_clean":"2024-04-26T21:27:30.689622+0000","last_became_active":"2024-04-26T21:27:02.306041+0000","last_became_peered":"2024-04-26T21:27:02.306041+0000","last_unstale":"2024-04-26T21:27:30.689622+0000","last_undegraded":"2024-04-26T21:27:30.689622+0000","last_fullsized":"2024-04-26T21:27:30.689622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:44.501916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397691+0000","last_change":"2024-04-26T21:27:02.859346+0000","last_active":"2024-04-26T21:27:08.397691+0000","last_peered":"2024-04-26T21:27:08.397691+0000","last_clean":"2024-04-26T21:27:08.397691+0000","last_became_active":"2024-04-26T21:27:02.859025+0000","last_became_peered":"2024-04-26T21:27:02.859025+0000","last_unstale":"2024-04-26T21:27:08.397691+0000","last_undegraded":"2024-04-26T21:27:08.397691+0000","last_fullsized":"2024-04-26T21:27:08.397691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:06.258512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337565+0000","last_change":"2024-04-26T21:27:07.382349+0000","last_active":"2024-04-26T21:27:08.337565+0000","last_peered":"2024-04-26T21:27:08.337565+0000","last_clean":"2024-04-26T21:27:08.337565+0000","last_became_active":"2024-04-26T21:27:07.381930+0000","last_became_peered":"2024-04-26T21:27:07.381930+0000","last_unstale":"2024-04-26T21:27:08.337565+0000","last_undegraded":"2024-04-26T21:27:08.337565+0000","last_fullsized":"2024-04-26T21:27:08.337565+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:15:25.970573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.477106+0000","last_change":"2024-04-26T21:27:02.305449+0000","last_active":"2024-04-26T21:27:28.477106+0000","last_peered":"2024-04-26T21:27:28.477106+0000","last_clean":"2024-04-26T21:27:28.477106+0000","last_became_active":"2024-04-26T21:27:02.305095+0000","last_became_peered":"2024-04-26T21:27:02.305095+0000","last_unstale":"2024-04-26T21:27:28.477106+0000","last_undegraded":"2024-04-26T21:27:28.477106+0000","last_fullsized":"2024-04-26T21:27:28.477106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:47:31.627826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397772+0000","last_change":"2024-04-26T21:27:02.865707+0000","last_active":"2024-04-26T21:27:08.397772+0000","last_peered":"2024-04-26T21:27:08.397772+0000","last_clean":"2024-04-26T21:27:08.397772+0000","last_became_active":"2024-04-26T21:27:02.865436+0000","last_became_peered":"2024-04-26T21:27:02.865436+0000","last_unstale":"2024-04-26T21:27:08.397772+0000","last_undegraded":"2024-04-26T21:27:08.397772+0000","last_fullsized":"2024-04-26T21:27:08.397772+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:53.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399105+0000","last_change":"2024-04-26T21:27:07.379092+0000","last_active":"2024-04-26T21:27:08.399105+0000","last_peered":"2024-04-26T21:27:08.399105+0000","last_clean":"2024-04-26T21:27:08.399105+0000","last_became_active":"2024-04-26T21:27:07.378716+0000","last_became_peered":"2024-04-26T21:27:07.378716+0000","last_unstale":"2024-04-26T21:27:08.399105+0000","last_undegraded":"2024-04-26T21:27:08.399105+0000","last_fullsized":"2024-04-26T21:27:08.399105+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:18.906224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.476718+0000","last_change":"2024-04-26T21:27:02.307510+0000","last_active":"2024-04-26T21:27:28.476718+0000","last_peered":"2024-04-26T21:27:28.476718+0000","last_clean":"2024-04-26T21:27:28.476718+0000","last_became_active":"2024-04-26T21:27:02.306932+0000","last_became_peered":"2024-04-26T21:27:02.306932+0000","last_unstale":"2024-04-26T21:27:28.476718+0000","last_undegraded":"2024-04-26T21:27:28.476718+0000","last_fullsized":"2024-04-26T21:27:28.476718+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:21:34.217767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399452+0000","last_change":"2024-04-26T21:27:02.874246+0000","last_active":"2024-04-26T21:27:08.399452+0000","last_peered":"2024-04-26T21:27:08.399452+0000","last_clean":"2024-04-26T21:27:08.399452+0000","last_became_active":"2024-04-26T21:27:02.873880+0000","last_became_peered":"2024-04-26T21:27:02.873880+0000","last_unstale":"2024-04-26T21:27:08.399452+0000","last_undegraded":"2024-04-26T21:27:08.399452+0000","last_fullsized":"2024-04-26T21:27:08.399452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:21.579832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.339493+0000","last_change":"2024-04-26T21:27:07.387201+0000","last_active":"2024-04-26T21:27:08.339493+0000","last_peered":"2024-04-26T21:27:08.339493+0000","last_clean":"2024-04-26T21:27:08.339493+0000","last_became_active":"2024-04-26T21:27:07.386327+0000","last_became_peered":"2024-04-26T21:27:07.386327+0000","last_unstale":"2024-04-26T21:27:08.339493+0000","last_undegraded":"2024-04-26T21:27:08.339493+0000","last_fullsized":"2024-04-26T21:27:08.339493+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:40.801572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397484+0000","last_change":"2024-04-26T21:27:02.304132+0000","last_active":"2024-04-26T21:27:08.397484+0000","last_peered":"2024-04-26T21:27:08.397484+0000","last_clean":"2024-04-26T21:27:08.397484+0000","last_became_active":"2024-04-26T21:27:02.303562+0000","last_became_peered":"2024-04-26T21:27:02.303562+0000","last_unstale":"2024-04-26T21:27:08.397484+0000","last_undegraded":"2024-04-26T21:27:08.397484+0000","last_fullsized":"2024-04-26T21:27:08.397484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:54.417590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399833+0000","last_change":"2024-04-26T21:27:02.866539+0000","last_active":"2024-04-26T21:27:08.399833+0000","last_peered":"2024-04-26T21:27:08.399833+0000","last_clean":"2024-04-26T21:27:08.399833+0000","last_became_active":"2024-04-26T21:27:02.866193+0000","last_became_peered":"2024-04-26T21:27:02.866193+0000","last_unstale":"2024-04-26T21:27:08.399833+0000","last_undegraded":"2024-04-26T21:27:08.399833+0000","last_fullsized":"2024-04-26T21:27:08.399833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:44.234261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400356+0000","last_change":"2024-04-26T21:27:07.368394+0000","last_active":"2024-04-26T21:27:08.400356+0000","last_peered":"2024-04-26T21:27:08.400356+0000","last_clean":"2024-04-26T21:27:08.400356+0000","last_became_active":"2024-04-26T21:27:07.367998+0000","last_became_peered":"2024-04-26T21:27:07.367998+0000","last_unstale":"2024-04-26T21:27:08.400356+0000","last_undegraded":"2024-04-26T21:27:08.400356+0000","last_fullsized":"2024-04-26T21:27:08.400356+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:31:28.459759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.466384+0000","last_change":"2024-04-26T21:27:02.304096+0000","last_active":"2024-04-26T21:27:27.466384+0000","last_peered":"2024-04-26T21:27:27.466384+0000","last_clean":"2024-04-26T21:27:27.466384+0000","last_became_active":"2024-04-26T21:27:02.303736+0000","last_became_peered":"2024-04-26T21:27:02.303736+0000","last_unstale":"2024-04-26T21:27:27.466384+0000","last_undegraded":"2024-04-26T21:27:27.466384+0000","last_fullsized":"2024-04-26T21:27:27.466384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:25.400913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398607+0000","last_change":"2024-04-26T21:27:02.863033+0000","last_active":"2024-04-26T21:27:08.398607+0000","last_peered":"2024-04-26T21:27:08.398607+0000","last_clean":"2024-04-26T21:27:08.398607+0000","last_became_active":"2024-04-26T21:27:02.862572+0000","last_became_peered":"2024-04-26T21:27:02.862572+0000","last_unstale":"2024-04-26T21:27:08.398607+0000","last_undegraded":"2024-04-26T21:27:08.398607+0000","last_fullsized":"2024-04-26T21:27:08.398607+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:02:55.574601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398874+0000","last_change":"2024-04-26T21:27:07.373626+0000","last_active":"2024-04-26T21:27:08.398874+0000","last_peered":"2024-04-26T21:27:08.398874+0000","last_clean":"2024-04-26T21:27:08.398874+0000","last_became_active":"2024-04-26T21:27:07.373175+0000","last_became_peered":"2024-04-26T21:27:07.373175+0000","last_unstale":"2024-04-26T21:27:08.398874+0000","last_undegraded":"2024-04-26T21:27:08.398874+0000","last_fullsized":"2024-04-26T21:27:08.398874+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:53:09.118035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.464716+0000","last_change":"2024-04-26T21:27:02.299267+0000","last_active":"2024-04-26T21:27:27.464716+0000","last_peered":"2024-04-26T21:27:27.464716+0000","last_clean":"2024-04-26T21:27:27.464716+0000","last_became_active":"2024-04-26T21:27:02.298842+0000","last_became_peered":"2024-04-26T21:27:02.298842+0000","last_unstale":"2024-04-26T21:27:27.464716+0000","last_undegraded":"2024-04-26T21:27:27.464716+0000","last_fullsized":"2024-04-26T21:27:27.464716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:08.431792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.340599+0000","last_change":"2024-04-26T21:27:02.858739+0000","last_active":"2024-04-26T21:27:08.340599+0000","last_peered":"2024-04-26T21:27:08.340599+0000","last_clean":"2024-04-26T21:27:08.340599+0000","last_became_active":"2024-04-26T21:27:02.857458+0000","last_became_peered":"2024-04-26T21:27:02.857458+0000","last_unstale":"2024-04-26T21:27:08.340599+0000","last_undegraded":"2024-04-26T21:27:08.340599+0000","last_fullsized":"2024-04-26T21:27:08.340599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:00.354052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341751+0000","last_change":"2024-04-26T21:27:07.377690+0000","last_active":"2024-04-26T21:27:08.341751+0000","last_peered":"2024-04-26T21:27:08.341751+0000","last_clean":"2024-04-26T21:27:08.341751+0000","last_became_active":"2024-04-26T21:27:07.377192+0000","last_became_peered":"2024-04-26T21:27:07.377192+0000","last_unstale":"2024-04-26T21:27:08.341751+0000","last_undegraded":"2024-04-26T21:27:08.341751+0000","last_fullsized":"2024-04-26T21:27:08.341751+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:48.069083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.465863+0000","last_change":"2024-04-26T21:27:02.316110+0000","last_active":"2024-04-26T21:27:27.465863+0000","last_peered":"2024-04-26T21:27:27.465863+0000","last_clean":"2024-04-26T21:27:27.465863+0000","last_became_active":"2024-04-26T21:27:02.315767+0000","last_became_peered":"2024-04-26T21:27:02.315767+0000","last_unstale":"2024-04-26T21:27:27.465863+0000","last_undegraded":"2024-04-26T21:27:27.465863+0000","last_fullsized":"2024-04-26T21:27:27.465863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:00.990872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397435+0000","last_change":"2024-04-26T21:27:02.855736+0000","last_active":"2024-04-26T21:27:08.397435+0000","last_peered":"2024-04-26T21:27:08.397435+0000","last_clean":"2024-04-26T21:27:08.397435+0000","last_became_active":"2024-04-26T21:27:02.855378+0000","last_became_peered":"2024-04-26T21:27:02.855378+0000","last_unstale":"2024-04-26T21:27:08.397435+0000","last_undegraded":"2024-04-26T21:27:08.397435+0000","last_fullsized":"2024-04-26T21:27:08.397435+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:55.272869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399118+0000","last_change":"2024-04-26T21:27:07.376804+0000","last_active":"2024-04-26T21:27:08.399118+0000","last_peered":"2024-04-26T21:27:08.399118+0000","last_clean":"2024-04-26T21:27:08.399118+0000","last_became_active":"2024-04-26T21:27:07.374378+0000","last_became_peered":"2024-04-26T21:27:07.374378+0000","last_unstale":"2024-04-26T21:27:08.399118+0000","last_undegraded":"2024-04-26T21:27:08.399118+0000","last_fullsized":"2024-04-26T21:27:08.399118+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:03.930253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.686307+0000","last_change":"2024-04-26T21:27:02.297467+0000","last_active":"2024-04-26T21:27:28.686307+0000","last_peered":"2024-04-26T21:27:28.686307+0000","last_clean":"2024-04-26T21:27:28.686307+0000","last_became_active":"2024-04-26T21:27:02.297077+0000","last_became_peered":"2024-04-26T21:27:02.297077+0000","last_unstale":"2024-04-26T21:27:28.686307+0000","last_undegraded":"2024-04-26T21:27:28.686307+0000","last_fullsized":"2024-04-26T21:27:28.686307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:47.047565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398238+0000","last_change":"2024-04-26T21:27:02.858161+0000","last_active":"2024-04-26T21:27:08.398238+0000","last_peered":"2024-04-26T21:27:08.398238+0000","last_clean":"2024-04-26T21:27:08.398238+0000","last_became_active":"2024-04-26T21:27:02.857747+0000","last_became_peered":"2024-04-26T21:27:02.857747+0000","last_unstale":"2024-04-26T21:27:08.398238+0000","last_undegraded":"2024-04-26T21:27:08.398238+0000","last_fullsized":"2024-04-26T21:27:08.398238+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:10:50.843195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398106+0000","last_change":"2024-04-26T21:27:07.376253+0000","last_active":"2024-04-26T21:27:08.398106+0000","last_peered":"2024-04-26T21:27:08.398106+0000","last_clean":"2024-04-26T21:27:08.398106+0000","last_became_active":"2024-04-26T21:27:07.375822+0000","last_became_peered":"2024-04-26T21:27:07.375822+0000","last_unstale":"2024-04-26T21:27:08.398106+0000","last_undegraded":"2024-04-26T21:27:08.398106+0000","last_fullsized":"2024-04-26T21:27:08.398106+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:22:01.784163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.476343+0000","last_change":"2024-04-26T21:27:02.300743+0000","last_active":"2024-04-26T21:27:28.476343+0000","last_peered":"2024-04-26T21:27:28.476343+0000","last_clean":"2024-04-26T21:27:28.476343+0000","last_became_active":"2024-04-26T21:27:02.300424+0000","last_became_peered":"2024-04-26T21:27:02.300424+0000","last_unstale":"2024-04-26T21:27:28.476343+0000","last_undegraded":"2024-04-26T21:27:28.476343+0000","last_fullsized":"2024-04-26T21:27:28.476343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:10.408234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398707+0000","last_change":"2024-04-26T21:27:02.872456+0000","last_active":"2024-04-26T21:27:08.398707+0000","last_peered":"2024-04-26T21:27:08.398707+0000","last_clean":"2024-04-26T21:27:08.398707+0000","last_became_active":"2024-04-26T21:27:02.872064+0000","last_became_peered":"2024-04-26T21:27:02.872064+0000","last_unstale":"2024-04-26T21:27:08.398707+0000","last_undegraded":"2024-04-26T21:27:08.398707+0000","last_fullsized":"2024-04-26T21:27:08.398707+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:33.511164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398819+0000","last_change":"2024-04-26T21:27:07.386148+0000","last_active":"2024-04-26T21:27:08.398819+0000","last_peered":"2024-04-26T21:27:08.398819+0000","last_clean":"2024-04-26T21:27:08.398819+0000","last_became_active":"2024-04-26T21:27:07.385787+0000","last_became_peered":"2024-04-26T21:27:07.385787+0000","last_unstale":"2024-04-26T21:27:08.398819+0000","last_undegraded":"2024-04-26T21:27:08.398819+0000","last_fullsized":"2024-04-26T21:27:08.398819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:57.776660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.467213+0000","last_change":"2024-04-26T21:27:02.313725+0000","last_active":"2024-04-26T21:27:27.467213+0000","last_peered":"2024-04-26T21:27:27.467213+0000","last_clean":"2024-04-26T21:27:27.467213+0000","last_became_active":"2024-04-26T21:27:02.313350+0000","last_became_peered":"2024-04-26T21:27:02.313350+0000","last_unstale":"2024-04-26T21:27:27.467213+0000","last_undegraded":"2024-04-26T21:27:27.467213+0000","last_fullsized":"2024-04-26T21:27:27.467213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:21:30.608067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398692+0000","last_change":"2024-04-26T21:27:02.876043+0000","last_active":"2024-04-26T21:27:08.398692+0000","last_peered":"2024-04-26T21:27:08.398692+0000","last_clean":"2024-04-26T21:27:08.398692+0000","last_became_active":"2024-04-26T21:27:02.875665+0000","last_became_peered":"2024-04-26T21:27:02.875665+0000","last_unstale":"2024-04-26T21:27:08.398692+0000","last_undegraded":"2024-04-26T21:27:08.398692+0000","last_fullsized":"2024-04-26T21:27:08.398692+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:53:47.205853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401003+0000","last_change":"2024-04-26T21:27:07.378091+0000","last_active":"2024-04-26T21:27:08.401003+0000","last_peered":"2024-04-26T21:27:08.401003+0000","last_clean":"2024-04-26T21:27:08.401003+0000","last_became_active":"2024-04-26T21:27:07.377671+0000","last_became_peered":"2024-04-26T21:27:07.377671+0000","last_unstale":"2024-04-26T21:27:08.401003+0000","last_undegraded":"2024-04-26T21:27:08.401003+0000","last_fullsized":"2024-04-26T21:27:08.401003+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:41:36.551655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:30.689730+0000","last_change":"2024-04-26T21:27:02.294738+0000","last_active":"2024-04-26T21:27:30.689730+0000","last_peered":"2024-04-26T21:27:30.689730+0000","last_clean":"2024-04-26T21:27:30.689730+0000","last_became_active":"2024-04-26T21:27:02.294338+0000","last_became_peered":"2024-04-26T21:27:02.294338+0000","last_unstale":"2024-04-26T21:27:30.689730+0000","last_undegraded":"2024-04-26T21:27:30.689730+0000","last_fullsized":"2024-04-26T21:27:30.689730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:25.309107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398442+0000","last_change":"2024-04-26T21:27:02.861332+0000","last_active":"2024-04-26T21:27:08.398442+0000","last_peered":"2024-04-26T21:27:08.398442+0000","last_clean":"2024-04-26T21:27:08.398442+0000","last_became_active":"2024-04-26T21:27:02.857865+0000","last_became_peered":"2024-04-26T21:27:02.857865+0000","last_unstale":"2024-04-26T21:27:08.398442+0000","last_undegraded":"2024-04-26T21:27:08.398442+0000","last_fullsized":"2024-04-26T21:27:08.398442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:48.493086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401726+0000","last_change":"2024-04-26T21:27:07.383219+0000","last_active":"2024-04-26T21:27:08.401726+0000","last_peered":"2024-04-26T21:27:08.401726+0000","last_clean":"2024-04-26T21:27:08.401726+0000","last_became_active":"2024-04-26T21:27:07.382820+0000","last_became_peered":"2024-04-26T21:27:07.382820+0000","last_unstale":"2024-04-26T21:27:08.401726+0000","last_undegraded":"2024-04-26T21:27:08.401726+0000","last_fullsized":"2024-04-26T21:27:08.401726+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:07.752124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.477768+0000","last_change":"2024-04-26T21:27:02.307746+0000","last_active":"2024-04-26T21:27:28.477768+0000","last_peered":"2024-04-26T21:27:28.477768+0000","last_clean":"2024-04-26T21:27:28.477768+0000","last_became_active":"2024-04-26T21:27:02.307406+0000","last_became_peered":"2024-04-26T21:27:02.307406+0000","last_unstale":"2024-04-26T21:27:28.477768+0000","last_undegraded":"2024-04-26T21:27:28.477768+0000","last_fullsized":"2024-04-26T21:27:28.477768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:13.799727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398177+0000","last_change":"2024-04-26T21:27:02.861194+0000","last_active":"2024-04-26T21:27:08.398177+0000","last_peered":"2024-04-26T21:27:08.398177+0000","last_clean":"2024-04-26T21:27:08.398177+0000","last_became_active":"2024-04-26T21:27:02.860871+0000","last_became_peered":"2024-04-26T21:27:02.860871+0000","last_unstale":"2024-04-26T21:27:08.398177+0000","last_undegraded":"2024-04-26T21:27:08.398177+0000","last_fullsized":"2024-04-26T21:27:08.398177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:56.829027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397336+0000","last_change":"2024-04-26T21:27:07.371265+0000","last_active":"2024-04-26T21:27:08.397336+0000","last_peered":"2024-04-26T21:27:08.397336+0000","last_clean":"2024-04-26T21:27:08.397336+0000","last_became_active":"2024-04-26T21:27:07.370928+0000","last_became_peered":"2024-04-26T21:27:07.370928+0000","last_unstale":"2024-04-26T21:27:08.397336+0000","last_undegraded":"2024-04-26T21:27:08.397336+0000","last_fullsized":"2024-04-26T21:27:08.397336+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:33:05.640493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.683557+0000","last_change":"2024-04-26T21:27:02.301711+0000","last_active":"2024-04-26T21:27:28.683557+0000","last_peered":"2024-04-26T21:27:28.683557+0000","last_clean":"2024-04-26T21:27:28.683557+0000","last_became_active":"2024-04-26T21:27:02.301429+0000","last_became_peered":"2024-04-26T21:27:02.301429+0000","last_unstale":"2024-04-26T21:27:28.683557+0000","last_undegraded":"2024-04-26T21:27:28.683557+0000","last_fullsized":"2024-04-26T21:27:28.683557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:50.876824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398948+0000","last_change":"2024-04-26T21:27:02.854372+0000","last_active":"2024-04-26T21:27:08.398948+0000","last_peered":"2024-04-26T21:27:08.398948+0000","last_clean":"2024-04-26T21:27:08.398948+0000","last_became_active":"2024-04-26T21:27:02.853826+0000","last_became_peered":"2024-04-26T21:27:02.853826+0000","last_unstale":"2024-04-26T21:27:08.398948+0000","last_undegraded":"2024-04-26T21:27:08.398948+0000","last_fullsized":"2024-04-26T21:27:08.398948+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:08.253384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397623+0000","last_change":"2024-04-26T21:27:07.506684+0000","last_active":"2024-04-26T21:27:08.397623+0000","last_peered":"2024-04-26T21:27:08.397623+0000","last_clean":"2024-04-26T21:27:08.397623+0000","last_became_active":"2024-04-26T21:27:07.506203+0000","last_became_peered":"2024-04-26T21:27:07.506203+0000","last_unstale":"2024-04-26T21:27:08.397623+0000","last_undegraded":"2024-04-26T21:27:08.397623+0000","last_fullsized":"2024-04-26T21:27:08.397623+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:42:48.580106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399612+0000","last_change":"2024-04-26T21:27:02.306265+0000","last_active":"2024-04-26T21:27:08.399612+0000","last_peered":"2024-04-26T21:27:08.399612+0000","last_clean":"2024-04-26T21:27:08.399612+0000","last_became_active":"2024-04-26T21:27:02.305941+0000","last_became_peered":"2024-04-26T21:27:02.305941+0000","last_unstale":"2024-04-26T21:27:08.399612+0000","last_undegraded":"2024-04-26T21:27:08.399612+0000","last_fullsized":"2024-04-26T21:27:08.399612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:29.076833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398899+0000","last_change":"2024-04-26T21:27:02.871916+0000","last_active":"2024-04-26T21:27:08.398899+0000","last_peered":"2024-04-26T21:27:08.398899+0000","last_clean":"2024-04-26T21:27:08.398899+0000","last_became_active":"2024-04-26T21:27:02.871517+0000","last_became_peered":"2024-04-26T21:27:02.871517+0000","last_unstale":"2024-04-26T21:27:08.398899+0000","last_undegraded":"2024-04-26T21:27:08.398899+0000","last_fullsized":"2024-04-26T21:27:08.398899+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:41.361757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400181+0000","last_change":"2024-04-26T21:27:07.383422+0000","last_active":"2024-04-26T21:27:08.400181+0000","last_peered":"2024-04-26T21:27:08.400181+0000","last_clean":"2024-04-26T21:27:08.400181+0000","last_became_active":"2024-04-26T21:27:07.382727+0000","last_became_peered":"2024-04-26T21:27:07.382727+0000","last_unstale":"2024-04-26T21:27:08.400181+0000","last_undegraded":"2024-04-26T21:27:08.400181+0000","last_fullsized":"2024-04-26T21:27:08.400181+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:22.002658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398824+0000","last_change":"2024-04-26T21:27:02.308081+0000","last_active":"2024-04-26T21:27:08.398824+0000","last_peered":"2024-04-26T21:27:08.398824+0000","last_clean":"2024-04-26T21:27:08.398824+0000","last_became_active":"2024-04-26T21:27:02.307715+0000","last_became_peered":"2024-04-26T21:27:02.307715+0000","last_unstale":"2024-04-26T21:27:08.398824+0000","last_undegraded":"2024-04-26T21:27:08.398824+0000","last_fullsized":"2024-04-26T21:27:08.398824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:28.967817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398685+0000","last_change":"2024-04-26T21:27:02.859527+0000","last_active":"2024-04-26T21:27:08.398685+0000","last_peered":"2024-04-26T21:27:08.398685+0000","last_clean":"2024-04-26T21:27:08.398685+0000","last_became_active":"2024-04-26T21:27:02.859173+0000","last_became_peered":"2024-04-26T21:27:02.859173+0000","last_unstale":"2024-04-26T21:27:08.398685+0000","last_undegraded":"2024-04-26T21:27:08.398685+0000","last_fullsized":"2024-04-26T21:27:08.398685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:24.811630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400540+0000","last_change":"2024-04-26T21:27:07.509778+0000","last_active":"2024-04-26T21:27:08.400540+0000","last_peered":"2024-04-26T21:27:08.400540+0000","last_clean":"2024-04-26T21:27:08.400540+0000","last_became_active":"2024-04-26T21:27:07.508344+0000","last_became_peered":"2024-04-26T21:27:07.508344+0000","last_unstale":"2024-04-26T21:27:08.400540+0000","last_undegraded":"2024-04-26T21:27:08.400540+0000","last_fullsized":"2024-04-26T21:27:08.400540+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:38:05.926717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397671+0000","last_change":"2024-04-26T21:27:02.862880+0000","last_active":"2024-04-26T21:27:08.397671+0000","last_peered":"2024-04-26T21:27:08.397671+0000","last_clean":"2024-04-26T21:27:08.397671+0000","last_became_active":"2024-04-26T21:27:02.862499+0000","last_became_peered":"2024-04-26T21:27:02.862499+0000","last_unstale":"2024-04-26T21:27:08.397671+0000","last_undegraded":"2024-04-26T21:27:08.397671+0000","last_fullsized":"2024-04-26T21:27:08.397671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:12:33.955508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.478159+0000","last_change":"2024-04-26T21:27:02.317413+0000","last_active":"2024-04-26T21:27:28.478159+0000","last_peered":"2024-04-26T21:27:28.478159+0000","last_clean":"2024-04-26T21:27:28.478159+0000","last_became_active":"2024-04-26T21:27:02.316510+0000","last_became_peered":"2024-04-26T21:27:02.316510+0000","last_unstale":"2024-04-26T21:27:28.478159+0000","last_undegraded":"2024-04-26T21:27:28.478159+0000","last_fullsized":"2024-04-26T21:27:28.478159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:49.156366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399332+0000","last_change":"2024-04-26T21:27:07.373434+0000","last_active":"2024-04-26T21:27:08.399332+0000","last_peered":"2024-04-26T21:27:08.399332+0000","last_clean":"2024-04-26T21:27:08.399332+0000","last_became_active":"2024-04-26T21:27:07.372973+0000","last_became_peered":"2024-04-26T21:27:07.372973+0000","last_unstale":"2024-04-26T21:27:08.399332+0000","last_undegraded":"2024-04-26T21:27:08.399332+0000","last_fullsized":"2024-04-26T21:27:08.399332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:04.022651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400567+0000","last_change":"2024-04-26T21:27:02.862734+0000","last_active":"2024-04-26T21:27:08.400567+0000","last_peered":"2024-04-26T21:27:08.400567+0000","last_clean":"2024-04-26T21:27:08.400567+0000","last_became_active":"2024-04-26T21:27:02.862334+0000","last_became_peered":"2024-04-26T21:27:02.862334+0000","last_unstale":"2024-04-26T21:27:08.400567+0000","last_undegraded":"2024-04-26T21:27:08.400567+0000","last_fullsized":"2024-04-26T21:27:08.400567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:09.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.681983+0000","last_change":"2024-04-26T21:27:02.302471+0000","last_active":"2024-04-26T21:27:28.681983+0000","last_peered":"2024-04-26T21:27:28.681983+0000","last_clean":"2024-04-26T21:27:28.681983+0000","last_became_active":"2024-04-26T21:27:02.302055+0000","last_became_peered":"2024-04-26T21:27:02.302055+0000","last_unstale":"2024-04-26T21:27:28.681983+0000","last_undegraded":"2024-04-26T21:27:28.681983+0000","last_fullsized":"2024-04-26T21:27:28.681983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:29:14.649670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398204+0000","last_change":"2024-04-26T21:25:41.647633+0000","last_active":"2024-04-26T21:27:08.398204+0000","last_peered":"2024-04-26T21:27:08.398204+0000","last_clean":"2024-04-26T21:27:08.398204+0000","last_became_active":"2024-04-26T21:25:41.647091+0000","last_became_peered":"2024-04-26T21:25:41.647091+0000","last_unstale":"2024-04-26T21:27:08.398204+0000","last_undegraded":"2024-04-26T21:27:08.398204+0000","last_fullsized":"2024-04-26T21:27:08.398204+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_clean_scrub_stamp":"2024-04-26T21:21:43.547929+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:37.666065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397305+0000","last_change":"2024-04-26T21:27:07.506944+0000","last_active":"2024-04-26T21:27:08.397305+0000","last_peered":"2024-04-26T21:27:08.397305+0000","last_clean":"2024-04-26T21:27:08.397305+0000","last_became_active":"2024-04-26T21:27:07.506451+0000","last_became_peered":"2024-04-26T21:27:07.506451+0000","last_unstale":"2024-04-26T21:27:08.397305+0000","last_undegraded":"2024-04-26T21:27:08.397305+0000","last_fullsized":"2024-04-26T21:27:08.397305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:21.916951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397908+0000","last_change":"2024-04-26T21:27:02.874116+0000","last_active":"2024-04-26T21:27:08.397908+0000","last_peered":"2024-04-26T21:27:08.397908+0000","last_clean":"2024-04-26T21:27:08.397908+0000","last_became_active":"2024-04-26T21:27:02.871334+0000","last_became_peered":"2024-04-26T21:27:02.871334+0000","last_unstale":"2024-04-26T21:27:08.397908+0000","last_undegraded":"2024-04-26T21:27:08.397908+0000","last_fullsized":"2024-04-26T21:27:08.397908+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:28.835061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.473757+0000","last_change":"2024-04-26T21:27:02.303089+0000","last_active":"2024-04-26T21:27:28.473757+0000","last_peered":"2024-04-26T21:27:28.473757+0000","last_clean":"2024-04-26T21:27:28.473757+0000","last_became_active":"2024-04-26T21:27:02.301793+0000","last_became_peered":"2024-04-26T21:27:02.301793+0000","last_unstale":"2024-04-26T21:27:28.473757+0000","last_undegraded":"2024-04-26T21:27:28.473757+0000","last_fullsized":"2024-04-26T21:27:28.473757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:29.599675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398914+0000","last_change":"2024-04-26T21:27:07.373894+0000","last_active":"2024-04-26T21:27:08.398914+0000","last_peered":"2024-04-26T21:27:08.398914+0000","last_clean":"2024-04-26T21:27:08.398914+0000","last_became_active":"2024-04-26T21:27:07.373338+0000","last_became_peered":"2024-04-26T21:27:07.373338+0000","last_unstale":"2024-04-26T21:27:08.398914+0000","last_undegraded":"2024-04-26T21:27:08.398914+0000","last_fullsized":"2024-04-26T21:27:08.398914+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:49.636493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.465810+0000","last_change":"2024-04-26T21:27:02.307754+0000","last_active":"2024-04-26T21:27:27.465810+0000","last_peered":"2024-04-26T21:27:27.465810+0000","last_clean":"2024-04-26T21:27:27.465810+0000","last_became_active":"2024-04-26T21:27:02.307423+0000","last_became_peered":"2024-04-26T21:27:02.307423+0000","last_unstale":"2024-04-26T21:27:27.465810+0000","last_undegraded":"2024-04-26T21:27:27.465810+0000","last_fullsized":"2024-04-26T21:27:27.465810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:26.701568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400337+0000","last_change":"2024-04-26T21:27:02.869831+0000","last_active":"2024-04-26T21:27:08.400337+0000","last_peered":"2024-04-26T21:27:08.400337+0000","last_clean":"2024-04-26T21:27:08.400337+0000","last_became_active":"2024-04-26T21:27:02.869375+0000","last_became_peered":"2024-04-26T21:27:02.869375+0000","last_unstale":"2024-04-26T21:27:08.400337+0000","last_undegraded":"2024-04-26T21:27:08.400337+0000","last_fullsized":"2024-04-26T21:27:08.400337+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:20.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399580+0000","last_change":"2024-04-26T21:27:07.507562+0000","last_active":"2024-04-26T21:27:08.399580+0000","last_peered":"2024-04-26T21:27:08.399580+0000","last_clean":"2024-04-26T21:27:08.399580+0000","last_became_active":"2024-04-26T21:27:07.507063+0000","last_became_peered":"2024-04-26T21:27:07.507063+0000","last_unstale":"2024-04-26T21:27:08.399580+0000","last_undegraded":"2024-04-26T21:27:08.399580+0000","last_fullsized":"2024-04-26T21:27:08.399580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:58.192617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.477333+0000","last_change":"2024-04-26T21:27:02.315360+0000","last_active":"2024-04-26T21:27:28.477333+0000","last_peered":"2024-04-26T21:27:28.477333+0000","last_clean":"2024-04-26T21:27:28.477333+0000","last_became_active":"2024-04-26T21:27:02.314817+0000","last_became_peered":"2024-04-26T21:27:02.314817+0000","last_unstale":"2024-04-26T21:27:28.477333+0000","last_undegraded":"2024-04-26T21:27:28.477333+0000","last_fullsized":"2024-04-26T21:27:28.477333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:12.045363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400867+0000","last_change":"2024-04-26T21:27:02.857006+0000","last_active":"2024-04-26T21:27:08.400867+0000","last_peered":"2024-04-26T21:27:08.400867+0000","last_clean":"2024-04-26T21:27:08.400867+0000","last_became_active":"2024-04-26T21:27:02.856589+0000","last_became_peered":"2024-04-26T21:27:02.856589+0000","last_unstale":"2024-04-26T21:27:08.400867+0000","last_undegraded":"2024-04-26T21:27:08.400867+0000","last_fullsized":"2024-04-26T21:27:08.400867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:10.292791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337336+0000","last_change":"2024-04-26T21:27:07.362016+0000","last_active":"2024-04-26T21:27:08.337336+0000","last_peered":"2024-04-26T21:27:08.337336+0000","last_clean":"2024-04-26T21:27:08.337336+0000","last_became_active":"2024-04-26T21:27:07.361360+0000","last_became_peered":"2024-04-26T21:27:07.361360+0000","last_unstale":"2024-04-26T21:27:08.337336+0000","last_undegraded":"2024-04-26T21:27:08.337336+0000","last_fullsized":"2024-04-26T21:27:08.337336+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:03.196293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.465015+0000","last_change":"2024-04-26T21:27:02.308088+0000","last_active":"2024-04-26T21:27:27.465015+0000","last_peered":"2024-04-26T21:27:27.465015+0000","last_clean":"2024-04-26T21:27:27.465015+0000","last_became_active":"2024-04-26T21:27:02.307727+0000","last_became_peered":"2024-04-26T21:27:02.307727+0000","last_unstale":"2024-04-26T21:27:27.465015+0000","last_undegraded":"2024-04-26T21:27:27.465015+0000","last_fullsized":"2024-04-26T21:27:27.465015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:02.132125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.340082+0000","last_change":"2024-04-26T21:27:02.847821+0000","last_active":"2024-04-26T21:27:08.340082+0000","last_peered":"2024-04-26T21:27:08.340082+0000","last_clean":"2024-04-26T21:27:08.340082+0000","last_became_active":"2024-04-26T21:27:02.847167+0000","last_became_peered":"2024-04-26T21:27:02.847167+0000","last_unstale":"2024-04-26T21:27:08.340082+0000","last_undegraded":"2024-04-26T21:27:08.340082+0000","last_fullsized":"2024-04-26T21:27:08.340082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:54:46.596138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.342017+0000","last_change":"2024-04-26T21:27:07.506609+0000","last_active":"2024-04-26T21:27:08.342017+0000","last_peered":"2024-04-26T21:27:08.342017+0000","last_clean":"2024-04-26T21:27:08.342017+0000","last_became_active":"2024-04-26T21:27:07.506100+0000","last_became_peered":"2024-04-26T21:27:07.506100+0000","last_unstale":"2024-04-26T21:27:08.342017+0000","last_undegraded":"2024-04-26T21:27:08.342017+0000","last_fullsized":"2024-04-26T21:27:08.342017+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:26.535809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.464939+0000","last_change":"2024-04-26T21:27:02.297432+0000","last_active":"2024-04-26T21:27:27.464939+0000","last_peered":"2024-04-26T21:27:27.464939+0000","last_clean":"2024-04-26T21:27:27.464939+0000","last_became_active":"2024-04-26T21:27:02.297081+0000","last_became_peered":"2024-04-26T21:27:02.297081+0000","last_unstale":"2024-04-26T21:27:27.464939+0000","last_undegraded":"2024-04-26T21:27:27.464939+0000","last_fullsized":"2024-04-26T21:27:27.464939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:08.139469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.396953+0000","last_change":"2024-04-26T21:27:02.866653+0000","last_active":"2024-04-26T21:27:08.396953+0000","last_peered":"2024-04-26T21:27:08.396953+0000","last_clean":"2024-04-26T21:27:08.396953+0000","last_became_active":"2024-04-26T21:27:02.866275+0000","last_became_peered":"2024-04-26T21:27:02.866275+0000","last_unstale":"2024-04-26T21:27:08.396953+0000","last_undegraded":"2024-04-26T21:27:08.396953+0000","last_fullsized":"2024-04-26T21:27:08.396953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:39.089972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401332+0000","last_change":"2024-04-26T21:27:07.377084+0000","last_active":"2024-04-26T21:27:08.401332+0000","last_peered":"2024-04-26T21:27:08.401332+0000","last_clean":"2024-04-26T21:27:08.401332+0000","last_became_active":"2024-04-26T21:27:07.374052+0000","last_became_peered":"2024-04-26T21:27:07.374052+0000","last_unstale":"2024-04-26T21:27:08.401332+0000","last_undegraded":"2024-04-26T21:27:08.401332+0000","last_fullsized":"2024-04-26T21:27:08.401332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:06:20.031170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.468113+0000","last_change":"2024-04-26T21:27:02.307250+0000","last_active":"2024-04-26T21:27:27.468113+0000","last_peered":"2024-04-26T21:27:27.468113+0000","last_clean":"2024-04-26T21:27:27.468113+0000","last_became_active":"2024-04-26T21:27:02.303532+0000","last_became_peered":"2024-04-26T21:27:02.303532+0000","last_unstale":"2024-04-26T21:27:27.468113+0000","last_undegraded":"2024-04-26T21:27:27.468113+0000","last_fullsized":"2024-04-26T21:27:27.468113+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:36.265188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398462+0000","last_change":"2024-04-26T21:27:02.858818+0000","last_active":"2024-04-26T21:27:08.398462+0000","last_peered":"2024-04-26T21:27:08.398462+0000","last_clean":"2024-04-26T21:27:08.398462+0000","last_became_active":"2024-04-26T21:27:02.858059+0000","last_became_peered":"2024-04-26T21:27:02.858059+0000","last_unstale":"2024-04-26T21:27:08.398462+0000","last_undegraded":"2024-04-26T21:27:08.398462+0000","last_fullsized":"2024-04-26T21:27:08.398462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:02.159334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341651+0000","last_change":"2024-04-26T21:27:07.381273+0000","last_active":"2024-04-26T21:27:08.341651+0000","last_peered":"2024-04-26T21:27:08.341651+0000","last_clean":"2024-04-26T21:27:08.341651+0000","last_became_active":"2024-04-26T21:27:07.380891+0000","last_became_peered":"2024-04-26T21:27:07.380891+0000","last_unstale":"2024-04-26T21:27:08.341651+0000","last_undegraded":"2024-04-26T21:27:08.341651+0000","last_fullsized":"2024-04-26T21:27:08.341651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:39.641577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337283+0000","last_change":"2024-04-26T21:27:02.301363+0000","last_active":"2024-04-26T21:27:08.337283+0000","last_peered":"2024-04-26T21:27:08.337283+0000","last_clean":"2024-04-26T21:27:08.337283+0000","last_became_active":"2024-04-26T21:27:02.301059+0000","last_became_peered":"2024-04-26T21:27:02.301059+0000","last_unstale":"2024-04-26T21:27:08.337283+0000","last_undegraded":"2024-04-26T21:27:08.337283+0000","last_fullsized":"2024-04-26T21:27:08.337283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:17.586190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.402014+0000","last_change":"2024-04-26T21:27:02.860612+0000","last_active":"2024-04-26T21:27:08.402014+0000","last_peered":"2024-04-26T21:27:08.402014+0000","last_clean":"2024-04-26T21:27:08.402014+0000","last_became_active":"2024-04-26T21:27:02.857285+0000","last_became_peered":"2024-04-26T21:27:02.857285+0000","last_unstale":"2024-04-26T21:27:08.402014+0000","last_undegraded":"2024-04-26T21:27:08.402014+0000","last_fullsized":"2024-04-26T21:27:08.402014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:11.968818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341818+0000","last_change":"2024-04-26T21:27:07.506425+0000","last_active":"2024-04-26T21:27:08.341818+0000","last_peered":"2024-04-26T21:27:08.341818+0000","last_clean":"2024-04-26T21:27:08.341818+0000","last_became_active":"2024-04-26T21:27:07.505968+0000","last_became_peered":"2024-04-26T21:27:07.505968+0000","last_unstale":"2024-04-26T21:27:08.341818+0000","last_undegraded":"2024-04-26T21:27:08.341818+0000","last_fullsized":"2024-04-26T21:27:08.341818+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:59.914560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.465016+0000","last_change":"2024-04-26T21:27:02.294635+0000","last_active":"2024-04-26T21:27:27.465016+0000","last_peered":"2024-04-26T21:27:27.465016+0000","last_clean":"2024-04-26T21:27:27.465016+0000","last_became_active":"2024-04-26T21:27:02.294311+0000","last_became_peered":"2024-04-26T21:27:02.294311+0000","last_unstale":"2024-04-26T21:27:27.465016+0000","last_undegraded":"2024-04-26T21:27:27.465016+0000","last_fullsized":"2024-04-26T21:27:27.465016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:47.906546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398900+0000","last_change":"2024-04-26T21:27:02.875475+0000","last_active":"2024-04-26T21:27:08.398900+0000","last_peered":"2024-04-26T21:27:08.398900+0000","last_clean":"2024-04-26T21:27:08.398900+0000","last_became_active":"2024-04-26T21:27:02.875137+0000","last_became_peered":"2024-04-26T21:27:02.875137+0000","last_unstale":"2024-04-26T21:27:08.398900+0000","last_undegraded":"2024-04-26T21:27:08.398900+0000","last_fullsized":"2024-04-26T21:27:08.398900+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:04.988973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397080+0000","last_change":"2024-04-26T21:27:07.372927+0000","last_active":"2024-04-26T21:27:08.397080+0000","last_peered":"2024-04-26T21:27:08.397080+0000","last_clean":"2024-04-26T21:27:08.397080+0000","last_became_active":"2024-04-26T21:27:07.372371+0000","last_became_peered":"2024-04-26T21:27:07.372371+0000","last_unstale":"2024-04-26T21:27:08.397080+0000","last_undegraded":"2024-04-26T21:27:08.397080+0000","last_fullsized":"2024-04-26T21:27:08.397080+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:12:41.607167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.477987+0000","last_change":"2024-04-26T21:27:02.295379+0000","last_active":"2024-04-26T21:27:28.477987+0000","last_peered":"2024-04-26T21:27:28.477987+0000","last_clean":"2024-04-26T21:27:28.477987+0000","last_became_active":"2024-04-26T21:27:02.295077+0000","last_became_peered":"2024-04-26T21:27:02.295077+0000","last_unstale":"2024-04-26T21:27:28.477987+0000","last_undegraded":"2024-04-26T21:27:28.477987+0000","last_fullsized":"2024-04-26T21:27:28.477987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:58.177960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399407+0000","last_change":"2024-04-26T21:27:02.855998+0000","last_active":"2024-04-26T21:27:08.399407+0000","last_peered":"2024-04-26T21:27:08.399407+0000","last_clean":"2024-04-26T21:27:08.399407+0000","last_became_active":"2024-04-26T21:27:02.855636+0000","last_became_peered":"2024-04-26T21:27:02.855636+0000","last_unstale":"2024-04-26T21:27:08.399407+0000","last_undegraded":"2024-04-26T21:27:08.399407+0000","last_fullsized":"2024-04-26T21:27:08.399407+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:19:02.988946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.339186+0000","last_change":"2024-04-26T21:27:07.378980+0000","last_active":"2024-04-26T21:27:08.339186+0000","last_peered":"2024-04-26T21:27:08.339186+0000","last_clean":"2024-04-26T21:27:08.339186+0000","last_became_active":"2024-04-26T21:27:07.378374+0000","last_became_peered":"2024-04-26T21:27:07.378374+0000","last_unstale":"2024-04-26T21:27:08.339186+0000","last_undegraded":"2024-04-26T21:27:08.339186+0000","last_fullsized":"2024-04-26T21:27:08.339186+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:31.845297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397898+0000","last_change":"2024-04-26T21:27:02.313801+0000","last_active":"2024-04-26T21:27:08.397898+0000","last_peered":"2024-04-26T21:27:08.397898+0000","last_clean":"2024-04-26T21:27:08.397898+0000","last_became_active":"2024-04-26T21:27:02.313540+0000","last_became_peered":"2024-04-26T21:27:02.313540+0000","last_unstale":"2024-04-26T21:27:08.397898+0000","last_undegraded":"2024-04-26T21:27:08.397898+0000","last_fullsized":"2024-04-26T21:27:08.397898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:38.612261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399083+0000","last_change":"2024-04-26T21:27:02.864151+0000","last_active":"2024-04-26T21:27:08.399083+0000","last_peered":"2024-04-26T21:27:08.399083+0000","last_clean":"2024-04-26T21:27:08.399083+0000","last_became_active":"2024-04-26T21:27:02.863818+0000","last_became_peered":"2024-04-26T21:27:02.863818+0000","last_unstale":"2024-04-26T21:27:08.399083+0000","last_undegraded":"2024-04-26T21:27:08.399083+0000","last_fullsized":"2024-04-26T21:27:08.399083+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:26.215370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397531+0000","last_change":"2024-04-26T21:27:07.365821+0000","last_active":"2024-04-26T21:27:08.397531+0000","last_peered":"2024-04-26T21:27:08.397531+0000","last_clean":"2024-04-26T21:27:08.397531+0000","last_became_active":"2024-04-26T21:27:07.365347+0000","last_became_peered":"2024-04-26T21:27:07.365347+0000","last_unstale":"2024-04-26T21:27:08.397531+0000","last_undegraded":"2024-04-26T21:27:08.397531+0000","last_fullsized":"2024-04-26T21:27:08.397531+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:40:11.596908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401414+0000","last_change":"2024-04-26T21:27:02.310641+0000","last_active":"2024-04-26T21:27:08.401414+0000","last_peered":"2024-04-26T21:27:08.401414+0000","last_clean":"2024-04-26T21:27:08.401414+0000","last_became_active":"2024-04-26T21:27:02.308569+0000","last_became_peered":"2024-04-26T21:27:02.308569+0000","last_unstale":"2024-04-26T21:27:08.401414+0000","last_undegraded":"2024-04-26T21:27:08.401414+0000","last_fullsized":"2024-04-26T21:27:08.401414+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:20.988098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399291+0000","last_change":"2024-04-26T21:27:02.854976+0000","last_active":"2024-04-26T21:27:08.399291+0000","last_peered":"2024-04-26T21:27:08.399291+0000","last_clean":"2024-04-26T21:27:08.399291+0000","last_became_active":"2024-04-26T21:27:02.854611+0000","last_became_peered":"2024-04-26T21:27:02.854611+0000","last_unstale":"2024-04-26T21:27:08.399291+0000","last_undegraded":"2024-04-26T21:27:08.399291+0000","last_fullsized":"2024-04-26T21:27:08.399291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:57.065657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398537+0000","last_change":"2024-04-26T21:27:02.859017+0000","last_active":"2024-04-26T21:27:08.398537+0000","last_peered":"2024-04-26T21:27:08.398537+0000","last_clean":"2024-04-26T21:27:08.398537+0000","last_became_active":"2024-04-26T21:27:02.858507+0000","last_became_peered":"2024-04-26T21:27:02.858507+0000","last_unstale":"2024-04-26T21:27:08.398537+0000","last_undegraded":"2024-04-26T21:27:08.398537+0000","last_fullsized":"2024-04-26T21:27:08.398537+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:14.850891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337861+0000","last_change":"2024-04-26T21:27:02.858247+0000","last_active":"2024-04-26T21:27:08.337861+0000","last_peered":"2024-04-26T21:27:08.337861+0000","last_clean":"2024-04-26T21:27:08.337861+0000","last_became_active":"2024-04-26T21:27:02.857900+0000","last_became_peered":"2024-04-26T21:27:02.857900+0000","last_unstale":"2024-04-26T21:27:08.337861+0000","last_undegraded":"2024-04-26T21:27:08.337861+0000","last_fullsized":"2024-04-26T21:27:08.337861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:02:43.998925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337151+0000","last_change":"2024-04-26T21:27:02.868817+0000","last_active":"2024-04-26T21:27:08.337151+0000","last_peered":"2024-04-26T21:27:08.337151+0000","last_clean":"2024-04-26T21:27:08.337151+0000","last_became_active":"2024-04-26T21:27:02.868557+0000","last_became_peered":"2024-04-26T21:27:02.868557+0000","last_unstale":"2024-04-26T21:27:08.337151+0000","last_undegraded":"2024-04-26T21:27:08.337151+0000","last_fullsized":"2024-04-26T21:27:08.337151+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:55.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.339735+0000","last_change":"2024-04-26T21:27:02.876029+0000","last_active":"2024-04-26T21:27:08.339735+0000","last_peered":"2024-04-26T21:27:08.339735+0000","last_clean":"2024-04-26T21:27:08.339735+0000","last_became_active":"2024-04-26T21:27:02.875656+0000","last_became_peered":"2024-04-26T21:27:02.875656+0000","last_unstale":"2024-04-26T21:27:08.339735+0000","last_undegraded":"2024-04-26T21:27:08.339735+0000","last_fullsized":"2024-04-26T21:27:08.339735+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:47.317581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400469+0000","last_change":"2024-04-26T21:27:02.873818+0000","last_active":"2024-04-26T21:27:08.400469+0000","last_peered":"2024-04-26T21:27:08.400469+0000","last_clean":"2024-04-26T21:27:08.400469+0000","last_became_active":"2024-04-26T21:27:02.873459+0000","last_became_peered":"2024-04-26T21:27:02.873459+0000","last_unstale":"2024-04-26T21:27:08.400469+0000","last_undegraded":"2024-04-26T21:27:08.400469+0000","last_fullsized":"2024-04-26T21:27:08.400469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:07.259187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398912+0000","last_change":"2024-04-26T21:27:02.854901+0000","last_active":"2024-04-26T21:27:08.398912+0000","last_peered":"2024-04-26T21:27:08.398912+0000","last_clean":"2024-04-26T21:27:08.398912+0000","last_became_active":"2024-04-26T21:27:02.854547+0000","last_became_peered":"2024-04-26T21:27:02.854547+0000","last_unstale":"2024-04-26T21:27:08.398912+0000","last_undegraded":"2024-04-26T21:27:08.398912+0000","last_fullsized":"2024-04-26T21:27:08.398912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:24.320868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337890+0000","last_change":"2024-04-26T21:27:02.856616+0000","last_active":"2024-04-26T21:27:08.337890+0000","last_peered":"2024-04-26T21:27:08.337890+0000","last_clean":"2024-04-26T21:27:08.337890+0000","last_became_active":"2024-04-26T21:27:02.856264+0000","last_became_peered":"2024-04-26T21:27:02.856264+0000","last_unstale":"2024-04-26T21:27:08.337890+0000","last_undegraded":"2024-04-26T21:27:08.337890+0000","last_fullsized":"2024-04-26T21:27:08.337890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:43.374534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401714+0000","last_change":"2024-04-26T21:27:02.859587+0000","last_active":"2024-04-26T21:27:08.401714+0000","last_peered":"2024-04-26T21:27:08.401714+0000","last_clean":"2024-04-26T21:27:08.401714+0000","last_became_active":"2024-04-26T21:27:02.859136+0000","last_became_peered":"2024-04-26T21:27:02.859136+0000","last_unstale":"2024-04-26T21:27:08.401714+0000","last_undegraded":"2024-04-26T21:27:08.401714+0000","last_fullsized":"2024-04-26T21:27:08.401714+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:09.370319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397549+0000","last_change":"2024-04-26T21:27:02.864265+0000","last_active":"2024-04-26T21:27:08.397549+0000","last_peered":"2024-04-26T21:27:08.397549+0000","last_clean":"2024-04-26T21:27:08.397549+0000","last_became_active":"2024-04-26T21:27:02.863920+0000","last_became_peered":"2024-04-26T21:27:02.863920+0000","last_unstale":"2024-04-26T21:27:08.397549+0000","last_undegraded":"2024-04-26T21:27:08.397549+0000","last_fullsized":"2024-04-26T21:27:08.397549+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:13.941544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400091+0000","last_change":"2024-04-26T21:27:02.860189+0000","last_active":"2024-04-26T21:27:08.400091+0000","last_peered":"2024-04-26T21:27:08.400091+0000","last_clean":"2024-04-26T21:27:08.400091+0000","last_became_active":"2024-04-26T21:27:02.859843+0000","last_became_peered":"2024-04-26T21:27:02.859843+0000","last_unstale":"2024-04-26T21:27:08.400091+0000","last_undegraded":"2024-04-26T21:27:08.400091+0000","last_fullsized":"2024-04-26T21:27:08.400091+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397337+0000","last_change":"2024-04-26T21:27:02.865459+0000","last_active":"2024-04-26T21:27:08.397337+0000","last_peered":"2024-04-26T21:27:08.397337+0000","last_clean":"2024-04-26T21:27:08.397337+0000","last_became_active":"2024-04-26T21:27:02.863602+0000","last_became_peered":"2024-04-26T21:27:02.863602+0000","last_unstale":"2024-04-26T21:27:08.397337+0000","last_undegraded":"2024-04-26T21:27:08.397337+0000","last_fullsized":"2024-04-26T21:27:08.397337+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:03.733858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398941+0000","last_change":"2024-04-26T21:27:02.873711+0000","last_active":"2024-04-26T21:27:08.398941+0000","last_peered":"2024-04-26T21:27:08.398941+0000","last_clean":"2024-04-26T21:27:08.398941+0000","last_became_active":"2024-04-26T21:27:02.873349+0000","last_became_peered":"2024-04-26T21:27:02.873349+0000","last_unstale":"2024-04-26T21:27:08.398941+0000","last_undegraded":"2024-04-26T21:27:08.398941+0000","last_fullsized":"2024-04-26T21:27:08.398941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:10.841536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398767+0000","last_change":"2024-04-26T21:27:02.852625+0000","last_active":"2024-04-26T21:27:08.398767+0000","last_peered":"2024-04-26T21:27:08.398767+0000","last_clean":"2024-04-26T21:27:08.398767+0000","last_became_active":"2024-04-26T21:27:02.852177+0000","last_became_peered":"2024-04-26T21:27:02.852177+0000","last_unstale":"2024-04-26T21:27:08.398767+0000","last_undegraded":"2024-04-26T21:27:08.398767+0000","last_fullsized":"2024-04-26T21:27:08.398767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:52.046678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7452,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":131,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":98484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318291,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":486206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579928,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1634304,"data_stored":1080790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":490538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005093,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":486362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266730,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32752,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712400,"statfs":{"total":95995035648,"available":95961497600,"internally_reserved":0,"allocated":1613824,"data_stored":1077784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462960,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":485624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":527530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":487102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":528880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":485908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888124,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":490364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724598,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36928,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708224,"statfs":{"total":95995035648,"available":95957221376,"internally_reserved":0,"allocated":1630208,"data_stored":1081564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":41906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":43256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:27:33.730 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T21:27:33.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.986+0000 7f48fd800640 1 -- 172.21.15.1:0/2151813076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48f8104bb0 msgr2=0x7f48f8104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:33.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.986+0000 7f48fd800640 1 --2- 172.21.15.1:0/2151813076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48f8104bb0 0x7f48f8104fd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f48e0009860 tx=0x7f48e002a340 comp rx=0 tx=0).stop 2024-04-26T21:27:33.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.986+0000 7f48fd800640 1 -- 172.21.15.1:0/2151813076 shutdown_connections 2024-04-26T21:27:33.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.986+0000 7f48fd800640 1 --2- 172.21.15.1:0/2151813076 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8106760 0x7f48f810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.986+0000 7f48fd800640 1 --2- 172.21.15.1:0/2151813076 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48f8105da0 0x7f48f8106220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.986+0000 7f48fd800640 1 --2- 172.21.15.1:0/2151813076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48f8104bb0 0x7f48f8104fd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.986+0000 7f48fd800640 1 -- 172.21.15.1:0/2151813076 >> 172.21.15.1:0/2151813076 conn(0x7f48f8100120 msgr2=0x7f48f8102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:33.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.986+0000 7f48fd800640 1 -- 172.21.15.1:0/2151813076 shutdown_connections 2024-04-26T21:27:33.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.986+0000 7f48fd800640 1 -- 172.21.15.1:0/2151813076 wait complete. 2024-04-26T21:27:33.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.987+0000 7f48fd800640 1 Processor -- start 2024-04-26T21:27:33.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48fd800640 1 -- start start 2024-04-26T21:27:33.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48fd800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48f8105da0 0x7f48f819b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:33.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48fd800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8106760 0x7f48f819b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:33.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48fd800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48f81a05b0 0x7f48f81a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:33.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48fd800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f48f81106d0 con 0x7f48f8106760 2024-04-26T21:27:33.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48fd800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f48f8110550 con 0x7f48f8105da0 2024-04-26T21:27:33.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48fd800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48f8110850 con 0x7f48f81a05b0 2024-04-26T21:27:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48f8105da0 0x7f48f819b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48f6000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8106760 0x7f48f819b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48f8105da0 0x7f48f819b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:46862/0 (socket says 172.21.15.1:46862) 2024-04-26T21:27:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48f6a00640 1 -- 172.21.15.1:0/82489797 learned_addr learned my addr 172.21.15.1:0/82489797 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.988+0000 7f48f7400640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48f81a05b0 0x7f48f81a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48f6a00640 1 -- 172.21.15.1:0/82489797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48f81a05b0 msgr2=0x7f48f81a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48f6a00640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48f81a05b0 0x7f48f81a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48f6a00640 1 -- 172.21.15.1:0/82489797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8106760 msgr2=0x7f48f819b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48f6a00640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8106760 0x7f48f819b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:33.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48f6a00640 1 -- 172.21.15.1:0/82489797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f81a3010 con 0x7f48f8105da0 2024-04-26T21:27:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48f6a00640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48f8105da0 0x7f48f819b010 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f48e0035040 tx=0x7f48e0032e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48ef400640 1 -- 172.21.15.1:0/82489797 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0037040 con 0x7f48f8105da0 2024-04-26T21:27:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48ef400640 1 -- 172.21.15.1:0/82489797 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48e0030e50 con 0x7f48f8105da0 2024-04-26T21:27:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48ef400640 1 -- 172.21.15.1:0/82489797 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e00053f0 con 0x7f48f8105da0 2024-04-26T21:27:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48fd800640 1 -- 172.21.15.1:0/82489797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48f81a32a0 con 0x7f48f8105da0 2024-04-26T21:27:33.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.989+0000 7f48fd800640 1 -- 172.21.15.1:0/82489797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48f81a3780 con 0x7f48f8105da0 2024-04-26T21:27:33.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.991+0000 7f48fd800640 1 -- 172.21.15.1:0/82489797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48f810c040 con 0x7f48f8105da0 2024-04-26T21:27:33.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.993+0000 7f48ef400640 1 -- 172.21.15.1:0/82489797 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f48e0043060 con 0x7f48f8105da0 2024-04-26T21:27:33.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.994+0000 7f48ef400640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f48dc0a3cc0 0x7f48dc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:33.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.994+0000 7f48ef400640 1 -- 172.21.15.1:0/82489797 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f48e00efa50 con 0x7f48f8105da0 2024-04-26T21:27:33.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.994+0000 7f48f6000640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f48dc0a3cc0 0x7f48dc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:33.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.995+0000 7f48f6000640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f48dc0a3cc0 0x7f48dc0a6180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f48f819c520 tx=0x7f48e8010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:33.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:33.996+0000 7f48ef400640 1 -- 172.21.15.1:0/82489797 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f48e00bc1c0 con 0x7f48f8105da0 2024-04-26T21:27:34.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:33 smithi001 ceph-mon[28054]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:34.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:33 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2695207121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:27:34.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:33 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/390356270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:33 smithi038 ceph-mon[32724]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:33 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2695207121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:27:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:33 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/390356270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:34.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.200+0000 7f48fd800640 1 -- 172.21.15.1:0/82489797 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f48f8062d70 con 0x7f48dc0a3cc0 2024-04-26T21:27:34.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:33 smithi186 ceph-mon[32991]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:27:34.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:33 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2695207121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:27:34.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:33 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/390356270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:34.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.217+0000 7f48ef400640 1 -- 172.21.15.1:0/82489797 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340383 (secure 0 0 0) 0x7f48f8062d70 con 0x7f48dc0a3cc0 2024-04-26T21:27:34.220 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:34.224 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.224+0000 7f48fd800640 1 -- 172.21.15.1:0/82489797 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f48dc0a3cc0 msgr2=0x7f48dc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.224+0000 7f48fd800640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f48dc0a3cc0 0x7f48dc0a6180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f48f819c520 tx=0x7f48e8010040 comp rx=0 tx=0).stop 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.224+0000 7f48fd800640 1 -- 172.21.15.1:0/82489797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48f8105da0 msgr2=0x7f48f819b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.224+0000 7f48fd800640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48f8105da0 0x7f48f819b010 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f48e0035040 tx=0x7f48e0032e90 comp rx=0 tx=0).stop 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.224+0000 7f48fd800640 1 -- 172.21.15.1:0/82489797 shutdown_connections 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.224+0000 7f48fd800640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f48dc0a3cc0 0x7f48dc0a6180 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.224+0000 7f48fd800640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48f81a05b0 0x7f48f81a29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.224+0000 7f48fd800640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48f8106760 0x7f48f819b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.224+0000 7f48fd800640 1 --2- 172.21.15.1:0/82489797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48f8105da0 0x7f48f819b010 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.224+0000 7f48fd800640 1 -- 172.21.15.1:0/82489797 >> 172.21.15.1:0/82489797 conn(0x7f48f8100120 msgr2=0x7f48f8109210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.225+0000 7f48fd800640 1 -- 172.21.15.1:0/82489797 shutdown_connections 2024-04-26T21:27:34.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:34.225+0000 7f48fd800640 1 -- 172.21.15.1:0/82489797 wait complete. 2024-04-26T21:27:34.247 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-04-26T21:27:33.310518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597937,"num_objects":63,"num_object_clones":0,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":167,"num_read_kb":144,"num_write":313,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":379612,"kb_used_data":13820,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124562212,"statfs":{"total":1151940427776,"available":1151551705088,"internally_reserved":0,"allocated":14151680,"data_stored":7708652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3070,"num_objects":39,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":84,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006175"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399037+0000","last_change":"2024-04-26T21:27:02.854724+0000","last_active":"2024-04-26T21:27:08.399037+0000","last_peered":"2024-04-26T21:27:08.399037+0000","last_clean":"2024-04-26T21:27:08.399037+0000","last_became_active":"2024-04-26T21:27:02.854125+0000","last_became_peered":"2024-04-26T21:27:02.854125+0000","last_unstale":"2024-04-26T21:27:08.399037+0000","last_undegraded":"2024-04-26T21:27:08.399037+0000","last_fullsized":"2024-04-26T21:27:08.399037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:16.500220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.339983+0000","last_change":"2024-04-26T21:27:02.852846+0000","last_active":"2024-04-26T21:27:08.339983+0000","last_peered":"2024-04-26T21:27:08.339983+0000","last_clean":"2024-04-26T21:27:08.339983+0000","last_became_active":"2024-04-26T21:27:02.851243+0000","last_became_peered":"2024-04-26T21:27:02.851243+0000","last_unstale":"2024-04-26T21:27:08.339983+0000","last_undegraded":"2024-04-26T21:27:08.339983+0000","last_fullsized":"2024-04-26T21:27:08.339983+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:09:11.240955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.339547+0000","last_change":"2024-04-26T21:27:02.855856+0000","last_active":"2024-04-26T21:27:08.339547+0000","last_peered":"2024-04-26T21:27:08.339547+0000","last_clean":"2024-04-26T21:27:08.339547+0000","last_became_active":"2024-04-26T21:27:02.855446+0000","last_became_peered":"2024-04-26T21:27:02.855446+0000","last_unstale":"2024-04-26T21:27:08.339547+0000","last_undegraded":"2024-04-26T21:27:08.339547+0000","last_fullsized":"2024-04-26T21:27:08.339547+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:35.183449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341803+0000","last_change":"2024-04-26T21:27:02.860949+0000","last_active":"2024-04-26T21:27:08.341803+0000","last_peered":"2024-04-26T21:27:08.341803+0000","last_clean":"2024-04-26T21:27:08.341803+0000","last_became_active":"2024-04-26T21:27:02.860567+0000","last_became_peered":"2024-04-26T21:27:02.860567+0000","last_unstale":"2024-04-26T21:27:08.341803+0000","last_undegraded":"2024-04-26T21:27:08.341803+0000","last_fullsized":"2024-04-26T21:27:08.341803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:09.851885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397220+0000","last_change":"2024-04-26T21:27:02.876019+0000","last_active":"2024-04-26T21:27:08.397220+0000","last_peered":"2024-04-26T21:27:08.397220+0000","last_clean":"2024-04-26T21:27:08.397220+0000","last_became_active":"2024-04-26T21:27:02.875652+0000","last_became_peered":"2024-04-26T21:27:02.875652+0000","last_unstale":"2024-04-26T21:27:08.397220+0000","last_undegraded":"2024-04-26T21:27:08.397220+0000","last_fullsized":"2024-04-26T21:27:08.397220+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:35.017787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397751+0000","last_change":"2024-04-26T21:27:02.860229+0000","last_active":"2024-04-26T21:27:08.397751+0000","last_peered":"2024-04-26T21:27:08.397751+0000","last_clean":"2024-04-26T21:27:08.397751+0000","last_became_active":"2024-04-26T21:27:02.859863+0000","last_became_peered":"2024-04-26T21:27:02.859863+0000","last_unstale":"2024-04-26T21:27:08.397751+0000","last_undegraded":"2024-04-26T21:27:08.397751+0000","last_fullsized":"2024-04-26T21:27:08.397751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:47.337073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398287+0000","last_change":"2024-04-26T21:27:02.863253+0000","last_active":"2024-04-26T21:27:08.398287+0000","last_peered":"2024-04-26T21:27:08.398287+0000","last_clean":"2024-04-26T21:27:08.398287+0000","last_became_active":"2024-04-26T21:27:02.861623+0000","last_became_peered":"2024-04-26T21:27:02.861623+0000","last_unstale":"2024-04-26T21:27:08.398287+0000","last_undegraded":"2024-04-26T21:27:08.398287+0000","last_fullsized":"2024-04-26T21:27:08.398287+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:57.642504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399585+0000","last_change":"2024-04-26T21:27:02.860822+0000","last_active":"2024-04-26T21:27:08.399585+0000","last_peered":"2024-04-26T21:27:08.399585+0000","last_clean":"2024-04-26T21:27:08.399585+0000","last_became_active":"2024-04-26T21:27:02.860482+0000","last_became_peered":"2024-04-26T21:27:02.860482+0000","last_unstale":"2024-04-26T21:27:08.399585+0000","last_undegraded":"2024-04-26T21:27:08.399585+0000","last_fullsized":"2024-04-26T21:27:08.399585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:59.224624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398297+0000","last_change":"2024-04-26T21:27:02.854782+0000","last_active":"2024-04-26T21:27:08.398297+0000","last_peered":"2024-04-26T21:27:08.398297+0000","last_clean":"2024-04-26T21:27:08.398297+0000","last_became_active":"2024-04-26T21:27:02.854423+0000","last_became_peered":"2024-04-26T21:27:02.854423+0000","last_unstale":"2024-04-26T21:27:08.398297+0000","last_undegraded":"2024-04-26T21:27:08.398297+0000","last_fullsized":"2024-04-26T21:27:08.398297+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:27:21.559775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399195+0000","last_change":"2024-04-26T21:27:02.860545+0000","last_active":"2024-04-26T21:27:08.399195+0000","last_peered":"2024-04-26T21:27:08.399195+0000","last_clean":"2024-04-26T21:27:08.399195+0000","last_became_active":"2024-04-26T21:27:02.858577+0000","last_became_peered":"2024-04-26T21:27:02.858577+0000","last_unstale":"2024-04-26T21:27:08.399195+0000","last_undegraded":"2024-04-26T21:27:08.399195+0000","last_fullsized":"2024-04-26T21:27:08.399195+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:43.390592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399281+0000","last_change":"2024-04-26T21:27:02.858685+0000","last_active":"2024-04-26T21:27:08.399281+0000","last_peered":"2024-04-26T21:27:08.399281+0000","last_clean":"2024-04-26T21:27:08.399281+0000","last_became_active":"2024-04-26T21:27:02.857054+0000","last_became_peered":"2024-04-26T21:27:02.857054+0000","last_unstale":"2024-04-26T21:27:08.399281+0000","last_undegraded":"2024-04-26T21:27:08.399281+0000","last_fullsized":"2024-04-26T21:27:08.399281+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:29.406132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341751+0000","last_change":"2024-04-26T21:27:02.852457+0000","last_active":"2024-04-26T21:27:08.341751+0000","last_peered":"2024-04-26T21:27:08.341751+0000","last_clean":"2024-04-26T21:27:08.341751+0000","last_became_active":"2024-04-26T21:27:02.852064+0000","last_became_peered":"2024-04-26T21:27:02.852064+0000","last_unstale":"2024-04-26T21:27:08.341751+0000","last_undegraded":"2024-04-26T21:27:08.341751+0000","last_fullsized":"2024-04-26T21:27:08.341751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:16.798413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399056+0000","last_change":"2024-04-26T21:27:02.872618+0000","last_active":"2024-04-26T21:27:08.399056+0000","last_peered":"2024-04-26T21:27:08.399056+0000","last_clean":"2024-04-26T21:27:08.399056+0000","last_became_active":"2024-04-26T21:27:02.872270+0000","last_became_peered":"2024-04-26T21:27:02.872270+0000","last_unstale":"2024-04-26T21:27:08.399056+0000","last_undegraded":"2024-04-26T21:27:08.399056+0000","last_fullsized":"2024-04-26T21:27:08.399056+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:45.828419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.338795+0000","last_change":"2024-04-26T21:27:02.868590+0000","last_active":"2024-04-26T21:27:08.338795+0000","last_peered":"2024-04-26T21:27:08.338795+0000","last_clean":"2024-04-26T21:27:08.338795+0000","last_became_active":"2024-04-26T21:27:02.866641+0000","last_became_peered":"2024-04-26T21:27:02.866641+0000","last_unstale":"2024-04-26T21:27:08.338795+0000","last_undegraded":"2024-04-26T21:27:08.338795+0000","last_fullsized":"2024-04-26T21:27:08.338795+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:17.313420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.338900+0000","last_change":"2024-04-26T21:27:02.867735+0000","last_active":"2024-04-26T21:27:08.338900+0000","last_peered":"2024-04-26T21:27:08.338900+0000","last_clean":"2024-04-26T21:27:08.338900+0000","last_became_active":"2024-04-26T21:27:02.867217+0000","last_became_peered":"2024-04-26T21:27:02.867217+0000","last_unstale":"2024-04-26T21:27:08.338900+0000","last_undegraded":"2024-04-26T21:27:08.338900+0000","last_fullsized":"2024-04-26T21:27:08.338900+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:04:45.533220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.396965+0000","last_change":"2024-04-26T21:27:02.857541+0000","last_active":"2024-04-26T21:27:08.396965+0000","last_peered":"2024-04-26T21:27:08.396965+0000","last_clean":"2024-04-26T21:27:08.396965+0000","last_became_active":"2024-04-26T21:27:02.855563+0000","last_became_peered":"2024-04-26T21:27:02.855563+0000","last_unstale":"2024-04-26T21:27:08.396965+0000","last_undegraded":"2024-04-26T21:27:08.396965+0000","last_fullsized":"2024-04-26T21:27:08.396965+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:39.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398898+0000","last_change":"2024-04-26T21:27:02.854461+0000","last_active":"2024-04-26T21:27:08.398898+0000","last_peered":"2024-04-26T21:27:08.398898+0000","last_clean":"2024-04-26T21:27:08.398898+0000","last_became_active":"2024-04-26T21:27:02.853691+0000","last_became_peered":"2024-04-26T21:27:02.853691+0000","last_unstale":"2024-04-26T21:27:08.398898+0000","last_undegraded":"2024-04-26T21:27:08.398898+0000","last_fullsized":"2024-04-26T21:27:08.398898+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:09:48.284180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400836+0000","last_change":"2024-04-26T21:27:02.864308+0000","last_active":"2024-04-26T21:27:08.400836+0000","last_peered":"2024-04-26T21:27:08.400836+0000","last_clean":"2024-04-26T21:27:08.400836+0000","last_became_active":"2024-04-26T21:27:02.863928+0000","last_became_peered":"2024-04-26T21:27:02.863928+0000","last_unstale":"2024-04-26T21:27:08.400836+0000","last_undegraded":"2024-04-26T21:27:08.400836+0000","last_fullsized":"2024-04-26T21:27:08.400836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:12.169394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337904+0000","last_change":"2024-04-26T21:27:02.876044+0000","last_active":"2024-04-26T21:27:08.337904+0000","last_peered":"2024-04-26T21:27:08.337904+0000","last_clean":"2024-04-26T21:27:08.337904+0000","last_became_active":"2024-04-26T21:27:02.875681+0000","last_became_peered":"2024-04-26T21:27:02.875681+0000","last_unstale":"2024-04-26T21:27:08.337904+0000","last_undegraded":"2024-04-26T21:27:08.337904+0000","last_fullsized":"2024-04-26T21:27:08.337904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:26.246916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337192+0000","last_change":"2024-04-26T21:27:07.374736+0000","last_active":"2024-04-26T21:27:08.337192+0000","last_peered":"2024-04-26T21:27:08.337192+0000","last_clean":"2024-04-26T21:27:08.337192+0000","last_became_active":"2024-04-26T21:27:07.374116+0000","last_became_peered":"2024-04-26T21:27:07.374116+0000","last_unstale":"2024-04-26T21:27:08.337192+0000","last_undegraded":"2024-04-26T21:27:08.337192+0000","last_fullsized":"2024-04-26T21:27:08.337192+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:11.450926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.478026+0000","last_change":"2024-04-26T21:27:02.302510+0000","last_active":"2024-04-26T21:27:28.478026+0000","last_peered":"2024-04-26T21:27:28.478026+0000","last_clean":"2024-04-26T21:27:28.478026+0000","last_became_active":"2024-04-26T21:27:02.302088+0000","last_became_peered":"2024-04-26T21:27:02.302088+0000","last_unstale":"2024-04-26T21:27:28.478026+0000","last_undegraded":"2024-04-26T21:27:28.478026+0000","last_fullsized":"2024-04-26T21:27:28.478026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:43.007722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401188+0000","last_change":"2024-04-26T21:27:02.861455+0000","last_active":"2024-04-26T21:27:08.401188+0000","last_peered":"2024-04-26T21:27:08.401188+0000","last_clean":"2024-04-26T21:27:08.401188+0000","last_became_active":"2024-04-26T21:27:02.861107+0000","last_became_peered":"2024-04-26T21:27:02.861107+0000","last_unstale":"2024-04-26T21:27:08.401188+0000","last_undegraded":"2024-04-26T21:27:08.401188+0000","last_fullsized":"2024-04-26T21:27:08.401188+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:10:18.382881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341432+0000","last_change":"2024-04-26T21:27:07.377797+0000","last_active":"2024-04-26T21:27:08.341432+0000","last_peered":"2024-04-26T21:27:08.341432+0000","last_clean":"2024-04-26T21:27:08.341432+0000","last_became_active":"2024-04-26T21:27:07.377314+0000","last_became_peered":"2024-04-26T21:27:07.377314+0000","last_unstale":"2024-04-26T21:27:08.341432+0000","last_undegraded":"2024-04-26T21:27:08.341432+0000","last_fullsized":"2024-04-26T21:27:08.341432+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:02:34.761034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401344+0000","last_change":"2024-04-26T21:27:02.309488+0000","last_active":"2024-04-26T21:27:08.401344+0000","last_peered":"2024-04-26T21:27:08.401344+0000","last_clean":"2024-04-26T21:27:08.401344+0000","last_became_active":"2024-04-26T21:27:02.308592+0000","last_became_peered":"2024-04-26T21:27:02.308592+0000","last_unstale":"2024-04-26T21:27:08.401344+0000","last_undegraded":"2024-04-26T21:27:08.401344+0000","last_fullsized":"2024-04-26T21:27:08.401344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:36.563847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.340948+0000","last_change":"2024-04-26T21:27:02.863204+0000","last_active":"2024-04-26T21:27:08.340948+0000","last_peered":"2024-04-26T21:27:08.340948+0000","last_clean":"2024-04-26T21:27:08.340948+0000","last_became_active":"2024-04-26T21:27:02.862466+0000","last_became_peered":"2024-04-26T21:27:02.862466+0000","last_unstale":"2024-04-26T21:27:08.340948+0000","last_undegraded":"2024-04-26T21:27:08.340948+0000","last_fullsized":"2024-04-26T21:27:08.340948+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:29.360830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341357+0000","last_change":"2024-04-26T21:27:07.378032+0000","last_active":"2024-04-26T21:27:08.341357+0000","last_peered":"2024-04-26T21:27:08.341357+0000","last_clean":"2024-04-26T21:27:08.341357+0000","last_became_active":"2024-04-26T21:27:07.377255+0000","last_became_peered":"2024-04-26T21:27:07.377255+0000","last_unstale":"2024-04-26T21:27:08.341357+0000","last_undegraded":"2024-04-26T21:27:08.341357+0000","last_fullsized":"2024-04-26T21:27:08.341357+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:18.989977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.472454+0000","last_change":"2024-04-26T21:27:02.294734+0000","last_active":"2024-04-26T21:27:28.472454+0000","last_peered":"2024-04-26T21:27:28.472454+0000","last_clean":"2024-04-26T21:27:28.472454+0000","last_became_active":"2024-04-26T21:27:02.294415+0000","last_became_peered":"2024-04-26T21:27:02.294415+0000","last_unstale":"2024-04-26T21:27:28.472454+0000","last_undegraded":"2024-04-26T21:27:28.472454+0000","last_fullsized":"2024-04-26T21:27:28.472454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:13.017537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397311+0000","last_change":"2024-04-26T21:27:02.876447+0000","last_active":"2024-04-26T21:27:08.397311+0000","last_peered":"2024-04-26T21:27:08.397311+0000","last_clean":"2024-04-26T21:27:08.397311+0000","last_became_active":"2024-04-26T21:27:02.876017+0000","last_became_peered":"2024-04-26T21:27:02.876017+0000","last_unstale":"2024-04-26T21:27:08.397311+0000","last_undegraded":"2024-04-26T21:27:08.397311+0000","last_fullsized":"2024-04-26T21:27:08.397311+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:57:05.453787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341317+0000","last_change":"2024-04-26T21:27:07.373700+0000","last_active":"2024-04-26T21:27:08.341317+0000","last_peered":"2024-04-26T21:27:08.341317+0000","last_clean":"2024-04-26T21:27:08.341317+0000","last_became_active":"2024-04-26T21:27:07.373255+0000","last_became_peered":"2024-04-26T21:27:07.373255+0000","last_unstale":"2024-04-26T21:27:08.341317+0000","last_undegraded":"2024-04-26T21:27:08.341317+0000","last_fullsized":"2024-04-26T21:27:08.341317+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:48.517197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397904+0000","last_change":"2024-04-26T21:27:02.300817+0000","last_active":"2024-04-26T21:27:08.397904+0000","last_peered":"2024-04-26T21:27:08.397904+0000","last_clean":"2024-04-26T21:27:08.397904+0000","last_became_active":"2024-04-26T21:27:02.300493+0000","last_became_peered":"2024-04-26T21:27:02.300493+0000","last_unstale":"2024-04-26T21:27:08.397904+0000","last_undegraded":"2024-04-26T21:27:08.397904+0000","last_fullsized":"2024-04-26T21:27:08.397904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:18.783104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397752+0000","last_change":"2024-04-26T21:27:02.859982+0000","last_active":"2024-04-26T21:27:08.397752+0000","last_peered":"2024-04-26T21:27:08.397752+0000","last_clean":"2024-04-26T21:27:08.397752+0000","last_became_active":"2024-04-26T21:27:02.857204+0000","last_became_peered":"2024-04-26T21:27:02.857204+0000","last_unstale":"2024-04-26T21:27:08.397752+0000","last_undegraded":"2024-04-26T21:27:08.397752+0000","last_fullsized":"2024-04-26T21:27:08.397752+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:22.423956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337711+0000","last_change":"2024-04-26T21:27:07.366039+0000","last_active":"2024-04-26T21:27:08.337711+0000","last_peered":"2024-04-26T21:27:08.337711+0000","last_clean":"2024-04-26T21:27:08.337711+0000","last_became_active":"2024-04-26T21:27:07.365560+0000","last_became_peered":"2024-04-26T21:27:07.365560+0000","last_unstale":"2024-04-26T21:27:08.337711+0000","last_undegraded":"2024-04-26T21:27:08.337711+0000","last_fullsized":"2024-04-26T21:27:08.337711+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:47.833476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:30.689622+0000","last_change":"2024-04-26T21:27:02.306364+0000","last_active":"2024-04-26T21:27:30.689622+0000","last_peered":"2024-04-26T21:27:30.689622+0000","last_clean":"2024-04-26T21:27:30.689622+0000","last_became_active":"2024-04-26T21:27:02.306041+0000","last_became_peered":"2024-04-26T21:27:02.306041+0000","last_unstale":"2024-04-26T21:27:30.689622+0000","last_undegraded":"2024-04-26T21:27:30.689622+0000","last_fullsized":"2024-04-26T21:27:30.689622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:44.501916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397691+0000","last_change":"2024-04-26T21:27:02.859346+0000","last_active":"2024-04-26T21:27:08.397691+0000","last_peered":"2024-04-26T21:27:08.397691+0000","last_clean":"2024-04-26T21:27:08.397691+0000","last_became_active":"2024-04-26T21:27:02.859025+0000","last_became_peered":"2024-04-26T21:27:02.859025+0000","last_unstale":"2024-04-26T21:27:08.397691+0000","last_undegraded":"2024-04-26T21:27:08.397691+0000","last_fullsized":"2024-04-26T21:27:08.397691+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:06.258512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337565+0000","last_change":"2024-04-26T21:27:07.382349+0000","last_active":"2024-04-26T21:27:08.337565+0000","last_peered":"2024-04-26T21:27:08.337565+0000","last_clean":"2024-04-26T21:27:08.337565+0000","last_became_active":"2024-04-26T21:27:07.381930+0000","last_became_peered":"2024-04-26T21:27:07.381930+0000","last_unstale":"2024-04-26T21:27:08.337565+0000","last_undegraded":"2024-04-26T21:27:08.337565+0000","last_fullsized":"2024-04-26T21:27:08.337565+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:15:25.970573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.477106+0000","last_change":"2024-04-26T21:27:02.305449+0000","last_active":"2024-04-26T21:27:28.477106+0000","last_peered":"2024-04-26T21:27:28.477106+0000","last_clean":"2024-04-26T21:27:28.477106+0000","last_became_active":"2024-04-26T21:27:02.305095+0000","last_became_peered":"2024-04-26T21:27:02.305095+0000","last_unstale":"2024-04-26T21:27:28.477106+0000","last_undegraded":"2024-04-26T21:27:28.477106+0000","last_fullsized":"2024-04-26T21:27:28.477106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:47:31.627826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397772+0000","last_change":"2024-04-26T21:27:02.865707+0000","last_active":"2024-04-26T21:27:08.397772+0000","last_peered":"2024-04-26T21:27:08.397772+0000","last_clean":"2024-04-26T21:27:08.397772+0000","last_became_active":"2024-04-26T21:27:02.865436+0000","last_became_peered":"2024-04-26T21:27:02.865436+0000","last_unstale":"2024-04-26T21:27:08.397772+0000","last_undegraded":"2024-04-26T21:27:08.397772+0000","last_fullsized":"2024-04-26T21:27:08.397772+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:53.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399105+0000","last_change":"2024-04-26T21:27:07.379092+0000","last_active":"2024-04-26T21:27:08.399105+0000","last_peered":"2024-04-26T21:27:08.399105+0000","last_clean":"2024-04-26T21:27:08.399105+0000","last_became_active":"2024-04-26T21:27:07.378716+0000","last_became_peered":"2024-04-26T21:27:07.378716+0000","last_unstale":"2024-04-26T21:27:08.399105+0000","last_undegraded":"2024-04-26T21:27:08.399105+0000","last_fullsized":"2024-04-26T21:27:08.399105+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:18.906224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.476718+0000","last_change":"2024-04-26T21:27:02.307510+0000","last_active":"2024-04-26T21:27:28.476718+0000","last_peered":"2024-04-26T21:27:28.476718+0000","last_clean":"2024-04-26T21:27:28.476718+0000","last_became_active":"2024-04-26T21:27:02.306932+0000","last_became_peered":"2024-04-26T21:27:02.306932+0000","last_unstale":"2024-04-26T21:27:28.476718+0000","last_undegraded":"2024-04-26T21:27:28.476718+0000","last_fullsized":"2024-04-26T21:27:28.476718+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:21:34.217767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399452+0000","last_change":"2024-04-26T21:27:02.874246+0000","last_active":"2024-04-26T21:27:08.399452+0000","last_peered":"2024-04-26T21:27:08.399452+0000","last_clean":"2024-04-26T21:27:08.399452+0000","last_became_active":"2024-04-26T21:27:02.873880+0000","last_became_peered":"2024-04-26T21:27:02.873880+0000","last_unstale":"2024-04-26T21:27:08.399452+0000","last_undegraded":"2024-04-26T21:27:08.399452+0000","last_fullsized":"2024-04-26T21:27:08.399452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:21.579832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.339493+0000","last_change":"2024-04-26T21:27:07.387201+0000","last_active":"2024-04-26T21:27:08.339493+0000","last_peered":"2024-04-26T21:27:08.339493+0000","last_clean":"2024-04-26T21:27:08.339493+0000","last_became_active":"2024-04-26T21:27:07.386327+0000","last_became_peered":"2024-04-26T21:27:07.386327+0000","last_unstale":"2024-04-26T21:27:08.339493+0000","last_undegraded":"2024-04-26T21:27:08.339493+0000","last_fullsized":"2024-04-26T21:27:08.339493+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:40.801572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397484+0000","last_change":"2024-04-26T21:27:02.304132+0000","last_active":"2024-04-26T21:27:08.397484+0000","last_peered":"2024-04-26T21:27:08.397484+0000","last_clean":"2024-04-26T21:27:08.397484+0000","last_became_active":"2024-04-26T21:27:02.303562+0000","last_became_peered":"2024-04-26T21:27:02.303562+0000","last_unstale":"2024-04-26T21:27:08.397484+0000","last_undegraded":"2024-04-26T21:27:08.397484+0000","last_fullsized":"2024-04-26T21:27:08.397484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:54.417590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399833+0000","last_change":"2024-04-26T21:27:02.866539+0000","last_active":"2024-04-26T21:27:08.399833+0000","last_peered":"2024-04-26T21:27:08.399833+0000","last_clean":"2024-04-26T21:27:08.399833+0000","last_became_active":"2024-04-26T21:27:02.866193+0000","last_became_peered":"2024-04-26T21:27:02.866193+0000","last_unstale":"2024-04-26T21:27:08.399833+0000","last_undegraded":"2024-04-26T21:27:08.399833+0000","last_fullsized":"2024-04-26T21:27:08.399833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:44.234261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400356+0000","last_change":"2024-04-26T21:27:07.368394+0000","last_active":"2024-04-26T21:27:08.400356+0000","last_peered":"2024-04-26T21:27:08.400356+0000","last_clean":"2024-04-26T21:27:08.400356+0000","last_became_active":"2024-04-26T21:27:07.367998+0000","last_became_peered":"2024-04-26T21:27:07.367998+0000","last_unstale":"2024-04-26T21:27:08.400356+0000","last_undegraded":"2024-04-26T21:27:08.400356+0000","last_fullsized":"2024-04-26T21:27:08.400356+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:31:28.459759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.466384+0000","last_change":"2024-04-26T21:27:02.304096+0000","last_active":"2024-04-26T21:27:27.466384+0000","last_peered":"2024-04-26T21:27:27.466384+0000","last_clean":"2024-04-26T21:27:27.466384+0000","last_became_active":"2024-04-26T21:27:02.303736+0000","last_became_peered":"2024-04-26T21:27:02.303736+0000","last_unstale":"2024-04-26T21:27:27.466384+0000","last_undegraded":"2024-04-26T21:27:27.466384+0000","last_fullsized":"2024-04-26T21:27:27.466384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:25.400913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398607+0000","last_change":"2024-04-26T21:27:02.863033+0000","last_active":"2024-04-26T21:27:08.398607+0000","last_peered":"2024-04-26T21:27:08.398607+0000","last_clean":"2024-04-26T21:27:08.398607+0000","last_became_active":"2024-04-26T21:27:02.862572+0000","last_became_peered":"2024-04-26T21:27:02.862572+0000","last_unstale":"2024-04-26T21:27:08.398607+0000","last_undegraded":"2024-04-26T21:27:08.398607+0000","last_fullsized":"2024-04-26T21:27:08.398607+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:02:55.574601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398874+0000","last_change":"2024-04-26T21:27:07.373626+0000","last_active":"2024-04-26T21:27:08.398874+0000","last_peered":"2024-04-26T21:27:08.398874+0000","last_clean":"2024-04-26T21:27:08.398874+0000","last_became_active":"2024-04-26T21:27:07.373175+0000","last_became_peered":"2024-04-26T21:27:07.373175+0000","last_unstale":"2024-04-26T21:27:08.398874+0000","last_undegraded":"2024-04-26T21:27:08.398874+0000","last_fullsized":"2024-04-26T21:27:08.398874+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:53:09.118035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.464716+0000","last_change":"2024-04-26T21:27:02.299267+0000","last_active":"2024-04-26T21:27:27.464716+0000","last_peered":"2024-04-26T21:27:27.464716+0000","last_clean":"2024-04-26T21:27:27.464716+0000","last_became_active":"2024-04-26T21:27:02.298842+0000","last_became_peered":"2024-04-26T21:27:02.298842+0000","last_unstale":"2024-04-26T21:27:27.464716+0000","last_undegraded":"2024-04-26T21:27:27.464716+0000","last_fullsized":"2024-04-26T21:27:27.464716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:08.431792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.340599+0000","last_change":"2024-04-26T21:27:02.858739+0000","last_active":"2024-04-26T21:27:08.340599+0000","last_peered":"2024-04-26T21:27:08.340599+0000","last_clean":"2024-04-26T21:27:08.340599+0000","last_became_active":"2024-04-26T21:27:02.857458+0000","last_became_peered":"2024-04-26T21:27:02.857458+0000","last_unstale":"2024-04-26T21:27:08.340599+0000","last_undegraded":"2024-04-26T21:27:08.340599+0000","last_fullsized":"2024-04-26T21:27:08.340599+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:00.354052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341751+0000","last_change":"2024-04-26T21:27:07.377690+0000","last_active":"2024-04-26T21:27:08.341751+0000","last_peered":"2024-04-26T21:27:08.341751+0000","last_clean":"2024-04-26T21:27:08.341751+0000","last_became_active":"2024-04-26T21:27:07.377192+0000","last_became_peered":"2024-04-26T21:27:07.377192+0000","last_unstale":"2024-04-26T21:27:08.341751+0000","last_undegraded":"2024-04-26T21:27:08.341751+0000","last_fullsized":"2024-04-26T21:27:08.341751+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:48.069083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.465863+0000","last_change":"2024-04-26T21:27:02.316110+0000","last_active":"2024-04-26T21:27:27.465863+0000","last_peered":"2024-04-26T21:27:27.465863+0000","last_clean":"2024-04-26T21:27:27.465863+0000","last_became_active":"2024-04-26T21:27:02.315767+0000","last_became_peered":"2024-04-26T21:27:02.315767+0000","last_unstale":"2024-04-26T21:27:27.465863+0000","last_undegraded":"2024-04-26T21:27:27.465863+0000","last_fullsized":"2024-04-26T21:27:27.465863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:00.990872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397435+0000","last_change":"2024-04-26T21:27:02.855736+0000","last_active":"2024-04-26T21:27:08.397435+0000","last_peered":"2024-04-26T21:27:08.397435+0000","last_clean":"2024-04-26T21:27:08.397435+0000","last_became_active":"2024-04-26T21:27:02.855378+0000","last_became_peered":"2024-04-26T21:27:02.855378+0000","last_unstale":"2024-04-26T21:27:08.397435+0000","last_undegraded":"2024-04-26T21:27:08.397435+0000","last_fullsized":"2024-04-26T21:27:08.397435+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:55.272869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399118+0000","last_change":"2024-04-26T21:27:07.376804+0000","last_active":"2024-04-26T21:27:08.399118+0000","last_peered":"2024-04-26T21:27:08.399118+0000","last_clean":"2024-04-26T21:27:08.399118+0000","last_became_active":"2024-04-26T21:27:07.374378+0000","last_became_peered":"2024-04-26T21:27:07.374378+0000","last_unstale":"2024-04-26T21:27:08.399118+0000","last_undegraded":"2024-04-26T21:27:08.399118+0000","last_fullsized":"2024-04-26T21:27:08.399118+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:03.930253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.686307+0000","last_change":"2024-04-26T21:27:02.297467+0000","last_active":"2024-04-26T21:27:28.686307+0000","last_peered":"2024-04-26T21:27:28.686307+0000","last_clean":"2024-04-26T21:27:28.686307+0000","last_became_active":"2024-04-26T21:27:02.297077+0000","last_became_peered":"2024-04-26T21:27:02.297077+0000","last_unstale":"2024-04-26T21:27:28.686307+0000","last_undegraded":"2024-04-26T21:27:28.686307+0000","last_fullsized":"2024-04-26T21:27:28.686307+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:47.047565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398238+0000","last_change":"2024-04-26T21:27:02.858161+0000","last_active":"2024-04-26T21:27:08.398238+0000","last_peered":"2024-04-26T21:27:08.398238+0000","last_clean":"2024-04-26T21:27:08.398238+0000","last_became_active":"2024-04-26T21:27:02.857747+0000","last_became_peered":"2024-04-26T21:27:02.857747+0000","last_unstale":"2024-04-26T21:27:08.398238+0000","last_undegraded":"2024-04-26T21:27:08.398238+0000","last_fullsized":"2024-04-26T21:27:08.398238+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:10:50.843195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398106+0000","last_change":"2024-04-26T21:27:07.376253+0000","last_active":"2024-04-26T21:27:08.398106+0000","last_peered":"2024-04-26T21:27:08.398106+0000","last_clean":"2024-04-26T21:27:08.398106+0000","last_became_active":"2024-04-26T21:27:07.375822+0000","last_became_peered":"2024-04-26T21:27:07.375822+0000","last_unstale":"2024-04-26T21:27:08.398106+0000","last_undegraded":"2024-04-26T21:27:08.398106+0000","last_fullsized":"2024-04-26T21:27:08.398106+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:22:01.784163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.476343+0000","last_change":"2024-04-26T21:27:02.300743+0000","last_active":"2024-04-26T21:27:28.476343+0000","last_peered":"2024-04-26T21:27:28.476343+0000","last_clean":"2024-04-26T21:27:28.476343+0000","last_became_active":"2024-04-26T21:27:02.300424+0000","last_became_peered":"2024-04-26T21:27:02.300424+0000","last_unstale":"2024-04-26T21:27:28.476343+0000","last_undegraded":"2024-04-26T21:27:28.476343+0000","last_fullsized":"2024-04-26T21:27:28.476343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:10.408234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398707+0000","last_change":"2024-04-26T21:27:02.872456+0000","last_active":"2024-04-26T21:27:08.398707+0000","last_peered":"2024-04-26T21:27:08.398707+0000","last_clean":"2024-04-26T21:27:08.398707+0000","last_became_active":"2024-04-26T21:27:02.872064+0000","last_became_peered":"2024-04-26T21:27:02.872064+0000","last_unstale":"2024-04-26T21:27:08.398707+0000","last_undegraded":"2024-04-26T21:27:08.398707+0000","last_fullsized":"2024-04-26T21:27:08.398707+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:33.511164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398819+0000","last_change":"2024-04-26T21:27:07.386148+0000","last_active":"2024-04-26T21:27:08.398819+0000","last_peered":"2024-04-26T21:27:08.398819+0000","last_clean":"2024-04-26T21:27:08.398819+0000","last_became_active":"2024-04-26T21:27:07.385787+0000","last_became_peered":"2024-04-26T21:27:07.385787+0000","last_unstale":"2024-04-26T21:27:08.398819+0000","last_undegraded":"2024-04-26T21:27:08.398819+0000","last_fullsized":"2024-04-26T21:27:08.398819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:57.776660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.467213+0000","last_change":"2024-04-26T21:27:02.313725+0000","last_active":"2024-04-26T21:27:27.467213+0000","last_peered":"2024-04-26T21:27:27.467213+0000","last_clean":"2024-04-26T21:27:27.467213+0000","last_became_active":"2024-04-26T21:27:02.313350+0000","last_became_peered":"2024-04-26T21:27:02.313350+0000","last_unstale":"2024-04-26T21:27:27.467213+0000","last_undegraded":"2024-04-26T21:27:27.467213+0000","last_fullsized":"2024-04-26T21:27:27.467213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:21:30.608067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398692+0000","last_change":"2024-04-26T21:27:02.876043+0000","last_active":"2024-04-26T21:27:08.398692+0000","last_peered":"2024-04-26T21:27:08.398692+0000","last_clean":"2024-04-26T21:27:08.398692+0000","last_became_active":"2024-04-26T21:27:02.875665+0000","last_became_peered":"2024-04-26T21:27:02.875665+0000","last_unstale":"2024-04-26T21:27:08.398692+0000","last_undegraded":"2024-04-26T21:27:08.398692+0000","last_fullsized":"2024-04-26T21:27:08.398692+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:53:47.205853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401003+0000","last_change":"2024-04-26T21:27:07.378091+0000","last_active":"2024-04-26T21:27:08.401003+0000","last_peered":"2024-04-26T21:27:08.401003+0000","last_clean":"2024-04-26T21:27:08.401003+0000","last_became_active":"2024-04-26T21:27:07.377671+0000","last_became_peered":"2024-04-26T21:27:07.377671+0000","last_unstale":"2024-04-26T21:27:08.401003+0000","last_undegraded":"2024-04-26T21:27:08.401003+0000","last_fullsized":"2024-04-26T21:27:08.401003+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:41:36.551655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:30.689730+0000","last_change":"2024-04-26T21:27:02.294738+0000","last_active":"2024-04-26T21:27:30.689730+0000","last_peered":"2024-04-26T21:27:30.689730+0000","last_clean":"2024-04-26T21:27:30.689730+0000","last_became_active":"2024-04-26T21:27:02.294338+0000","last_became_peered":"2024-04-26T21:27:02.294338+0000","last_unstale":"2024-04-26T21:27:30.689730+0000","last_undegraded":"2024-04-26T21:27:30.689730+0000","last_fullsized":"2024-04-26T21:27:30.689730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:25.309107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398442+0000","last_change":"2024-04-26T21:27:02.861332+0000","last_active":"2024-04-26T21:27:08.398442+0000","last_peered":"2024-04-26T21:27:08.398442+0000","last_clean":"2024-04-26T21:27:08.398442+0000","last_became_active":"2024-04-26T21:27:02.857865+0000","last_became_peered":"2024-04-26T21:27:02.857865+0000","last_unstale":"2024-04-26T21:27:08.398442+0000","last_undegraded":"2024-04-26T21:27:08.398442+0000","last_fullsized":"2024-04-26T21:27:08.398442+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:48.493086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401726+0000","last_change":"2024-04-26T21:27:07.383219+0000","last_active":"2024-04-26T21:27:08.401726+0000","last_peered":"2024-04-26T21:27:08.401726+0000","last_clean":"2024-04-26T21:27:08.401726+0000","last_became_active":"2024-04-26T21:27:07.382820+0000","last_became_peered":"2024-04-26T21:27:07.382820+0000","last_unstale":"2024-04-26T21:27:08.401726+0000","last_undegraded":"2024-04-26T21:27:08.401726+0000","last_fullsized":"2024-04-26T21:27:08.401726+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:07.752124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.477768+0000","last_change":"2024-04-26T21:27:02.307746+0000","last_active":"2024-04-26T21:27:28.477768+0000","last_peered":"2024-04-26T21:27:28.477768+0000","last_clean":"2024-04-26T21:27:28.477768+0000","last_became_active":"2024-04-26T21:27:02.307406+0000","last_became_peered":"2024-04-26T21:27:02.307406+0000","last_unstale":"2024-04-26T21:27:28.477768+0000","last_undegraded":"2024-04-26T21:27:28.477768+0000","last_fullsized":"2024-04-26T21:27:28.477768+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:13.799727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398177+0000","last_change":"2024-04-26T21:27:02.861194+0000","last_active":"2024-04-26T21:27:08.398177+0000","last_peered":"2024-04-26T21:27:08.398177+0000","last_clean":"2024-04-26T21:27:08.398177+0000","last_became_active":"2024-04-26T21:27:02.860871+0000","last_became_peered":"2024-04-26T21:27:02.860871+0000","last_unstale":"2024-04-26T21:27:08.398177+0000","last_undegraded":"2024-04-26T21:27:08.398177+0000","last_fullsized":"2024-04-26T21:27:08.398177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:56.829027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397336+0000","last_change":"2024-04-26T21:27:07.371265+0000","last_active":"2024-04-26T21:27:08.397336+0000","last_peered":"2024-04-26T21:27:08.397336+0000","last_clean":"2024-04-26T21:27:08.397336+0000","last_became_active":"2024-04-26T21:27:07.370928+0000","last_became_peered":"2024-04-26T21:27:07.370928+0000","last_unstale":"2024-04-26T21:27:08.397336+0000","last_undegraded":"2024-04-26T21:27:08.397336+0000","last_fullsized":"2024-04-26T21:27:08.397336+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:33:05.640493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.683557+0000","last_change":"2024-04-26T21:27:02.301711+0000","last_active":"2024-04-26T21:27:28.683557+0000","last_peered":"2024-04-26T21:27:28.683557+0000","last_clean":"2024-04-26T21:27:28.683557+0000","last_became_active":"2024-04-26T21:27:02.301429+0000","last_became_peered":"2024-04-26T21:27:02.301429+0000","last_unstale":"2024-04-26T21:27:28.683557+0000","last_undegraded":"2024-04-26T21:27:28.683557+0000","last_fullsized":"2024-04-26T21:27:28.683557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:50.876824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398948+0000","last_change":"2024-04-26T21:27:02.854372+0000","last_active":"2024-04-26T21:27:08.398948+0000","last_peered":"2024-04-26T21:27:08.398948+0000","last_clean":"2024-04-26T21:27:08.398948+0000","last_became_active":"2024-04-26T21:27:02.853826+0000","last_became_peered":"2024-04-26T21:27:02.853826+0000","last_unstale":"2024-04-26T21:27:08.398948+0000","last_undegraded":"2024-04-26T21:27:08.398948+0000","last_fullsized":"2024-04-26T21:27:08.398948+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:08.253384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397623+0000","last_change":"2024-04-26T21:27:07.506684+0000","last_active":"2024-04-26T21:27:08.397623+0000","last_peered":"2024-04-26T21:27:08.397623+0000","last_clean":"2024-04-26T21:27:08.397623+0000","last_became_active":"2024-04-26T21:27:07.506203+0000","last_became_peered":"2024-04-26T21:27:07.506203+0000","last_unstale":"2024-04-26T21:27:08.397623+0000","last_undegraded":"2024-04-26T21:27:08.397623+0000","last_fullsized":"2024-04-26T21:27:08.397623+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:42:48.580106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399612+0000","last_change":"2024-04-26T21:27:02.306265+0000","last_active":"2024-04-26T21:27:08.399612+0000","last_peered":"2024-04-26T21:27:08.399612+0000","last_clean":"2024-04-26T21:27:08.399612+0000","last_became_active":"2024-04-26T21:27:02.305941+0000","last_became_peered":"2024-04-26T21:27:02.305941+0000","last_unstale":"2024-04-26T21:27:08.399612+0000","last_undegraded":"2024-04-26T21:27:08.399612+0000","last_fullsized":"2024-04-26T21:27:08.399612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:29.076833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398899+0000","last_change":"2024-04-26T21:27:02.871916+0000","last_active":"2024-04-26T21:27:08.398899+0000","last_peered":"2024-04-26T21:27:08.398899+0000","last_clean":"2024-04-26T21:27:08.398899+0000","last_became_active":"2024-04-26T21:27:02.871517+0000","last_became_peered":"2024-04-26T21:27:02.871517+0000","last_unstale":"2024-04-26T21:27:08.398899+0000","last_undegraded":"2024-04-26T21:27:08.398899+0000","last_fullsized":"2024-04-26T21:27:08.398899+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:41.361757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400181+0000","last_change":"2024-04-26T21:27:07.383422+0000","last_active":"2024-04-26T21:27:08.400181+0000","last_peered":"2024-04-26T21:27:08.400181+0000","last_clean":"2024-04-26T21:27:08.400181+0000","last_became_active":"2024-04-26T21:27:07.382727+0000","last_became_peered":"2024-04-26T21:27:07.382727+0000","last_unstale":"2024-04-26T21:27:08.400181+0000","last_undegraded":"2024-04-26T21:27:08.400181+0000","last_fullsized":"2024-04-26T21:27:08.400181+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:22.002658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398824+0000","last_change":"2024-04-26T21:27:02.308081+0000","last_active":"2024-04-26T21:27:08.398824+0000","last_peered":"2024-04-26T21:27:08.398824+0000","last_clean":"2024-04-26T21:27:08.398824+0000","last_became_active":"2024-04-26T21:27:02.307715+0000","last_became_peered":"2024-04-26T21:27:02.307715+0000","last_unstale":"2024-04-26T21:27:08.398824+0000","last_undegraded":"2024-04-26T21:27:08.398824+0000","last_fullsized":"2024-04-26T21:27:08.398824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:28.967817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398685+0000","last_change":"2024-04-26T21:27:02.859527+0000","last_active":"2024-04-26T21:27:08.398685+0000","last_peered":"2024-04-26T21:27:08.398685+0000","last_clean":"2024-04-26T21:27:08.398685+0000","last_became_active":"2024-04-26T21:27:02.859173+0000","last_became_peered":"2024-04-26T21:27:02.859173+0000","last_unstale":"2024-04-26T21:27:08.398685+0000","last_undegraded":"2024-04-26T21:27:08.398685+0000","last_fullsized":"2024-04-26T21:27:08.398685+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:24.811630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400540+0000","last_change":"2024-04-26T21:27:07.509778+0000","last_active":"2024-04-26T21:27:08.400540+0000","last_peered":"2024-04-26T21:27:08.400540+0000","last_clean":"2024-04-26T21:27:08.400540+0000","last_became_active":"2024-04-26T21:27:07.508344+0000","last_became_peered":"2024-04-26T21:27:07.508344+0000","last_unstale":"2024-04-26T21:27:08.400540+0000","last_undegraded":"2024-04-26T21:27:08.400540+0000","last_fullsized":"2024-04-26T21:27:08.400540+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:38:05.926717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397671+0000","last_change":"2024-04-26T21:27:02.862880+0000","last_active":"2024-04-26T21:27:08.397671+0000","last_peered":"2024-04-26T21:27:08.397671+0000","last_clean":"2024-04-26T21:27:08.397671+0000","last_became_active":"2024-04-26T21:27:02.862499+0000","last_became_peered":"2024-04-26T21:27:02.862499+0000","last_unstale":"2024-04-26T21:27:08.397671+0000","last_undegraded":"2024-04-26T21:27:08.397671+0000","last_fullsized":"2024-04-26T21:27:08.397671+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:12:33.955508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.478159+0000","last_change":"2024-04-26T21:27:02.317413+0000","last_active":"2024-04-26T21:27:28.478159+0000","last_peered":"2024-04-26T21:27:28.478159+0000","last_clean":"2024-04-26T21:27:28.478159+0000","last_became_active":"2024-04-26T21:27:02.316510+0000","last_became_peered":"2024-04-26T21:27:02.316510+0000","last_unstale":"2024-04-26T21:27:28.478159+0000","last_undegraded":"2024-04-26T21:27:28.478159+0000","last_fullsized":"2024-04-26T21:27:28.478159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:49.156366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399332+0000","last_change":"2024-04-26T21:27:07.373434+0000","last_active":"2024-04-26T21:27:08.399332+0000","last_peered":"2024-04-26T21:27:08.399332+0000","last_clean":"2024-04-26T21:27:08.399332+0000","last_became_active":"2024-04-26T21:27:07.372973+0000","last_became_peered":"2024-04-26T21:27:07.372973+0000","last_unstale":"2024-04-26T21:27:08.399332+0000","last_undegraded":"2024-04-26T21:27:08.399332+0000","last_fullsized":"2024-04-26T21:27:08.399332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:04.022651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400567+0000","last_change":"2024-04-26T21:27:02.862734+0000","last_active":"2024-04-26T21:27:08.400567+0000","last_peered":"2024-04-26T21:27:08.400567+0000","last_clean":"2024-04-26T21:27:08.400567+0000","last_became_active":"2024-04-26T21:27:02.862334+0000","last_became_peered":"2024-04-26T21:27:02.862334+0000","last_unstale":"2024-04-26T21:27:08.400567+0000","last_undegraded":"2024-04-26T21:27:08.400567+0000","last_fullsized":"2024-04-26T21:27:08.400567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:09.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.681983+0000","last_change":"2024-04-26T21:27:02.302471+0000","last_active":"2024-04-26T21:27:28.681983+0000","last_peered":"2024-04-26T21:27:28.681983+0000","last_clean":"2024-04-26T21:27:28.681983+0000","last_became_active":"2024-04-26T21:27:02.302055+0000","last_became_peered":"2024-04-26T21:27:02.302055+0000","last_unstale":"2024-04-26T21:27:28.681983+0000","last_undegraded":"2024-04-26T21:27:28.681983+0000","last_fullsized":"2024-04-26T21:27:28.681983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:29:14.649670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398204+0000","last_change":"2024-04-26T21:25:41.647633+0000","last_active":"2024-04-26T21:27:08.398204+0000","last_peered":"2024-04-26T21:27:08.398204+0000","last_clean":"2024-04-26T21:27:08.398204+0000","last_became_active":"2024-04-26T21:25:41.647091+0000","last_became_peered":"2024-04-26T21:25:41.647091+0000","last_unstale":"2024-04-26T21:27:08.398204+0000","last_undegraded":"2024-04-26T21:27:08.398204+0000","last_fullsized":"2024-04-26T21:27:08.398204+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_clean_scrub_stamp":"2024-04-26T21:21:43.547929+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:37.666065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397305+0000","last_change":"2024-04-26T21:27:07.506944+0000","last_active":"2024-04-26T21:27:08.397305+0000","last_peered":"2024-04-26T21:27:08.397305+0000","last_clean":"2024-04-26T21:27:08.397305+0000","last_became_active":"2024-04-26T21:27:07.506451+0000","last_became_peered":"2024-04-26T21:27:07.506451+0000","last_unstale":"2024-04-26T21:27:08.397305+0000","last_undegraded":"2024-04-26T21:27:08.397305+0000","last_fullsized":"2024-04-26T21:27:08.397305+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:15:21.916951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397908+0000","last_change":"2024-04-26T21:27:02.874116+0000","last_active":"2024-04-26T21:27:08.397908+0000","last_peered":"2024-04-26T21:27:08.397908+0000","last_clean":"2024-04-26T21:27:08.397908+0000","last_became_active":"2024-04-26T21:27:02.871334+0000","last_became_peered":"2024-04-26T21:27:02.871334+0000","last_unstale":"2024-04-26T21:27:08.397908+0000","last_undegraded":"2024-04-26T21:27:08.397908+0000","last_fullsized":"2024-04-26T21:27:08.397908+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:28.835061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.473757+0000","last_change":"2024-04-26T21:27:02.303089+0000","last_active":"2024-04-26T21:27:28.473757+0000","last_peered":"2024-04-26T21:27:28.473757+0000","last_clean":"2024-04-26T21:27:28.473757+0000","last_became_active":"2024-04-26T21:27:02.301793+0000","last_became_peered":"2024-04-26T21:27:02.301793+0000","last_unstale":"2024-04-26T21:27:28.473757+0000","last_undegraded":"2024-04-26T21:27:28.473757+0000","last_fullsized":"2024-04-26T21:27:28.473757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:51:29.599675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398914+0000","last_change":"2024-04-26T21:27:07.373894+0000","last_active":"2024-04-26T21:27:08.398914+0000","last_peered":"2024-04-26T21:27:08.398914+0000","last_clean":"2024-04-26T21:27:08.398914+0000","last_became_active":"2024-04-26T21:27:07.373338+0000","last_became_peered":"2024-04-26T21:27:07.373338+0000","last_unstale":"2024-04-26T21:27:08.398914+0000","last_undegraded":"2024-04-26T21:27:08.398914+0000","last_fullsized":"2024-04-26T21:27:08.398914+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:49.636493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.465810+0000","last_change":"2024-04-26T21:27:02.307754+0000","last_active":"2024-04-26T21:27:27.465810+0000","last_peered":"2024-04-26T21:27:27.465810+0000","last_clean":"2024-04-26T21:27:27.465810+0000","last_became_active":"2024-04-26T21:27:02.307423+0000","last_became_peered":"2024-04-26T21:27:02.307423+0000","last_unstale":"2024-04-26T21:27:27.465810+0000","last_undegraded":"2024-04-26T21:27:27.465810+0000","last_fullsized":"2024-04-26T21:27:27.465810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:43:26.701568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400337+0000","last_change":"2024-04-26T21:27:02.869831+0000","last_active":"2024-04-26T21:27:08.400337+0000","last_peered":"2024-04-26T21:27:08.400337+0000","last_clean":"2024-04-26T21:27:08.400337+0000","last_became_active":"2024-04-26T21:27:02.869375+0000","last_became_peered":"2024-04-26T21:27:02.869375+0000","last_unstale":"2024-04-26T21:27:08.400337+0000","last_undegraded":"2024-04-26T21:27:08.400337+0000","last_fullsized":"2024-04-26T21:27:08.400337+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:20.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399580+0000","last_change":"2024-04-26T21:27:07.507562+0000","last_active":"2024-04-26T21:27:08.399580+0000","last_peered":"2024-04-26T21:27:08.399580+0000","last_clean":"2024-04-26T21:27:08.399580+0000","last_became_active":"2024-04-26T21:27:07.507063+0000","last_became_peered":"2024-04-26T21:27:07.507063+0000","last_unstale":"2024-04-26T21:27:08.399580+0000","last_undegraded":"2024-04-26T21:27:08.399580+0000","last_fullsized":"2024-04-26T21:27:08.399580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:58.192617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.477333+0000","last_change":"2024-04-26T21:27:02.315360+0000","last_active":"2024-04-26T21:27:28.477333+0000","last_peered":"2024-04-26T21:27:28.477333+0000","last_clean":"2024-04-26T21:27:28.477333+0000","last_became_active":"2024-04-26T21:27:02.314817+0000","last_became_peered":"2024-04-26T21:27:02.314817+0000","last_unstale":"2024-04-26T21:27:28.477333+0000","last_undegraded":"2024-04-26T21:27:28.477333+0000","last_fullsized":"2024-04-26T21:27:28.477333+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:12.045363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400867+0000","last_change":"2024-04-26T21:27:02.857006+0000","last_active":"2024-04-26T21:27:08.400867+0000","last_peered":"2024-04-26T21:27:08.400867+0000","last_clean":"2024-04-26T21:27:08.400867+0000","last_became_active":"2024-04-26T21:27:02.856589+0000","last_became_peered":"2024-04-26T21:27:02.856589+0000","last_unstale":"2024-04-26T21:27:08.400867+0000","last_undegraded":"2024-04-26T21:27:08.400867+0000","last_fullsized":"2024-04-26T21:27:08.400867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:51:10.292791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337336+0000","last_change":"2024-04-26T21:27:07.362016+0000","last_active":"2024-04-26T21:27:08.337336+0000","last_peered":"2024-04-26T21:27:08.337336+0000","last_clean":"2024-04-26T21:27:08.337336+0000","last_became_active":"2024-04-26T21:27:07.361360+0000","last_became_peered":"2024-04-26T21:27:07.361360+0000","last_unstale":"2024-04-26T21:27:08.337336+0000","last_undegraded":"2024-04-26T21:27:08.337336+0000","last_fullsized":"2024-04-26T21:27:08.337336+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:03.196293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.465015+0000","last_change":"2024-04-26T21:27:02.308088+0000","last_active":"2024-04-26T21:27:27.465015+0000","last_peered":"2024-04-26T21:27:27.465015+0000","last_clean":"2024-04-26T21:27:27.465015+0000","last_became_active":"2024-04-26T21:27:02.307727+0000","last_became_peered":"2024-04-26T21:27:02.307727+0000","last_unstale":"2024-04-26T21:27:27.465015+0000","last_undegraded":"2024-04-26T21:27:27.465015+0000","last_fullsized":"2024-04-26T21:27:27.465015+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:02.132125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.340082+0000","last_change":"2024-04-26T21:27:02.847821+0000","last_active":"2024-04-26T21:27:08.340082+0000","last_peered":"2024-04-26T21:27:08.340082+0000","last_clean":"2024-04-26T21:27:08.340082+0000","last_became_active":"2024-04-26T21:27:02.847167+0000","last_became_peered":"2024-04-26T21:27:02.847167+0000","last_unstale":"2024-04-26T21:27:08.340082+0000","last_undegraded":"2024-04-26T21:27:08.340082+0000","last_fullsized":"2024-04-26T21:27:08.340082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:54:46.596138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.342017+0000","last_change":"2024-04-26T21:27:07.506609+0000","last_active":"2024-04-26T21:27:08.342017+0000","last_peered":"2024-04-26T21:27:08.342017+0000","last_clean":"2024-04-26T21:27:08.342017+0000","last_became_active":"2024-04-26T21:27:07.506100+0000","last_became_peered":"2024-04-26T21:27:07.506100+0000","last_unstale":"2024-04-26T21:27:08.342017+0000","last_undegraded":"2024-04-26T21:27:08.342017+0000","last_fullsized":"2024-04-26T21:27:08.342017+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:26.535809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.464939+0000","last_change":"2024-04-26T21:27:02.297432+0000","last_active":"2024-04-26T21:27:27.464939+0000","last_peered":"2024-04-26T21:27:27.464939+0000","last_clean":"2024-04-26T21:27:27.464939+0000","last_became_active":"2024-04-26T21:27:02.297081+0000","last_became_peered":"2024-04-26T21:27:02.297081+0000","last_unstale":"2024-04-26T21:27:27.464939+0000","last_undegraded":"2024-04-26T21:27:27.464939+0000","last_fullsized":"2024-04-26T21:27:27.464939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:08.139469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.396953+0000","last_change":"2024-04-26T21:27:02.866653+0000","last_active":"2024-04-26T21:27:08.396953+0000","last_peered":"2024-04-26T21:27:08.396953+0000","last_clean":"2024-04-26T21:27:08.396953+0000","last_became_active":"2024-04-26T21:27:02.866275+0000","last_became_peered":"2024-04-26T21:27:02.866275+0000","last_unstale":"2024-04-26T21:27:08.396953+0000","last_undegraded":"2024-04-26T21:27:08.396953+0000","last_fullsized":"2024-04-26T21:27:08.396953+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:39.089972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401332+0000","last_change":"2024-04-26T21:27:07.377084+0000","last_active":"2024-04-26T21:27:08.401332+0000","last_peered":"2024-04-26T21:27:08.401332+0000","last_clean":"2024-04-26T21:27:08.401332+0000","last_became_active":"2024-04-26T21:27:07.374052+0000","last_became_peered":"2024-04-26T21:27:07.374052+0000","last_unstale":"2024-04-26T21:27:08.401332+0000","last_undegraded":"2024-04-26T21:27:08.401332+0000","last_fullsized":"2024-04-26T21:27:08.401332+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:06:20.031170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.468113+0000","last_change":"2024-04-26T21:27:02.307250+0000","last_active":"2024-04-26T21:27:27.468113+0000","last_peered":"2024-04-26T21:27:27.468113+0000","last_clean":"2024-04-26T21:27:27.468113+0000","last_became_active":"2024-04-26T21:27:02.303532+0000","last_became_peered":"2024-04-26T21:27:02.303532+0000","last_unstale":"2024-04-26T21:27:27.468113+0000","last_undegraded":"2024-04-26T21:27:27.468113+0000","last_fullsized":"2024-04-26T21:27:27.468113+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:36.265188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398462+0000","last_change":"2024-04-26T21:27:02.858818+0000","last_active":"2024-04-26T21:27:08.398462+0000","last_peered":"2024-04-26T21:27:08.398462+0000","last_clean":"2024-04-26T21:27:08.398462+0000","last_became_active":"2024-04-26T21:27:02.858059+0000","last_became_peered":"2024-04-26T21:27:02.858059+0000","last_unstale":"2024-04-26T21:27:08.398462+0000","last_undegraded":"2024-04-26T21:27:08.398462+0000","last_fullsized":"2024-04-26T21:27:08.398462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:02.159334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341651+0000","last_change":"2024-04-26T21:27:07.381273+0000","last_active":"2024-04-26T21:27:08.341651+0000","last_peered":"2024-04-26T21:27:08.341651+0000","last_clean":"2024-04-26T21:27:08.341651+0000","last_became_active":"2024-04-26T21:27:07.380891+0000","last_became_peered":"2024-04-26T21:27:07.380891+0000","last_unstale":"2024-04-26T21:27:08.341651+0000","last_undegraded":"2024-04-26T21:27:08.341651+0000","last_fullsized":"2024-04-26T21:27:08.341651+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:39.641577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337283+0000","last_change":"2024-04-26T21:27:02.301363+0000","last_active":"2024-04-26T21:27:08.337283+0000","last_peered":"2024-04-26T21:27:08.337283+0000","last_clean":"2024-04-26T21:27:08.337283+0000","last_became_active":"2024-04-26T21:27:02.301059+0000","last_became_peered":"2024-04-26T21:27:02.301059+0000","last_unstale":"2024-04-26T21:27:08.337283+0000","last_undegraded":"2024-04-26T21:27:08.337283+0000","last_fullsized":"2024-04-26T21:27:08.337283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:17.586190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.402014+0000","last_change":"2024-04-26T21:27:02.860612+0000","last_active":"2024-04-26T21:27:08.402014+0000","last_peered":"2024-04-26T21:27:08.402014+0000","last_clean":"2024-04-26T21:27:08.402014+0000","last_became_active":"2024-04-26T21:27:02.857285+0000","last_became_peered":"2024-04-26T21:27:02.857285+0000","last_unstale":"2024-04-26T21:27:08.402014+0000","last_undegraded":"2024-04-26T21:27:08.402014+0000","last_fullsized":"2024-04-26T21:27:08.402014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:11.968818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.341818+0000","last_change":"2024-04-26T21:27:07.506425+0000","last_active":"2024-04-26T21:27:08.341818+0000","last_peered":"2024-04-26T21:27:08.341818+0000","last_clean":"2024-04-26T21:27:08.341818+0000","last_became_active":"2024-04-26T21:27:07.505968+0000","last_became_peered":"2024-04-26T21:27:07.505968+0000","last_unstale":"2024-04-26T21:27:08.341818+0000","last_undegraded":"2024-04-26T21:27:08.341818+0000","last_fullsized":"2024-04-26T21:27:08.341818+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:59.914560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:27.465016+0000","last_change":"2024-04-26T21:27:02.294635+0000","last_active":"2024-04-26T21:27:27.465016+0000","last_peered":"2024-04-26T21:27:27.465016+0000","last_clean":"2024-04-26T21:27:27.465016+0000","last_became_active":"2024-04-26T21:27:02.294311+0000","last_became_peered":"2024-04-26T21:27:02.294311+0000","last_unstale":"2024-04-26T21:27:27.465016+0000","last_undegraded":"2024-04-26T21:27:27.465016+0000","last_fullsized":"2024-04-26T21:27:27.465016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:47.906546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398900+0000","last_change":"2024-04-26T21:27:02.875475+0000","last_active":"2024-04-26T21:27:08.398900+0000","last_peered":"2024-04-26T21:27:08.398900+0000","last_clean":"2024-04-26T21:27:08.398900+0000","last_became_active":"2024-04-26T21:27:02.875137+0000","last_became_peered":"2024-04-26T21:27:02.875137+0000","last_unstale":"2024-04-26T21:27:08.398900+0000","last_undegraded":"2024-04-26T21:27:08.398900+0000","last_fullsized":"2024-04-26T21:27:08.398900+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:04.988973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397080+0000","last_change":"2024-04-26T21:27:07.372927+0000","last_active":"2024-04-26T21:27:08.397080+0000","last_peered":"2024-04-26T21:27:08.397080+0000","last_clean":"2024-04-26T21:27:08.397080+0000","last_became_active":"2024-04-26T21:27:07.372371+0000","last_became_peered":"2024-04-26T21:27:07.372371+0000","last_unstale":"2024-04-26T21:27:08.397080+0000","last_undegraded":"2024-04-26T21:27:08.397080+0000","last_fullsized":"2024-04-26T21:27:08.397080+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:12:41.607167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:28.477987+0000","last_change":"2024-04-26T21:27:02.295379+0000","last_active":"2024-04-26T21:27:28.477987+0000","last_peered":"2024-04-26T21:27:28.477987+0000","last_clean":"2024-04-26T21:27:28.477987+0000","last_became_active":"2024-04-26T21:27:02.295077+0000","last_became_peered":"2024-04-26T21:27:02.295077+0000","last_unstale":"2024-04-26T21:27:28.477987+0000","last_undegraded":"2024-04-26T21:27:28.477987+0000","last_fullsized":"2024-04-26T21:27:28.477987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:58.177960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399407+0000","last_change":"2024-04-26T21:27:02.855998+0000","last_active":"2024-04-26T21:27:08.399407+0000","last_peered":"2024-04-26T21:27:08.399407+0000","last_clean":"2024-04-26T21:27:08.399407+0000","last_became_active":"2024-04-26T21:27:02.855636+0000","last_became_peered":"2024-04-26T21:27:02.855636+0000","last_unstale":"2024-04-26T21:27:08.399407+0000","last_undegraded":"2024-04-26T21:27:08.399407+0000","last_fullsized":"2024-04-26T21:27:08.399407+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:19:02.988946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.339186+0000","last_change":"2024-04-26T21:27:07.378980+0000","last_active":"2024-04-26T21:27:08.339186+0000","last_peered":"2024-04-26T21:27:08.339186+0000","last_clean":"2024-04-26T21:27:08.339186+0000","last_became_active":"2024-04-26T21:27:07.378374+0000","last_became_peered":"2024-04-26T21:27:07.378374+0000","last_unstale":"2024-04-26T21:27:08.339186+0000","last_undegraded":"2024-04-26T21:27:08.339186+0000","last_fullsized":"2024-04-26T21:27:08.339186+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:31.845297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397898+0000","last_change":"2024-04-26T21:27:02.313801+0000","last_active":"2024-04-26T21:27:08.397898+0000","last_peered":"2024-04-26T21:27:08.397898+0000","last_clean":"2024-04-26T21:27:08.397898+0000","last_became_active":"2024-04-26T21:27:02.313540+0000","last_became_peered":"2024-04-26T21:27:02.313540+0000","last_unstale":"2024-04-26T21:27:08.397898+0000","last_undegraded":"2024-04-26T21:27:08.397898+0000","last_fullsized":"2024-04-26T21:27:08.397898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:38.612261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399083+0000","last_change":"2024-04-26T21:27:02.864151+0000","last_active":"2024-04-26T21:27:08.399083+0000","last_peered":"2024-04-26T21:27:08.399083+0000","last_clean":"2024-04-26T21:27:08.399083+0000","last_became_active":"2024-04-26T21:27:02.863818+0000","last_became_peered":"2024-04-26T21:27:02.863818+0000","last_unstale":"2024-04-26T21:27:08.399083+0000","last_undegraded":"2024-04-26T21:27:08.399083+0000","last_fullsized":"2024-04-26T21:27:08.399083+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:26.215370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397531+0000","last_change":"2024-04-26T21:27:07.365821+0000","last_active":"2024-04-26T21:27:08.397531+0000","last_peered":"2024-04-26T21:27:08.397531+0000","last_clean":"2024-04-26T21:27:08.397531+0000","last_became_active":"2024-04-26T21:27:07.365347+0000","last_became_peered":"2024-04-26T21:27:07.365347+0000","last_unstale":"2024-04-26T21:27:08.397531+0000","last_undegraded":"2024-04-26T21:27:08.397531+0000","last_fullsized":"2024-04-26T21:27:08.397531+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:40:11.596908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401414+0000","last_change":"2024-04-26T21:27:02.310641+0000","last_active":"2024-04-26T21:27:08.401414+0000","last_peered":"2024-04-26T21:27:08.401414+0000","last_clean":"2024-04-26T21:27:08.401414+0000","last_became_active":"2024-04-26T21:27:02.308569+0000","last_became_peered":"2024-04-26T21:27:02.308569+0000","last_unstale":"2024-04-26T21:27:08.401414+0000","last_undegraded":"2024-04-26T21:27:08.401414+0000","last_fullsized":"2024-04-26T21:27:08.401414+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:20.988098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.399291+0000","last_change":"2024-04-26T21:27:02.854976+0000","last_active":"2024-04-26T21:27:08.399291+0000","last_peered":"2024-04-26T21:27:08.399291+0000","last_clean":"2024-04-26T21:27:08.399291+0000","last_became_active":"2024-04-26T21:27:02.854611+0000","last_became_peered":"2024-04-26T21:27:02.854611+0000","last_unstale":"2024-04-26T21:27:08.399291+0000","last_undegraded":"2024-04-26T21:27:08.399291+0000","last_fullsized":"2024-04-26T21:27:08.399291+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:57.065657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398537+0000","last_change":"2024-04-26T21:27:02.859017+0000","last_active":"2024-04-26T21:27:08.398537+0000","last_peered":"2024-04-26T21:27:08.398537+0000","last_clean":"2024-04-26T21:27:08.398537+0000","last_became_active":"2024-04-26T21:27:02.858507+0000","last_became_peered":"2024-04-26T21:27:02.858507+0000","last_unstale":"2024-04-26T21:27:08.398537+0000","last_undegraded":"2024-04-26T21:27:08.398537+0000","last_fullsized":"2024-04-26T21:27:08.398537+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:14.850891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337861+0000","last_change":"2024-04-26T21:27:02.858247+0000","last_active":"2024-04-26T21:27:08.337861+0000","last_peered":"2024-04-26T21:27:08.337861+0000","last_clean":"2024-04-26T21:27:08.337861+0000","last_became_active":"2024-04-26T21:27:02.857900+0000","last_became_peered":"2024-04-26T21:27:02.857900+0000","last_unstale":"2024-04-26T21:27:08.337861+0000","last_undegraded":"2024-04-26T21:27:08.337861+0000","last_fullsized":"2024-04-26T21:27:08.337861+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:02:43.998925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337151+0000","last_change":"2024-04-26T21:27:02.868817+0000","last_active":"2024-04-26T21:27:08.337151+0000","last_peered":"2024-04-26T21:27:08.337151+0000","last_clean":"2024-04-26T21:27:08.337151+0000","last_became_active":"2024-04-26T21:27:02.868557+0000","last_became_peered":"2024-04-26T21:27:02.868557+0000","last_unstale":"2024-04-26T21:27:08.337151+0000","last_undegraded":"2024-04-26T21:27:08.337151+0000","last_fullsized":"2024-04-26T21:27:08.337151+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:55.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.339735+0000","last_change":"2024-04-26T21:27:02.876029+0000","last_active":"2024-04-26T21:27:08.339735+0000","last_peered":"2024-04-26T21:27:08.339735+0000","last_clean":"2024-04-26T21:27:08.339735+0000","last_became_active":"2024-04-26T21:27:02.875656+0000","last_became_peered":"2024-04-26T21:27:02.875656+0000","last_unstale":"2024-04-26T21:27:08.339735+0000","last_undegraded":"2024-04-26T21:27:08.339735+0000","last_fullsized":"2024-04-26T21:27:08.339735+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:47.317581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400469+0000","last_change":"2024-04-26T21:27:02.873818+0000","last_active":"2024-04-26T21:27:08.400469+0000","last_peered":"2024-04-26T21:27:08.400469+0000","last_clean":"2024-04-26T21:27:08.400469+0000","last_became_active":"2024-04-26T21:27:02.873459+0000","last_became_peered":"2024-04-26T21:27:02.873459+0000","last_unstale":"2024-04-26T21:27:08.400469+0000","last_undegraded":"2024-04-26T21:27:08.400469+0000","last_fullsized":"2024-04-26T21:27:08.400469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:07.259187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398912+0000","last_change":"2024-04-26T21:27:02.854901+0000","last_active":"2024-04-26T21:27:08.398912+0000","last_peered":"2024-04-26T21:27:08.398912+0000","last_clean":"2024-04-26T21:27:08.398912+0000","last_became_active":"2024-04-26T21:27:02.854547+0000","last_became_peered":"2024-04-26T21:27:02.854547+0000","last_unstale":"2024-04-26T21:27:08.398912+0000","last_undegraded":"2024-04-26T21:27:08.398912+0000","last_fullsized":"2024-04-26T21:27:08.398912+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:24.320868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.337890+0000","last_change":"2024-04-26T21:27:02.856616+0000","last_active":"2024-04-26T21:27:08.337890+0000","last_peered":"2024-04-26T21:27:08.337890+0000","last_clean":"2024-04-26T21:27:08.337890+0000","last_became_active":"2024-04-26T21:27:02.856264+0000","last_became_peered":"2024-04-26T21:27:02.856264+0000","last_unstale":"2024-04-26T21:27:08.337890+0000","last_undegraded":"2024-04-26T21:27:08.337890+0000","last_fullsized":"2024-04-26T21:27:08.337890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:43.374534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.401714+0000","last_change":"2024-04-26T21:27:02.859587+0000","last_active":"2024-04-26T21:27:08.401714+0000","last_peered":"2024-04-26T21:27:08.401714+0000","last_clean":"2024-04-26T21:27:08.401714+0000","last_became_active":"2024-04-26T21:27:02.859136+0000","last_became_peered":"2024-04-26T21:27:02.859136+0000","last_unstale":"2024-04-26T21:27:08.401714+0000","last_undegraded":"2024-04-26T21:27:08.401714+0000","last_fullsized":"2024-04-26T21:27:08.401714+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:09.370319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397549+0000","last_change":"2024-04-26T21:27:02.864265+0000","last_active":"2024-04-26T21:27:08.397549+0000","last_peered":"2024-04-26T21:27:08.397549+0000","last_clean":"2024-04-26T21:27:08.397549+0000","last_became_active":"2024-04-26T21:27:02.863920+0000","last_became_peered":"2024-04-26T21:27:02.863920+0000","last_unstale":"2024-04-26T21:27:08.397549+0000","last_undegraded":"2024-04-26T21:27:08.397549+0000","last_fullsized":"2024-04-26T21:27:08.397549+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:13.941544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.400091+0000","last_change":"2024-04-26T21:27:02.860189+0000","last_active":"2024-04-26T21:27:08.400091+0000","last_peered":"2024-04-26T21:27:08.400091+0000","last_clean":"2024-04-26T21:27:08.400091+0000","last_became_active":"2024-04-26T21:27:02.859843+0000","last_became_peered":"2024-04-26T21:27:02.859843+0000","last_unstale":"2024-04-26T21:27:08.400091+0000","last_undegraded":"2024-04-26T21:27:08.400091+0000","last_fullsized":"2024-04-26T21:27:08.400091+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.397337+0000","last_change":"2024-04-26T21:27:02.865459+0000","last_active":"2024-04-26T21:27:08.397337+0000","last_peered":"2024-04-26T21:27:08.397337+0000","last_clean":"2024-04-26T21:27:08.397337+0000","last_became_active":"2024-04-26T21:27:02.863602+0000","last_became_peered":"2024-04-26T21:27:02.863602+0000","last_unstale":"2024-04-26T21:27:08.397337+0000","last_undegraded":"2024-04-26T21:27:08.397337+0000","last_fullsized":"2024-04-26T21:27:08.397337+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:03.733858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398941+0000","last_change":"2024-04-26T21:27:02.873711+0000","last_active":"2024-04-26T21:27:08.398941+0000","last_peered":"2024-04-26T21:27:08.398941+0000","last_clean":"2024-04-26T21:27:08.398941+0000","last_became_active":"2024-04-26T21:27:02.873349+0000","last_became_peered":"2024-04-26T21:27:02.873349+0000","last_unstale":"2024-04-26T21:27:08.398941+0000","last_undegraded":"2024-04-26T21:27:08.398941+0000","last_fullsized":"2024-04-26T21:27:08.398941+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:10.841536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-26T21:27:08.398767+0000","last_change":"2024-04-26T21:27:02.852625+0000","last_active":"2024-04-26T21:27:08.398767+0000","last_peered":"2024-04-26T21:27:08.398767+0000","last_clean":"2024-04-26T21:27:08.398767+0000","last_became_active":"2024-04-26T21:27:02.852177+0000","last_became_peered":"2024-04-26T21:27:02.852177+0000","last_unstale":"2024-04-26T21:27:08.398767+0000","last_undegraded":"2024-04-26T21:27:08.398767+0000","last_fullsized":"2024-04-26T21:27:08.398767+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:52.046678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7452,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":131,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":98484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318291,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":486206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579928,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1634304,"data_stored":1080790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":490538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005093,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1024000,"data_stored":486362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266730,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32752,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712400,"statfs":{"total":95995035648,"available":95961497600,"internally_reserved":0,"allocated":1613824,"data_stored":1077784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462960,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":485624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":527530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1032192,"data_stored":487102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1081344,"data_stored":528880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":485908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888124,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":490364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724598,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36928,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708224,"statfs":{"total":95995035648,"available":95957221376,"internally_reserved":0,"allocated":1630208,"data_stored":1081564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":41906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":43256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:27:34.252 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-26T21:27:34.252 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T21:27:34.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T21:27:34.749 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:35.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:34 smithi001 ceph-mon[28054]: pgmap v340: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-26T21:27:35.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:34 smithi001 ceph-mon[28054]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:27:35.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:34 smithi038 ceph-mon[32724]: pgmap v340: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-26T21:27:35.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:34 smithi038 ceph-mon[32724]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:27:35.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:34 smithi186 ceph-mon[32991]: pgmap v340: 129 pgs: 129 active+clean; 584 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-04-26T21:27:35.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:34 smithi186 ceph-mon[32991]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:27:36.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:35 smithi001 ceph-mon[28054]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:27:36.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:35 smithi038 ceph-mon[32724]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:27:36.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:35 smithi186 ceph-mon[32991]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:27:36.980 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:36 smithi001 ceph-mon[28054]: pgmap v341: 129 pgs: 129 active+clean; 621 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.8 KiB/s wr, 12 op/s 2024-04-26T21:27:37.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:36 smithi038 ceph-mon[32724]: pgmap v341: 129 pgs: 129 active+clean; 621 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.8 KiB/s wr, 12 op/s 2024-04-26T21:27:37.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.165+0000 7f902d800700 1 -- 172.21.15.1:0/149807155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90281068d0 msgr2=0x7f9028106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:37.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.165+0000 7f902d800700 1 --2- 172.21.15.1:0/149807155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90281068d0 0x7f9028106d30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9014009920 tx=0x7f9014009c30 comp rx=0 tx=0).stop 2024-04-26T21:27:37.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.166+0000 7f902d800700 1 -- 172.21.15.1:0/149807155 shutdown_connections 2024-04-26T21:27:37.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.166+0000 7f902d800700 1 --2- 172.21.15.1:0/149807155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9028107270 0x7f902810db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:37.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.166+0000 7f902d800700 1 --2- 172.21.15.1:0/149807155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f90281056d0 0x7f9028105af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:37.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.166+0000 7f902d800700 1 --2- 172.21.15.1:0/149807155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90281068d0 0x7f9028106d30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:37.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.166+0000 7f902d800700 1 -- 172.21.15.1:0/149807155 >> 172.21.15.1:0/149807155 conn(0x7f9028100c70 msgr2=0x7f90281030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:37.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.166+0000 7f902d800700 1 -- 172.21.15.1:0/149807155 shutdown_connections 2024-04-26T21:27:37.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.166+0000 7f902d800700 1 -- 172.21.15.1:0/149807155 wait complete. 2024-04-26T21:27:37.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.167+0000 7f902d800700 1 Processor -- start 2024-04-26T21:27:37.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.167+0000 7f902d800700 1 -- start start 2024-04-26T21:27:37.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.167+0000 7f902d800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90281056d0 0x7f902819b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:37.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.167+0000 7f902d800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90281068d0 0x7f902819bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:37.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.167+0000 7f902d800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9028107270 0x7f90281a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:37.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.167+0000 7f902d800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f90281111c0 con 0x7f9028107270 2024-04-26T21:27:37.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.167+0000 7f902d800700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9028111040 con 0x7f90281056d0 2024-04-26T21:27:37.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.167+0000 7f902d800700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9028111340 con 0x7f90281068d0 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9026a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90281056d0 0x7f902819b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9027400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9028107270 0x7f90281a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9026a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90281056d0 0x7f902819b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:46880/0 (socket says 172.21.15.1:46880) 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9026a00700 1 -- 172.21.15.1:0/91100063 learned_addr learned my addr 172.21.15.1:0/91100063 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9026000700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90281068d0 0x7f902819bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9027400700 1 -- 172.21.15.1:0/91100063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90281068d0 msgr2=0x7f902819bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9027400700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90281068d0 0x7f902819bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9027400700 1 -- 172.21.15.1:0/91100063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90281056d0 msgr2=0x7f902819b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9027400700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90281056d0 0x7f902819b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9027400700 1 -- 172.21.15.1:0/91100063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90281a3440 con 0x7f9028107270 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.168+0000 7f9026a00700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90281056d0 0x7f902819b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.169+0000 7f9027400700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9028107270 0x7f90281a2da0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9020009f60 tx=0x7f9020009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:37.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.169+0000 7f901f400700 1 -- 172.21.15.1:0/91100063 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90200092e0 con 0x7f9028107270 2024-04-26T21:27:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.169+0000 7f901f400700 1 -- 172.21.15.1:0/91100063 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9020004bb0 con 0x7f9028107270 2024-04-26T21:27:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.169+0000 7f902d800700 1 -- 172.21.15.1:0/91100063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90281a36d0 con 0x7f9028107270 2024-04-26T21:27:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.169+0000 7f901f400700 1 -- 172.21.15.1:0/91100063 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9020004020 con 0x7f9028107270 2024-04-26T21:27:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.169+0000 7f902d800700 1 -- 172.21.15.1:0/91100063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90281a3c10 con 0x7f9028107270 2024-04-26T21:27:37.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.170+0000 7f902d800700 1 -- 172.21.15.1:0/91100063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9028051960 con 0x7f9028107270 2024-04-26T21:27:37.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.172+0000 7f901f400700 1 -- 172.21.15.1:0/91100063 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f902001b030 con 0x7f9028107270 2024-04-26T21:27:37.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.174+0000 7f901f400700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f900c0a3d80 0x7f900c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:37.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.174+0000 7f901f400700 1 -- 172.21.15.1:0/91100063 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f90200d01f0 con 0x7f9028107270 2024-04-26T21:27:37.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.174+0000 7f9026a00700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f900c0a3d80 0x7f900c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:37.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.175+0000 7f9026a00700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f900c0a3d80 0x7f900c0a6240 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9018009fd0 tx=0x7f9018009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:37.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.179+0000 7f901f400700 1 -- 172.21.15.1:0/91100063 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9020022020 con 0x7f9028107270 2024-04-26T21:27:37.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:36 smithi186 ceph-mon[32991]: pgmap v341: 129 pgs: 129 active+clean; 621 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 5.8 KiB/s wr, 12 op/s 2024-04-26T21:27:37.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.506+0000 7f902d800700 1 -- 172.21.15.1:0/91100063 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f902802e530 con 0x7f9028107270 2024-04-26T21:27:37.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.510+0000 7f901f400700 1 -- 172.21.15.1:0/91100063 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f902802e530 con 0x7f9028107270 2024-04-26T21:27:37.511 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-04-26T21:27:37.511 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQBJHCxmIKJGHhAAlTX2bckqL0Miskw0cPW69g== 2024-04-26T21:27:37.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 -- 172.21.15.1:0/91100063 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f900c0a3d80 msgr2=0x7f900c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:37.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f900c0a3d80 0x7f900c0a6240 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9018009fd0 tx=0x7f9018009310 comp rx=0 tx=0).stop 2024-04-26T21:27:37.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 -- 172.21.15.1:0/91100063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9028107270 msgr2=0x7f90281a2da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:37.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9028107270 0x7f90281a2da0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9020009f60 tx=0x7f9020009f90 comp rx=0 tx=0).stop 2024-04-26T21:27:37.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 -- 172.21.15.1:0/91100063 shutdown_connections 2024-04-26T21:27:37.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f900c0a3d80 0x7f900c0a6240 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:37.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9028107270 0x7f90281a2da0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:37.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90281056d0 0x7f902819b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:37.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 --2- 172.21.15.1:0/91100063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90281068d0 0x7f902819bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:37.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 -- 172.21.15.1:0/91100063 >> 172.21.15.1:0/91100063 conn(0x7f9028100c70 msgr2=0x7f9028109c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:37.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.513+0000 7f902d800700 1 -- 172.21.15.1:0/91100063 shutdown_connections 2024-04-26T21:27:37.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:37.514+0000 7f902d800700 1 -- 172.21.15.1:0/91100063 wait complete. 2024-04-26T21:27:37.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:37 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/91100063' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T21:27:37.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:37 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/91100063' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T21:27:38.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:37 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/91100063' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T21:27:38.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:37 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/91100063' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T21:27:38.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:37 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/91100063' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T21:27:38.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:37 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/91100063' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T21:27:38.369 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:27:38.369 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T21:27:38.369 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T21:27:38.516 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T21:27:38.517 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T21:27:38.517 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T21:27:39.019 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:39.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:38 smithi001 ceph-mon[28054]: pgmap v342: 129 pgs: 129 active+clean; 621 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-04-26T21:27:39.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:38 smithi038 ceph-mon[32724]: pgmap v342: 129 pgs: 129 active+clean; 621 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-04-26T21:27:39.105 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:38 smithi186 ceph-mon[32991]: pgmap v342: 129 pgs: 129 active+clean; 621 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-04-26T21:27:41.049 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:40 smithi001 ceph-mon[28054]: pgmap v343: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2024-04-26T21:27:41.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:40 smithi038 ceph-mon[32724]: pgmap v343: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2024-04-26T21:27:41.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:40 smithi186 ceph-mon[32991]: pgmap v343: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2024-04-26T21:27:41.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.220+0000 7f2308a00700 1 -- 172.21.15.1:0/2975491502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2304078270 msgr2=0x7f2304113dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:41.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.220+0000 7f2308a00700 1 --2- 172.21.15.1:0/2975491502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2304078270 0x7f2304113dc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009990 tx=0x7f22ec009ca0 comp rx=0 tx=0).stop 2024-04-26T21:27:41.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.220+0000 7f2308a00700 1 -- 172.21.15.1:0/2975491502 shutdown_connections 2024-04-26T21:27:41.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.220+0000 7f2308a00700 1 --2- 172.21.15.1:0/2975491502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2304077950 0x7f2304077d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:41.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.220+0000 7f2308a00700 1 --2- 172.21.15.1:0/2975491502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2304114300 0x7f23041166f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:41.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.220+0000 7f2308a00700 1 --2- 172.21.15.1:0/2975491502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2304078270 0x7f2304113dc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:41.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.220+0000 7f2308a00700 1 -- 172.21.15.1:0/2975491502 >> 172.21.15.1:0/2975491502 conn(0x7f2304101150 msgr2=0x7f2304103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:41.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.221+0000 7f2308a00700 1 -- 172.21.15.1:0/2975491502 shutdown_connections 2024-04-26T21:27:41.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.221+0000 7f2308a00700 1 -- 172.21.15.1:0/2975491502 wait complete. 2024-04-26T21:27:41.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.222+0000 7f2308a00700 1 Processor -- start 2024-04-26T21:27:41.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.222+0000 7f2308a00700 1 -- start start 2024-04-26T21:27:41.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.222+0000 7f2308a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2304077950 0x7f23041a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:41.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.222+0000 7f2308a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2304078270 0x7f23041a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:41.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.222+0000 7f2308a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2304114300 0x7f23041a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:41.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.222+0000 7f2308a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2304119ce0 con 0x7f2304077950 2024-04-26T21:27:41.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.222+0000 7f2308a00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2304119b60 con 0x7f2304114300 2024-04-26T21:27:41.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.222+0000 7f2308a00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2304119e60 con 0x7f2304078270 2024-04-26T21:27:41.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.222+0000 7f2301e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2304077950 0x7f23041a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:41.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f2301e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2304077950 0x7f23041a01b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46030/0 (socket says 172.21.15.1:46030) 2024-04-26T21:27:41.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f2301e00700 1 -- 172.21.15.1:0/1124554340 learned_addr learned my addr 172.21.15.1:0/1124554340 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:41.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f2302800700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2304114300 0x7f23041a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:41.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f22fbe00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2304078270 0x7f23041a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f2302800700 1 -- 172.21.15.1:0/1124554340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2304078270 msgr2=0x7f23041a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f2302800700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2304078270 0x7f23041a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f2302800700 1 -- 172.21.15.1:0/1124554340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2304077950 msgr2=0x7f23041a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f2302800700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2304077950 0x7f23041a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f2302800700 1 -- 172.21.15.1:0/1124554340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23041a51c0 con 0x7f2304114300 2024-04-26T21:27:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f2301e00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2304077950 0x7f23041a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.223+0000 7f2302800700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2304114300 0x7f23041a4a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f230407bc40 tx=0x7f22fc00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.224+0000 7f22faa00700 1 -- 172.21.15.1:0/1124554340 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0092e0 con 0x7f2304114300 2024-04-26T21:27:41.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.224+0000 7f2308a00700 1 -- 172.21.15.1:0/1124554340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23041a5450 con 0x7f2304114300 2024-04-26T21:27:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.224+0000 7f22faa00700 1 -- 172.21.15.1:0/1124554340 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22fc004bb0 con 0x7f2304114300 2024-04-26T21:27:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.224+0000 7f22faa00700 1 -- 172.21.15.1:0/1124554340 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc004020 con 0x7f2304114300 2024-04-26T21:27:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.224+0000 7f2308a00700 1 -- 172.21.15.1:0/1124554340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23041acd30 con 0x7f2304114300 2024-04-26T21:27:41.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.225+0000 7f2308a00700 1 -- 172.21.15.1:0/1124554340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f230406a110 con 0x7f2304114300 2024-04-26T21:27:41.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.227+0000 7f22faa00700 1 -- 172.21.15.1:0/1124554340 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f22fc01b030 con 0x7f2304114300 2024-04-26T21:27:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.228+0000 7f22faa00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22e80a3cd0 0x7f22e80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.228+0000 7f22faa00700 1 -- 172.21.15.1:0/1124554340 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f22fc0d0230 con 0x7f2304114300 2024-04-26T21:27:41.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.229+0000 7f2301e00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22e80a3cd0 0x7f22e80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:41.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.229+0000 7f2301e00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22e80a3cd0 0x7f22e80a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f22f4011590 tx=0x7f22f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:41.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.233+0000 7f22faa00700 1 -- 172.21.15.1:0/1124554340 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f22fc022020 con 0x7f2304114300 2024-04-26T21:27:41.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.527+0000 7f2308a00700 1 -- 172.21.15.1:0/1124554340 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f230402e490 con 0x7f2304114300 2024-04-26T21:27:41.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.534+0000 7f22faa00700 1 -- 172.21.15.1:0/1124554340 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264988 (secure 0 0 0) 0x7f22fc010040 con 0x7f2304114300 2024-04-26T21:27:41.536 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.538+0000 7f2308a00700 1 -- 172.21.15.1:0/1124554340 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22e80a3cd0 msgr2=0x7f22e80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:41.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.538+0000 7f2308a00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22e80a3cd0 0x7f22e80a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f22f4011590 tx=0x7f22f4009260 comp rx=0 tx=0).stop 2024-04-26T21:27:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.539+0000 7f2308a00700 1 -- 172.21.15.1:0/1124554340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2304114300 msgr2=0x7f23041a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.539+0000 7f2308a00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2304114300 0x7f23041a4a80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f230407bc40 tx=0x7f22fc00ef90 comp rx=0 tx=0).stop 2024-04-26T21:27:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.539+0000 7f2308a00700 1 -- 172.21.15.1:0/1124554340 shutdown_connections 2024-04-26T21:27:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.539+0000 7f2308a00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22e80a3cd0 0x7f22e80a6190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.539+0000 7f2308a00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2304114300 0x7f23041a4a80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.539+0000 7f2308a00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2304077950 0x7f23041a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.539+0000 7f2308a00700 1 --2- 172.21.15.1:0/1124554340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2304078270 0x7f23041a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.539+0000 7f2308a00700 1 -- 172.21.15.1:0/1124554340 >> 172.21.15.1:0/1124554340 conn(0x7f2304101150 msgr2=0x7f2304103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.539+0000 7f2308a00700 1 -- 172.21.15.1:0/1124554340 shutdown_connections 2024-04-26T21:27:41.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:41.539+0000 7f2308a00700 1 -- 172.21.15.1:0/1124554340 wait complete. 2024-04-26T21:27:41.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:41 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1124554340' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T21:27:42.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:41 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1124554340' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T21:27:42.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:41 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1124554340' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T21:27:42.377 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":2786715792},{"type":"v1","addr":"172.21.15.1:6801","nonce":2786715792}]},"active_addr":"172.21.15.1:6801/2786715792","active_change":"2024-04-26T21:18:55.152062+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.1:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":590615206}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2275413710}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":4024107554}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2080672016}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2563183900}]}]} 2024-04-26T21:27:42.383 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T21:27:42.383 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T21:27:42.383 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T21:27:42.889 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:43.035 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:42 smithi001 ceph-mon[28054]: pgmap v344: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-04-26T21:27:43.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:42 smithi038 ceph-mon[32724]: pgmap v344: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-04-26T21:27:43.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:42 smithi186 ceph-mon[32991]: pgmap v344: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.3 KiB/s wr, 13 op/s 2024-04-26T21:27:44.887 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:44 smithi001 ceph-mon[28054]: pgmap v345: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-04-26T21:27:45.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:44 smithi038 ceph-mon[32724]: pgmap v345: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-04-26T21:27:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.173+0000 7f272cc00700 1 -- 172.21.15.1:0/2859071953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2728105560 msgr2=0x7f27281059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.173+0000 7f272cc00700 1 --2- 172.21.15.1:0/2859071953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2728105560 0x7f27281059e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f27140099c0 tx=0x7f2714009cd0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.174+0000 7f272cc00700 1 -- 172.21.15.1:0/2859071953 shutdown_connections 2024-04-26T21:27:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.174+0000 7f272cc00700 1 --2- 172.21.15.1:0/2859071953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2728105f20 0x7f27281123e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.174+0000 7f272cc00700 1 --2- 172.21.15.1:0/2859071953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272810b610 0x7f272810b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.174+0000 7f272cc00700 1 --2- 172.21.15.1:0/2859071953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2728105560 0x7f27281059e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.174+0000 7f272cc00700 1 -- 172.21.15.1:0/2859071953 >> 172.21.15.1:0/2859071953 conn(0x7f2728101010 msgr2=0x7f2728103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.174+0000 7f272cc00700 1 -- 172.21.15.1:0/2859071953 shutdown_connections 2024-04-26T21:27:45.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.174+0000 7f272cc00700 1 -- 172.21.15.1:0/2859071953 wait complete. 2024-04-26T21:27:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.175+0000 7f272cc00700 1 Processor -- start 2024-04-26T21:27:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.175+0000 7f272cc00700 1 -- start start 2024-04-26T21:27:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.175+0000 7f272cc00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2728105f20 0x7f272819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f272cc00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272810b610 0x7f272819c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f272cc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27281a0680 0x7f27281a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f272cc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2728115780 con 0x7f27281a0680 2024-04-26T21:27:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f272cc00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2728115600 con 0x7f272810b610 2024-04-26T21:27:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f272cc00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2728115900 con 0x7f2728105f20 2024-04-26T21:27:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2726a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272810b610 0x7f272819c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2726a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272810b610 0x7f272819c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54222/0 (socket says 172.21.15.1:54222) 2024-04-26T21:27:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2726a00700 1 -- 172.21.15.1:0/3510605229 learned_addr learned my addr 172.21.15.1:0/3510605229 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2727400700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2728105f20 0x7f272819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:45.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2727e00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27281a0680 0x7f27281a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2727400700 1 -- 172.21.15.1:0/3510605229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272810b610 msgr2=0x7f272819c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:45.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2727400700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272810b610 0x7f272819c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2727400700 1 -- 172.21.15.1:0/3510605229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27281a0680 msgr2=0x7f27281a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2727400700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27281a0680 0x7f27281a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2727400700 1 -- 172.21.15.1:0/3510605229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27281a11a0 con 0x7f2728105f20 2024-04-26T21:27:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.176+0000 7f2726a00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272810b610 0x7f272819c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.177+0000 7f2727400700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2728105f20 0x7f272819bdb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f271800e9f0 tx=0x7f271800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.177+0000 7f271fe00700 1 -- 172.21.15.1:0/3510605229 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271800c9c0 con 0x7f2728105f20 2024-04-26T21:27:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.177+0000 7f272cc00700 1 -- 172.21.15.1:0/3510605229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27281a1490 con 0x7f2728105f20 2024-04-26T21:27:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.177+0000 7f271fe00700 1 -- 172.21.15.1:0/3510605229 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f271800cb60 con 0x7f2728105f20 2024-04-26T21:27:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.177+0000 7f271fe00700 1 -- 172.21.15.1:0/3510605229 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718018680 con 0x7f2728105f20 2024-04-26T21:27:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.177+0000 7f272cc00700 1 -- 172.21.15.1:0/3510605229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27281a8cc0 con 0x7f2728105f20 2024-04-26T21:27:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.178+0000 7f272cc00700 1 -- 172.21.15.1:0/3510605229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f272806a110 con 0x7f2728105f20 2024-04-26T21:27:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.180+0000 7f271fe00700 1 -- 172.21.15.1:0/3510605229 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f2718010430 con 0x7f2728105f20 2024-04-26T21:27:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.182+0000 7f271fe00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f270c0a3d50 0x7f270c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.182+0000 7f271fe00700 1 -- 172.21.15.1:0/3510605229 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f27180d0910 con 0x7f2728105f20 2024-04-26T21:27:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.182+0000 7f2726a00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f270c0a3d50 0x7f270c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.183+0000 7f2726a00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f270c0a3d50 0x7f270c0a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2714003760 tx=0x7f2714005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.186+0000 7f271fe00700 1 -- 172.21.15.1:0/3510605229 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f271809cef0 con 0x7f2728105f20 2024-04-26T21:27:45.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:44 smithi186 ceph-mon[32991]: pgmap v345: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 5.4 KiB/s wr, 10 op/s 2024-04-26T21:27:45.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.424+0000 7f272cc00700 1 -- 172.21.15.1:0/3510605229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f272802e490 con 0x7f2728105f20 2024-04-26T21:27:45.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.427+0000 7f271fe00700 1 -- 172.21.15.1:0/3510605229 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23972 (secure 0 0 0) 0x7f271809c7f0 con 0x7f2728105f20 2024-04-26T21:27:45.428 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:45.428 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:27:07.816919+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:27:45.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.430+0000 7f272cc00700 1 -- 172.21.15.1:0/3510605229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f270c0a3d50 msgr2=0x7f270c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:45.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.430+0000 7f272cc00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f270c0a3d50 0x7f270c0a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2714003760 tx=0x7f2714005fb0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.430+0000 7f272cc00700 1 -- 172.21.15.1:0/3510605229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2728105f20 msgr2=0x7f272819bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:45.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.430+0000 7f272cc00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2728105f20 0x7f272819bdb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f271800e9f0 tx=0x7f271800ed00 comp rx=0 tx=0).stop 2024-04-26T21:27:45.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.430+0000 7f272cc00700 1 -- 172.21.15.1:0/3510605229 shutdown_connections 2024-04-26T21:27:45.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.430+0000 7f272cc00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f270c0a3d50 0x7f270c0a6210 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.430+0000 7f272cc00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2728105f20 0x7f272819bdb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.430+0000 7f272cc00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27281a0680 0x7f27281a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.430+0000 7f272cc00700 1 --2- 172.21.15.1:0/3510605229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272810b610 0x7f272819c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:45.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.430+0000 7f272cc00700 1 -- 172.21.15.1:0/3510605229 >> 172.21.15.1:0/3510605229 conn(0x7f2728101010 msgr2=0x7f2728102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:45.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.431+0000 7f272cc00700 1 -- 172.21.15.1:0/3510605229 shutdown_connections 2024-04-26T21:27:45.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:45.431+0000 7f272cc00700 1 -- 172.21.15.1:0/3510605229 wait complete. 2024-04-26T21:27:46.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:45 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3510605229' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:46.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:45 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3510605229' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:46.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:45 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3510605229' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:46.302 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T21:27:46.303 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T21:27:46.785 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:47.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:46 smithi001 ceph-mon[28054]: pgmap v346: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.8 KiB/s wr, 9 op/s 2024-04-26T21:27:47.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:46 smithi038 ceph-mon[32724]: pgmap v346: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.8 KiB/s wr, 9 op/s 2024-04-26T21:27:47.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:46 smithi186 ceph-mon[32991]: pgmap v346: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.8 KiB/s wr, 9 op/s 2024-04-26T21:27:49.048 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:48 smithi001 ceph-mon[28054]: pgmap v347: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T21:27:49.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:48 smithi038 ceph-mon[32724]: pgmap v347: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T21:27:49.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.110+0000 7f76fce00700 1 -- 172.21.15.1:0/785353913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f8105640 msgr2=0x7f76f8105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:49.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.110+0000 7f76fce00700 1 --2- 172.21.15.1:0/785353913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f8105640 0x7f76f8105ac0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f76e00098f0 tx=0x7f76e0009c00 comp rx=0 tx=0).stop 2024-04-26T21:27:49.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.110+0000 7f76fce00700 1 -- 172.21.15.1:0/785353913 shutdown_connections 2024-04-26T21:27:49.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.110+0000 7f76fce00700 1 --2- 172.21.15.1:0/785353913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8106000 0x7f76f8112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:49.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.110+0000 7f76fce00700 1 --2- 172.21.15.1:0/785353913 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76f810b640 0x7f76f810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:49.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.110+0000 7f76fce00700 1 --2- 172.21.15.1:0/785353913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f8105640 0x7f76f8105ac0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:49.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.110+0000 7f76fce00700 1 -- 172.21.15.1:0/785353913 >> 172.21.15.1:0/785353913 conn(0x7f76f8101130 msgr2=0x7f76f8103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:49.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.110+0000 7f76fce00700 1 -- 172.21.15.1:0/785353913 shutdown_connections 2024-04-26T21:27:49.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.110+0000 7f76fce00700 1 -- 172.21.15.1:0/785353913 wait complete. 2024-04-26T21:27:49.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.112+0000 7f76fce00700 1 Processor -- start 2024-04-26T21:27:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.112+0000 7f76fce00700 1 -- start start 2024-04-26T21:27:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76fce00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76f8105640 0x7f76f819bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76fce00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8106000 0x7f76f819c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76fce00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f810b640 0x7f76f81a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76fce00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f76f8115840 con 0x7f76f8105640 2024-04-26T21:27:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76fce00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81156c0 con 0x7f76f810b640 2024-04-26T21:27:49.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76fce00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f76f81159c0 con 0x7f76f8106000 2024-04-26T21:27:49.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76f5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76f8105640 0x7f76f819bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:49.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76f5e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76f8105640 0x7f76f819bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39412/0 (socket says 172.21.15.1:39412) 2024-04-26T21:27:49.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76efe00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8106000 0x7f76f819c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:49.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76f5e00700 1 -- 172.21.15.1:0/4205383848 learned_addr learned my addr 172.21.15.1:0/4205383848 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:49.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76f6800700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f810b640 0x7f76f81a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:49.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76efe00700 1 -- 172.21.15.1:0/4205383848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f810b640 msgr2=0x7f76f81a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:49.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76efe00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f810b640 0x7f76f81a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:49.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76efe00700 1 -- 172.21.15.1:0/4205383848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76f8105640 msgr2=0x7f76f819bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:49.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76efe00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76f8105640 0x7f76f819bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:49.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.113+0000 7f76efe00700 1 -- 172.21.15.1:0/4205383848 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f81a0d80 con 0x7f76f8106000 2024-04-26T21:27:49.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.114+0000 7f76efe00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8106000 0x7f76f819c2b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009230 tx=0x7f76e000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:49.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.114+0000 7f76eea00700 1 -- 172.21.15.1:0/4205383848 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e0005040 con 0x7f76f8106000 2024-04-26T21:27:49.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.114+0000 7f76eea00700 1 -- 172.21.15.1:0/4205383848 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76e0005690 con 0x7f76f8106000 2024-04-26T21:27:49.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.114+0000 7f76fce00700 1 -- 172.21.15.1:0/4205383848 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f81a1010 con 0x7f76f8106000 2024-04-26T21:27:49.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.114+0000 7f76eea00700 1 -- 172.21.15.1:0/4205383848 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e00038e0 con 0x7f76f8106000 2024-04-26T21:27:49.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.114+0000 7f76fce00700 1 -- 172.21.15.1:0/4205383848 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f81a88f0 con 0x7f76f8106000 2024-04-26T21:27:49.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.115+0000 7f76fce00700 1 -- 172.21.15.1:0/4205383848 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76f806a110 con 0x7f76f8106000 2024-04-26T21:27:49.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.117+0000 7f76eea00700 1 -- 172.21.15.1:0/4205383848 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f76e000b760 con 0x7f76f8106000 2024-04-26T21:27:49.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.119+0000 7f76eea00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f76dc0a3d50 0x7f76dc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:49.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.119+0000 7f76eea00700 1 -- 172.21.15.1:0/4205383848 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f76e00d1550 con 0x7f76f8106000 2024-04-26T21:27:49.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.119+0000 7f76f5e00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f76dc0a3d50 0x7f76dc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:49.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.120+0000 7f76f5e00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f76dc0a3d50 0x7f76dc0a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009750 tx=0x7f76e8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:49.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.123+0000 7f76eea00700 1 -- 172.21.15.1:0/4205383848 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f76e0013030 con 0x7f76f8106000 2024-04-26T21:27:49.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:48 smithi186 ceph-mon[32991]: pgmap v347: 129 pgs: 129 active+clean; 630 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-04-26T21:27:49.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.360+0000 7f76fce00700 1 -- 172.21.15.1:0/4205383848 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76f802e490 con 0x7f76f8106000 2024-04-26T21:27:49.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.362+0000 7f76eea00700 1 -- 172.21.15.1:0/4205383848 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23972 (secure 0 0 0) 0x7f76e0021380 con 0x7f76f8106000 2024-04-26T21:27:49.364 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:27:49.364 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":91,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:27:07.816919+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:27:49.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 -- 172.21.15.1:0/4205383848 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f76dc0a3d50 msgr2=0x7f76dc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:49.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f76dc0a3d50 0x7f76dc0a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f76e8009750 tx=0x7f76e8009310 comp rx=0 tx=0).stop 2024-04-26T21:27:49.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 -- 172.21.15.1:0/4205383848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8106000 msgr2=0x7f76f819c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:49.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8106000 0x7f76f819c2b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009230 tx=0x7f76e000fa00 comp rx=0 tx=0).stop 2024-04-26T21:27:49.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 -- 172.21.15.1:0/4205383848 shutdown_connections 2024-04-26T21:27:49.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f76dc0a3d50 0x7f76dc0a6210 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:49.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f76f810b640 0x7f76f81a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:49.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76f8105640 0x7f76f819bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:49.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 --2- 172.21.15.1:0/4205383848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76f8106000 0x7f76f819c2b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:49.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 -- 172.21.15.1:0/4205383848 >> 172.21.15.1:0/4205383848 conn(0x7f76f8101130 msgr2=0x7f76f8101870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:49.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 -- 172.21.15.1:0/4205383848 shutdown_connections 2024-04-26T21:27:49.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:49.366+0000 7f76fce00700 1 -- 172.21.15.1:0/4205383848 wait complete. 2024-04-26T21:27:50.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:49 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4205383848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:50.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:49 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4205383848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:50.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T21:27:50.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T21:27:50.193 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T21:27:50.193 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T21:27:50.194 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T21:27:50.194 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T21:27:50.194 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T21:27:50.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T21:27:50.195 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-26T21:27:50.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-26T21:27:50.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-26T21:27:50.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-26T21:27:50.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:49 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4205383848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:27:50.814 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:50 smithi001 ceph-mon[28054]: pgmap v348: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-26T21:27:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:50 smithi038 ceph-mon[32724]: pgmap v348: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-26T21:27:51.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:50 smithi186 ceph-mon[32991]: pgmap v348: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-26T21:27:51.415 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:51.471 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:51.482 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:51.925 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:51.926 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:51.928 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:51.935 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:27:53.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:52 smithi001 ceph-mon[28054]: pgmap v349: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T21:27:53.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:52 smithi038 ceph-mon[32724]: pgmap v349: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T21:27:53.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:52 smithi186 ceph-mon[32991]: pgmap v349: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T21:27:55.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:54 smithi001 ceph-mon[28054]: pgmap v350: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T21:27:55.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:54 smithi038 ceph-mon[32724]: pgmap v350: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T21:27:55.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:54 smithi186 ceph-mon[32991]: pgmap v350: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T21:27:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-26T21:27:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-26T21:27:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-26T21:27:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-26T21:27:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-26T21:27:56.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-26T21:27:56.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2024-04-26T21:27:56.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-26T21:27:56.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-26T21:27:56.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:55 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-04-26T21:27:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-26T21:27:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-26T21:27:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-26T21:27:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-26T21:27:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-26T21:27:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-26T21:27:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2024-04-26T21:27:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-26T21:27:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-26T21:27:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:55 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-04-26T21:27:56.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-04-26T21:27:56.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-04-26T21:27:56.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-26T21:27:56.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-04-26T21:27:56.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-04-26T21:27:56.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-26T21:27:56.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2024-04-26T21:27:56.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-04-26T21:27:56.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-04-26T21:27:56.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:55 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: pgmap v351: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-04-26T21:27:57.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:56 smithi001 ceph-mon[28054]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T21:27:57.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: pgmap v351: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-04-26T21:27:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:56 smithi186 ceph-mon[32991]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T21:27:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: pgmap v351: 129 pgs: 129 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 341 B/s wr, 6 op/s 2024-04-26T21:27:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-04-26T21:27:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-04-26T21:27:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-26T21:27:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-04-26T21:27:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-04-26T21:27:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-26T21:27:57.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2024-04-26T21:27:57.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-04-26T21:27:57.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-04-26T21:27:57.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2024-04-26T21:27:57.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:56 smithi038 ceph-mon[32724]: osdmap e92: 12 total, 12 up, 12 in 2024-04-26T21:27:58.179 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:57 smithi001 ceph-mon[28054]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T21:27:58.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:57 smithi186 ceph-mon[32991]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T21:27:58.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:57 smithi038 ceph-mon[32724]: osdmap e93: 12 total, 12 up, 12 in 2024-04-26T21:27:59.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:27:58 smithi186 ceph-mon[32991]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T21:27:59.228 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:27:58 smithi001 ceph-mon[28054]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T21:27:59.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:27:58 smithi038 ceph-mon[32724]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 633 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T21:27:59.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.821+0000 7fbb35200700 1 -- 172.21.15.1:0/3647816087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb30103e30 msgr2=0x7fbb30110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:59.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.821+0000 7fbb35200700 1 --2- 172.21.15.1:0/3647816087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb30103e30 0x7fbb30110290 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800b0a0 tx=0x7fbb2800b3b0 comp rx=0 tx=0).stop 2024-04-26T21:27:59.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.821+0000 7fbb35200700 1 -- 172.21.15.1:0/3647816087 shutdown_connections 2024-04-26T21:27:59.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.821+0000 7fbb35200700 1 --2- 172.21.15.1:0/3647816087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb301094c0 0x7fbb301098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:59.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.821+0000 7fbb35200700 1 --2- 172.21.15.1:0/3647816087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb30103e30 0x7fbb30110290 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:59.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.821+0000 7fbb35200700 1 --2- 172.21.15.1:0/3647816087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb30103470 0x7fbb301038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:59.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.821+0000 7fbb35200700 1 -- 172.21.15.1:0/3647816087 >> 172.21.15.1:0/3647816087 conn(0x7fbb3007a580 msgr2=0x7fbb3007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:59.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.821+0000 7fbb35200700 1 -- 172.21.15.1:0/3647816087 shutdown_connections 2024-04-26T21:27:59.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.821+0000 7fbb35200700 1 -- 172.21.15.1:0/3647816087 wait complete. 2024-04-26T21:27:59.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.822+0000 7fbb35200700 1 Processor -- start 2024-04-26T21:27:59.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.822+0000 7fbb35200700 1 -- start start 2024-04-26T21:27:59.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb35200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb30103470 0x7fbb30075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:59.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb35200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb30103e30 0x7fbb30075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:59.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb35200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb301094c0 0x7fbb300761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:59.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb35200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30113610 con 0x7fbb30103470 2024-04-26T21:27:59.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb35200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30113490 con 0x7fbb301094c0 2024-04-26T21:27:59.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb35200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30113790 con 0x7fbb30103e30 2024-04-26T21:27:59.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb2de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb30103470 0x7fbb30075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:59.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb2de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb30103470 0x7fbb30075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47224/0 (socket says 172.21.15.1:47224) 2024-04-26T21:27:59.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb2de00700 1 -- 172.21.15.1:0/937640973 learned_addr learned my addr 172.21.15.1:0/937640973 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:27:59.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb2d400700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb30103e30 0x7fbb30075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.823+0000 7fbb2e800700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb301094c0 0x7fbb300761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.824+0000 7fbb2d400700 1 -- 172.21.15.1:0/937640973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb301094c0 msgr2=0x7fbb300761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.824+0000 7fbb2d400700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb301094c0 0x7fbb300761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.824+0000 7fbb2d400700 1 -- 172.21.15.1:0/937640973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb30103470 msgr2=0x7fbb30075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.824+0000 7fbb2d400700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb30103470 0x7fbb30075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.824+0000 7fbb2d400700 1 -- 172.21.15.1:0/937640973 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb301a8030 con 0x7fbb30103e30 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.824+0000 7fbb2de00700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb30103470 0x7fbb30075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.824+0000 7fbb2e800700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb301094c0 0x7fbb300761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.824+0000 7fbb2d400700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb30103e30 0x7fbb30075c60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c00e9f0 tx=0x7fbb1c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.824+0000 7fbb26a00700 1 -- 172.21.15.1:0/937640973 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c00c9c0 con 0x7fbb30103e30 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.825+0000 7fbb26a00700 1 -- 172.21.15.1:0/937640973 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb1c00cb60 con 0x7fbb30103e30 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.825+0000 7fbb26a00700 1 -- 172.21.15.1:0/937640973 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c018680 con 0x7fbb30103e30 2024-04-26T21:27:59.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.825+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb301a8320 con 0x7fbb30103e30 2024-04-26T21:27:59.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.825+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb301a88b0 con 0x7fbb30103e30 2024-04-26T21:27:59.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.826+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbb30063b30 con 0x7fbb30103e30 2024-04-26T21:27:59.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.829+0000 7fbb26a00700 1 -- 172.21.15.1:0/937640973 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fbb1c010430 con 0x7fbb30103e30 2024-04-26T21:27:59.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.830+0000 7fbb26a00700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbb140a3d50 0x7fbb140a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:59.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.830+0000 7fbb26a00700 1 -- 172.21.15.1:0/937640973 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fbb1c0d0f90 con 0x7fbb30103e30 2024-04-26T21:27:59.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.830+0000 7fbb2de00700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbb140a3d50 0x7fbb140a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:59.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.831+0000 7fbb26a00700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] conn(0x7fbb140aeeb0 0x7fbb140b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:27:59.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.831+0000 7fbb26a00700 1 -- 172.21.15.1:0/937640973 --> [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb1c00be50 con 0x7fbb140aeeb0 2024-04-26T21:27:59.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.831+0000 7fbb26a00700 1 -- 172.21.15.1:0/937640973 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fbb1c0d1340 con 0x7fbb30103e30 2024-04-26T21:27:59.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.831+0000 7fbb2e800700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] conn(0x7fbb140aeeb0 0x7fbb140b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:27:59.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.831+0000 7fbb2de00700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbb140a3d50 0x7fbb140a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbb20005950 tx=0x7fbb2000b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:59.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.832+0000 7fbb2e800700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] conn(0x7fbb140aeeb0 0x7fbb140b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:27:59.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.832+0000 7fbb26a00700 1 -- 172.21.15.1:0/937640973 <== osd.4 v2:172.21.15.38:6808/2096224 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbb1c00be50 con 0x7fbb140aeeb0 2024-04-26T21:27:59.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.851+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 --> [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbb30063f30 con 0x7fbb140aeeb0 2024-04-26T21:27:59.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.852+0000 7fbb26a00700 1 -- 172.21.15.1:0/937640973 <== osd.4 v2:172.21.15.38:6808/2096224 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbb30063f30 con 0x7fbb140aeeb0 2024-04-26T21:27:59.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.852+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 >> [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] conn(0x7fbb140aeeb0 msgr2=0x7fbb140b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:59.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.852+0000 7fbb35200700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] conn(0x7fbb140aeeb0 0x7fbb140b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:27:59.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.853+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbb140a3d50 msgr2=0x7fbb140a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:59.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.853+0000 7fbb35200700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fbb140a3d50 0x7fbb140a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbb20005950 tx=0x7fbb2000b500 comp rx=0 tx=0).stop 2024-04-26T21:27:59.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.853+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb30103e30 msgr2=0x7fbb30075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:27:59.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.853+0000 7fbb35200700 1 --2- 172.21.15.1:0/937640973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb30103e30 0x7fbb30075c60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c00e9f0 tx=0x7fbb1c00ed00 comp rx=0 tx=0).stop 2024-04-26T21:27:59.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.853+0000 7fbb2e800700 1 -- 172.21.15.1:0/937640973 reap_dead start 2024-04-26T21:27:59.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.853+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 shutdown_connections 2024-04-26T21:27:59.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.853+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 >> 172.21.15.1:0/937640973 conn(0x7fbb3007a580 msgr2=0x7fbb30102270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:27:59.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.853+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 shutdown_connections 2024-04-26T21:27:59.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:27:59.853+0000 7fbb35200700 1 -- 172.21.15.1:0/937640973 wait complete. 2024-04-26T21:28:00.968 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:00 smithi001 ceph-mon[28054]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T21:28:01.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.139+0000 7f9377a00700 1 -- 172.21.15.1:0/2218961463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9370103470 msgr2=0x7f93701038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.139+0000 7f9377a00700 1 --2- 172.21.15.1:0/2218961463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9370103470 0x7f93701038f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f935c0098f0 tx=0x7f935c009c00 comp rx=0 tx=0).stop 2024-04-26T21:28:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.139+0000 7f9377a00700 1 -- 172.21.15.1:0/2218961463 shutdown_connections 2024-04-26T21:28:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.139+0000 7f9377a00700 1 --2- 172.21.15.1:0/2218961463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9370103e30 0x7f9370110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.139+0000 7f9377a00700 1 --2- 172.21.15.1:0/2218961463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f93701094c0 0x7f93701098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.140+0000 7f9377a00700 1 --2- 172.21.15.1:0/2218961463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9370103470 0x7f93701038f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.140+0000 7f9377a00700 1 -- 172.21.15.1:0/2218961463 >> 172.21.15.1:0/2218961463 conn(0x7f937007a580 msgr2=0x7f937007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.140+0000 7f9377a00700 1 -- 172.21.15.1:0/2218961463 shutdown_connections 2024-04-26T21:28:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.140+0000 7f9377a00700 1 -- 172.21.15.1:0/2218961463 wait complete. 2024-04-26T21:28:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.141+0000 7f9377a00700 1 Processor -- start 2024-04-26T21:28:01.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.141+0000 7f9377a00700 1 -- start start 2024-04-26T21:28:01.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.141+0000 7f9377a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370103470 0x7f9370075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:01.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.141+0000 7f9377a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9370103e30 0x7f9370075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:01.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.141+0000 7f9377a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93701094c0 0x7f93700761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:01.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.141+0000 7f9377a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9370113640 con 0x7f9370103470 2024-04-26T21:28:01.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.141+0000 7f9377a00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f93701134c0 con 0x7f9370103e30 2024-04-26T21:28:01.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.141+0000 7f9377a00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f93701137c0 con 0x7f93701094c0 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.142+0000 7f9375000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370103470 0x7f9370075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.142+0000 7f9375000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370103470 0x7f9370075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47232/0 (socket says 172.21.15.1:47232) 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.142+0000 7f9375000700 1 -- 172.21.15.1:0/2941992316 learned_addr learned my addr 172.21.15.1:0/2941992316 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.142+0000 7f936fe00700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9370103e30 0x7f9370075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.142+0000 7f9375a00700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93701094c0 0x7f93700761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.142+0000 7f936fe00700 1 -- 172.21.15.1:0/2941992316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93701094c0 msgr2=0x7f93700761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.142+0000 7f936fe00700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93701094c0 0x7f93700761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.142+0000 7f936fe00700 1 -- 172.21.15.1:0/2941992316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370103470 msgr2=0x7f9370075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.142+0000 7f936fe00700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9370103470 0x7f9370075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.142+0000 7f936fe00700 1 -- 172.21.15.1:0/2941992316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93701a8120 con 0x7f9370103e30 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.143+0000 7f936fe00700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9370103e30 0x7f9370075c60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f935c00c010 tx=0x7f935c00fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.143+0000 7f936d600700 1 -- 172.21.15.1:0/2941992316 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c005140 con 0x7f9370103e30 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.143+0000 7f936d600700 1 -- 172.21.15.1:0/2941992316 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f935c00b560 con 0x7f9370103e30 2024-04-26T21:28:01.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.143+0000 7f936d600700 1 -- 172.21.15.1:0/2941992316 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c003a80 con 0x7f9370103e30 2024-04-26T21:28:01.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.143+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93701a83b0 con 0x7f9370103e30 2024-04-26T21:28:01.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.143+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93701a88e0 con 0x7f9370103e30 2024-04-26T21:28:01.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.144+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9370063af0 con 0x7f9370103e30 2024-04-26T21:28:01.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.148+0000 7f936d600700 1 -- 172.21.15.1:0/2941992316 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f935c00ba50 con 0x7f9370103e30 2024-04-26T21:28:01.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.150+0000 7f936d600700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f93580a3d50 0x7f93580a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:01.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.150+0000 7f936d600700 1 -- 172.21.15.1:0/2941992316 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f935c0d1bf0 con 0x7f9370103e30 2024-04-26T21:28:01.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.150+0000 7f9375000700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f93580a3d50 0x7f93580a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:01.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.150+0000 7f936d600700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] conn(0x7f93580aeeb0 0x7f93580b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:01.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.150+0000 7f936d600700 1 -- 172.21.15.1:0/2941992316 --> [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f935c02ee50 con 0x7f93580aeeb0 2024-04-26T21:28:01.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.150+0000 7f936d600700 1 -- 172.21.15.1:0/2941992316 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f935c0d1fa0 con 0x7f9370103e30 2024-04-26T21:28:01.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.151+0000 7f9375a00700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] conn(0x7f93580aeeb0 0x7f93580b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:01.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.151+0000 7f9375000700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f93580a3d50 0x7f93580a6210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9364005950 tx=0x7f936400b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:01.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.151+0000 7f9375a00700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] conn(0x7f93580aeeb0 0x7f93580b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:01.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.152+0000 7f936d600700 1 -- 172.21.15.1:0/2941992316 <== osd.2 v2:172.21.15.186:6800/2555417177 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f935c02ee50 con 0x7f93580aeeb0 2024-04-26T21:28:01.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.171+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 --> [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9370063f20 con 0x7f93580aeeb0 2024-04-26T21:28:01.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.172+0000 7f936d600700 1 -- 172.21.15.1:0/2941992316 <== osd.2 v2:172.21.15.186:6800/2555417177 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9370063f20 con 0x7f93580aeeb0 2024-04-26T21:28:01.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.172+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 >> [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] conn(0x7f93580aeeb0 msgr2=0x7f93580b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:01.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.172+0000 7f9377a00700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] conn(0x7f93580aeeb0 0x7f93580b1350 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:01.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.172+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f93580a3d50 msgr2=0x7f93580a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:01.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.172+0000 7f9377a00700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f93580a3d50 0x7f93580a6210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9364005950 tx=0x7f936400b500 comp rx=0 tx=0).stop 2024-04-26T21:28:01.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.172+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9370103e30 msgr2=0x7f9370075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:01.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.172+0000 7f9377a00700 1 --2- 172.21.15.1:0/2941992316 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9370103e30 0x7f9370075c60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f935c00c010 tx=0x7f935c00fa70 comp rx=0 tx=0).stop 2024-04-26T21:28:01.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.172+0000 7f9375a00700 1 -- 172.21.15.1:0/2941992316 reap_dead start 2024-04-26T21:28:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.173+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 shutdown_connections 2024-04-26T21:28:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.173+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 >> 172.21.15.1:0/2941992316 conn(0x7f937007a580 msgr2=0x7f9370102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.173+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 shutdown_connections 2024-04-26T21:28:01.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:01.173+0000 7f9377a00700 1 -- 172.21.15.1:0/2941992316 wait complete. 2024-04-26T21:28:01.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:00 smithi186 ceph-mon[32991]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T21:28:01.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:00 smithi038 ceph-mon[32724]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 0 B/s, 0 objects/s recovering 2024-04-26T21:28:01.377 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:01.379 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:02.566 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:03.146 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:02 smithi001 ceph-mon[28054]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T21:28:03.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:02 smithi186 ceph-mon[32991]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T21:28:03.306 INFO:teuthology.orchestra.run.smithi001.stdout:146028888130 2024-04-26T21:28:03.306 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T21:28:03.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:02 smithi038 ceph-mon[32724]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 383 B/s wr, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T21:28:03.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.328+0000 7f0281400700 1 -- 172.21.15.1:0/3006192282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f027c1145f0 msgr2=0x7f027c1169e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:03.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.328+0000 7f0281400700 1 --2- 172.21.15.1:0/3006192282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f027c1145f0 0x7f027c1169e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0274009f90 tx=0x7f027400b330 comp rx=0 tx=0).stop 2024-04-26T21:28:03.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.329+0000 7f0281400700 1 -- 172.21.15.1:0/3006192282 shutdown_connections 2024-04-26T21:28:03.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.329+0000 7f0281400700 1 --2- 172.21.15.1:0/3006192282 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c075650 0x7f027c075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:03.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.329+0000 7f0281400700 1 --2- 172.21.15.1:0/3006192282 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f027c1145f0 0x7f027c1169e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:03.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.329+0000 7f0281400700 1 --2- 172.21.15.1:0/3006192282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f027c075f70 0x7f027c1140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:03.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.329+0000 7f0281400700 1 -- 172.21.15.1:0/3006192282 >> 172.21.15.1:0/3006192282 conn(0x7f027c07a580 msgr2=0x7f027c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:03.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.329+0000 7f0281400700 1 -- 172.21.15.1:0/3006192282 shutdown_connections 2024-04-26T21:28:03.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.329+0000 7f0281400700 1 -- 172.21.15.1:0/3006192282 wait complete. 2024-04-26T21:28:03.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.330+0000 7f0281400700 1 Processor -- start 2024-04-26T21:28:03.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.330+0000 7f0281400700 1 -- start start 2024-04-26T21:28:03.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.330+0000 7f0281400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c075650 0x7f027c1a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:03.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f0281400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f027c075f70 0x7f027c1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:03.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f0281400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f027c1145f0 0x7f027c1a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:03.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f0281400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f027c119c10 con 0x7f027c1145f0 2024-04-26T21:28:03.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f0281400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f027c119a90 con 0x7f027c075f70 2024-04-26T21:28:03.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f0281400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f027c119d90 con 0x7f027c075650 2024-04-26T21:28:03.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f0279e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f027c075f70 0x7f027c1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:03.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f027a800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c075650 0x7f027c1a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:03.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f0279e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f027c075f70 0x7f027c1a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54252/0 (socket says 172.21.15.1:54252) 2024-04-26T21:28:03.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f0279e00700 1 -- 172.21.15.1:0/1618275650 learned_addr learned my addr 172.21.15.1:0/1618275650 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:03.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f027b200700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f027c1145f0 0x7f027c1a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:03.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f027a800700 1 -- 172.21.15.1:0/1618275650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f027c075f70 msgr2=0x7f027c1a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:03.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f027a800700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f027c075f70 0x7f027c1a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.331+0000 7f027a800700 1 -- 172.21.15.1:0/1618275650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f027c1145f0 msgr2=0x7f027c1a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.332+0000 7f027a800700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f027c1145f0 0x7f027c1a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.332+0000 7f027a800700 1 -- 172.21.15.1:0/1618275650 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f027c1a5170 con 0x7f027c075650 2024-04-26T21:28:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.332+0000 7f0279e00700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f027c075f70 0x7f027c1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.332+0000 7f027a800700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c075650 0x7f027c1a0160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f026c00e9f0 tx=0x7f026c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:03.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.332+0000 7f0273400700 1 -- 172.21.15.1:0/1618275650 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c00c9c0 con 0x7f027c075650 2024-04-26T21:28:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.333+0000 7f0273400700 1 -- 172.21.15.1:0/1618275650 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f026c00cb60 con 0x7f027c075650 2024-04-26T21:28:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.333+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f027c1a5460 con 0x7f027c075650 2024-04-26T21:28:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.333+0000 7f0273400700 1 -- 172.21.15.1:0/1618275650 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c018680 con 0x7f027c075650 2024-04-26T21:28:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.333+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f027c1acce0 con 0x7f027c075650 2024-04-26T21:28:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.334+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f027c113e10 con 0x7f027c075650 2024-04-26T21:28:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.337+0000 7f0273400700 1 -- 172.21.15.1:0/1618275650 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f026c010430 con 0x7f027c075650 2024-04-26T21:28:03.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.338+0000 7f0273400700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f02600a3d50 0x7f02600a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.338+0000 7f0273400700 1 -- 172.21.15.1:0/1618275650 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f026c0d0a30 con 0x7f027c075650 2024-04-26T21:28:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.338+0000 7f0279e00700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f02600a3d50 0x7f02600a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.338+0000 7f0273400700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] conn(0x7f02600aeeb0 0x7f02600b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.339+0000 7f0273400700 1 -- 172.21.15.1:0/1618275650 --> [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f026c015e80 con 0x7f02600aeeb0 2024-04-26T21:28:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.339+0000 7f0273400700 1 -- 172.21.15.1:0/1618275650 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f026c0d52a0 con 0x7f027c075650 2024-04-26T21:28:03.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.339+0000 7f027b200700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] conn(0x7f02600aeeb0 0x7f02600b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:03.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.339+0000 7f0279e00700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f02600a3d50 0x7f02600a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0268009750 tx=0x7f0268009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:03.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.339+0000 7f027b200700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] conn(0x7f02600aeeb0 0x7f02600b1350 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:03.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.341+0000 7f0273400700 1 -- 172.21.15.1:0/1618275650 <== osd.11 v2:172.21.15.186:6824/2966559392 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f026c015e80 con 0x7f02600aeeb0 2024-04-26T21:28:03.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.359+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 --> [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f027c076d40 con 0x7f02600aeeb0 2024-04-26T21:28:03.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.360+0000 7f0273400700 1 -- 172.21.15.1:0/1618275650 <== osd.11 v2:172.21.15.186:6824/2966559392 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f027c076d40 con 0x7f02600aeeb0 2024-04-26T21:28:03.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.360+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 >> [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] conn(0x7f02600aeeb0 msgr2=0x7f02600b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:03.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.360+0000 7f0281400700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] conn(0x7f02600aeeb0 0x7f02600b1350 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.360+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f02600a3d50 msgr2=0x7f02600a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.360+0000 7f0281400700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f02600a3d50 0x7f02600a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0268009750 tx=0x7f0268009310 comp rx=0 tx=0).stop 2024-04-26T21:28:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.361+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c075650 msgr2=0x7f027c1a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.361+0000 7f0281400700 1 --2- 172.21.15.1:0/1618275650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c075650 0x7f027c1a0160 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f026c00e9f0 tx=0x7f026c00ed00 comp rx=0 tx=0).stop 2024-04-26T21:28:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.361+0000 7f027b200700 1 -- 172.21.15.1:0/1618275650 reap_dead start 2024-04-26T21:28:03.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.361+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 shutdown_connections 2024-04-26T21:28:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.361+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 >> 172.21.15.1:0/1618275650 conn(0x7f027c07a580 msgr2=0x7f027c113430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.361+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 shutdown_connections 2024-04-26T21:28:03.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:03.361+0000 7f0281400700 1 -- 172.21.15.1:0/1618275650 wait complete. 2024-04-26T21:28:04.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:03 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:04.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:03 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:04.268 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:03 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:28:04.678 INFO:teuthology.orchestra.run.smithi001.stdout:352187318298 2024-04-26T21:28:04.678 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-26T21:28:05.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:04 smithi186 ceph-mon[32991]: pgmap v357: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 383 B/s wr, 5 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T21:28:05.260 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:04 smithi001 ceph-mon[28054]: pgmap v357: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 383 B/s wr, 5 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T21:28:05.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:04 smithi038 ceph-mon[32724]: pgmap v357: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 383 B/s wr, 5 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T21:28:06.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:05 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:05 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:06.328 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:05 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:07.202 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:06 smithi001 ceph-mon[28054]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 323 B/s wr, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T21:28:07.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:06 smithi186 ceph-mon[32991]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 323 B/s wr, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T21:28:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:06 smithi038 ceph-mon[32724]: pgmap v358: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 323 B/s wr, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-04-26T21:28:07.992 INFO:teuthology.orchestra.run.smithi001.stdout:77309411407 2024-04-26T21:28:07.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T21:28:09.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:08 smithi186 ceph-mon[32991]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 292 B/s wr, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T21:28:09.303 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:08 smithi001 ceph-mon[28054]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 292 B/s wr, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T21:28:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:08 smithi038 ceph-mon[32724]: pgmap v359: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 292 B/s wr, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T21:28:09.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.488+0000 7f7759400700 1 -- 172.21.15.1:0/2117565884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7754075f70 msgr2=0x7f77541140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.488+0000 7f7759400700 1 --2- 172.21.15.1:0/2117565884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7754075f70 0x7f77541140b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7740009990 tx=0x7f7740009ca0 comp rx=0 tx=0).stop 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.489+0000 7f7759400700 1 -- 172.21.15.1:0/2117565884 shutdown_connections 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.489+0000 7f7759400700 1 --2- 172.21.15.1:0/2117565884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77541145f0 0x7f77541169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.489+0000 7f7759400700 1 --2- 172.21.15.1:0/2117565884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7754075650 0x7f7754075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.489+0000 7f7759400700 1 --2- 172.21.15.1:0/2117565884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7754075f70 0x7f77541140b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.489+0000 7f7759400700 1 -- 172.21.15.1:0/2117565884 >> 172.21.15.1:0/2117565884 conn(0x7f775407a580 msgr2=0x7f775407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.489+0000 7f7759400700 1 -- 172.21.15.1:0/2117565884 shutdown_connections 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.489+0000 7f7759400700 1 -- 172.21.15.1:0/2117565884 wait complete. 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.490+0000 7f7759400700 1 Processor -- start 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.490+0000 7f7759400700 1 -- start start 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.490+0000 7f7759400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7754075650 0x7f77541a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.490+0000 7f7759400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7754075f70 0x7f77541a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:09.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.490+0000 7f7759400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77541145f0 0x7f77541a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.490+0000 7f7759400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7754119c50 con 0x7f7754075650 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.490+0000 7f7759400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7754119ad0 con 0x7f7754075f70 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.490+0000 7f7759400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7754119dd0 con 0x7f77541145f0 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7751e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7754075f70 0x7f77541a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7751e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7754075f70 0x7f77541a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42764/0 (socket says 172.21.15.1:42764) 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7751e00700 1 -- 172.21.15.1:0/1058660344 learned_addr learned my addr 172.21.15.1:0/1058660344 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7752800700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7754075650 0x7f77541a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7753200700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77541145f0 0x7f77541a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7751e00700 1 -- 172.21.15.1:0/1058660344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77541145f0 msgr2=0x7f77541a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7751e00700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77541145f0 0x7f77541a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7751e00700 1 -- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7754075650 msgr2=0x7f77541a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7751e00700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7754075650 0x7f77541a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7751e00700 1 -- 172.21.15.1:0/1058660344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77541a5170 con 0x7f7754075f70 2024-04-26T21:28:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7753200700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77541145f0 0x7f77541a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.491+0000 7f7751e00700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7754075f70 0x7f77541a06a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f77400057c0 tx=0x7f77400057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.492+0000 7f774b400700 1 -- 172.21.15.1:0/1058660344 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774000fad0 con 0x7f7754075f70 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.492+0000 7f774b400700 1 -- 172.21.15.1:0/1058660344 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f774000fc70 con 0x7f7754075f70 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.492+0000 7f774b400700 1 -- 172.21.15.1:0/1058660344 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7740017650 con 0x7f7754075f70 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.492+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77541a5400 con 0x7f7754075f70 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.492+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77541a5830 con 0x7f7754075f70 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.493+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7754110af0 con 0x7f7754075f70 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.495+0000 7f774b400700 1 -- 172.21.15.1:0/1058660344 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f7740009d10 con 0x7f7754075f70 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.496+0000 7f774b400700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f77380a3d50 0x7f77380a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.497+0000 7f774b400700 1 -- 172.21.15.1:0/1058660344 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f77400d19e0 con 0x7f7754075f70 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.497+0000 7f7752800700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f77380a3d50 0x7f77380a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.497+0000 7f774b400700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] conn(0x7f77380aeeb0 0x7f77380b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:09.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.497+0000 7f774b400700 1 -- 172.21.15.1:0/1058660344 --> [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77380b1a20 con 0x7f77380aeeb0 2024-04-26T21:28:09.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.497+0000 7f774b400700 1 -- 172.21.15.1:0/1058660344 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f77400d1d90 con 0x7f7754075f70 2024-04-26T21:28:09.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.497+0000 7f7753200700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] conn(0x7f77380aeeb0 0x7f77380b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:09.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.498+0000 7f7752800700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f77380a3d50 0x7f77380a6210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7744009c60 tx=0x7f7744009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:09.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.498+0000 7f7753200700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] conn(0x7f77380aeeb0 0x7f77380b1350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:09.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.499+0000 7f774b400700 1 -- 172.21.15.1:0/1058660344 <== osd.3 v2:172.21.15.1:6810/851547730 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77380b1a20 con 0x7f77380aeeb0 2024-04-26T21:28:09.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.518+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 --> [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7754076d40 con 0x7f77380aeeb0 2024-04-26T21:28:09.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.518+0000 7f774b400700 1 -- 172.21.15.1:0/1058660344 <== osd.3 v2:172.21.15.1:6810/851547730 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7754076d40 con 0x7f77380aeeb0 2024-04-26T21:28:09.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.519+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] conn(0x7f77380aeeb0 msgr2=0x7f77380b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:09.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.519+0000 7f7759400700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] conn(0x7f77380aeeb0 0x7f77380b1350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:09.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.519+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f77380a3d50 msgr2=0x7f77380a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:09.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.519+0000 7f7759400700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f77380a3d50 0x7f77380a6210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7744009c60 tx=0x7f7744009260 comp rx=0 tx=0).stop 2024-04-26T21:28:09.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.519+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7754075f70 msgr2=0x7f77541a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:09.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.519+0000 7f7759400700 1 --2- 172.21.15.1:0/1058660344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7754075f70 0x7f77541a06a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f77400057c0 tx=0x7f77400057f0 comp rx=0 tx=0).stop 2024-04-26T21:28:09.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.519+0000 7f7753200700 1 -- 172.21.15.1:0/1058660344 reap_dead start 2024-04-26T21:28:09.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.519+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 shutdown_connections 2024-04-26T21:28:09.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.519+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 >> 172.21.15.1:0/1058660344 conn(0x7f775407a580 msgr2=0x7f7754110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:09.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.520+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 shutdown_connections 2024-04-26T21:28:09.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:09.520+0000 7f7759400700 1 -- 172.21.15.1:0/1058660344 wait complete. 2024-04-26T21:28:10.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.843+0000 7fb0d3a00700 1 -- 172.21.15.1:0/3547392849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc10b660 msgr2=0x7fb0cc10ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:10.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.843+0000 7fb0d3a00700 1 --2- 172.21.15.1:0/3547392849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc10b660 0x7fb0cc10ba40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00098f0 tx=0x7fb0c0009c00 comp rx=0 tx=0).stop 2024-04-26T21:28:10.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.843+0000 7fb0d3a00700 1 -- 172.21.15.1:0/3547392849 shutdown_connections 2024-04-26T21:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.843+0000 7fb0d3a00700 1 --2- 172.21.15.1:0/3547392849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc10b660 0x7fb0cc10ba40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.843+0000 7fb0d3a00700 1 --2- 172.21.15.1:0/3547392849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0cc106020 0x7fb0cc112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.843+0000 7fb0d3a00700 1 --2- 172.21.15.1:0/3547392849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0cc105660 0x7fb0cc105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.843+0000 7fb0d3a00700 1 -- 172.21.15.1:0/3547392849 >> 172.21.15.1:0/3547392849 conn(0x7fb0cc101150 msgr2=0x7fb0cc103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.843+0000 7fb0d3a00700 1 -- 172.21.15.1:0/3547392849 shutdown_connections 2024-04-26T21:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.844+0000 7fb0d3a00700 1 -- 172.21.15.1:0/3547392849 wait complete. 2024-04-26T21:28:10.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.845+0000 7fb0d3a00700 1 Processor -- start 2024-04-26T21:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.845+0000 7fb0d3a00700 1 -- start start 2024-04-26T21:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0d3a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0cc105660 0x7fb0cc19bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0d3a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0cc106020 0x7fb0cc19c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0d3a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc10b660 0x7fb0cc1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0d3a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc115890 con 0x7fb0cc105660 2024-04-26T21:28:10.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0d3a00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc115710 con 0x7fb0cc106020 2024-04-26T21:28:10.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0d3a00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0cc115a10 con 0x7fb0cc10b660 2024-04-26T21:28:10.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0d1000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0cc105660 0x7fb0cc19bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:10.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0cbe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0cc106020 0x7fb0cc19c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0cbe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0cc106020 0x7fb0cc19c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42778/0 (socket says 172.21.15.1:42778) 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0d1000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0cc105660 0x7fb0cc19bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43338/0 (socket says 172.21.15.1:43338) 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0cbe00700 1 -- 172.21.15.1:0/2763644004 learned_addr learned my addr 172.21.15.1:0/2763644004 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.846+0000 7fb0d1a00700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc10b660 0x7fb0cc1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0cbe00700 1 -- 172.21.15.1:0/2763644004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc10b660 msgr2=0x7fb0cc1a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0cbe00700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc10b660 0x7fb0cc1a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0cbe00700 1 -- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0cc105660 msgr2=0x7fb0cc19bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0cbe00700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0cc105660 0x7fb0cc19bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0cbe00700 1 -- 172.21.15.1:0/2763644004 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0cc1a0db0 con 0x7fb0cc106020 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0d1a00700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0cc10b660 0x7fb0cc1a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0d1000700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0cc105660 0x7fb0cc19bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:10.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0cbe00700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0cc106020 0x7fb0cc19c2e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800eb30 tx=0x7fb0b800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:10.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0c9600700 1 -- 172.21.15.1:0/2763644004 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b800cbd0 con 0x7fb0cc106020 2024-04-26T21:28:10.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0c9600700 1 -- 172.21.15.1:0/2763644004 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0b80044d0 con 0x7fb0cc106020 2024-04-26T21:28:10.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0cc1a10a0 con 0x7fb0cc106020 2024-04-26T21:28:10.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.847+0000 7fb0c9600700 1 -- 172.21.15.1:0/2763644004 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8003e30 con 0x7fb0cc106020 2024-04-26T21:28:10.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.848+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0cc1a8920 con 0x7fb0cc106020 2024-04-26T21:28:10.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.848+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb0cc063bb0 con 0x7fb0cc106020 2024-04-26T21:28:10.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.851+0000 7fb0c9600700 1 -- 172.21.15.1:0/2763644004 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fb0b8004000 con 0x7fb0cc106020 2024-04-26T21:28:10.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.852+0000 7fb0c9600700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb0b40a3cd0 0x7fb0b40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:10.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.852+0000 7fb0c9600700 1 -- 172.21.15.1:0/2763644004 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb0b80d1120 con 0x7fb0cc106020 2024-04-26T21:28:10.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.852+0000 7fb0d1000700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb0b40a3cd0 0x7fb0b40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:10.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.853+0000 7fb0c9600700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] conn(0x7fb0b40aee30 0x7fb0b40b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:10.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.853+0000 7fb0c9600700 1 -- 172.21.15.1:0/2763644004 --> [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0b8021e50 con 0x7fb0b40aee30 2024-04-26T21:28:10.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.853+0000 7fb0c9600700 1 -- 172.21.15.1:0/2763644004 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fb0b80d14d0 con 0x7fb0cc106020 2024-04-26T21:28:10.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.853+0000 7fb0d1a00700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] conn(0x7fb0b40aee30 0x7fb0b40b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:10.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.853+0000 7fb0d1000700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb0b40a3cd0 0x7fb0b40a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00098c0 tx=0x7fb0c0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:10.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.854+0000 7fb0d1a00700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] conn(0x7fb0b40aee30 0x7fb0b40b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:10.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.855+0000 7fb0c9600700 1 -- 172.21.15.1:0/2763644004 <== osd.0 v2:172.21.15.1:6802/3716675555 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb0b8021e50 con 0x7fb0b40aee30 2024-04-26T21:28:10.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.873+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 --> [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb0cc064030 con 0x7fb0b40aee30 2024-04-26T21:28:10.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.874+0000 7fb0c9600700 1 -- 172.21.15.1:0/2763644004 <== osd.0 v2:172.21.15.1:6802/3716675555 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb0cc064030 con 0x7fb0b40aee30 2024-04-26T21:28:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.874+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] conn(0x7fb0b40aee30 msgr2=0x7fb0b40b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:10.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.874+0000 7fb0d3a00700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] conn(0x7fb0b40aee30 0x7fb0b40b12d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.874+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb0b40a3cd0 msgr2=0x7fb0b40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.875+0000 7fb0d3a00700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb0b40a3cd0 0x7fb0b40a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00098c0 tx=0x7fb0c0019040 comp rx=0 tx=0).stop 2024-04-26T21:28:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.875+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0cc106020 msgr2=0x7fb0cc19c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.875+0000 7fb0d3a00700 1 --2- 172.21.15.1:0/2763644004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0cc106020 0x7fb0cc19c2e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800eb30 tx=0x7fb0b800ee40 comp rx=0 tx=0).stop 2024-04-26T21:28:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.875+0000 7fb0d1a00700 1 -- 172.21.15.1:0/2763644004 reap_dead start 2024-04-26T21:28:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.875+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 shutdown_connections 2024-04-26T21:28:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.875+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 >> 172.21.15.1:0/2763644004 conn(0x7fb0cc101150 msgr2=0x7fb0cc101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.875+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 shutdown_connections 2024-04-26T21:28:10.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:10.875+0000 7fb0d3a00700 1 -- 172.21.15.1:0/2763644004 wait complete. 2024-04-26T21:28:11.002 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:10 smithi001 ceph-mon[28054]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T21:28:11.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:10 smithi186 ceph-mon[32991]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T21:28:11.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:10 smithi038 ceph-mon[32724]: pgmap v360: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T21:28:12.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.109+0000 7f3278e00700 1 -- 172.21.15.1:0/3718755956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274103e30 msgr2=0x7f32741102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:12.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.109+0000 7f3278e00700 1 --2- 172.21.15.1:0/3718755956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274103e30 0x7f32741102c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f326c009990 tx=0x7f326c009ca0 comp rx=0 tx=0).stop 2024-04-26T21:28:12.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.109+0000 7f3278e00700 1 -- 172.21.15.1:0/3718755956 shutdown_connections 2024-04-26T21:28:12.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.109+0000 7f3278e00700 1 --2- 172.21.15.1:0/3718755956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274103e30 0x7f32741102c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:12.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.109+0000 7f3278e00700 1 --2- 172.21.15.1:0/3718755956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32741094f0 0x7f32741098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:12.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.109+0000 7f3278e00700 1 --2- 172.21.15.1:0/3718755956 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3274103470 0x7f32741038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:12.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.109+0000 7f3278e00700 1 -- 172.21.15.1:0/3718755956 >> 172.21.15.1:0/3718755956 conn(0x7f327407a5d0 msgr2=0x7f327407a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:12.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.109+0000 7f3278e00700 1 -- 172.21.15.1:0/3718755956 shutdown_connections 2024-04-26T21:28:12.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.109+0000 7f3278e00700 1 -- 172.21.15.1:0/3718755956 wait complete. 2024-04-26T21:28:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.111+0000 7f3278e00700 1 Processor -- start 2024-04-26T21:28:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.111+0000 7f3278e00700 1 -- start start 2024-04-26T21:28:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.111+0000 7f3278e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3274103470 0x7f32741a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:12.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.111+0000 7f3278e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274103e30 0x7f32741a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:12.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.111+0000 7f3278e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32741094f0 0x7f327419bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:12.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.111+0000 7f3278e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3274113640 con 0x7f32741094f0 2024-04-26T21:28:12.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.111+0000 7f3278e00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f32741134c0 con 0x7f3274103470 2024-04-26T21:28:12.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.111+0000 7f3278e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f32741137c0 con 0x7f3274103e30 2024-04-26T21:28:12.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f326be00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274103e30 0x7f32741a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:12.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f3271e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3274103470 0x7f32741a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:12.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f3271e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3274103470 0x7f32741a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42788/0 (socket says 172.21.15.1:42788) 2024-04-26T21:28:12.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f3271e00700 1 -- 172.21.15.1:0/3066028354 learned_addr learned my addr 172.21.15.1:0/3066028354 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:12.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f3272800700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32741094f0 0x7f327419bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:12.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f3271e00700 1 -- 172.21.15.1:0/3066028354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274103e30 msgr2=0x7f32741a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:12.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f3271e00700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274103e30 0x7f32741a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:12.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f3271e00700 1 -- 172.21.15.1:0/3066028354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32741094f0 msgr2=0x7f327419bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:12.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f3271e00700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32741094f0 0x7f327419bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:12.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f3271e00700 1 -- 172.21.15.1:0/3066028354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327419c230 con 0x7f3274103470 2024-04-26T21:28:12.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.112+0000 7f3271e00700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3274103470 0x7f32741a1910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f32640060b0 tx=0x7f326400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:12.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.113+0000 7f326aa00700 1 -- 172.21.15.1:0/3066028354 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326401a070 con 0x7f3274103470 2024-04-26T21:28:12.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.113+0000 7f326aa00700 1 -- 172.21.15.1:0/3066028354 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f326400dd20 con 0x7f3274103470 2024-04-26T21:28:12.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.113+0000 7f326aa00700 1 -- 172.21.15.1:0/3066028354 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3264014600 con 0x7f3274103470 2024-04-26T21:28:12.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.113+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f327419c520 con 0x7f3274103470 2024-04-26T21:28:12.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.113+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32741a87e0 con 0x7f3274103470 2024-04-26T21:28:12.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.114+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f32741102d0 con 0x7f3274103470 2024-04-26T21:28:12.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.118+0000 7f326aa00700 1 -- 172.21.15.1:0/3066028354 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f326401d070 con 0x7f3274103470 2024-04-26T21:28:12.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.119+0000 7f326aa00700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f32580a3d50 0x7f32580a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:12.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.119+0000 7f326aa00700 1 -- 172.21.15.1:0/3066028354 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f32640d0860 con 0x7f3274103470 2024-04-26T21:28:12.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.119+0000 7f326be00700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f32580a3d50 0x7f32580a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:12.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.120+0000 7f326aa00700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] conn(0x7f32580aeeb0 0x7f32580b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:12.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.120+0000 7f326aa00700 1 -- 172.21.15.1:0/3066028354 --> [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3264013ec0 con 0x7f32580aeeb0 2024-04-26T21:28:12.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.120+0000 7f326aa00700 1 -- 172.21.15.1:0/3066028354 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f32640d0c10 con 0x7f3274103470 2024-04-26T21:28:12.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.120+0000 7f3272800700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] conn(0x7f32580aeeb0 0x7f32580b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:12.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.120+0000 7f326be00700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f32580a3d50 0x7f32580a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f327407aed0 tx=0x7f325c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:12.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.121+0000 7f3272800700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] conn(0x7f32580aeeb0 0x7f32580b1350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:12.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.122+0000 7f326aa00700 1 -- 172.21.15.1:0/3066028354 <== osd.7 v2:172.21.15.38:6816/826797783 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3264013ec0 con 0x7f32580aeeb0 2024-04-26T21:28:12.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.140+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 --> [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3274104b90 con 0x7f32580aeeb0 2024-04-26T21:28:12.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.141+0000 7f326aa00700 1 -- 172.21.15.1:0/3066028354 <== osd.7 v2:172.21.15.38:6816/826797783 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3274104b90 con 0x7f32580aeeb0 2024-04-26T21:28:12.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.141+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 >> [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] conn(0x7f32580aeeb0 msgr2=0x7f32580b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:12.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.141+0000 7f3278e00700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] conn(0x7f32580aeeb0 0x7f32580b1350 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:12.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.142+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f32580a3d50 msgr2=0x7f32580a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:12.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.142+0000 7f3278e00700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f32580a3d50 0x7f32580a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f327407aed0 tx=0x7f325c009260 comp rx=0 tx=0).stop 2024-04-26T21:28:12.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.142+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3274103470 msgr2=0x7f32741a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:12.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.142+0000 7f3278e00700 1 --2- 172.21.15.1:0/3066028354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3274103470 0x7f32741a1910 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f32640060b0 tx=0x7f326400d3b0 comp rx=0 tx=0).stop 2024-04-26T21:28:12.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.142+0000 7f3272800700 1 -- 172.21.15.1:0/3066028354 reap_dead start 2024-04-26T21:28:12.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.142+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 shutdown_connections 2024-04-26T21:28:12.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.142+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 >> 172.21.15.1:0/3066028354 conn(0x7f327407a5d0 msgr2=0x7f327410e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:12.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.142+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 shutdown_connections 2024-04-26T21:28:12.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:12.142+0000 7f3278e00700 1 -- 172.21.15.1:0/3066028354 wait complete. 2024-04-26T21:28:13.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:12 smithi001 ceph-mon[28054]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T21:28:13.138 INFO:teuthology.orchestra.run.smithi001.stdout:111669149770 2024-04-26T21:28:13.138 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T21:28:13.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:12 smithi186 ceph-mon[32991]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T21:28:13.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:12 smithi038 ceph-mon[32724]: pgmap v361: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-04-26T21:28:14.102 INFO:teuthology.orchestra.run.smithi001.stdout:34359738459 2024-04-26T21:28:14.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T21:28:14.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.211+0000 7f5eb2600700 1 -- 172.21.15.1:0/2493279142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac10ff60 msgr2=0x7f5eac112350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:14.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.211+0000 7f5eb2600700 1 --2- 172.21.15.1:0/2493279142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac10ff60 0x7f5eac112350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400b0a0 tx=0x7f5ea400b3b0 comp rx=0 tx=0).stop 2024-04-26T21:28:14.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.211+0000 7f5eb2600700 1 -- 172.21.15.1:0/2493279142 shutdown_connections 2024-04-26T21:28:14.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.211+0000 7f5eb2600700 1 --2- 172.21.15.1:0/2493279142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5eac06bb30 0x7f5eac06bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:14.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.211+0000 7f5eb2600700 1 --2- 172.21.15.1:0/2493279142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac10ff60 0x7f5eac112350 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:14.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.211+0000 7f5eb2600700 1 --2- 172.21.15.1:0/2493279142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5eac06c450 0x7f5eac10fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:14.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.211+0000 7f5eb2600700 1 -- 172.21.15.1:0/2493279142 >> 172.21.15.1:0/2493279142 conn(0x7f5eac078370 msgr2=0x7f5eac078780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:14.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.211+0000 7f5eb2600700 1 -- 172.21.15.1:0/2493279142 shutdown_connections 2024-04-26T21:28:14.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.211+0000 7f5eb2600700 1 -- 172.21.15.1:0/2493279142 wait complete. 2024-04-26T21:28:14.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.213+0000 7f5eb2600700 1 Processor -- start 2024-04-26T21:28:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.213+0000 7f5eb2600700 1 -- start start 2024-04-26T21:28:14.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.213+0000 7f5eb2600700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5eac06bb30 0x7f5eac1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.213+0000 7f5eb2600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac06c450 0x7f5eac1a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.213+0000 7f5eb2600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5eac10ff60 0x7f5eac19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.213+0000 7f5eb2600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5eac1158b0 con 0x7f5eac10ff60 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.213+0000 7f5eb2600700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5eac115730 con 0x7f5eac06c450 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.213+0000 7f5eb2600700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5eac115a30 con 0x7f5eac06bb30 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eab400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5eac06bb30 0x7f5eac1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eaaa00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac06c450 0x7f5eac1a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eaaa00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac06c450 0x7f5eac1a1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42806/0 (socket says 172.21.15.1:42806) 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eaaa00700 1 -- 172.21.15.1:0/1888570294 learned_addr learned my addr 172.21.15.1:0/1888570294 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eab400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5eac06bb30 0x7f5eac1a1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37452/0 (socket says 172.21.15.1:37452) 2024-04-26T21:28:14.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eabe00700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5eac10ff60 0x7f5eac19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:14.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eab400700 1 -- 172.21.15.1:0/1888570294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac06c450 msgr2=0x7f5eac1a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:14.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eab400700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5eac06c450 0x7f5eac1a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:14.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eab400700 1 -- 172.21.15.1:0/1888570294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5eac10ff60 msgr2=0x7f5eac19bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:14.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eab400700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5eac10ff60 0x7f5eac19bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.214+0000 7f5eab400700 1 -- 172.21.15.1:0/1888570294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eac19c290 con 0x7f5eac06bb30 2024-04-26T21:28:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.215+0000 7f5eab400700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5eac06bb30 0x7f5eac1a1920 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00e9f0 tx=0x7f5e9c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.215+0000 7f5ea3e00700 1 -- 172.21.15.1:0/1888570294 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c004220 con 0x7f5eac06bb30 2024-04-26T21:28:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.215+0000 7f5ea3e00700 1 -- 172.21.15.1:0/1888570294 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e9c0043c0 con 0x7f5eac06bb30 2024-04-26T21:28:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.215+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eac19c580 con 0x7f5eac06bb30 2024-04-26T21:28:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.215+0000 7f5ea3e00700 1 -- 172.21.15.1:0/1888570294 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c005020 con 0x7f5eac06bb30 2024-04-26T21:28:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.215+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eac1a87f0 con 0x7f5eac06bb30 2024-04-26T21:28:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.216+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5eac063b70 con 0x7f5eac06bb30 2024-04-26T21:28:14.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.219+0000 7f5ea3e00700 1 -- 172.21.15.1:0/1888570294 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f5e9c004750 con 0x7f5eac06bb30 2024-04-26T21:28:14.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.220+0000 7f5ea3e00700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e900a3cd0 0x7f5e900a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:14.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.220+0000 7f5ea3e00700 1 -- 172.21.15.1:0/1888570294 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f5e9c0d08d0 con 0x7f5eac06bb30 2024-04-26T21:28:14.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.220+0000 7f5eaaa00700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e900a3cd0 0x7f5e900a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:14.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.220+0000 7f5ea3e00700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] conn(0x7f5e900aee30 0x7f5e900b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:14.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.220+0000 7f5ea3e00700 1 -- 172.21.15.1:0/1888570294 --> [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e9c00be50 con 0x7f5e900aee30 2024-04-26T21:28:14.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.221+0000 7f5ea3e00700 1 -- 172.21.15.1:0/1888570294 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f5e9c0d52a0 con 0x7f5eac06bb30 2024-04-26T21:28:14.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.221+0000 7f5eabe00700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] conn(0x7f5e900aee30 0x7f5e900b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:14.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.221+0000 7f5eaaa00700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e900a3cd0 0x7f5e900a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5eac101f20 tx=0x7f5e98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:14.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.222+0000 7f5eabe00700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] conn(0x7f5e900aee30 0x7f5e900b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:14.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.223+0000 7f5ea3e00700 1 -- 172.21.15.1:0/1888570294 <== osd.1 v2:172.21.15.38:6800/2314678612 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5e9c00be50 con 0x7f5e900aee30 2024-04-26T21:28:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.241+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 --> [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5eac063ff0 con 0x7f5e900aee30 2024-04-26T21:28:14.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.242+0000 7f5ea3e00700 1 -- 172.21.15.1:0/1888570294 <== osd.1 v2:172.21.15.38:6800/2314678612 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5eac063ff0 con 0x7f5e900aee30 2024-04-26T21:28:14.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.242+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 >> [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] conn(0x7f5e900aee30 msgr2=0x7f5e900b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:14.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.242+0000 7f5eb2600700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] conn(0x7f5e900aee30 0x7f5e900b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:14.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.242+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e900a3cd0 msgr2=0x7f5e900a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:14.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.242+0000 7f5eb2600700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5e900a3cd0 0x7f5e900a6190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5eac101f20 tx=0x7f5e98009260 comp rx=0 tx=0).stop 2024-04-26T21:28:14.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.242+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5eac06bb30 msgr2=0x7f5eac1a1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.242+0000 7f5eb2600700 1 --2- 172.21.15.1:0/1888570294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5eac06bb30 0x7f5eac1a1920 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00e9f0 tx=0x7f5e9c00ed00 comp rx=0 tx=0).stop 2024-04-26T21:28:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.242+0000 7f5eabe00700 1 -- 172.21.15.1:0/1888570294 reap_dead start 2024-04-26T21:28:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.243+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 shutdown_connections 2024-04-26T21:28:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.243+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 >> 172.21.15.1:0/1888570294 conn(0x7f5eac078370 msgr2=0x7f5eac1114c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.243+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 shutdown_connections 2024-04-26T21:28:14.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:14.243+0000 7f5eb2600700 1 -- 172.21.15.1:0/1888570294 wait complete. 2024-04-26T21:28:15.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.125+0000 7f73fba00700 1 -- 172.21.15.1:0/3880897162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4101b80 msgr2=0x7f73f4102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.125+0000 7f73fba00700 1 --2- 172.21.15.1:0/3880897162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4101b80 0x7f73f4102000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f73ec00b0a0 tx=0x7f73ec00b3b0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.125+0000 7f73fba00700 1 -- 172.21.15.1:0/3880897162 shutdown_connections 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.125+0000 7f73fba00700 1 --2- 172.21.15.1:0/3880897162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f4107640 0x7f73f4107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.125+0000 7f73fba00700 1 --2- 172.21.15.1:0/3880897162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f4101b80 0x7f73f4102000 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.125+0000 7f73fba00700 1 --2- 172.21.15.1:0/3880897162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f406a420 0x7f73f4101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.125+0000 7f73fba00700 1 -- 172.21.15.1:0/3880897162 >> 172.21.15.1:0/3880897162 conn(0x7f73f4078520 msgr2=0x7f73f40ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.125+0000 7f73fba00700 1 -- 172.21.15.1:0/3880897162 shutdown_connections 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.125+0000 7f73fba00700 1 -- 172.21.15.1:0/3880897162 wait complete. 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.126+0000 7f73fba00700 1 Processor -- start 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73fba00700 1 -- start start 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73fba00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f406a420 0x7f73f419f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73fba00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f4101b80 0x7f73f419fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:15.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73fba00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f4107640 0x7f73f4199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73fba00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41124f0 con 0x7f73f406a420 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73fba00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f73f4112370 con 0x7f73f4107640 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73fba00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73f4112670 con 0x7f73f4101b80 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73f9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f406a420 0x7f73f419f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73f9000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f406a420 0x7f73f419f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43392/0 (socket says 172.21.15.1:43392) 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73f9000700 1 -- 172.21.15.1:0/1177677464 learned_addr learned my addr 172.21.15.1:0/1177677464 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.127+0000 7f73f3e00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f4101b80 0x7f73f419fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.128+0000 7f73f9a00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f4107640 0x7f73f4199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.128+0000 7f73f9000700 1 -- 172.21.15.1:0/1177677464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f4101b80 msgr2=0x7f73f419fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.128+0000 7f73f9000700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f4101b80 0x7f73f419fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.128+0000 7f73f9000700 1 -- 172.21.15.1:0/1177677464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f4107640 msgr2=0x7f73f4199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.128+0000 7f73f9000700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f4107640 0x7f73f4199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.128+0000 7f73f9000700 1 -- 172.21.15.1:0/1177677464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f419a1e0 con 0x7f73f406a420 2024-04-26T21:28:15.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.128+0000 7f73f3e00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f4101b80 0x7f73f419fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.128+0000 7f73f9a00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f4107640 0x7f73f4199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.128+0000 7f73f9000700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f406a420 0x7f73f419f8c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f73e800e9a0 tx=0x7f73e800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.129+0000 7f73f1600700 1 -- 172.21.15.1:0/1177677464 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e800ca10 con 0x7f73f406a420 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.129+0000 7f73f1600700 1 -- 172.21.15.1:0/1177677464 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73e800cbb0 con 0x7f73f406a420 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.129+0000 7f73f1600700 1 -- 172.21.15.1:0/1177677464 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8018680 con 0x7f73f406a420 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.129+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f419a4d0 con 0x7f73f406a420 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.129+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f41a6790 con 0x7f73f406a420 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.130+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f73f410af70 con 0x7f73f406a420 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.132+0000 7f73f1600700 1 -- 172.21.15.1:0/1177677464 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f73e8010430 con 0x7f73f406a420 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.134+0000 7f73f1600700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73dc0a3cd0 0x7f73dc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.134+0000 7f73f1600700 1 -- 172.21.15.1:0/1177677464 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f73e80d0fb0 con 0x7f73f406a420 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.134+0000 7f73f3e00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73dc0a3cd0 0x7f73dc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.134+0000 7f73f1600700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] conn(0x7f73dc0aee30 0x7f73dc0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:15.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.135+0000 7f73f1600700 1 -- 172.21.15.1:0/1177677464 --> [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73e8019ec0 con 0x7f73dc0aee30 2024-04-26T21:28:15.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.135+0000 7f73f1600700 1 -- 172.21.15.1:0/1177677464 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f73e80d1360 con 0x7f73f406a420 2024-04-26T21:28:15.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.135+0000 7f73f9a00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] conn(0x7f73dc0aee30 0x7f73dc0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:15.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.135+0000 7f73f3e00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73dc0a3cd0 0x7f73dc0a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f73f4106cd0 tx=0x7f73e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:15.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.136+0000 7f73f9a00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] conn(0x7f73dc0aee30 0x7f73dc0b12d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:15.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.137+0000 7f73f1600700 1 -- 172.21.15.1:0/1177677464 <== osd.10 v2:172.21.15.38:6824/4252488524 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73e8019ec0 con 0x7f73dc0aee30 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.155+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 --> [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f73f4102d80 con 0x7f73dc0aee30 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.156+0000 7f73f1600700 1 -- 172.21.15.1:0/1177677464 <== osd.10 v2:172.21.15.38:6824/4252488524 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f73f4102d80 con 0x7f73dc0aee30 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.156+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 >> [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] conn(0x7f73dc0aee30 msgr2=0x7f73dc0b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.156+0000 7f73fba00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] conn(0x7f73dc0aee30 0x7f73dc0b12d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.157+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73dc0a3cd0 msgr2=0x7f73dc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.157+0000 7f73fba00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f73dc0a3cd0 0x7f73dc0a6190 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f73f4106cd0 tx=0x7f73e4009260 comp rx=0 tx=0).stop 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.157+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f406a420 msgr2=0x7f73f419f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.157+0000 7f73fba00700 1 --2- 172.21.15.1:0/1177677464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f73f406a420 0x7f73f419f8c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f73e800e9a0 tx=0x7f73e800ecb0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.157+0000 7f73f9a00700 1 -- 172.21.15.1:0/1177677464 reap_dead start 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.157+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 shutdown_connections 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.157+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 >> 172.21.15.1:0/1177677464 conn(0x7f73f4078520 msgr2=0x7f73f410a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.157+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 shutdown_connections 2024-04-26T21:28:15.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.157+0000 7f73fba00700 1 -- 172.21.15.1:0/1177677464 wait complete. 2024-04-26T21:28:15.175 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:14 smithi001 ceph-mon[28054]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:15.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:14 smithi186 ceph-mon[32991]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:15.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:14 smithi038 ceph-mon[32724]: pgmap v362: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.847+0000 7f4232400700 1 -- 172.21.15.1:0/752372921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f422c1094c0 msgr2=0x7f422c1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.847+0000 7f4232400700 1 --2- 172.21.15.1:0/752372921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f422c1094c0 0x7f422c1098a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f421c00b0a0 tx=0x7f421c00b3b0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.847+0000 7f4232400700 1 -- 172.21.15.1:0/752372921 shutdown_connections 2024-04-26T21:28:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.847+0000 7f4232400700 1 --2- 172.21.15.1:0/752372921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f422c103e30 0x7f422c110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.847+0000 7f4232400700 1 --2- 172.21.15.1:0/752372921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f422c1094c0 0x7f422c1098a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.847+0000 7f4232400700 1 --2- 172.21.15.1:0/752372921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f422c103470 0x7f422c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.847+0000 7f4232400700 1 -- 172.21.15.1:0/752372921 >> 172.21.15.1:0/752372921 conn(0x7f422c07a580 msgr2=0x7f422c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.847+0000 7f4232400700 1 -- 172.21.15.1:0/752372921 shutdown_connections 2024-04-26T21:28:15.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.847+0000 7f4232400700 1 -- 172.21.15.1:0/752372921 wait complete. 2024-04-26T21:28:15.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.848+0000 7f4232400700 1 Processor -- start 2024-04-26T21:28:15.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.848+0000 7f4232400700 1 -- start start 2024-04-26T21:28:15.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f4232400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f422c103470 0x7f422c075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:15.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f4232400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f422c103e30 0x7f422c075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:15.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f4232400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f422c1094c0 0x7f422c0761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:15.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f4232400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f422c113610 con 0x7f422c1094c0 2024-04-26T21:28:15.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f4232400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f422c113490 con 0x7f422c103e30 2024-04-26T21:28:15.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f4232400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f422c113790 con 0x7f422c103470 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f422b400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f422c103470 0x7f422c075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f422b400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f422c103470 0x7f422c075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37496/0 (socket says 172.21.15.1:37496) 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f422b400700 1 -- 172.21.15.1:0/1360185387 learned_addr learned my addr 172.21.15.1:0/1360185387 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f4222a00700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f422c103e30 0x7f422c075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f422be00700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f422c1094c0 0x7f422c0761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f422b400700 1 -- 172.21.15.1:0/1360185387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f422c103e30 msgr2=0x7f422c075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f422b400700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f422c103e30 0x7f422c075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f422b400700 1 -- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f422c1094c0 msgr2=0x7f422c0761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.849+0000 7f422b400700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f422c1094c0 0x7f422c0761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.850+0000 7f422b400700 1 -- 172.21.15.1:0/1360185387 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422c1a80b0 con 0x7f422c103470 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.850+0000 7f422b400700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f422c103470 0x7f422c075720 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f421c00d010 tx=0x7f421c004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:15.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.850+0000 7f4228c00700 1 -- 172.21.15.1:0/1360185387 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c021070 con 0x7f422c103470 2024-04-26T21:28:15.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.850+0000 7f4228c00700 1 -- 172.21.15.1:0/1360185387 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f421c007c40 con 0x7f422c103470 2024-04-26T21:28:15.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.850+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f422c1a8340 con 0x7f422c103470 2024-04-26T21:28:15.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.850+0000 7f4228c00700 1 -- 172.21.15.1:0/1360185387 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c004710 con 0x7f422c103470 2024-04-26T21:28:15.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.850+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f422c1a87f0 con 0x7f422c103470 2024-04-26T21:28:15.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.851+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f422c063b30 con 0x7f422c103470 2024-04-26T21:28:15.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.854+0000 7f4228c00700 1 -- 172.21.15.1:0/1360185387 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f421c003a00 con 0x7f422c103470 2024-04-26T21:28:15.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.855+0000 7f4228c00700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42100a3cd0 0x7f42100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:15.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.855+0000 7f4228c00700 1 -- 172.21.15.1:0/1360185387 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f421c00f020 con 0x7f422c103470 2024-04-26T21:28:15.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.855+0000 7f4222a00700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42100a3cd0 0x7f42100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:15.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.856+0000 7f4228c00700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] conn(0x7f42100aed00 0x7f42100b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:15.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.856+0000 7f4228c00700 1 -- 172.21.15.1:0/1360185387 --> [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f421c00ce60 con 0x7f42100aed00 2024-04-26T21:28:15.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.856+0000 7f4228c00700 1 -- 172.21.15.1:0/1360185387 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f421c0d23d0 con 0x7f422c103470 2024-04-26T21:28:15.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.856+0000 7f422be00700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] conn(0x7f42100aed00 0x7f42100b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:15.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.856+0000 7f4222a00700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42100a3cd0 0x7f42100a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f422c07ac60 tx=0x7f4214009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:15.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.857+0000 7f422be00700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] conn(0x7f42100aed00 0x7f42100b11a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:15.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.858+0000 7f4228c00700 1 -- 172.21.15.1:0/1360185387 <== osd.9 v2:172.21.15.1:6826/4290308342 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f421c00ce60 con 0x7f42100aed00 2024-04-26T21:28:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.876+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 --> [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f422c063fb0 con 0x7f42100aed00 2024-04-26T21:28:15.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.877+0000 7f4228c00700 1 -- 172.21.15.1:0/1360185387 <== osd.9 v2:172.21.15.1:6826/4290308342 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f422c063fb0 con 0x7f42100aed00 2024-04-26T21:28:15.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.877+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] conn(0x7f42100aed00 msgr2=0x7f42100b11a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.877+0000 7f4232400700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] conn(0x7f42100aed00 0x7f42100b11a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.877+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42100a3cd0 msgr2=0x7f42100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.877+0000 7f4232400700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f42100a3cd0 0x7f42100a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f422c07ac60 tx=0x7f4214009260 comp rx=0 tx=0).stop 2024-04-26T21:28:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.877+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f422c103470 msgr2=0x7f422c075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.877+0000 7f4232400700 1 --2- 172.21.15.1:0/1360185387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f422c103470 0x7f422c075720 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f421c00d010 tx=0x7f421c004d00 comp rx=0 tx=0).stop 2024-04-26T21:28:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.877+0000 7f422be00700 1 -- 172.21.15.1:0/1360185387 reap_dead start 2024-04-26T21:28:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.878+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 shutdown_connections 2024-04-26T21:28:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.878+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 >> 172.21.15.1:0/1360185387 conn(0x7f422c07a580 msgr2=0x7f422c1022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.878+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 shutdown_connections 2024-04-26T21:28:15.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:15.878+0000 7f4232400700 1 -- 172.21.15.1:0/1360185387 wait complete. 2024-04-26T21:28:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.382+0000 7f153e800700 1 -- 172.21.15.1:0/3344909615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15381094c0 msgr2=0x7f15381098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.382+0000 7f153e800700 1 --2- 172.21.15.1:0/3344909615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15381094c0 0x7f15381098a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1528009990 tx=0x7f1528009ca0 comp rx=0 tx=0).stop 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.382+0000 7f153e800700 1 -- 172.21.15.1:0/3344909615 shutdown_connections 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.382+0000 7f153e800700 1 --2- 172.21.15.1:0/3344909615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1538103e30 0x7f1538110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.382+0000 7f153e800700 1 --2- 172.21.15.1:0/3344909615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15381094c0 0x7f15381098a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.382+0000 7f153e800700 1 --2- 172.21.15.1:0/3344909615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1538103470 0x7f15381038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.382+0000 7f153e800700 1 -- 172.21.15.1:0/3344909615 >> 172.21.15.1:0/3344909615 conn(0x7f153807a580 msgr2=0x7f153807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.382+0000 7f153e800700 1 -- 172.21.15.1:0/3344909615 shutdown_connections 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.382+0000 7f153e800700 1 -- 172.21.15.1:0/3344909615 wait complete. 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.383+0000 7f153e800700 1 Processor -- start 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f153e800700 1 -- start start 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f153e800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1538103470 0x7f153819bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f153e800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1538103e30 0x7f153819c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f153e800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15381094c0 0x7f15381a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f153e800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1538113610 con 0x7f1538103470 2024-04-26T21:28:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f153e800700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1538113490 con 0x7f15381094c0 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f153e800700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1538113790 con 0x7f1538103e30 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f1537400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1538103470 0x7f153819bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f1537400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1538103470 0x7f153819bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43424/0 (socket says 172.21.15.1:43424) 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f1537400700 1 -- 172.21.15.1:0/1535052762 learned_addr learned my addr 172.21.15.1:0/1535052762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f1536a00700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1538103e30 0x7f153819c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.384+0000 7f1537e00700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15381094c0 0x7f15381a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.385+0000 7f1537400700 1 -- 172.21.15.1:0/1535052762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1538103e30 msgr2=0x7f153819c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.385+0000 7f1537400700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1538103e30 0x7f153819c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.385+0000 7f1537400700 1 -- 172.21.15.1:0/1535052762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15381094c0 msgr2=0x7f15381a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.385+0000 7f1537400700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15381094c0 0x7f15381a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.385+0000 7f1537400700 1 -- 172.21.15.1:0/1535052762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15381a0d70 con 0x7f1538103470 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.385+0000 7f1537e00700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15381094c0 0x7f15381a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.385+0000 7f1537400700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1538103470 0x7f153819bd60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1528000c00 tx=0x7f15280057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:16.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.385+0000 7f152fe00700 1 -- 172.21.15.1:0/1535052762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1528027070 con 0x7f1538103470 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.386+0000 7f152fe00700 1 -- 172.21.15.1:0/1535052762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1528022470 con 0x7f1538103470 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.386+0000 7f152fe00700 1 -- 172.21.15.1:0/1535052762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1528017650 con 0x7f1538103470 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.386+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15381a1000 con 0x7f1538103470 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.386+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15381a88e0 con 0x7f1538103470 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.387+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1538110350 con 0x7f1538103470 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.389+0000 7f152fe00700 1 -- 172.21.15.1:0/1535052762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f152802a070 con 0x7f1538103470 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.390+0000 7f152fe00700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f151c0a3cd0 0x7f151c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.391+0000 7f152fe00700 1 -- 172.21.15.1:0/1535052762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f15280d1cf0 con 0x7f1538103470 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.391+0000 7f1536a00700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f151c0a3cd0 0x7f151c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.391+0000 7f152fe00700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] conn(0x7f151c0aee30 0x7f151c0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.391+0000 7f152fe00700 1 -- 172.21.15.1:0/1535052762 --> [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1528009030 con 0x7f151c0aee30 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.391+0000 7f152fe00700 1 -- 172.21.15.1:0/1535052762 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f15280d20a0 con 0x7f1538103470 2024-04-26T21:28:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.391+0000 7f1537e00700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] conn(0x7f151c0aee30 0x7f151c0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:16.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.391+0000 7f1536a00700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f151c0a3cd0 0x7f151c0a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f153807af00 tx=0x7f15240073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:16.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.392+0000 7f1537e00700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] conn(0x7f151c0aee30 0x7f151c0b12d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:16.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.393+0000 7f152fe00700 1 -- 172.21.15.1:0/1535052762 <== osd.8 v2:172.21.15.186:6816/758273463 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1528009030 con 0x7f151c0aee30 2024-04-26T21:28:16.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.411+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 --> [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1538104b60 con 0x7f151c0aee30 2024-04-26T21:28:16.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.412+0000 7f152fe00700 1 -- 172.21.15.1:0/1535052762 <== osd.8 v2:172.21.15.186:6816/758273463 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1538104b60 con 0x7f151c0aee30 2024-04-26T21:28:16.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.413+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 >> [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] conn(0x7f151c0aee30 msgr2=0x7f151c0b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:16.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.413+0000 7f153e800700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] conn(0x7f151c0aee30 0x7f151c0b12d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:16.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.413+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f151c0a3cd0 msgr2=0x7f151c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:16.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.413+0000 7f153e800700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f151c0a3cd0 0x7f151c0a6190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f153807af00 tx=0x7f15240073d0 comp rx=0 tx=0).stop 2024-04-26T21:28:16.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.413+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1538103470 msgr2=0x7f153819bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:16.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.413+0000 7f153e800700 1 --2- 172.21.15.1:0/1535052762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1538103470 0x7f153819bd60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1528000c00 tx=0x7f15280057c0 comp rx=0 tx=0).stop 2024-04-26T21:28:16.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.413+0000 7f1537e00700 1 -- 172.21.15.1:0/1535052762 reap_dead start 2024-04-26T21:28:16.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.413+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 shutdown_connections 2024-04-26T21:28:16.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.413+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 >> 172.21.15.1:0/1535052762 conn(0x7f153807a580 msgr2=0x7f153810e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:16.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.414+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 shutdown_connections 2024-04-26T21:28:16.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:16.414+0000 7f153e800700 1 -- 172.21.15.1:0/1535052762 wait complete. 2024-04-26T21:28:17.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:16 smithi186 ceph-mon[32991]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T21:28:17.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:16 smithi001 ceph-mon[28054]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T21:28:17.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:16 smithi038 ceph-mon[32724]: pgmap v363: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-04-26T21:28:18.646 INFO:teuthology.orchestra.run.smithi001.stdout:227633266739 2024-04-26T21:28:18.646 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.029+0000 7f2263200700 1 -- 172.21.15.1:0/2100532413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2264103f50 msgr2=0x7f2264107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.029+0000 7f2263200700 1 --2- 172.21.15.1:0/2100532413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2264103f50 0x7f2264107fa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f22500099c0 tx=0x7f2250009cd0 comp rx=0 tx=0).stop 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.029+0000 7f2263200700 1 -- 172.21.15.1:0/2100532413 shutdown_connections 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.029+0000 7f2263200700 1 --2- 172.21.15.1:0/2100532413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22641035a0 0x7f2264103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.029+0000 7f2263200700 1 --2- 172.21.15.1:0/2100532413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2264108680 0x7f2264110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.029+0000 7f2263200700 1 --2- 172.21.15.1:0/2100532413 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2264103f50 0x7f2264107fa0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.029+0000 7f2263200700 1 -- 172.21.15.1:0/2100532413 >> 172.21.15.1:0/2100532413 conn(0x7f22640fee30 msgr2=0x7f2264101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.029+0000 7f2263200700 1 -- 172.21.15.1:0/2100532413 shutdown_connections 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.029+0000 7f2263200700 1 -- 172.21.15.1:0/2100532413 wait complete. 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.030+0000 7f2263200700 1 Processor -- start 2024-04-26T21:28:18.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2263200700 1 -- start start 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2263200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22641035a0 0x7f2264075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2263200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2264103f50 0x7f2264075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2263200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2264108680 0x7f22640761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2263200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2264113560 con 0x7f2264103f50 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2263200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22641133e0 con 0x7f2264108680 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2263200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22641136e0 con 0x7f22641035a0 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2262800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2264108680 0x7f22640761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2261400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2264103f50 0x7f2264075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2262800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2264108680 0x7f22640761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42890/0 (socket says 172.21.15.1:42890) 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2261e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22641035a0 0x7f2264075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:18.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.031+0000 7f2262800700 1 -- 172.21.15.1:0/273806508 learned_addr learned my addr 172.21.15.1:0/273806508 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f2262800700 1 -- 172.21.15.1:0/273806508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22641035a0 msgr2=0x7f2264075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f2262800700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22641035a0 0x7f2264075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f2262800700 1 -- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2264103f50 msgr2=0x7f2264075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f2262800700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2264103f50 0x7f2264075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f2262800700 1 -- 172.21.15.1:0/273806508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22641a7f90 con 0x7f2264108680 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f2261e00700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22641035a0 0x7f2264075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f2262800700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2264108680 0x7f22640761a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f225c00ef30 tx=0x7f225c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f225aa00700 1 -- 172.21.15.1:0/273806508 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225c0092e0 con 0x7f2264108680 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22641a8220 con 0x7f2264108680 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f225aa00700 1 -- 172.21.15.1:0/273806508 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f225c004bb0 con 0x7f2264108680 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.032+0000 7f225aa00700 1 -- 172.21.15.1:0/273806508 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225c004020 con 0x7f2264108680 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.033+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22641a8730 con 0x7f2264108680 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.033+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2264063b30 con 0x7f2264108680 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.035+0000 7f225aa00700 1 -- 172.21.15.1:0/273806508 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f225c01b030 con 0x7f2264108680 2024-04-26T21:28:18.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.037+0000 7f225aa00700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22480a3cd0 0x7f22480a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.037+0000 7f225aa00700 1 -- 172.21.15.1:0/273806508 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f225c0d0d10 con 0x7f2264108680 2024-04-26T21:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.037+0000 7f2261e00700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22480a3cd0 0x7f22480a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.037+0000 7f225aa00700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] conn(0x7f22480aee30 0x7f22480b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.037+0000 7f225aa00700 1 -- 172.21.15.1:0/273806508 --> [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22480b19a0 con 0x7f22480aee30 2024-04-26T21:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.037+0000 7f225aa00700 1 -- 172.21.15.1:0/273806508 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f225c0d10c0 con 0x7f2264108680 2024-04-26T21:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.037+0000 7f2261400700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] conn(0x7f22480aee30 0x7f22480b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.038+0000 7f2261e00700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22480a3cd0 0x7f22480a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2254011490 tx=0x7f2254009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.038+0000 7f2261400700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] conn(0x7f22480aee30 0x7f22480b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:18.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.039+0000 7f225aa00700 1 -- 172.21.15.1:0/273806508 <== osd.6 v2:172.21.15.1:6818/1579861250 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22480b19a0 con 0x7f22480aee30 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.057+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 --> [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2264063fb0 con 0x7f22480aee30 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.058+0000 7f225aa00700 1 -- 172.21.15.1:0/273806508 <== osd.6 v2:172.21.15.1:6818/1579861250 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2264063fb0 con 0x7f22480aee30 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] conn(0x7f22480aee30 msgr2=0x7f22480b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2263200700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] conn(0x7f22480aee30 0x7f22480b12d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22480a3cd0 msgr2=0x7f22480a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2263200700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22480a3cd0 0x7f22480a6190 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2254011490 tx=0x7f2254009260 comp rx=0 tx=0).stop 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2264108680 msgr2=0x7f22640761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2263200700 1 --2- 172.21.15.1:0/273806508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2264108680 0x7f22640761a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f225c00ef30 tx=0x7f225c00c3a0 comp rx=0 tx=0).stop 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2262800700 1 -- 172.21.15.1:0/273806508 reap_dead start 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 shutdown_connections 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 >> 172.21.15.1:0/273806508 conn(0x7f22640fee30 msgr2=0x7f2264101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 shutdown_connections 2024-04-26T21:28:18.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:17.059+0000 7f2263200700 1 -- 172.21.15.1:0/273806508 wait complete. 2024-04-26T21:28:19.195 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:18 smithi001 ceph-mon[28054]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T21:28:19.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:18 smithi186 ceph-mon[32991]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T21:28:19.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:18 smithi038 ceph-mon[32724]: pgmap v364: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-26T21:28:20.071 INFO:teuthology.orchestra.run.smithi001.stdout:317827579938 2024-04-26T21:28:20.071 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-26T21:28:20.317 INFO:teuthology.orchestra.run.smithi001.stdout:55834574935 2024-04-26T21:28:20.317 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T21:28:20.433 INFO:teuthology.orchestra.run.smithi001.stdout:201863462970 2024-04-26T21:28:20.434 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T21:28:20.928 INFO:teuthology.orchestra.run.smithi001.stdout:296352743463 2024-04-26T21:28:20.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-26T21:28:21.154 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:20 smithi001 ceph-mon[28054]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-26T21:28:21.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:20 smithi186 ceph-mon[32991]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-26T21:28:21.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:20 smithi038 ceph-mon[32724]: pgmap v365: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-26T21:28:21.800 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:21.809 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:21.817 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:21.819 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:21.879 INFO:teuthology.orchestra.run.smithi001.stdout:261993005103 2024-04-26T21:28:22.423 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-26T21:28:22.717 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:22.721 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:22.727 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.769+0000 7f9abec00700 1 -- 172.21.15.1:0/1528204818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806b130 msgr2=0x7f9ab806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.769+0000 7f9abec00700 1 --2- 172.21.15.1:0/1528204818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806b130 0x7f9ab806b590 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009990 tx=0x7f9aa0009ca0 comp rx=0 tx=0).stop 2024-04-26T21:28:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.769+0000 7f9abec00700 1 -- 172.21.15.1:0/1528204818 shutdown_connections 2024-04-26T21:28:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.769+0000 7f9abec00700 1 --2- 172.21.15.1:0/1528204818 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ab806bad0 0x7f9ab81102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.769+0000 7f9abec00700 1 --2- 172.21.15.1:0/1528204818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab81094a0 0x7f9ab8109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.769+0000 7f9abec00700 1 --2- 172.21.15.1:0/1528204818 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806b130 0x7f9ab806b590 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:22.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.769+0000 7f9abec00700 1 -- 172.21.15.1:0/1528204818 >> 172.21.15.1:0/1528204818 conn(0x7f9ab80792d0 msgr2=0x7f9ab80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.771+0000 7f9abec00700 1 -- 172.21.15.1:0/1528204818 shutdown_connections 2024-04-26T21:28:22.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.771+0000 7f9abec00700 1 -- 172.21.15.1:0/1528204818 wait complete. 2024-04-26T21:28:22.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.773+0000 7f9abec00700 1 Processor -- start 2024-04-26T21:28:22.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.773+0000 7f9abec00700 1 -- start start 2024-04-26T21:28:22.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.773+0000 7f9abec00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806b130 0x7f9ab81a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:22.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.773+0000 7f9abec00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ab806bad0 0x7f9ab81a6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:22.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.773+0000 7f9abec00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab81094a0 0x7f9ab810f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:22.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.773+0000 7f9abec00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab81136f0 con 0x7f9ab806bad0 2024-04-26T21:28:22.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9abec00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab8113570 con 0x7f9ab81094a0 2024-04-26T21:28:22.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9abec00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab8113870 con 0x7f9ab806b130 2024-04-26T21:28:22.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9ab7e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806b130 0x7f9ab81a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:22.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9aaf400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ab806bad0 0x7f9ab81a6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:22.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9ab7e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806b130 0x7f9ab81a5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58752/0 (socket says 172.21.15.1:58752) 2024-04-26T21:28:22.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9ab7e00700 1 -- 172.21.15.1:0/2901809144 learned_addr learned my addr 172.21.15.1:0/2901809144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:22.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9abcc00700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab81094a0 0x7f9ab810f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:22.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9ab7e00700 1 -- 172.21.15.1:0/2901809144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab81094a0 msgr2=0x7f9ab810f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:22.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9ab7e00700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab81094a0 0x7f9ab810f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9ab7e00700 1 -- 172.21.15.1:0/2901809144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ab806bad0 msgr2=0x7f9ab81a6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9ab7e00700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ab806bad0 0x7f9ab81a6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9ab7e00700 1 -- 172.21.15.1:0/2901809144 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab810fd50 con 0x7f9ab806b130 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.774+0000 7f9abcc00700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ab81094a0 0x7f9ab810f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.775+0000 7f9ab7e00700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806b130 0x7f9ab81a5f10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800c370 tx=0x7f9aa800c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.775+0000 7f9ab5600700 1 -- 172.21.15.1:0/2901809144 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa80092e0 con 0x7f9ab806b130 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.775+0000 7f9ab5600700 1 -- 172.21.15.1:0/2901809144 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aa8003a00 con 0x7f9ab806b130 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.776+0000 7f9ab5600700 1 -- 172.21.15.1:0/2901809144 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8004650 con 0x7f9ab806b130 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.776+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab810ffe0 con 0x7f9ab806b130 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.776+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab8198b80 con 0x7f9ab806b130 2024-04-26T21:28:22.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.778+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9ab8063b30 con 0x7f9ab806b130 2024-04-26T21:28:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.782+0000 7f9ab5600700 1 -- 172.21.15.1:0/2901809144 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9aa802b020 con 0x7f9ab806b130 2024-04-26T21:28:22.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.783+0000 7f9ab5600700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9a9c0a3e50 0x7f9a9c0a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:22.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.784+0000 7f9ab5600700 1 -- 172.21.15.1:0/2901809144 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f9aa80e1220 con 0x7f9ab806b130 2024-04-26T21:28:22.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.784+0000 7f9ab5600700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] conn(0x7f9a9c0aefb0 0x7f9a9c0b1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:22.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.784+0000 7f9ab5600700 1 -- 172.21.15.1:0/2901809144 --> [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9aa8025ec0 con 0x7f9a9c0aefb0 2024-04-26T21:28:22.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.784+0000 7f9ab5600700 1 -- 172.21.15.1:0/2901809144 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f9aa80e15d0 con 0x7f9ab806b130 2024-04-26T21:28:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.785+0000 7f9aaf400700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9a9c0a3e50 0x7f9a9c0a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.785+0000 7f9abcc00700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] conn(0x7f9a9c0aefb0 0x7f9a9c0b1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.786+0000 7f9aaf400700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9a9c0a3e50 0x7f9a9c0a6310 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9ab81107b0 tx=0x7f9aa0005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:22.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.786+0000 7f9abcc00700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] conn(0x7f9a9c0aefb0 0x7f9a9c0b1450 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:22.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.787+0000 7f9ab5600700 1 -- 172.21.15.1:0/2901809144 <== osd.5 v2:172.21.15.186:6808/577720361 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9aa8025ec0 con 0x7f9a9c0aefb0 2024-04-26T21:28:22.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.811+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 --> [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9ab8064000 con 0x7f9a9c0aefb0 2024-04-26T21:28:22.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.812+0000 7f9ab5600700 1 -- 172.21.15.1:0/2901809144 <== osd.5 v2:172.21.15.186:6808/577720361 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9ab8064000 con 0x7f9a9c0aefb0 2024-04-26T21:28:22.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.812+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 >> [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] conn(0x7f9a9c0aefb0 msgr2=0x7f9a9c0b1450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:22.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.812+0000 7f9abec00700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] conn(0x7f9a9c0aefb0 0x7f9a9c0b1450 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:22.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.812+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9a9c0a3e50 msgr2=0x7f9a9c0a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:22.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.812+0000 7f9abec00700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9a9c0a3e50 0x7f9a9c0a6310 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9ab81107b0 tx=0x7f9aa0005a90 comp rx=0 tx=0).stop 2024-04-26T21:28:22.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.813+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806b130 msgr2=0x7f9ab81a5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:22.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.813+0000 7f9abec00700 1 --2- 172.21.15.1:0/2901809144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806b130 0x7f9ab81a5f10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9aa800c370 tx=0x7f9aa800c680 comp rx=0 tx=0).stop 2024-04-26T21:28:22.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.813+0000 7f9abcc00700 1 -- 172.21.15.1:0/2901809144 reap_dead start 2024-04-26T21:28:22.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.813+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 shutdown_connections 2024-04-26T21:28:22.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.813+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 >> 172.21.15.1:0/2901809144 conn(0x7f9ab80792d0 msgr2=0x7f9ab8100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:22.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.813+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 shutdown_connections 2024-04-26T21:28:22.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:22.813+0000 7f9abec00700 1 -- 172.21.15.1:0/2901809144 wait complete. 2024-04-26T21:28:23.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:22 smithi186 ceph-mon[32991]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:28:23.310 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:22 smithi001 ceph-mon[28054]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:28:23.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:22 smithi038 ceph-mon[32724]: pgmap v366: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-04-26T21:28:24.934 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:25.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:24 smithi186 ceph-mon[32991]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T21:28:25.246 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:24 smithi001 ceph-mon[28054]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T21:28:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:24 smithi038 ceph-mon[32724]: pgmap v367: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-04-26T21:28:27.115 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:26 smithi001 ceph-mon[28054]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-26T21:28:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:26 smithi038 ceph-mon[32724]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-26T21:28:27.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:26 smithi186 ceph-mon[32991]: pgmap v368: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-04-26T21:28:28.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.123+0000 7f10bfe00700 1 -- 172.21.15.1:0/2141846660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10b8106000 msgr2=0x7f10b8112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.123+0000 7f10bfe00700 1 --2- 172.21.15.1:0/2141846660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10b8106000 0x7f10b8112510 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f10b0009990 tx=0x7f10b0009ca0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.123+0000 7f10bfe00700 1 -- 172.21.15.1:0/2141846660 shutdown_connections 2024-04-26T21:28:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.123+0000 7f10bfe00700 1 --2- 172.21.15.1:0/2141846660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10b8106000 0x7f10b8112510 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.123+0000 7f10bfe00700 1 --2- 172.21.15.1:0/2141846660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b810b640 0x7f10b810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.123+0000 7f10bfe00700 1 --2- 172.21.15.1:0/2141846660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b8105640 0x7f10b8105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.123+0000 7f10bfe00700 1 -- 172.21.15.1:0/2141846660 >> 172.21.15.1:0/2141846660 conn(0x7f10b8101130 msgr2=0x7f10b8103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.123+0000 7f10bfe00700 1 -- 172.21.15.1:0/2141846660 shutdown_connections 2024-04-26T21:28:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.124+0000 7f10bfe00700 1 -- 172.21.15.1:0/2141846660 wait complete. 2024-04-26T21:28:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.125+0000 7f10bfe00700 1 Processor -- start 2024-04-26T21:28:28.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.125+0000 7f10bfe00700 1 -- start start 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bfe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10b8105640 0x7f10b81a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bfe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b8106000 0x7f10b81a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bfe00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b810b640 0x7f10b819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bfe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f10b8115890 con 0x7f10b8106000 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bfe00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f10b8115710 con 0x7f10b8105640 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bfe00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f10b8115a10 con 0x7f10b810b640 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bd400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10b8105640 0x7f10b81a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bd400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10b8105640 0x7f10b81a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:35114/0 (socket says 172.21.15.1:35114) 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bd400700 1 -- 172.21.15.1:0/237179142 learned_addr learned my addr 172.21.15.1:0/237179142 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bca00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b8106000 0x7f10b81a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:28.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bde00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b810b640 0x7f10b819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bd400700 1 -- 172.21.15.1:0/237179142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b810b640 msgr2=0x7f10b819bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bd400700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b810b640 0x7f10b819bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bd400700 1 -- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b8106000 msgr2=0x7f10b81a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.126+0000 7f10bd400700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b8106000 0x7f10b81a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.127+0000 7f10bd400700 1 -- 172.21.15.1:0/237179142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b819c230 con 0x7f10b8105640 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.127+0000 7f10bca00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b8106000 0x7f10b81a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.127+0000 7f10bd400700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10b8105640 0x7f10b81a1910 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00ef10 tx=0x7f10ac00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.127+0000 7f10b6000700 1 -- 172.21.15.1:0/237179142 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac00ca70 con 0x7f10b8105640 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.127+0000 7f10b6000700 1 -- 172.21.15.1:0/237179142 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10ac019070 con 0x7f10b8105640 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.127+0000 7f10bfe00700 1 -- 172.21.15.1:0/237179142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b819c4c0 con 0x7f10b8105640 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.127+0000 7f10b6000700 1 -- 172.21.15.1:0/237179142 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac014480 con 0x7f10b8105640 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.127+0000 7f10bfe00700 1 -- 172.21.15.1:0/237179142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b81a87e0 con 0x7f10b8105640 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.129+0000 7f10bfe00700 1 -- 172.21.15.1:0/237179142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10b806a110 con 0x7f10b8105640 2024-04-26T21:28:28.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.130+0000 7f10b6000700 1 -- 172.21.15.1:0/237179142 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f10ac012050 con 0x7f10b8105640 2024-04-26T21:28:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.132+0000 7f10b6000700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f10a00a3d50 0x7f10a00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.132+0000 7f10b6000700 1 -- 172.21.15.1:0/237179142 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f10ac0d08f0 con 0x7f10b8105640 2024-04-26T21:28:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.132+0000 7f10bca00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f10a00a3d50 0x7f10a00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.133+0000 7f10bca00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f10a00a3d50 0x7f10a00a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f10b810ac60 tx=0x7f10a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.136+0000 7f10b6000700 1 -- 172.21.15.1:0/237179142 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f10ac010040 con 0x7f10b8105640 2024-04-26T21:28:28.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.371+0000 7f10bfe00700 1 -- 172.21.15.1:0/237179142 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f10b802e490 con 0x7f10b8105640 2024-04-26T21:28:28.488 INFO:teuthology.orchestra.run.smithi001.stdout:77309411412 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.372+0000 7f10b6000700 1 -- 172.21.15.1:0/237179142 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f10ac09cdb0 con 0x7f10b8105640 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 -- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f10a00a3d50 msgr2=0x7f10a00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f10a00a3d50 0x7f10a00a6210 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f10b810ac60 tx=0x7f10a4009260 comp rx=0 tx=0).stop 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 -- 172.21.15.1:0/237179142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10b8105640 msgr2=0x7f10b81a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10b8105640 0x7f10b81a1910 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00ef10 tx=0x7f10ac00c3a0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 -- 172.21.15.1:0/237179142 shutdown_connections 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f10a00a3d50 0x7f10a00a6210 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10b810b640 0x7f10b819bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10b8105640 0x7f10b81a1910 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 --2- 172.21.15.1:0/237179142 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10b8106000 0x7f10b81a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 -- 172.21.15.1:0/237179142 >> 172.21.15.1:0/237179142 conn(0x7f10b8101130 msgr2=0x7f10b81016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.375+0000 7f10bfe00700 1 -- 172.21.15.1:0/237179142 shutdown_connections 2024-04-26T21:28:28.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:28.376+0000 7f10bfe00700 1 -- 172.21.15.1:0/237179142 wait complete. 2024-04-26T21:28:29.193 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:28 smithi001 ceph-mon[28054]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:28:29.193 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:28 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/237179142' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T21:28:29.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:28 smithi038 ceph-mon[32724]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:28:29.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:28 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/237179142' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T21:28:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:28 smithi186 ceph-mon[32991]: pgmap v369: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-04-26T21:28:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:28 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/237179142' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T21:28:29.819 INFO:teuthology.orchestra.run.smithi001.stdout:167503724609 2024-04-26T21:28:29.819 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T21:28:31.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:30 smithi001 ceph-mon[28054]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-26T21:28:31.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:30 smithi038 ceph-mon[32724]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-26T21:28:31.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:30 smithi186 ceph-mon[32991]: pgmap v370: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-26T21:28:32.168 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:33.047 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:32 smithi001 ceph-mon[28054]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:33.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:32 smithi038 ceph-mon[32724]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:33.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:32 smithi186 ceph-mon[32991]: pgmap v371: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:34.577 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411412 for osd.2 2024-04-26T21:28:34.578 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:35.122 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:34 smithi001 ceph-mon[28054]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:34 smithi038 ceph-mon[32724]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:35.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:34 smithi186 ceph-mon[32991]: pgmap v372: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:36.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.608+0000 7f25f0200700 1 -- 172.21.15.1:0/2238152175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e806b130 msgr2=0x7f25e806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.608+0000 7f25f0200700 1 --2- 172.21.15.1:0/2238152175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e806b130 0x7f25e806b590 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f25d40098f0 tx=0x7f25d4009c00 comp rx=0 tx=0).stop 2024-04-26T21:28:36.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.609+0000 7f25f0200700 1 -- 172.21.15.1:0/2238152175 shutdown_connections 2024-04-26T21:28:36.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.609+0000 7f25f0200700 1 --2- 172.21.15.1:0/2238152175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25e806bad0 0x7f25e81102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.609+0000 7f25f0200700 1 --2- 172.21.15.1:0/2238152175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25e81094a0 0x7f25e8109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.609+0000 7f25f0200700 1 --2- 172.21.15.1:0/2238152175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e806b130 0x7f25e806b590 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.609+0000 7f25f0200700 1 -- 172.21.15.1:0/2238152175 >> 172.21.15.1:0/2238152175 conn(0x7f25e80792d0 msgr2=0x7f25e80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:36.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.609+0000 7f25f0200700 1 -- 172.21.15.1:0/2238152175 shutdown_connections 2024-04-26T21:28:36.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.609+0000 7f25f0200700 1 -- 172.21.15.1:0/2238152175 wait complete. 2024-04-26T21:28:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.610+0000 7f25f0200700 1 Processor -- start 2024-04-26T21:28:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.610+0000 7f25f0200700 1 -- start start 2024-04-26T21:28:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25f0200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25e806b130 0x7f25e819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25f0200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e806bad0 0x7f25e819fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25f0200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25e81094a0 0x7f25e81999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25f0200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f25e8113620 con 0x7f25e81094a0 2024-04-26T21:28:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25f0200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f25e81134a0 con 0x7f25e806bad0 2024-04-26T21:28:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25f0200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f25e81137a0 con 0x7f25e806b130 2024-04-26T21:28:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25ece00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e806bad0 0x7f25e819fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25ee200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25e81094a0 0x7f25e81999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25ed800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25e806b130 0x7f25e819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25ece00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e806bad0 0x7f25e819fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:35158/0 (socket says 172.21.15.1:35158) 2024-04-26T21:28:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25ece00700 1 -- 172.21.15.1:0/4248387430 learned_addr learned my addr 172.21.15.1:0/4248387430 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.611+0000 7f25ece00700 1 -- 172.21.15.1:0/4248387430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25e806b130 msgr2=0x7f25e819f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.612+0000 7f25ece00700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25e806b130 0x7f25e819f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.612+0000 7f25ece00700 1 -- 172.21.15.1:0/4248387430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25e81094a0 msgr2=0x7f25e81999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.612+0000 7f25ece00700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25e81094a0 0x7f25e81999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.612+0000 7f25ece00700 1 -- 172.21.15.1:0/4248387430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e819a130 con 0x7f25e806bad0 2024-04-26T21:28:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.612+0000 7f25ece00700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e806bad0 0x7f25e819fd50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009230 tx=0x7f25d400fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.612+0000 7f25e6000700 1 -- 172.21.15.1:0/4248387430 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d40050c0 con 0x7f25e806bad0 2024-04-26T21:28:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.613+0000 7f25f0200700 1 -- 172.21.15.1:0/4248387430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e819a3c0 con 0x7f25e806bad0 2024-04-26T21:28:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.613+0000 7f25e6000700 1 -- 172.21.15.1:0/4248387430 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25d400b560 con 0x7f25e806bad0 2024-04-26T21:28:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.613+0000 7f25e6000700 1 -- 172.21.15.1:0/4248387430 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4003a00 con 0x7f25e806bad0 2024-04-26T21:28:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.613+0000 7f25f0200700 1 -- 172.21.15.1:0/4248387430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e81a66e0 con 0x7f25e806bad0 2024-04-26T21:28:36.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.614+0000 7f25f0200700 1 -- 172.21.15.1:0/4248387430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e806a110 con 0x7f25e806bad0 2024-04-26T21:28:36.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.616+0000 7f25e6000700 1 -- 172.21.15.1:0/4248387430 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f25d400b710 con 0x7f25e806bad0 2024-04-26T21:28:36.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.618+0000 7f25e6000700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25d00a3cd0 0x7f25d00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.618+0000 7f25e6000700 1 -- 172.21.15.1:0/4248387430 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f25d40d1810 con 0x7f25e806bad0 2024-04-26T21:28:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.618+0000 7f25ed800700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25d00a3cd0 0x7f25d00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.619+0000 7f25ed800700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25d00a3cd0 0x7f25d00a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f25dc009750 tx=0x7f25dc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.622+0000 7f25e6000700 1 -- 172.21.15.1:0/4248387430 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f25d4013030 con 0x7f25e806bad0 2024-04-26T21:28:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.494+0000 7f7c7b200700 1 -- 172.21.15.1:0/3628113726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c741094c0 msgr2=0x7f7c741098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.494+0000 7f7c7b200700 1 --2- 172.21.15.1:0/3628113726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c741094c0 0x7f7c741098a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009990 tx=0x7f7c64009ca0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.494+0000 7f7c7b200700 1 -- 172.21.15.1:0/3628113726 shutdown_connections 2024-04-26T21:28:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.494+0000 7f7c7b200700 1 --2- 172.21.15.1:0/3628113726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c74103e30 0x7f7c74110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.494+0000 7f7c7b200700 1 --2- 172.21.15.1:0/3628113726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c741094c0 0x7f7c741098a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.494+0000 7f7c7b200700 1 --2- 172.21.15.1:0/3628113726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c74103470 0x7f7c741038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.494+0000 7f7c7b200700 1 -- 172.21.15.1:0/3628113726 >> 172.21.15.1:0/3628113726 conn(0x7f7c7407a580 msgr2=0x7f7c7407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.495+0000 7f7c7b200700 1 -- 172.21.15.1:0/3628113726 shutdown_connections 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.495+0000 7f7c7b200700 1 -- 172.21.15.1:0/3628113726 wait complete. 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.496+0000 7f7c7b200700 1 Processor -- start 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.497+0000 7f7c7b200700 1 -- start start 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.497+0000 7f7c7b200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c74103470 0x7f7c7419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.497+0000 7f7c7b200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c74103e30 0x7f7c7419fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.497+0000 7f7c7b200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c741094c0 0x7f7c74199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.497+0000 7f7c7b200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74113610 con 0x7f7c74103e30 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.497+0000 7f7c7b200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74113490 con 0x7f7c741094c0 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.497+0000 7f7c7b200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74113790 con 0x7f7c74103470 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c79200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c741094c0 0x7f7c74199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c79200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c741094c0 0x7f7c74199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:35146/0 (socket says 172.21.15.1:35146) 2024-04-26T21:28:36.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c79200700 1 -- 172.21.15.1:0/3350874789 learned_addr learned my addr 172.21.15.1:0/3350874789 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c73e00700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c74103470 0x7f7c7419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c79200700 1 -- 172.21.15.1:0/3350874789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c74103470 msgr2=0x7f7c7419f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c79200700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c74103470 0x7f7c7419f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c79200700 1 -- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c74103e30 msgr2=0x7f7c7419fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c73400700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c74103e30 0x7f7c7419fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c79200700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c74103e30 0x7f7c7419fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c79200700 1 -- 172.21.15.1:0/3350874789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7419a0f0 con 0x7f7c741094c0 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.498+0000 7f7c73400700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c74103e30 0x7f7c7419fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.499+0000 7f7c79200700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c741094c0 0x7f7c74199990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c007ee0 tx=0x7f7c6c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.499+0000 7f7c70c00700 1 -- 172.21.15.1:0/3350874789 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00dbd0 con 0x7f7c741094c0 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.499+0000 7f7c70c00700 1 -- 172.21.15.1:0/3350874789 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c6c00dd70 con 0x7f7c741094c0 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.499+0000 7f7c7b200700 1 -- 172.21.15.1:0/3350874789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c7419a3e0 con 0x7f7c741094c0 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.499+0000 7f7c70c00700 1 -- 172.21.15.1:0/3350874789 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c005990 con 0x7f7c741094c0 2024-04-26T21:28:36.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.499+0000 7f7c7b200700 1 -- 172.21.15.1:0/3350874789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c741a66a0 con 0x7f7c741094c0 2024-04-26T21:28:36.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.501+0000 7f7c7b200700 1 -- 172.21.15.1:0/3350874789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c54005330 con 0x7f7c741094c0 2024-04-26T21:28:36.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.505+0000 7f7c70c00700 1 -- 172.21.15.1:0/3350874789 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f7c6c01a020 con 0x7f7c741094c0 2024-04-26T21:28:36.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.507+0000 7f7c70c00700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c580a3d50 0x7f7c580a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.507+0000 7f7c70c00700 1 -- 172.21.15.1:0/3350874789 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f7c6c0cff20 con 0x7f7c741094c0 2024-04-26T21:28:36.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.507+0000 7f7c73e00700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c580a3d50 0x7f7c580a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.508+0000 7f7c73e00700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c580a3d50 0x7f7c580a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7c7419aad0 tx=0x7f7c6400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:36.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.510+0000 7f7c70c00700 1 -- 172.21.15.1:0/3350874789 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7c6c010070 con 0x7f7c741094c0 2024-04-26T21:28:36.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.752+0000 7f7c7b200700 1 -- 172.21.15.1:0/3350874789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f7c540058f0 con 0x7f7c741094c0 2024-04-26T21:28:36.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.753+0000 7f7c70c00700 1 -- 172.21.15.1:0/3350874789 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7c6c022250 con 0x7f7c741094c0 2024-04-26T21:28:36.754 INFO:teuthology.orchestra.run.smithi001.stdout:201863462974 2024-04-26T21:28:36.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.755+0000 7f7c7b200700 1 -- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c580a3d50 msgr2=0x7f7c580a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c580a3d50 0x7f7c580a6210 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7c7419aad0 tx=0x7f7c6400b260 comp rx=0 tx=0).stop 2024-04-26T21:28:36.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 -- 172.21.15.1:0/3350874789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c741094c0 msgr2=0x7f7c74199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c741094c0 0x7f7c74199990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c007ee0 tx=0x7f7c6c00d3b0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 -- 172.21.15.1:0/3350874789 shutdown_connections 2024-04-26T21:28:36.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7c580a3d50 0x7f7c580a6210 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c74103470 0x7f7c7419f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7c741094c0 0x7f7c74199990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 --2- 172.21.15.1:0/3350874789 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c74103e30 0x7f7c7419fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 -- 172.21.15.1:0/3350874789 >> 172.21.15.1:0/3350874789 conn(0x7f7c7407a580 msgr2=0x7f7c741022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:36.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 -- 172.21.15.1:0/3350874789 shutdown_connections 2024-04-26T21:28:36.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.756+0000 7f7c7b200700 1 -- 172.21.15.1:0/3350874789 wait complete. 2024-04-26T21:28:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.201+0000 7f98e2400700 1 -- 172.21.15.1:0/2116435341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98dc1083b0 msgr2=0x7f98dc108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.201+0000 7f98e2400700 1 --2- 172.21.15.1:0/2116435341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98dc1083b0 0x7f98dc108790 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0099c0 tx=0x7f98cc009cd0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.202+0000 7f98e2400700 1 -- 172.21.15.1:0/2116435341 shutdown_connections 2024-04-26T21:28:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.202+0000 7f98e2400700 1 --2- 172.21.15.1:0/2116435341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98dc103e90 0x7f98dc110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.202+0000 7f98e2400700 1 --2- 172.21.15.1:0/2116435341 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98dc1083b0 0x7f98dc108790 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.202+0000 7f98e2400700 1 --2- 172.21.15.1:0/2116435341 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98dc1034d0 0x7f98dc103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.202+0000 7f98e2400700 1 -- 172.21.15.1:0/2116435341 >> 172.21.15.1:0/2116435341 conn(0x7f98dc07a580 msgr2=0x7f98dc07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.202+0000 7f98e2400700 1 -- 172.21.15.1:0/2116435341 shutdown_connections 2024-04-26T21:28:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.202+0000 7f98e2400700 1 -- 172.21.15.1:0/2116435341 wait complete. 2024-04-26T21:28:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.203+0000 7f98e2400700 1 Processor -- start 2024-04-26T21:28:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.203+0000 7f98e2400700 1 -- start start 2024-04-26T21:28:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.204+0000 7f98e2400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98dc1034d0 0x7f98dc19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.204+0000 7f98e2400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98dc103e90 0x7f98dc19c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.204+0000 7f98e2400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98dc1083b0 0x7f98dc1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.781 INFO:teuthology.orchestra.run.smithi001.stdout:34359738464 2024-04-26T21:28:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.204+0000 7f98e2400700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f98dc113610 con 0x7f98dc1083b0 2024-04-26T21:28:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.204+0000 7f98e2400700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98dc113490 con 0x7f98dc1034d0 2024-04-26T21:28:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.204+0000 7f98e2400700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98dc113790 con 0x7f98dc103e90 2024-04-26T21:28:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.204+0000 7f98db400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98dc1034d0 0x7f98dc19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.204+0000 7f98db400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98dc1034d0 0x7f98dc19bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:35126/0 (socket says 172.21.15.1:35126) 2024-04-26T21:28:36.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.204+0000 7f98db400700 1 -- 172.21.15.1:0/984363452 learned_addr learned my addr 172.21.15.1:0/984363452 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.204+0000 7f98daa00700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98dc103e90 0x7f98dc19c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.205+0000 7f98dbe00700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98dc1083b0 0x7f98dc1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.205+0000 7f98db400700 1 -- 172.21.15.1:0/984363452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98dc103e90 msgr2=0x7f98dc19c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.205+0000 7f98db400700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98dc103e90 0x7f98dc19c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.205+0000 7f98db400700 1 -- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98dc1083b0 msgr2=0x7f98dc1a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.205+0000 7f98db400700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98dc1083b0 0x7f98dc1a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.205+0000 7f98db400700 1 -- 172.21.15.1:0/984363452 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98dc1a0d30 con 0x7f98dc1034d0 2024-04-26T21:28:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.205+0000 7f98dbe00700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98dc1083b0 0x7f98dc1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:28:36.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.205+0000 7f98db400700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98dc1034d0 0x7f98dc19bd20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f98cc006010 tx=0x7f98cc004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.205+0000 7f98d3e00700 1 -- 172.21.15.1:0/984363452 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc026070 con 0x7f98dc1034d0 2024-04-26T21:28:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.206+0000 7f98d3e00700 1 -- 172.21.15.1:0/984363452 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98cc003c70 con 0x7f98dc1034d0 2024-04-26T21:28:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.206+0000 7f98e2400700 1 -- 172.21.15.1:0/984363452 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98dc1a0fc0 con 0x7f98dc1034d0 2024-04-26T21:28:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.206+0000 7f98d3e00700 1 -- 172.21.15.1:0/984363452 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc00fbe0 con 0x7f98dc1034d0 2024-04-26T21:28:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.206+0000 7f98e2400700 1 -- 172.21.15.1:0/984363452 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98dc1a88a0 con 0x7f98dc1034d0 2024-04-26T21:28:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.207+0000 7f98e2400700 1 -- 172.21.15.1:0/984363452 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98dc06a110 con 0x7f98dc1034d0 2024-04-26T21:28:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.210+0000 7f98d3e00700 1 -- 172.21.15.1:0/984363452 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f98cc029070 con 0x7f98dc1034d0 2024-04-26T21:28:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.212+0000 7f98d3e00700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98c00a3d50 0x7f98c00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.212+0000 7f98d3e00700 1 -- 172.21.15.1:0/984363452 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f98cc0d10a0 con 0x7f98dc1034d0 2024-04-26T21:28:36.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.212+0000 7f98daa00700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98c00a3d50 0x7f98c00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.213+0000 7f98daa00700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98c00a3d50 0x7f98c00a6210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98dc110670 tx=0x7f98c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.216+0000 7f98d3e00700 1 -- 172.21.15.1:0/984363452 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f98cc05e060 con 0x7f98dc1034d0 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.462+0000 7f98e2400700 1 -- 172.21.15.1:0/984363452 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f98dc1a8a70 con 0x7f98dc1034d0 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.462+0000 7f98d3e00700 1 -- 172.21.15.1:0/984363452 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f98cc013070 con 0x7f98dc1034d0 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.467+0000 7f98e2400700 1 -- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98c00a3d50 msgr2=0x7f98c00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.467+0000 7f98e2400700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98c00a3d50 0x7f98c00a6210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f98dc110670 tx=0x7f98c8009260 comp rx=0 tx=0).stop 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.467+0000 7f98e2400700 1 -- 172.21.15.1:0/984363452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98dc1034d0 msgr2=0x7f98dc19bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.467+0000 7f98e2400700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98dc1034d0 0x7f98dc19bd20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f98cc006010 tx=0x7f98cc004270 comp rx=0 tx=0).stop 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.467+0000 7f98e2400700 1 -- 172.21.15.1:0/984363452 shutdown_connections 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.467+0000 7f98e2400700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98c00a3d50 0x7f98c00a6210 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.467+0000 7f98e2400700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98dc1083b0 0x7f98dc1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.467+0000 7f98e2400700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98dc1034d0 0x7f98dc19bd20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.467+0000 7f98e2400700 1 --2- 172.21.15.1:0/984363452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98dc103e90 0x7f98dc19c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.467+0000 7f98e2400700 1 -- 172.21.15.1:0/984363452 >> 172.21.15.1:0/984363452 conn(0x7f98dc07a580 msgr2=0x7f98dc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:36.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.468+0000 7f98e2400700 1 -- 172.21.15.1:0/984363452 shutdown_connections 2024-04-26T21:28:36.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.468+0000 7f98e2400700 1 -- 172.21.15.1:0/984363452 wait complete. 2024-04-26T21:28:36.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.864+0000 7f25f0200700 1 -- 172.21.15.1:0/4248387430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f25e819af00 con 0x7f25e806bad0 2024-04-26T21:28:36.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.865+0000 7f25e6000700 1 -- 172.21.15.1:0/4248387430 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f25d4021380 con 0x7f25e806bad0 2024-04-26T21:28:36.867 INFO:teuthology.orchestra.run.smithi001.stdout:227633266744 2024-04-26T21:28:36.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 -- 172.21.15.1:0/4248387430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25d00a3cd0 msgr2=0x7f25d00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25d00a3cd0 0x7f25d00a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f25dc009750 tx=0x7f25dc009310 comp rx=0 tx=0).stop 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 -- 172.21.15.1:0/4248387430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e806bad0 msgr2=0x7f25e819fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e806bad0 0x7f25e819fd50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009230 tx=0x7f25d400fa70 comp rx=0 tx=0).stop 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 -- 172.21.15.1:0/4248387430 shutdown_connections 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f25d00a3cd0 0x7f25d00a6190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25e806b130 0x7f25e819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25e81094a0 0x7f25e81999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 --2- 172.21.15.1:0/4248387430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25e806bad0 0x7f25e819fd50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 -- 172.21.15.1:0/4248387430 >> 172.21.15.1:0/4248387430 conn(0x7f25e80792d0 msgr2=0x7f25e81004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 -- 172.21.15.1:0/4248387430 shutdown_connections 2024-04-26T21:28:36.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:36.868+0000 7f25f0200700 1 -- 172.21.15.1:0/4248387430 wait complete. 2024-04-26T21:28:37.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:36 smithi001 ceph-mon[28054]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:28:37.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:36 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/984363452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T21:28:37.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:36 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3350874789' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T21:28:37.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:36 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4248387430' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T21:28:37.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.305+0000 7f0995a00700 1 -- 172.21.15.1:0/4077595553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099006c5a0 msgr2=0x7f0990108020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:37.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.305+0000 7f0995a00700 1 --2- 172.21.15.1:0/4077595553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099006c5a0 0x7f0990108020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f097800b0a0 tx=0x7f097800b3b0 comp rx=0 tx=0).stop 2024-04-26T21:28:37.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.305+0000 7f0995a00700 1 -- 172.21.15.1:0/4077595553 shutdown_connections 2024-04-26T21:28:37.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.305+0000 7f0995a00700 1 --2- 172.21.15.1:0/4077595553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f099006bbf0 0x7f099006bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:37.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.305+0000 7f0995a00700 1 --2- 172.21.15.1:0/4077595553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0990108750 0x7f0990110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:37.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.305+0000 7f0995a00700 1 --2- 172.21.15.1:0/4077595553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099006c5a0 0x7f0990108020 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:37.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.305+0000 7f0995a00700 1 -- 172.21.15.1:0/4077595553 >> 172.21.15.1:0/4077595553 conn(0x7f0990079160 msgr2=0x7f0990079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:37.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.305+0000 7f0995a00700 1 -- 172.21.15.1:0/4077595553 shutdown_connections 2024-04-26T21:28:37.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.305+0000 7f0995a00700 1 -- 172.21.15.1:0/4077595553 wait complete. 2024-04-26T21:28:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.306+0000 7f0995a00700 1 Processor -- start 2024-04-26T21:28:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.306+0000 7f0995a00700 1 -- start start 2024-04-26T21:28:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.306+0000 7f0995a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f099006bbf0 0x7f099019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.306+0000 7f0995a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099006c5a0 0x7f099019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.306+0000 7f0995a00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0990108750 0x7f0990199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.306+0000 7f0995a00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0990113510 con 0x7f099006c5a0 2024-04-26T21:28:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.306+0000 7f0995a00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0990113390 con 0x7f099006bbf0 2024-04-26T21:28:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.306+0000 7f0995a00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0990113690 con 0x7f0990108750 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.307+0000 7f0987400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099006c5a0 0x7f099019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.307+0000 7f098fe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f099006bbf0 0x7f099019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.307+0000 7f0987400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099006c5a0 0x7f099019fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45240/0 (socket says 172.21.15.1:45240) 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.307+0000 7f0987400700 1 -- 172.21.15.1:0/727532507 learned_addr learned my addr 172.21.15.1:0/727532507 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.307+0000 7f0995000700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0990108750 0x7f0990199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.307+0000 7f0987400700 1 -- 172.21.15.1:0/727532507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0990108750 msgr2=0x7f0990199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.307+0000 7f0987400700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0990108750 0x7f0990199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.307+0000 7f0987400700 1 -- 172.21.15.1:0/727532507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f099006bbf0 msgr2=0x7f099019f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.307+0000 7f0987400700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f099006bbf0 0x7f099019f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.307+0000 7f0987400700 1 -- 172.21.15.1:0/727532507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0990199ff0 con 0x7f099006c5a0 2024-04-26T21:28:37.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.308+0000 7f0987400700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099006c5a0 0x7f099019fbd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f097800b070 tx=0x7f0978004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.308+0000 7f098d600700 1 -- 172.21.15.1:0/727532507 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0978021070 con 0x7f099006c5a0 2024-04-26T21:28:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.308+0000 7f098d600700 1 -- 172.21.15.1:0/727532507 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0978007e30 con 0x7f099006c5a0 2024-04-26T21:28:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.308+0000 7f098d600700 1 -- 172.21.15.1:0/727532507 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0978004590 con 0x7f099006c5a0 2024-04-26T21:28:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.308+0000 7f0995a00700 1 -- 172.21.15.1:0/727532507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f099019a280 con 0x7f099006c5a0 2024-04-26T21:28:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.308+0000 7f0995a00700 1 -- 172.21.15.1:0/727532507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09901a64f0 con 0x7f099006c5a0 2024-04-26T21:28:37.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.309+0000 7f0995a00700 1 -- 172.21.15.1:0/727532507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f099010b8e0 con 0x7f099006c5a0 2024-04-26T21:28:37.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.311+0000 7f098d600700 1 -- 172.21.15.1:0/727532507 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f0978004730 con 0x7f099006c5a0 2024-04-26T21:28:37.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.313+0000 7f098d600700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09740a3d50 0x7f09740a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:37.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.313+0000 7f098d600700 1 -- 172.21.15.1:0/727532507 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f097800f020 con 0x7f099006c5a0 2024-04-26T21:28:37.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.313+0000 7f098fe00700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09740a3d50 0x7f09740a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:37.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.314+0000 7f098fe00700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09740a3d50 0x7f09740a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0980009c60 tx=0x7f0980009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:37.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.318+0000 7f098d600700 1 -- 172.21.15.1:0/727532507 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f097802a020 con 0x7f099006c5a0 2024-04-26T21:28:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:36 smithi038 ceph-mon[32724]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:28:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:36 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/984363452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T21:28:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:36 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3350874789' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T21:28:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:36 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4248387430' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T21:28:37.437 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:37.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:36 smithi186 ceph-mon[32991]: pgmap v373: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:28:37.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:36 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/984363452' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T21:28:37.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:36 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3350874789' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T21:28:37.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:36 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4248387430' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T21:28:37.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.558+0000 7f0995a00700 1 -- 172.21.15.1:0/727532507 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f099019ad40 con 0x7f099006c5a0 2024-04-26T21:28:37.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.560+0000 7f098d600700 1 -- 172.21.15.1:0/727532507 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f097809e2c0 con 0x7f099006c5a0 2024-04-26T21:28:37.563 INFO:teuthology.orchestra.run.smithi001.stdout:317827579942 2024-04-26T21:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 -- 172.21.15.1:0/727532507 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09740a3d50 msgr2=0x7f09740a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:37.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09740a3d50 0x7f09740a6210 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0980009c60 tx=0x7f0980009260 comp rx=0 tx=0).stop 2024-04-26T21:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 -- 172.21.15.1:0/727532507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099006c5a0 msgr2=0x7f099019fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099006c5a0 0x7f099019fbd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f097800b070 tx=0x7f0978004d00 comp rx=0 tx=0).stop 2024-04-26T21:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 -- 172.21.15.1:0/727532507 shutdown_connections 2024-04-26T21:28:37.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f09740a3d50 0x7f09740a6210 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0990108750 0x7f0990199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f099006bbf0 0x7f099019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 --2- 172.21.15.1:0/727532507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f099006c5a0 0x7f099019fbd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 -- 172.21.15.1:0/727532507 >> 172.21.15.1:0/727532507 conn(0x7f0990079160 msgr2=0x7f0990100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 -- 172.21.15.1:0/727532507 shutdown_connections 2024-04-26T21:28:37.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:37.563+0000 7f0995a00700 1 -- 172.21.15.1:0/727532507 wait complete. 2024-04-26T21:28:38.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:37 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/727532507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T21:28:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:37 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/727532507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T21:28:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:37 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/727532507' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-26T21:28:38.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.595+0000 7ff88fe00700 1 -- 172.21.15.1:0/846295842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff888105600 msgr2=0x7ff888105a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:38.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.595+0000 7ff88fe00700 1 --2- 172.21.15.1:0/846295842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff888105600 0x7ff888105a80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff874009990 tx=0x7ff874009ca0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.595+0000 7ff88fe00700 1 -- 172.21.15.1:0/846295842 shutdown_connections 2024-04-26T21:28:38.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.595+0000 7ff88fe00700 1 --2- 172.21.15.1:0/846295842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff88810b690 0x7ff88810ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.595+0000 7ff88fe00700 1 --2- 172.21.15.1:0/846295842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff888105fc0 0x7ff8881124d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.595+0000 7ff88fe00700 1 --2- 172.21.15.1:0/846295842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff888105600 0x7ff888105a80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.595+0000 7ff88fe00700 1 -- 172.21.15.1:0/846295842 >> 172.21.15.1:0/846295842 conn(0x7ff8881010d0 msgr2=0x7ff8881034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:38.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.595+0000 7ff88fe00700 1 -- 172.21.15.1:0/846295842 shutdown_connections 2024-04-26T21:28:38.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.595+0000 7ff88fe00700 1 -- 172.21.15.1:0/846295842 wait complete. 2024-04-26T21:28:38.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.596+0000 7ff88fe00700 1 Processor -- start 2024-04-26T21:28:38.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.596+0000 7ff88fe00700 1 -- start start 2024-04-26T21:28:38.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.596+0000 7ff88fe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff888105600 0x7ff8881a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:38.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.596+0000 7ff88fe00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff888105fc0 0x7ff8881a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:38.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.596+0000 7ff88fe00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff88810b690 0x7ff88819bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:38.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.596+0000 7ff88fe00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff888115850 con 0x7ff888105fc0 2024-04-26T21:28:38.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88fe00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff8881156d0 con 0x7ff888105600 2024-04-26T21:28:38.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88fe00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff8881159d0 con 0x7ff88810b690 2024-04-26T21:28:38.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff888105fc0 0x7ff8881a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:38.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88d400700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff888105600 0x7ff8881a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:38.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88ca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff888105fc0 0x7ff8881a1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52982/0 (socket says 172.21.15.1:52982) 2024-04-26T21:28:38.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88ca00700 1 -- 172.21.15.1:0/3459857328 learned_addr learned my addr 172.21.15.1:0/3459857328 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:38.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88de00700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff88810b690 0x7ff88819bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:38.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88d400700 1 -- 172.21.15.1:0/3459857328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff88810b690 msgr2=0x7ff88819bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:38.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88d400700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff88810b690 0x7ff88819bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88d400700 1 -- 172.21.15.1:0/3459857328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff888105fc0 msgr2=0x7ff8881a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:38.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88d400700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff888105fc0 0x7ff8881a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88d400700 1 -- 172.21.15.1:0/3459857328 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88819c210 con 0x7ff888105600 2024-04-26T21:28:38.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.597+0000 7ff88ca00700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff888105fc0 0x7ff8881a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.598+0000 7ff88d400700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff888105600 0x7ff8881a18f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00e9c0 tx=0x7ff87c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.598+0000 7ff886000700 1 -- 172.21.15.1:0/3459857328 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c004230 con 0x7ff888105600 2024-04-26T21:28:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.598+0000 7ff886000700 1 -- 172.21.15.1:0/3459857328 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff87c0043d0 con 0x7ff888105600 2024-04-26T21:28:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.598+0000 7ff886000700 1 -- 172.21.15.1:0/3459857328 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87c003d90 con 0x7ff888105600 2024-04-26T21:28:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.598+0000 7ff88fe00700 1 -- 172.21.15.1:0/3459857328 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88819c500 con 0x7ff888105600 2024-04-26T21:28:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.598+0000 7ff88fe00700 1 -- 172.21.15.1:0/3459857328 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8881a87c0 con 0x7ff888105600 2024-04-26T21:28:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.599+0000 7ff88fe00700 1 -- 172.21.15.1:0/3459857328 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff88806a110 con 0x7ff888105600 2024-04-26T21:28:38.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.601+0000 7ff886000700 1 -- 172.21.15.1:0/3459857328 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7ff87c00c7c0 con 0x7ff888105600 2024-04-26T21:28:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.603+0000 7ff886000700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8700a3cd0 0x7ff8700a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.603+0000 7ff886000700 1 -- 172.21.15.1:0/3459857328 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff87c0d0c20 con 0x7ff888105600 2024-04-26T21:28:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.603+0000 7ff88ca00700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8700a3cd0 0x7ff8700a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.604+0000 7ff88ca00700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8700a3cd0 0x7ff8700a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff88810ad20 tx=0x7ff87400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.608+0000 7ff886000700 1 -- 172.21.15.1:0/3459857328 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff87c09d0e0 con 0x7ff888105600 2024-04-26T21:28:38.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.861+0000 7ff88fe00700 1 -- 172.21.15.1:0/3459857328 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff88819cfe0 con 0x7ff888105600 2024-04-26T21:28:38.884 INFO:teuthology.orchestra.run.smithi001.stdout:146028888137 2024-04-26T21:28:38.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.861+0000 7ff886000700 1 -- 172.21.15.1:0/3459857328 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff87c09c9e0 con 0x7ff888105600 2024-04-26T21:28:38.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 -- 172.21.15.1:0/3459857328 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8700a3cd0 msgr2=0x7ff8700a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:38.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8700a3cd0 0x7ff8700a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff88810ad20 tx=0x7ff87400b280 comp rx=0 tx=0).stop 2024-04-26T21:28:38.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 -- 172.21.15.1:0/3459857328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff888105600 msgr2=0x7ff8881a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:38.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff888105600 0x7ff8881a18f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00e9c0 tx=0x7ff87c00ecd0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 -- 172.21.15.1:0/3459857328 shutdown_connections 2024-04-26T21:28:38.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff8700a3cd0 0x7ff8700a6190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff88810b690 0x7ff88819bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff888105600 0x7ff8881a18f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 --2- 172.21.15.1:0/3459857328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff888105fc0 0x7ff8881a1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:38.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 -- 172.21.15.1:0/3459857328 >> 172.21.15.1:0/3459857328 conn(0x7ff8881010d0 msgr2=0x7ff888102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:38.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 -- 172.21.15.1:0/3459857328 shutdown_connections 2024-04-26T21:28:38.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:38.864+0000 7ff88fe00700 1 -- 172.21.15.1:0/3459857328 wait complete. 2024-04-26T21:28:39.042 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:38 smithi001 ceph-mon[28054]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:39.042 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:38 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3459857328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T21:28:39.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:38 smithi038 ceph-mon[32724]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:39.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:38 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3459857328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T21:28:39.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:38 smithi186 ceph-mon[32991]: pgmap v374: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:39.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:38 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3459857328' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T21:28:40.787 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738459 got 34359738464 for osd.0 2024-04-26T21:28:40.787 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.369+0000 7f80f1600700 1 -- 172.21.15.1:0/678767777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80ec10fbe0 msgr2=0x7f80ec112010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.369+0000 7f80f1600700 1 --2- 172.21.15.1:0/678767777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80ec10fbe0 0x7f80ec112010 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f80e400b0a0 tx=0x7f80e400b3b0 comp rx=0 tx=0).stop 2024-04-26T21:28:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.369+0000 7f80f1600700 1 -- 172.21.15.1:0/678767777 shutdown_connections 2024-04-26T21:28:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.369+0000 7f80f1600700 1 --2- 172.21.15.1:0/678767777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80ec1042e0 0x7f80ec06b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.369+0000 7f80f1600700 1 --2- 172.21.15.1:0/678767777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80ec10fbe0 0x7f80ec112010 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.369+0000 7f80f1600700 1 --2- 172.21.15.1:0/678767777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ec06bed0 0x7f80ec06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.369+0000 7f80f1600700 1 -- 172.21.15.1:0/678767777 >> 172.21.15.1:0/678767777 conn(0x7f80ec079160 msgr2=0x7f80ec079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.369+0000 7f80f1600700 1 -- 172.21.15.1:0/678767777 shutdown_connections 2024-04-26T21:28:41.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.369+0000 7f80f1600700 1 -- 172.21.15.1:0/678767777 wait complete. 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.370+0000 7f80f1600700 1 Processor -- start 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.370+0000 7f80f1600700 1 -- start start 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80f1600700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ec06bed0 0x7f80ec1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80f1600700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80ec1042e0 0x7f80ec1a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80f1600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80ec10fbe0 0x7f80ec19b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80f1600700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f80ec115750 con 0x7f80ec10fbe0 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80f1600700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f80ec1155d0 con 0x7f80ec06bed0 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80f1600700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f80ec1158d0 con 0x7f80ec1042e0 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80e3e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80ec1042e0 0x7f80ec1a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80f0c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80ec10fbe0 0x7f80ec19b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80ebe00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ec06bed0 0x7f80ec1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80e3e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80ec1042e0 0x7f80ec1a1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:55536/0 (socket says 172.21.15.1:55536) 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.371+0000 7f80e3e00700 1 -- 172.21.15.1:0/27535636 learned_addr learned my addr 172.21.15.1:0/27535636 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:41.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80ebe00700 1 -- 172.21.15.1:0/27535636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80ec1042e0 msgr2=0x7f80ec1a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80ebe00700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80ec1042e0 0x7f80ec1a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80ebe00700 1 -- 172.21.15.1:0/27535636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80ec10fbe0 msgr2=0x7f80ec19b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80ebe00700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80ec10fbe0 0x7f80ec19b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80ebe00700 1 -- 172.21.15.1:0/27535636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80ec19c0d0 con 0x7f80ec06bed0 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80ebe00700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ec06bed0 0x7f80ec1a17b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00e9f0 tx=0x7f80dc00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80e9600700 1 -- 172.21.15.1:0/27535636 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc00c9c0 con 0x7f80ec06bed0 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80e9600700 1 -- 172.21.15.1:0/27535636 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80dc00cb60 con 0x7f80ec06bed0 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80f1600700 1 -- 172.21.15.1:0/27535636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80ec19c3c0 con 0x7f80ec06bed0 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80e9600700 1 -- 172.21.15.1:0/27535636 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80dc018680 con 0x7f80ec06bed0 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.372+0000 7f80f1600700 1 -- 172.21.15.1:0/27535636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80ec1a8680 con 0x7f80ec06bed0 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.373+0000 7f80f1600700 1 -- 172.21.15.1:0/27535636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80ec06a110 con 0x7f80ec06bed0 2024-04-26T21:28:41.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.376+0000 7f80e9600700 1 -- 172.21.15.1:0/27535636 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f80dc010430 con 0x7f80ec06bed0 2024-04-26T21:28:41.039 INFO:teuthology.orchestra.run.smithi001.stdout:352187318305 2024-04-26T21:28:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.377+0000 7f80e9600700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f80d00a3cd0 0x7f80d00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.377+0000 7f80e9600700 1 -- 172.21.15.1:0/27535636 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f80dc0d0b90 con 0x7f80ec06bed0 2024-04-26T21:28:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.377+0000 7f80e3e00700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f80d00a3cd0 0x7f80d00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.378+0000 7f80e3e00700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f80d00a3cd0 0x7f80d00a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009680 tx=0x7f80d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.380+0000 7f80e9600700 1 -- 172.21.15.1:0/27535636 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f80dc09d050 con 0x7f80ec06bed0 2024-04-26T21:28:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.614+0000 7f80f1600700 1 -- 172.21.15.1:0/27535636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f80ec1a88b0 con 0x7f80ec06bed0 2024-04-26T21:28:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.615+0000 7f80e9600700 1 -- 172.21.15.1:0/27535636 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f80dc09c950 con 0x7f80ec06bed0 2024-04-26T21:28:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.617+0000 7f80f1600700 1 -- 172.21.15.1:0/27535636 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f80d00a3cd0 msgr2=0x7f80d00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:41.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f80d00a3cd0 0x7f80d00a6190 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f80d4009680 tx=0x7f80d4009260 comp rx=0 tx=0).stop 2024-04-26T21:28:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 -- 172.21.15.1:0/27535636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ec06bed0 msgr2=0x7f80ec1a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ec06bed0 0x7f80ec1a17b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f80dc00e9f0 tx=0x7f80dc00ed00 comp rx=0 tx=0).stop 2024-04-26T21:28:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 -- 172.21.15.1:0/27535636 shutdown_connections 2024-04-26T21:28:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f80d00a3cd0 0x7f80d00a6190 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80ec10fbe0 0x7f80ec19b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80ec06bed0 0x7f80ec1a17b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 --2- 172.21.15.1:0/27535636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80ec1042e0 0x7f80ec1a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 -- 172.21.15.1:0/27535636 >> 172.21.15.1:0/27535636 conn(0x7f80ec079160 msgr2=0x7f80ec1014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 -- 172.21.15.1:0/27535636 shutdown_connections 2024-04-26T21:28:41.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:40.618+0000 7f80f1600700 1 -- 172.21.15.1:0/27535636 wait complete. 2024-04-26T21:28:41.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:41 smithi001 ceph-mon[28054]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:28:41.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:41 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/27535636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T21:28:41.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:41 smithi038 ceph-mon[32724]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:28:41.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:41 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/27535636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T21:28:41.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:41 smithi186 ceph-mon[32991]: pgmap v375: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:28:41.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:41 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/27535636' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-26T21:28:41.908 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888130 got 146028888137 for osd.4 2024-04-26T21:28:41.908 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:42.292 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462970 got 201863462974 for osd.6 2024-04-26T21:28:42.292 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:43.107 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579938 got 317827579942 for osd.10 2024-04-26T21:28:43.107 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:43.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:43 smithi001 ceph-mon[28054]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:43.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:43 smithi038 ceph-mon[32724]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:43.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:43 smithi186 ceph-mon[32991]: pgmap v376: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.794+0000 7f1b2aa00700 1 -- 172.21.15.1:0/3082236199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b2410fe20 msgr2=0x7f1b24112210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.794+0000 7f1b2aa00700 1 --2- 172.21.15.1:0/3082236199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b2410fe20 0x7f1b24112210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00b0a0 tx=0x7f1b1c00b3b0 comp rx=0 tx=0).stop 2024-04-26T21:28:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.794+0000 7f1b2aa00700 1 -- 172.21.15.1:0/3082236199 shutdown_connections 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.794+0000 7f1b2aa00700 1 --2- 172.21.15.1:0/3082236199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b2410fe20 0x7f1b24112210 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.794+0000 7f1b2aa00700 1 --2- 172.21.15.1:0/3082236199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24075650 0x7f1b24075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.794+0000 7f1b2aa00700 1 --2- 172.21.15.1:0/3082236199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b24075f70 0x7f1b2410f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.794+0000 7f1b2aa00700 1 -- 172.21.15.1:0/3082236199 >> 172.21.15.1:0/3082236199 conn(0x7f1b2407a580 msgr2=0x7f1b2407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.794+0000 7f1b2aa00700 1 -- 172.21.15.1:0/3082236199 shutdown_connections 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.794+0000 7f1b2aa00700 1 -- 172.21.15.1:0/3082236199 wait complete. 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.795+0000 7f1b2aa00700 1 Processor -- start 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.795+0000 7f1b2aa00700 1 -- start start 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.795+0000 7f1b2aa00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24075650 0x7f1b241a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.795+0000 7f1b2aa00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b24075f70 0x7f1b241a1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b2aa00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b2410fe20 0x7f1b2419bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b2aa00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1b241157e0 con 0x7f1b2410fe20 2024-04-26T21:28:43.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b2aa00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24115660 con 0x7f1b24075650 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b2aa00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1b24115960 con 0x7f1b24075f70 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b23e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24075650 0x7f1b241a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b23e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24075650 0x7f1b241a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54304/0 (socket says 172.21.15.1:54304) 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b28a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b2410fe20 0x7f1b2419bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b23e00700 1 -- 172.21.15.1:0/390298662 learned_addr learned my addr 172.21.15.1:0/390298662 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b23400700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b24075f70 0x7f1b241a1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b23e00700 1 -- 172.21.15.1:0/390298662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b24075f70 msgr2=0x7f1b241a1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b23e00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b24075f70 0x7f1b241a1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b23e00700 1 -- 172.21.15.1:0/390298662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b2410fe20 msgr2=0x7f1b2419bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b23e00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b2410fe20 0x7f1b2419bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b23e00700 1 -- 172.21.15.1:0/390298662 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2419c240 con 0x7f1b24075650 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.796+0000 7f1b28a00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b2410fe20 0x7f1b2419bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:43.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.797+0000 7f1b23e00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24075650 0x7f1b241a18d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400eb30 tx=0x7f1b1400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.797+0000 7f1b20c00700 1 -- 172.21.15.1:0/390298662 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1400cbd0 con 0x7f1b24075650 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.797+0000 7f1b2aa00700 1 -- 172.21.15.1:0/390298662 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b2419c530 con 0x7f1b24075650 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.797+0000 7f1b20c00700 1 -- 172.21.15.1:0/390298662 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b140044d0 con 0x7f1b24075650 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.797+0000 7f1b20c00700 1 -- 172.21.15.1:0/390298662 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b14003e30 con 0x7f1b24075650 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.797+0000 7f1b2aa00700 1 -- 172.21.15.1:0/390298662 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b241a87a0 con 0x7f1b24075650 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.798+0000 7f1b2aa00700 1 -- 172.21.15.1:0/390298662 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b2406a110 con 0x7f1b24075650 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.801+0000 7f1b20c00700 1 -- 172.21.15.1:0/390298662 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f1b14004000 con 0x7f1b24075650 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.802+0000 7f1b20c00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1b080a3d50 0x7f1b080a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.802+0000 7f1b20c00700 1 -- 172.21.15.1:0/390298662 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1b140d0e50 con 0x7f1b24075650 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.802+0000 7f1b23400700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1b080a3d50 0x7f1b080a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.803+0000 7f1b23400700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1b080a3d50 0x7f1b080a6210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1b24111ab0 tx=0x7f1b10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:43.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:43.804+0000 7f1b20c00700 1 -- 172.21.15.1:0/390298662 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1b14054060 con 0x7f1b24075650 2024-04-26T21:28:44.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.049+0000 7f1b2aa00700 1 -- 172.21.15.1:0/390298662 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f1b241a8a30 con 0x7f1b24075650 2024-04-26T21:28:44.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.050+0000 7f1b20c00700 1 -- 172.21.15.1:0/390298662 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1b140042f0 con 0x7f1b24075650 2024-04-26T21:28:44.051 INFO:teuthology.orchestra.run.smithi001.stdout:111669149777 2024-04-26T21:28:44.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.053+0000 7f1b2aa00700 1 -- 172.21.15.1:0/390298662 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1b080a3d50 msgr2=0x7f1b080a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.053+0000 7f1b2aa00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1b080a3d50 0x7f1b080a6210 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1b24111ab0 tx=0x7f1b10009260 comp rx=0 tx=0).stop 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.053+0000 7f1b2aa00700 1 -- 172.21.15.1:0/390298662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24075650 msgr2=0x7f1b241a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.053+0000 7f1b2aa00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24075650 0x7f1b241a18d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400eb30 tx=0x7f1b1400ee40 comp rx=0 tx=0).stop 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.054+0000 7f1b2aa00700 1 -- 172.21.15.1:0/390298662 shutdown_connections 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.054+0000 7f1b2aa00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1b080a3d50 0x7f1b080a6210 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.054+0000 7f1b2aa00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b2410fe20 0x7f1b2419bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.054+0000 7f1b2aa00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b24075650 0x7f1b241a18d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.054+0000 7f1b2aa00700 1 --2- 172.21.15.1:0/390298662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b24075f70 0x7f1b241a1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.054+0000 7f1b2aa00700 1 -- 172.21.15.1:0/390298662 >> 172.21.15.1:0/390298662 conn(0x7f1b2407a580 msgr2=0x7f1b24100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.054+0000 7f1b2aa00700 1 -- 172.21.15.1:0/390298662 shutdown_connections 2024-04-26T21:28:44.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:44.054+0000 7f1b2aa00700 1 -- 172.21.15.1:0/390298662 wait complete. 2024-04-26T21:28:44.419 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266739 got 227633266744 for osd.7 2024-04-26T21:28:44.419 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:45.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:45 smithi001 ceph-mon[28054]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:45.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:45 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/390298662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T21:28:45.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:45 smithi038 ceph-mon[32724]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:45.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:45 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/390298662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T21:28:45.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:45 smithi186 ceph-mon[32991]: pgmap v377: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:45.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:45 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/390298662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T21:28:46.070 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318298 got 352187318305 for osd.11 2024-04-26T21:28:46.071 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:46.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.836+0000 7f3b91e00700 1 -- 172.21.15.1:0/3703497375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8c10fa50 msgr2=0x7f3b8c10fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.836+0000 7f3b91e00700 1 --2- 172.21.15.1:0/3703497375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8c10fa50 0x7f3b8c10fe30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009990 tx=0x7f3b7c009ca0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.836+0000 7f3b91e00700 1 -- 172.21.15.1:0/3703497375 shutdown_connections 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.836+0000 7f3b91e00700 1 --2- 172.21.15.1:0/3703497375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b8c10a410 0x7f3b8c116920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.836+0000 7f3b91e00700 1 --2- 172.21.15.1:0/3703497375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8c10fa50 0x7f3b8c10fe30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.836+0000 7f3b91e00700 1 --2- 172.21.15.1:0/3703497375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b8c109a50 0x7f3b8c109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.836+0000 7f3b91e00700 1 -- 172.21.15.1:0/3703497375 >> 172.21.15.1:0/3703497375 conn(0x7f3b8c076bf0 msgr2=0x7f3b8c108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.836+0000 7f3b91e00700 1 -- 172.21.15.1:0/3703497375 shutdown_connections 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.836+0000 7f3b91e00700 1 -- 172.21.15.1:0/3703497375 wait complete. 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.837+0000 7f3b91e00700 1 Processor -- start 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.838+0000 7f3b91e00700 1 -- start start 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.838+0000 7f3b91e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b8c109a50 0x7f3b8c1a00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.838+0000 7f3b91e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b8c10a410 0x7f3b8c1a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:46.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.838+0000 7f3b91e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8c10fa50 0x7f3b8c1a4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.838+0000 7f3b91e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c119c90 con 0x7f3b8c10fa50 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.838+0000 7f3b91e00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c119b10 con 0x7f3b8c10a410 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.838+0000 7f3b91e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8c119e10 con 0x7f3b8c109a50 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8aa00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b8c109a50 0x7f3b8c1a00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.838+0000 7f3b8b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8c10fa50 0x7f3b8c1a4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8aa00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b8c109a50 0x7f3b8c1a00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:55584/0 (socket says 172.21.15.1:55584) 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8aa00700 1 -- 172.21.15.1:0/2712086118 learned_addr learned my addr 172.21.15.1:0/2712086118 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8a000700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b8c10a410 0x7f3b8c1a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8aa00700 1 -- 172.21.15.1:0/2712086118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b8c10a410 msgr2=0x7f3b8c1a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8aa00700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b8c10a410 0x7f3b8c1a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8aa00700 1 -- 172.21.15.1:0/2712086118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8c10fa50 msgr2=0x7f3b8c1a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8aa00700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8c10fa50 0x7f3b8c1a4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8aa00700 1 -- 172.21.15.1:0/2712086118 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8c1a50b0 con 0x7f3b8c109a50 2024-04-26T21:28:46.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8b400700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8c10fa50 0x7f3b8c1a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.839+0000 7f3b8aa00700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b8c109a50 0x7f3b8c1a00a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c005910 tx=0x7f3b7c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.840+0000 7f3b90a00700 1 -- 172.21.15.1:0/2712086118 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c027070 con 0x7f3b8c109a50 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.840+0000 7f3b90a00700 1 -- 172.21.15.1:0/2712086118 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b7c003f20 con 0x7f3b8c109a50 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.840+0000 7f3b91e00700 1 -- 172.21.15.1:0/2712086118 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8c1a5340 con 0x7f3b8c109a50 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.840+0000 7f3b90a00700 1 -- 172.21.15.1:0/2712086118 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c022c60 con 0x7f3b8c109a50 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.840+0000 7f3b91e00700 1 -- 172.21.15.1:0/2712086118 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b8c1acc20 con 0x7f3b8c109a50 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.841+0000 7f3b91e00700 1 -- 172.21.15.1:0/2712086118 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b8c06a110 con 0x7f3b8c109a50 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.843+0000 7f3b90a00700 1 -- 172.21.15.1:0/2712086118 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f3b7c02a070 con 0x7f3b8c109a50 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.845+0000 7f3b90a00700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f3b700a3d50 0x7f3b700a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.845+0000 7f3b90a00700 1 -- 172.21.15.1:0/2712086118 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f3b7c022e00 con 0x7f3b8c109a50 2024-04-26T21:28:46.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.845+0000 7f3b8a000700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f3b700a3d50 0x7f3b700a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:46.607 INFO:teuthology.orchestra.run.smithi001.stdout:55834574941 2024-04-26T21:28:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.846+0000 7f3b8a000700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f3b700a3d50 0x7f3b700a6210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c078a30 tx=0x7f3b7800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:45.849+0000 7f3b90a00700 1 -- 172.21.15.1:0/2712086118 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3b7c05f060 con 0x7f3b8c109a50 2024-04-26T21:28:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.103+0000 7f3b91e00700 1 -- 172.21.15.1:0/2712086118 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f3b8c02e490 con 0x7f3b8c109a50 2024-04-26T21:28:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.104+0000 7f3b90a00700 1 -- 172.21.15.1:0/2712086118 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3b7c013070 con 0x7f3b8c109a50 2024-04-26T21:28:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 -- 172.21.15.1:0/2712086118 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f3b700a3d50 msgr2=0x7f3b700a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f3b700a3d50 0x7f3b700a6210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c078a30 tx=0x7f3b7800a380 comp rx=0 tx=0).stop 2024-04-26T21:28:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 -- 172.21.15.1:0/2712086118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b8c109a50 msgr2=0x7f3b8c1a00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b8c109a50 0x7f3b8c1a00a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c005910 tx=0x7f3b7c0047b0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 -- 172.21.15.1:0/2712086118 shutdown_connections 2024-04-26T21:28:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f3b700a3d50 0x7f3b700a6210 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b8c109a50 0x7f3b8c1a00a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8c10fa50 0x7f3b8c1a4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 --2- 172.21.15.1:0/2712086118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b8c10a410 0x7f3b8c1a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 -- 172.21.15.1:0/2712086118 >> 172.21.15.1:0/2712086118 conn(0x7f3b8c076bf0 msgr2=0x7f3b8c077170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 -- 172.21.15.1:0/2712086118 shutdown_connections 2024-04-26T21:28:46.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:46.107+0000 7f3b91e00700 1 -- 172.21.15.1:0/2712086118 wait complete. 2024-04-26T21:28:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:47.246 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149770 got 111669149777 for osd.3 2024-04-26T21:28:47.246 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:47.313 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:47 smithi001 ceph-mon[28054]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:28:47.313 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:47 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2712086118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T21:28:47.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:47 smithi038 ceph-mon[32724]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:28:47.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:47 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2712086118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T21:28:47.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:47 smithi186 ceph-mon[32991]: pgmap v378: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:28:47.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:47 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2712086118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T21:28:48.011 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574935 got 55834574941 for osd.1 2024-04-26T21:28:48.011 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:49.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.822+0000 7f4edca00700 1 -- 172.21.15.1:0/2380754930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed8075650 msgr2=0x7f4ed8075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:49.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.822+0000 7f4edca00700 1 --2- 172.21.15.1:0/2380754930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed8075650 0x7f4ed8075a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4ec80098f0 tx=0x7f4ec8009c00 comp rx=0 tx=0).stop 2024-04-26T21:28:49.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.822+0000 7f4edca00700 1 -- 172.21.15.1:0/2380754930 shutdown_connections 2024-04-26T21:28:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.822+0000 7f4edca00700 1 --2- 172.21.15.1:0/2380754930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed81145f0 0x7f4ed81169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.822+0000 7f4edca00700 1 --2- 172.21.15.1:0/2380754930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed8075650 0x7f4ed8075a30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.822+0000 7f4edca00700 1 --2- 172.21.15.1:0/2380754930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed8075f70 0x7f4ed81140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.822+0000 7f4edca00700 1 -- 172.21.15.1:0/2380754930 >> 172.21.15.1:0/2380754930 conn(0x7f4ed807a580 msgr2=0x7f4ed807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.822+0000 7f4edca00700 1 -- 172.21.15.1:0/2380754930 shutdown_connections 2024-04-26T21:28:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.823+0000 7f4edca00700 1 -- 172.21.15.1:0/2380754930 wait complete. 2024-04-26T21:28:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.824+0000 7f4edca00700 1 Processor -- start 2024-04-26T21:28:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.824+0000 7f4edca00700 1 -- start start 2024-04-26T21:28:49.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.825+0000 7f4edca00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed8075650 0x7f4ed81a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.825+0000 7f4edca00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed8075f70 0x7f4ed81a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.825+0000 7f4edca00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed81145f0 0x7f4ed81a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.825+0000 7f4edca00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed8119c50 con 0x7f4ed81145f0 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.825+0000 7f4edca00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed8119ad0 con 0x7f4ed8075f70 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.825+0000 7f4edca00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed8119dd0 con 0x7f4ed8075650 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.826+0000 7f4ed5e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed8075650 0x7f4ed81a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.826+0000 7f4ed6800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed81145f0 0x7f4ed81a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.826+0000 7f4ed5e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed8075650 0x7f4ed81a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:53824/0 (socket says 172.21.15.1:53824) 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.826+0000 7f4ed5e00700 1 -- 172.21.15.1:0/3041764229 learned_addr learned my addr 172.21.15.1:0/3041764229 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.826+0000 7f4ed5400700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed8075f70 0x7f4ed81a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:49.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.826+0000 7f4ed5400700 1 -- 172.21.15.1:0/3041764229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed8075650 msgr2=0x7f4ed81a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.826+0000 7f4ed5400700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed8075650 0x7f4ed81a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.826+0000 7f4ed5400700 1 -- 172.21.15.1:0/3041764229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed81145f0 msgr2=0x7f4ed81a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.826+0000 7f4ed5400700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed81145f0 0x7f4ed81a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.826+0000 7f4ed5400700 1 -- 172.21.15.1:0/3041764229 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed81a5170 con 0x7f4ed8075f70 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.827+0000 7f4ed5400700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed8075f70 0x7f4ed81a06a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e9c0 tx=0x7f4ec400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.827+0000 7f4ecea00700 1 -- 172.21.15.1:0/3041764229 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec400c9c0 con 0x7f4ed8075f70 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.827+0000 7f4ecea00700 1 -- 172.21.15.1:0/3041764229 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ec400cb60 con 0x7f4ed8075f70 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.827+0000 7f4edca00700 1 -- 172.21.15.1:0/3041764229 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed81a5460 con 0x7f4ed8075f70 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.827+0000 7f4ecea00700 1 -- 172.21.15.1:0/3041764229 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4018420 con 0x7f4ed8075f70 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.827+0000 7f4edca00700 1 -- 172.21.15.1:0/3041764229 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed81acce0 con 0x7f4ed8075f70 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.828+0000 7f4edca00700 1 -- 172.21.15.1:0/3041764229 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed806a110 con 0x7f4ed8075f70 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.831+0000 7f4ecea00700 1 -- 172.21.15.1:0/3041764229 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f4ec4010430 con 0x7f4ed8075f70 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.832+0000 7f4ecea00700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4ebc0a3d50 0x7f4ebc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:49.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.832+0000 7f4ecea00700 1 -- 172.21.15.1:0/3041764229 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4ec40d0b30 con 0x7f4ed8075f70 2024-04-26T21:28:49.186 INFO:teuthology.orchestra.run.smithi001.stdout:296352743470 2024-04-26T21:28:49.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.833+0000 7f4ed5e00700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4ebc0a3d50 0x7f4ebc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:49.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.833+0000 7f4ed5e00700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4ebc0a3d50 0x7f4ebc0a6210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8009680 tx=0x7f4ec8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:49.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:48.834+0000 7f4ecea00700 1 -- 172.21.15.1:0/3041764229 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f4ec409cff0 con 0x7f4ed8075f70 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.071+0000 7f4edca00700 1 -- 172.21.15.1:0/3041764229 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f4ed81acf10 con 0x7f4ed8075f70 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.072+0000 7f4ecea00700 1 -- 172.21.15.1:0/3041764229 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4ec409c8f0 con 0x7f4ed8075f70 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.074+0000 7f4edca00700 1 -- 172.21.15.1:0/3041764229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4ebc0a3d50 msgr2=0x7f4ebc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.074+0000 7f4edca00700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4ebc0a3d50 0x7f4ebc0a6210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8009680 tx=0x7f4ec8019040 comp rx=0 tx=0).stop 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.074+0000 7f4edca00700 1 -- 172.21.15.1:0/3041764229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed8075f70 msgr2=0x7f4ed81a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.074+0000 7f4edca00700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed8075f70 0x7f4ed81a06a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e9c0 tx=0x7f4ec400ecd0 comp rx=0 tx=0).stop 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.074+0000 7f4edca00700 1 -- 172.21.15.1:0/3041764229 shutdown_connections 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.074+0000 7f4edca00700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f4ebc0a3d50 0x7f4ebc0a6210 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.074+0000 7f4edca00700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ed8075650 0x7f4ed81a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.075+0000 7f4edca00700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ed81145f0 0x7f4ed81a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.075+0000 7f4edca00700 1 --2- 172.21.15.1:0/3041764229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ed8075f70 0x7f4ed81a06a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.075+0000 7f4edca00700 1 -- 172.21.15.1:0/3041764229 >> 172.21.15.1:0/3041764229 conn(0x7f4ed807a580 msgr2=0x7f4ed8110090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.075+0000 7f4edca00700 1 -- 172.21.15.1:0/3041764229 shutdown_connections 2024-04-26T21:28:49.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:49.075+0000 7f4edca00700 1 -- 172.21.15.1:0/3041764229 wait complete. 2024-04-26T21:28:49.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:49 smithi001 ceph-mon[28054]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:49.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:49 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:49.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:49 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:49.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:49 smithi038 ceph-mon[32724]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:49.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:49 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:49.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:49 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:49.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:49 smithi186 ceph-mon[32991]: pgmap v379: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:49.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:49 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:49.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:49 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:50.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:50 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3041764229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T21:28:50.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:50 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3041764229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T21:28:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:50 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3041764229' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-26T21:28:51.041 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743463 got 296352743470 for osd.9 2024-04-26T21:28:51.041 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:51.312 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:51 smithi001 ceph-mon[28054]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:28:51.312 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:51 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:51.312 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:51 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:51.312 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:51 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:51.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:51 smithi038 ceph-mon[32724]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:28:51.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:51 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:51.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:51 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:51.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:51 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:51.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:51 smithi186 ceph-mon[32991]: pgmap v380: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:28:51.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:51 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:28:51.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:51 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:28:51.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:51 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:28:51.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.640+0000 7f05f4800700 1 -- 172.21.15.1:0/3883827832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05ec10fe20 msgr2=0x7f05ec112210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:51.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.640+0000 7f05f4800700 1 --2- 172.21.15.1:0/3883827832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05ec10fe20 0x7f05ec112210 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009f90 tx=0x7f05e800b330 comp rx=0 tx=0).stop 2024-04-26T21:28:51.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.640+0000 7f05f4800700 1 -- 172.21.15.1:0/3883827832 shutdown_connections 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.640+0000 7f05f4800700 1 --2- 172.21.15.1:0/3883827832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05ec10fe20 0x7f05ec112210 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.640+0000 7f05f4800700 1 --2- 172.21.15.1:0/3883827832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ec075650 0x7f05ec075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.640+0000 7f05f4800700 1 --2- 172.21.15.1:0/3883827832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05ec075f70 0x7f05ec10f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.640+0000 7f05f4800700 1 -- 172.21.15.1:0/3883827832 >> 172.21.15.1:0/3883827832 conn(0x7f05ec07a5d0 msgr2=0x7f05ec07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.640+0000 7f05f4800700 1 -- 172.21.15.1:0/3883827832 shutdown_connections 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.640+0000 7f05f4800700 1 -- 172.21.15.1:0/3883827832 wait complete. 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.642+0000 7f05f4800700 1 Processor -- start 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.642+0000 7f05f4800700 1 -- start start 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.642+0000 7f05f4800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ec075650 0x7f05ec1a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.642+0000 7f05f4800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05ec075f70 0x7f05ec1a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.642+0000 7f05f4800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05ec10fe20 0x7f05ec19bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.642+0000 7f05f4800700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec1158b0 con 0x7f05ec10fe20 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.642+0000 7f05f4800700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec115730 con 0x7f05ec075650 2024-04-26T21:28:51.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.642+0000 7f05f4800700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec115a30 con 0x7f05ec075f70 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05ec075f70 0x7f05ec1a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ec075650 0x7f05ec1a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05ec075f70 0x7f05ec1a1f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:53846/0 (socket says 172.21.15.1:53846) 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f2800700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05ec10fe20 0x7f05ec19bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1400700 1 -- 172.21.15.1:0/2751213277 learned_addr learned my addr 172.21.15.1:0/2751213277 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ec075650 0x7f05ec1a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59410/0 (socket says 172.21.15.1:59410) 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1400700 1 -- 172.21.15.1:0/2751213277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ec075650 msgr2=0x7f05ec1a1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1400700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ec075650 0x7f05ec1a1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1400700 1 -- 172.21.15.1:0/2751213277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05ec10fe20 msgr2=0x7f05ec19bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1400700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05ec10fe20 0x7f05ec19bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1400700 1 -- 172.21.15.1:0/2751213277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ec19c450 con 0x7f05ec075f70 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.643+0000 7f05f1400700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05ec075f70 0x7f05ec1a1f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00ec10 tx=0x7f05dc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.644+0000 7f05e6a00700 1 -- 172.21.15.1:0/2751213277 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc00cc90 con 0x7f05ec075f70 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.644+0000 7f05e6a00700 1 -- 172.21.15.1:0/2751213277 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05dc00ce30 con 0x7f05ec075f70 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.644+0000 7f05f4800700 1 -- 172.21.15.1:0/2751213277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05ec19c740 con 0x7f05ec075f70 2024-04-26T21:28:51.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.644+0000 7f05e6a00700 1 -- 172.21.15.1:0/2751213277 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc010430 con 0x7f05ec075f70 2024-04-26T21:28:51.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.644+0000 7f05f4800700 1 -- 172.21.15.1:0/2751213277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05ec1a89d0 con 0x7f05ec075f70 2024-04-26T21:28:51.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.645+0000 7f05f4800700 1 -- 172.21.15.1:0/2751213277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05ec076d70 con 0x7f05ec075f70 2024-04-26T21:28:51.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.648+0000 7f05e6a00700 1 -- 172.21.15.1:0/2751213277 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f05dc004000 con 0x7f05ec075f70 2024-04-26T21:28:51.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.649+0000 7f05e6a00700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f05d40a3d50 0x7f05d40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:51.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.649+0000 7f05e6a00700 1 -- 172.21.15.1:0/2751213277 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f05dc0d0c80 con 0x7f05ec075f70 2024-04-26T21:28:51.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.649+0000 7f05f1e00700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f05d40a3d50 0x7f05d40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:51.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.650+0000 7f05f1e00700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f05d40a3d50 0x7f05d40a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009c60 tx=0x7f05e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:51.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.651+0000 7f05e6a00700 1 -- 172.21.15.1:0/2751213277 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f05dc09d140 con 0x7f05ec075f70 2024-04-26T21:28:51.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.888+0000 7f05f4800700 1 -- 172.21.15.1:0/2751213277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f05ec02e490 con 0x7f05ec075f70 2024-04-26T21:28:51.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.889+0000 7f05e6a00700 1 -- 172.21.15.1:0/2751213277 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f05dc09ca40 con 0x7f05ec075f70 2024-04-26T21:28:51.891 INFO:teuthology.orchestra.run.smithi001.stdout:167503724614 2024-04-26T21:28:51.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 -- 172.21.15.1:0/2751213277 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f05d40a3d50 msgr2=0x7f05d40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f05d40a3d50 0x7f05d40a6210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f05e0009c60 tx=0x7f05e0009260 comp rx=0 tx=0).stop 2024-04-26T21:28:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 -- 172.21.15.1:0/2751213277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05ec075f70 msgr2=0x7f05ec1a1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05ec075f70 0x7f05ec1a1f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00ec10 tx=0x7f05dc00ef20 comp rx=0 tx=0).stop 2024-04-26T21:28:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 -- 172.21.15.1:0/2751213277 shutdown_connections 2024-04-26T21:28:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f05d40a3d50 0x7f05d40a6210 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05ec10fe20 0x7f05ec19bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05ec075650 0x7f05ec1a1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 --2- 172.21.15.1:0/2751213277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05ec075f70 0x7f05ec1a1f90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:51.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 -- 172.21.15.1:0/2751213277 >> 172.21.15.1:0/2751213277 conn(0x7f05ec07a5d0 msgr2=0x7f05ec1006d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:51.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 -- 172.21.15.1:0/2751213277 shutdown_connections 2024-04-26T21:28:51.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:51.893+0000 7f05f4800700 1 -- 172.21.15.1:0/2751213277 wait complete. 2024-04-26T21:28:52.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:52 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2751213277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T21:28:52.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:52 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2751213277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T21:28:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:52 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2751213277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T21:28:52.863 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724609 got 167503724614 for osd.5 2024-04-26T21:28:52.863 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:53.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:53 smithi001 ceph-mon[28054]: pgmap v381: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:53.795 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:53 smithi186 ceph-mon[32991]: pgmap v381: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:53.796 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:53 smithi038 ceph-mon[32724]: pgmap v381: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:28:53.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.445+0000 7f8e9be00700 1 -- 172.21.15.1:0/3980121984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e9c10fb70 msgr2=0x7f8e9c111fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:53.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.445+0000 7f8e9be00700 1 --2- 172.21.15.1:0/3980121984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e9c10fb70 0x7f8e9c111fe0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8e94009f90 tx=0x7f8e9400b330 comp rx=0 tx=0).stop 2024-04-26T21:28:53.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.446+0000 7f8e9be00700 1 -- 172.21.15.1:0/3980121984 shutdown_connections 2024-04-26T21:28:53.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.446+0000 7f8e9be00700 1 --2- 172.21.15.1:0/3980121984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9c075850 0x7f8e9c075c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:53.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.446+0000 7f8e9be00700 1 --2- 172.21.15.1:0/3980121984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e9c10fb70 0x7f8e9c111fe0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:53.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.446+0000 7f8e9be00700 1 --2- 172.21.15.1:0/3980121984 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c076170 0x7f8e9c10f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:53.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.446+0000 7f8e9be00700 1 -- 172.21.15.1:0/3980121984 >> 172.21.15.1:0/3980121984 conn(0x7f8e9c0fee40 msgr2=0x7f8e9c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:53.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.446+0000 7f8e9be00700 1 -- 172.21.15.1:0/3980121984 shutdown_connections 2024-04-26T21:28:53.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.446+0000 7f8e9be00700 1 -- 172.21.15.1:0/3980121984 wait complete. 2024-04-26T21:28:53.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.447+0000 7f8e9be00700 1 Processor -- start 2024-04-26T21:28:53.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.447+0000 7f8e9be00700 1 -- start start 2024-04-26T21:28:53.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.447+0000 7f8e9be00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e9c075850 0x7f8e9c103290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:53.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.447+0000 7f8e9be00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9c076170 0x7f8e9c1037d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c10fb70 0x7f8e9c103d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9be00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c115710 con 0x7f8e9c10fb70 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9be00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c115590 con 0x7f8e9c075850 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9be00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e9c115890 con 0x7f8e9c076170 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9aa00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e9c075850 0x7f8e9c103290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9aa00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e9c075850 0x7f8e9c103290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59436/0 (socket says 172.21.15.1:59436) 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9aa00700 1 -- 172.21.15.1:0/1635899473 learned_addr learned my addr 172.21.15.1:0/1635899473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9b400700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c10fb70 0x7f8e9c103d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9a000700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9c076170 0x7f8e9c1037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9aa00700 1 -- 172.21.15.1:0/1635899473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9c076170 msgr2=0x7f8e9c1037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:53.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9aa00700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9c076170 0x7f8e9c1037d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9aa00700 1 -- 172.21.15.1:0/1635899473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c10fb70 msgr2=0x7f8e9c103d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9aa00700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c10fb70 0x7f8e9c103d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.448+0000 7f8e9aa00700 1 -- 172.21.15.1:0/1635899473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e9c1ac290 con 0x7f8e9c075850 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.449+0000 7f8e9aa00700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e9c075850 0x7f8e9c103290 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00e9f0 tx=0x7f8e8c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.449+0000 7f8e93400700 1 -- 172.21.15.1:0/1635899473 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c00cbd0 con 0x7f8e9c075850 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.449+0000 7f8e93400700 1 -- 172.21.15.1:0/1635899473 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e8c00ee90 con 0x7f8e9c075850 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.449+0000 7f8e93400700 1 -- 172.21.15.1:0/1635899473 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8c005590 con 0x7f8e9c075850 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.449+0000 7f8e9be00700 1 -- 172.21.15.1:0/1635899473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e9c1ac580 con 0x7f8e9c075850 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.449+0000 7f8e9be00700 1 -- 172.21.15.1:0/1635899473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e9c1aca70 con 0x7f8e9c075850 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.450+0000 7f8e9be00700 1 -- 172.21.15.1:0/1635899473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e9c076f00 con 0x7f8e9c075850 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.452+0000 7f8e93400700 1 -- 172.21.15.1:0/1635899473 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8e8c004750 con 0x7f8e9c075850 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.454+0000 7f8e93400700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e800a3cd0 0x7f8e800a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:53.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.454+0000 7f8e93400700 1 -- 172.21.15.1:0/1635899473 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f8e8c0d0d30 con 0x7f8e9c075850 2024-04-26T21:28:53.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.454+0000 7f8e9a000700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e800a3cd0 0x7f8e800a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:53.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.455+0000 7f8e9a000700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e800a3cd0 0x7f8e800a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c104a50 tx=0x7f8e880093e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:53.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.456+0000 7f8e93400700 1 -- 172.21.15.1:0/1635899473 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f8e8c09d1f0 con 0x7f8e9c075850 2024-04-26T21:28:53.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.689+0000 7f8e9be00700 1 -- 172.21.15.1:0/1635899473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f8e9c1047c0 con 0x7f8e9c075850 2024-04-26T21:28:53.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.690+0000 7f8e93400700 1 -- 172.21.15.1:0/1635899473 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8e8c09caf0 con 0x7f8e9c075850 2024-04-26T21:28:53.805 INFO:teuthology.orchestra.run.smithi001.stdout:261993005110 2024-04-26T21:28:53.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 -- 172.21.15.1:0/1635899473 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e800a3cd0 msgr2=0x7f8e800a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:53.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e800a3cd0 0x7f8e800a6190 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c104a50 tx=0x7f8e880093e0 comp rx=0 tx=0).stop 2024-04-26T21:28:53.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 -- 172.21.15.1:0/1635899473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e9c075850 msgr2=0x7f8e9c103290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:53.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e9c075850 0x7f8e9c103290 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c00e9f0 tx=0x7f8e8c00ed00 comp rx=0 tx=0).stop 2024-04-26T21:28:53.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 -- 172.21.15.1:0/1635899473 shutdown_connections 2024-04-26T21:28:53.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f8e800a3cd0 0x7f8e800a6190 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:53.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e9c10fb70 0x7f8e9c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:53.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8e9c075850 0x7f8e9c103290 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:53.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 --2- 172.21.15.1:0/1635899473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e9c076170 0x7f8e9c1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:53.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 -- 172.21.15.1:0/1635899473 >> 172.21.15.1:0/1635899473 conn(0x7f8e9c0fee40 msgr2=0x7f8e9c0ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:53.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 -- 172.21.15.1:0/1635899473 shutdown_connections 2024-04-26T21:28:53.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:53.693+0000 7f8e9be00700 1 -- 172.21.15.1:0/1635899473 wait complete. 2024-04-26T21:28:54.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:54 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1635899473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T21:28:54.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:54 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1635899473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T21:28:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:54 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1635899473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-26T21:28:54.553 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005103 got 261993005110 for osd.8 2024-04-26T21:28:54.553 DEBUG:teuthology.parallel:result is None 2024-04-26T21:28:54.553 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T21:28:54.553 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T21:28:55.036 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:55.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:55 smithi001 ceph-mon[28054]: pgmap v382: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:55.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:55 smithi038 ceph-mon[32724]: pgmap v382: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:55.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:55 smithi186 ceph-mon[32991]: pgmap v382: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:28:56.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]: dispatch 2024-04-26T21:28:56.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7, 4, 6]}]: dispatch 2024-04-26T21:28:56.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-04-26T21:28:56.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-26T21:28:56.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]: dispatch 2024-04-26T21:28:56.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-26T21:28:56.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-04-26T21:28:56.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]: dispatch 2024-04-26T21:28:56.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7, 4, 6]}]: dispatch 2024-04-26T21:28:56.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-04-26T21:28:56.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-26T21:28:56.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]: dispatch 2024-04-26T21:28:56.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-26T21:28:56.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-04-26T21:28:56.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]: dispatch 2024-04-26T21:28:56.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7, 4, 6]}]: dispatch 2024-04-26T21:28:56.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-04-26T21:28:56.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-04-26T21:28:56.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]: dispatch 2024-04-26T21:28:56.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-04-26T21:28:56.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-04-26T21:28:57.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.379+0000 7fe471e00700 1 -- 172.21.15.1:0/2876087722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe46c105630 msgr2=0x7fe46c105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:57.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.379+0000 7fe471e00700 1 --2- 172.21.15.1:0/2876087722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe46c105630 0x7fe46c105ab0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe458009990 tx=0x7fe458009ca0 comp rx=0 tx=0).stop 2024-04-26T21:28:57.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.379+0000 7fe471e00700 1 -- 172.21.15.1:0/2876087722 shutdown_connections 2024-04-26T21:28:57.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.379+0000 7fe471e00700 1 --2- 172.21.15.1:0/2876087722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe46c105ff0 0x7fe46c112500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:57.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.379+0000 7fe471e00700 1 --2- 172.21.15.1:0/2876087722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe46c10b630 0x7fe46c10ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:57.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.379+0000 7fe471e00700 1 --2- 172.21.15.1:0/2876087722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe46c105630 0x7fe46c105ab0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:57.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.379+0000 7fe471e00700 1 -- 172.21.15.1:0/2876087722 >> 172.21.15.1:0/2876087722 conn(0x7fe46c101120 msgr2=0x7fe46c103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:57.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.379+0000 7fe471e00700 1 -- 172.21.15.1:0/2876087722 shutdown_connections 2024-04-26T21:28:57.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:57 smithi001 ceph-mon[28054]: pgmap v383: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:28:57.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]': finished 2024-04-26T21:28:57.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7, 4, 6]}]': finished 2024-04-26T21:28:57.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-04-26T21:28:57.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-26T21:28:57.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]': finished 2024-04-26T21:28:57.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-26T21:28:57.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-04-26T21:28:57.381 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:57 smithi001 ceph-mon[28054]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T21:28:57.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.379+0000 7fe471e00700 1 -- 172.21.15.1:0/2876087722 wait complete. 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.381+0000 7fe471e00700 1 Processor -- start 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.381+0000 7fe471e00700 1 -- start start 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe471e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe46c105630 0x7fe46c1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe471e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe46c105ff0 0x7fe46c1a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe471e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe46c10b630 0x7fe46c19bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe471e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c115810 con 0x7fe46c10b630 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe471e00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c115690 con 0x7fe46c105630 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe471e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe46c115990 con 0x7fe46c105ff0 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe46b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe46c10b630 0x7fe46c19bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe46aa00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe46c105630 0x7fe46c1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe46a000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe46c105ff0 0x7fe46c1a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:57.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe46a000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe46c105ff0 0x7fe46c1a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:53878/0 (socket says 172.21.15.1:53878) 2024-04-26T21:28:57.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe46aa00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe46c105630 0x7fe46c1a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59448/0 (socket says 172.21.15.1:59448) 2024-04-26T21:28:57.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe46a000700 1 -- 172.21.15.1:0/1237322500 learned_addr learned my addr 172.21.15.1:0/1237322500 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:28:57.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe46b400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe46c10b630 0x7fe46c19bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50754/0 (socket says 172.21.15.1:50754) 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe46aa00700 1 -- 172.21.15.1:0/1237322500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe46c105ff0 msgr2=0x7fe46c1a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe46aa00700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe46c105ff0 0x7fe46c1a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.382+0000 7fe46aa00700 1 -- 172.21.15.1:0/1237322500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe46c10b630 msgr2=0x7fe46c19bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.383+0000 7fe46aa00700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe46c10b630 0x7fe46c19bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.383+0000 7fe46aa00700 1 -- 172.21.15.1:0/1237322500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe46c19c2b0 con 0x7fe46c105630 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.383+0000 7fe46b400700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe46c10b630 0x7fe46c19bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.383+0000 7fe46aa00700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe46c105630 0x7fe46c1a1940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe45c00e9f0 tx=0x7fe45c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.383+0000 7fe470a00700 1 -- 172.21.15.1:0/1237322500 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c00c9c0 con 0x7fe46c105630 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.383+0000 7fe470a00700 1 -- 172.21.15.1:0/1237322500 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe45c00cb60 con 0x7fe46c105630 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.383+0000 7fe470a00700 1 -- 172.21.15.1:0/1237322500 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe45c018680 con 0x7fe46c105630 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.383+0000 7fe471e00700 1 -- 172.21.15.1:0/1237322500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe46c19c5a0 con 0x7fe46c105630 2024-04-26T21:28:57.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.383+0000 7fe471e00700 1 -- 172.21.15.1:0/1237322500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe46c1a8810 con 0x7fe46c105630 2024-04-26T21:28:57.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.385+0000 7fe471e00700 1 -- 172.21.15.1:0/1237322500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe46c06a110 con 0x7fe46c105630 2024-04-26T21:28:57.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.387+0000 7fe470a00700 1 -- 172.21.15.1:0/1237322500 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fe45c010430 con 0x7fe46c105630 2024-04-26T21:28:57.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.389+0000 7fe470a00700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe4500a3d50 0x7fe4500a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:28:57.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.389+0000 7fe470a00700 1 -- 172.21.15.1:0/1237322500 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fe45c0d0d80 con 0x7fe46c105630 2024-04-26T21:28:57.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.389+0000 7fe46a000700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe4500a3d50 0x7fe4500a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:28:57.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.390+0000 7fe46a000700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe4500a3d50 0x7fe4500a6210 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe46c1037b0 tx=0x7fe458009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:28:57.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.391+0000 7fe470a00700 1 -- 172.21.15.1:0/1237322500 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe45c09d1a0 con 0x7fe46c105630 2024-04-26T21:28:57.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:57 smithi186 ceph-mon[32991]: pgmap v383: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:28:57.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]': finished 2024-04-26T21:28:57.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7, 4, 6]}]': finished 2024-04-26T21:28:57.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-04-26T21:28:57.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-26T21:28:57.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]': finished 2024-04-26T21:28:57.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-26T21:28:57.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-04-26T21:28:57.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:57 smithi186 ceph-mon[32991]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T21:28:57.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:57 smithi038 ceph-mon[32724]: pgmap v383: 129 pgs: 129 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:28:57.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8, 5, 9]}]': finished 2024-04-26T21:28:57.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7, 4, 6]}]': finished 2024-04-26T21:28:57.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-04-26T21:28:57.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-04-26T21:28:57.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [5, 10]}]': finished 2024-04-26T21:28:57.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-04-26T21:28:57.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-04-26T21:28:57.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:57 smithi038 ceph-mon[32724]: osdmap e94: 12 total, 12 up, 12 in 2024-04-26T21:28:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.621+0000 7fe471e00700 1 -- 172.21.15.1:0/1237322500 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe46c063860 con 0x7fe4500a3d50 2024-04-26T21:28:57.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.641+0000 7fe470a00700 1 -- 172.21.15.1:0/1237322500 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340276 (secure 0 0 0) 0x7fe46c063860 con 0x7fe4500a3d50 2024-04-26T21:28:57.645 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:28:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 -- 172.21.15.1:0/1237322500 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe4500a3d50 msgr2=0x7fe4500a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe4500a3d50 0x7fe4500a6210 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe46c1037b0 tx=0x7fe458009510 comp rx=0 tx=0).stop 2024-04-26T21:28:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 -- 172.21.15.1:0/1237322500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe46c105630 msgr2=0x7fe46c1a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:28:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe46c105630 0x7fe46c1a1940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe45c00e9f0 tx=0x7fe45c00ed00 comp rx=0 tx=0).stop 2024-04-26T21:28:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 -- 172.21.15.1:0/1237322500 shutdown_connections 2024-04-26T21:28:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe4500a3d50 0x7fe4500a6210 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe46c10b630 0x7fe46c19bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe46c105630 0x7fe46c1a1940 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 --2- 172.21.15.1:0/1237322500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe46c105ff0 0x7fe46c1a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:28:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 -- 172.21.15.1:0/1237322500 >> 172.21.15.1:0/1237322500 conn(0x7fe46c101120 msgr2=0x7fe46c101930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:28:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 -- 172.21.15.1:0/1237322500 shutdown_connections 2024-04-26T21:28:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:28:57.649+0000 7fe471e00700 1 -- 172.21.15.1:0/1237322500 wait complete. 2024-04-26T21:28:57.653 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T21:28:58.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:58 smithi001 ceph-mon[28054]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T21:28:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:58 smithi186 ceph-mon[32991]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T21:28:58.461 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-04-26T21:28:57.352806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649888,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":674,"num_read_kb":449,"num_write":344,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180758,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380212,"kb_used_data":14344,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561612,"statfs":{"total":1151940427776,"available":1151551090688,"internally_reserved":0,"allocated":14688256,"data_stored":8055818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004118"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839803+0000","last_change":"2024-04-26T21:27:02.854724+0000","last_active":"2024-04-26T21:27:56.839803+0000","last_peered":"2024-04-26T21:27:56.839803+0000","last_clean":"2024-04-26T21:27:56.839803+0000","last_became_active":"2024-04-26T21:27:02.854125+0000","last_became_peered":"2024-04-26T21:27:02.854125+0000","last_unstale":"2024-04-26T21:27:56.839803+0000","last_undegraded":"2024-04-26T21:27:56.839803+0000","last_fullsized":"2024-04-26T21:27:56.839803+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:16.500220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.085070+0000","last_change":"2024-04-26T21:27:02.852846+0000","last_active":"2024-04-26T21:28:56.085070+0000","last_peered":"2024-04-26T21:28:56.085070+0000","last_clean":"2024-04-26T21:28:56.085070+0000","last_became_active":"2024-04-26T21:27:02.851243+0000","last_became_peered":"2024-04-26T21:27:02.851243+0000","last_unstale":"2024-04-26T21:28:56.085070+0000","last_undegraded":"2024-04-26T21:28:56.085070+0000","last_fullsized":"2024-04-26T21:28:56.085070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:09:11.240955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.086854+0000","last_change":"2024-04-26T21:27:02.855856+0000","last_active":"2024-04-26T21:28:56.086854+0000","last_peered":"2024-04-26T21:28:56.086854+0000","last_clean":"2024-04-26T21:28:56.086854+0000","last_became_active":"2024-04-26T21:27:02.855446+0000","last_became_peered":"2024-04-26T21:27:02.855446+0000","last_unstale":"2024-04-26T21:28:56.086854+0000","last_undegraded":"2024-04-26T21:28:56.086854+0000","last_fullsized":"2024-04-26T21:28:56.086854+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:35.183449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833873+0000","last_change":"2024-04-26T21:27:02.860949+0000","last_active":"2024-04-26T21:27:56.833873+0000","last_peered":"2024-04-26T21:27:56.833873+0000","last_clean":"2024-04-26T21:27:56.833873+0000","last_became_active":"2024-04-26T21:27:02.860567+0000","last_became_peered":"2024-04-26T21:27:02.860567+0000","last_unstale":"2024-04-26T21:27:56.833873+0000","last_undegraded":"2024-04-26T21:27:56.833873+0000","last_fullsized":"2024-04-26T21:27:56.833873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:09.851885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839655+0000","last_change":"2024-04-26T21:27:02.876019+0000","last_active":"2024-04-26T21:27:56.839655+0000","last_peered":"2024-04-26T21:27:56.839655+0000","last_clean":"2024-04-26T21:27:56.839655+0000","last_became_active":"2024-04-26T21:27:02.875652+0000","last_became_peered":"2024-04-26T21:27:02.875652+0000","last_unstale":"2024-04-26T21:27:56.839655+0000","last_undegraded":"2024-04-26T21:27:56.839655+0000","last_fullsized":"2024-04-26T21:27:56.839655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:35.017787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.843939+0000","last_change":"2024-04-26T21:27:02.860229+0000","last_active":"2024-04-26T21:27:56.843939+0000","last_peered":"2024-04-26T21:27:56.843939+0000","last_clean":"2024-04-26T21:27:56.843939+0000","last_became_active":"2024-04-26T21:27:02.859863+0000","last_became_peered":"2024-04-26T21:27:02.859863+0000","last_unstale":"2024-04-26T21:27:56.843939+0000","last_undegraded":"2024-04-26T21:27:56.843939+0000","last_fullsized":"2024-04-26T21:27:56.843939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:47.337073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835778+0000","last_change":"2024-04-26T21:27:02.863253+0000","last_active":"2024-04-26T21:27:56.835778+0000","last_peered":"2024-04-26T21:27:56.835778+0000","last_clean":"2024-04-26T21:27:56.835778+0000","last_became_active":"2024-04-26T21:27:02.861623+0000","last_became_peered":"2024-04-26T21:27:02.861623+0000","last_unstale":"2024-04-26T21:27:56.835778+0000","last_undegraded":"2024-04-26T21:27:56.835778+0000","last_fullsized":"2024-04-26T21:27:56.835778+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:57.642504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838500+0000","last_change":"2024-04-26T21:27:02.860822+0000","last_active":"2024-04-26T21:27:56.838500+0000","last_peered":"2024-04-26T21:27:56.838500+0000","last_clean":"2024-04-26T21:27:56.838500+0000","last_became_active":"2024-04-26T21:27:02.860482+0000","last_became_peered":"2024-04-26T21:27:02.860482+0000","last_unstale":"2024-04-26T21:27:56.838500+0000","last_undegraded":"2024-04-26T21:27:56.838500+0000","last_fullsized":"2024-04-26T21:27:56.838500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:59.224624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.856389+0000","last_change":"2024-04-26T21:27:56.856389+0000","last_active":"2024-04-26T21:27:56.856389+0000","last_peered":"2024-04-26T21:27:56.856389+0000","last_clean":"2024-04-26T21:27:56.856389+0000","last_became_active":"2024-04-26T21:27:56.856008+0000","last_became_peered":"2024-04-26T21:27:56.856008+0000","last_unstale":"2024-04-26T21:27:56.856389+0000","last_undegraded":"2024-04-26T21:27:56.856389+0000","last_fullsized":"2024-04-26T21:27:56.856389+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:04.770535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.844413+0000","last_change":"2024-04-26T21:27:02.860545+0000","last_active":"2024-04-26T21:27:56.844413+0000","last_peered":"2024-04-26T21:27:56.844413+0000","last_clean":"2024-04-26T21:27:56.844413+0000","last_became_active":"2024-04-26T21:27:02.858577+0000","last_became_peered":"2024-04-26T21:27:02.858577+0000","last_unstale":"2024-04-26T21:27:56.844413+0000","last_undegraded":"2024-04-26T21:27:56.844413+0000","last_fullsized":"2024-04-26T21:27:56.844413+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:43.390592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836456+0000","last_change":"2024-04-26T21:27:02.858685+0000","last_active":"2024-04-26T21:27:56.836456+0000","last_peered":"2024-04-26T21:27:56.836456+0000","last_clean":"2024-04-26T21:27:56.836456+0000","last_became_active":"2024-04-26T21:27:02.857054+0000","last_became_peered":"2024-04-26T21:27:02.857054+0000","last_unstale":"2024-04-26T21:27:56.836456+0000","last_undegraded":"2024-04-26T21:27:56.836456+0000","last_fullsized":"2024-04-26T21:27:56.836456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:29.406132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833813+0000","last_change":"2024-04-26T21:27:02.852457+0000","last_active":"2024-04-26T21:27:56.833813+0000","last_peered":"2024-04-26T21:27:56.833813+0000","last_clean":"2024-04-26T21:27:56.833813+0000","last_became_active":"2024-04-26T21:27:02.852064+0000","last_became_peered":"2024-04-26T21:27:02.852064+0000","last_unstale":"2024-04-26T21:27:56.833813+0000","last_undegraded":"2024-04-26T21:27:56.833813+0000","last_fullsized":"2024-04-26T21:27:56.833813+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:16.798413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.855344+0000","last_change":"2024-04-26T21:27:02.872618+0000","last_active":"2024-04-26T21:27:56.855344+0000","last_peered":"2024-04-26T21:27:56.855344+0000","last_clean":"2024-04-26T21:27:56.855344+0000","last_became_active":"2024-04-26T21:27:02.872270+0000","last_became_peered":"2024-04-26T21:27:02.872270+0000","last_unstale":"2024-04-26T21:27:56.855344+0000","last_undegraded":"2024-04-26T21:27:56.855344+0000","last_fullsized":"2024-04-26T21:27:56.855344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:45.828419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.084143+0000","last_change":"2024-04-26T21:27:02.868590+0000","last_active":"2024-04-26T21:28:56.084143+0000","last_peered":"2024-04-26T21:28:56.084143+0000","last_clean":"2024-04-26T21:28:56.084143+0000","last_became_active":"2024-04-26T21:27:02.866641+0000","last_became_peered":"2024-04-26T21:27:02.866641+0000","last_unstale":"2024-04-26T21:28:56.084143+0000","last_undegraded":"2024-04-26T21:28:56.084143+0000","last_fullsized":"2024-04-26T21:28:56.084143+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:17.313420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.084420+0000","last_change":"2024-04-26T21:27:02.867735+0000","last_active":"2024-04-26T21:28:56.084420+0000","last_peered":"2024-04-26T21:28:56.084420+0000","last_clean":"2024-04-26T21:28:56.084420+0000","last_became_active":"2024-04-26T21:27:02.867217+0000","last_became_peered":"2024-04-26T21:27:02.867217+0000","last_unstale":"2024-04-26T21:28:56.084420+0000","last_undegraded":"2024-04-26T21:28:56.084420+0000","last_fullsized":"2024-04-26T21:28:56.084420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:04:45.533220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838463+0000","last_change":"2024-04-26T21:27:02.857541+0000","last_active":"2024-04-26T21:27:56.838463+0000","last_peered":"2024-04-26T21:27:56.838463+0000","last_clean":"2024-04-26T21:27:56.838463+0000","last_became_active":"2024-04-26T21:27:02.855563+0000","last_became_peered":"2024-04-26T21:27:02.855563+0000","last_unstale":"2024-04-26T21:27:56.838463+0000","last_undegraded":"2024-04-26T21:27:56.838463+0000","last_fullsized":"2024-04-26T21:27:56.838463+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:39.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839687+0000","last_change":"2024-04-26T21:27:02.854461+0000","last_active":"2024-04-26T21:27:56.839687+0000","last_peered":"2024-04-26T21:27:56.839687+0000","last_clean":"2024-04-26T21:27:56.839687+0000","last_became_active":"2024-04-26T21:27:02.853691+0000","last_became_peered":"2024-04-26T21:27:02.853691+0000","last_unstale":"2024-04-26T21:27:56.839687+0000","last_undegraded":"2024-04-26T21:27:56.839687+0000","last_fullsized":"2024-04-26T21:27:56.839687+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:09:48.284180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.096142+0000","last_change":"2024-04-26T21:27:02.864308+0000","last_active":"2024-04-26T21:28:56.096142+0000","last_peered":"2024-04-26T21:28:56.096142+0000","last_clean":"2024-04-26T21:28:56.096142+0000","last_became_active":"2024-04-26T21:27:02.863928+0000","last_became_peered":"2024-04-26T21:27:02.863928+0000","last_unstale":"2024-04-26T21:28:56.096142+0000","last_undegraded":"2024-04-26T21:28:56.096142+0000","last_fullsized":"2024-04-26T21:28:56.096142+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:12.169394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836060+0000","last_change":"2024-04-26T21:27:02.876044+0000","last_active":"2024-04-26T21:27:56.836060+0000","last_peered":"2024-04-26T21:27:56.836060+0000","last_clean":"2024-04-26T21:27:56.836060+0000","last_became_active":"2024-04-26T21:27:02.875681+0000","last_became_peered":"2024-04-26T21:27:02.875681+0000","last_unstale":"2024-04-26T21:27:56.836060+0000","last_undegraded":"2024-04-26T21:27:56.836060+0000","last_fullsized":"2024-04-26T21:27:56.836060+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:26.246916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835772+0000","last_change":"2024-04-26T21:27:07.374736+0000","last_active":"2024-04-26T21:27:56.835772+0000","last_peered":"2024-04-26T21:27:56.835772+0000","last_clean":"2024-04-26T21:27:56.835772+0000","last_became_active":"2024-04-26T21:27:07.374116+0000","last_became_peered":"2024-04-26T21:27:07.374116+0000","last_unstale":"2024-04-26T21:27:56.835772+0000","last_undegraded":"2024-04-26T21:27:56.835772+0000","last_fullsized":"2024-04-26T21:27:56.835772+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:11.450926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.093657+0000","last_change":"2024-04-26T21:27:02.302510+0000","last_active":"2024-04-26T21:28:56.093657+0000","last_peered":"2024-04-26T21:28:56.093657+0000","last_clean":"2024-04-26T21:28:56.093657+0000","last_became_active":"2024-04-26T21:27:02.302088+0000","last_became_peered":"2024-04-26T21:27:02.302088+0000","last_unstale":"2024-04-26T21:28:56.093657+0000","last_undegraded":"2024-04-26T21:28:56.093657+0000","last_fullsized":"2024-04-26T21:28:56.093657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:43.007722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840502+0000","last_change":"2024-04-26T21:27:02.861455+0000","last_active":"2024-04-26T21:27:56.840502+0000","last_peered":"2024-04-26T21:27:56.840502+0000","last_clean":"2024-04-26T21:27:56.840502+0000","last_became_active":"2024-04-26T21:27:02.861107+0000","last_became_peered":"2024-04-26T21:27:02.861107+0000","last_unstale":"2024-04-26T21:27:56.840502+0000","last_undegraded":"2024-04-26T21:27:56.840502+0000","last_fullsized":"2024-04-26T21:27:56.840502+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:10:18.382881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833137+0000","last_change":"2024-04-26T21:27:07.377797+0000","last_active":"2024-04-26T21:27:56.833137+0000","last_peered":"2024-04-26T21:27:56.833137+0000","last_clean":"2024-04-26T21:27:56.833137+0000","last_became_active":"2024-04-26T21:27:07.377314+0000","last_became_peered":"2024-04-26T21:27:07.377314+0000","last_unstale":"2024-04-26T21:27:56.833137+0000","last_undegraded":"2024-04-26T21:27:56.833137+0000","last_fullsized":"2024-04-26T21:27:56.833137+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:02:34.761034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840568+0000","last_change":"2024-04-26T21:27:02.309488+0000","last_active":"2024-04-26T21:27:56.840568+0000","last_peered":"2024-04-26T21:27:56.840568+0000","last_clean":"2024-04-26T21:27:56.840568+0000","last_became_active":"2024-04-26T21:27:02.308592+0000","last_became_peered":"2024-04-26T21:27:02.308592+0000","last_unstale":"2024-04-26T21:27:56.840568+0000","last_undegraded":"2024-04-26T21:27:56.840568+0000","last_fullsized":"2024-04-26T21:27:56.840568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:36.563847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.087039+0000","last_change":"2024-04-26T21:27:02.863204+0000","last_active":"2024-04-26T21:28:56.087039+0000","last_peered":"2024-04-26T21:28:56.087039+0000","last_clean":"2024-04-26T21:28:56.087039+0000","last_became_active":"2024-04-26T21:27:02.862466+0000","last_became_peered":"2024-04-26T21:27:02.862466+0000","last_unstale":"2024-04-26T21:28:56.087039+0000","last_undegraded":"2024-04-26T21:28:56.087039+0000","last_fullsized":"2024-04-26T21:28:56.087039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:29.360830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833314+0000","last_change":"2024-04-26T21:27:07.378032+0000","last_active":"2024-04-26T21:27:56.833314+0000","last_peered":"2024-04-26T21:27:56.833314+0000","last_clean":"2024-04-26T21:27:56.833314+0000","last_became_active":"2024-04-26T21:27:07.377255+0000","last_became_peered":"2024-04-26T21:27:07.377255+0000","last_unstale":"2024-04-26T21:27:56.833314+0000","last_undegraded":"2024-04-26T21:27:56.833314+0000","last_fullsized":"2024-04-26T21:27:56.833314+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:18.989977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.850816+0000","last_change":"2024-04-26T21:27:56.850816+0000","last_active":"2024-04-26T21:27:56.850816+0000","last_peered":"2024-04-26T21:27:56.850816+0000","last_clean":"2024-04-26T21:27:56.850816+0000","last_became_active":"2024-04-26T21:27:56.843299+0000","last_became_peered":"2024-04-26T21:27:56.843299+0000","last_unstale":"2024-04-26T21:27:56.850816+0000","last_undegraded":"2024-04-26T21:27:56.850816+0000","last_fullsized":"2024-04-26T21:27:56.850816+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:21:38.864693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839003+0000","last_change":"2024-04-26T21:27:02.876447+0000","last_active":"2024-04-26T21:27:56.839003+0000","last_peered":"2024-04-26T21:27:56.839003+0000","last_clean":"2024-04-26T21:27:56.839003+0000","last_became_active":"2024-04-26T21:27:02.876017+0000","last_became_peered":"2024-04-26T21:27:02.876017+0000","last_unstale":"2024-04-26T21:27:56.839003+0000","last_undegraded":"2024-04-26T21:27:56.839003+0000","last_fullsized":"2024-04-26T21:27:56.839003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:57:05.453787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833168+0000","last_change":"2024-04-26T21:27:07.373700+0000","last_active":"2024-04-26T21:27:56.833168+0000","last_peered":"2024-04-26T21:27:56.833168+0000","last_clean":"2024-04-26T21:27:56.833168+0000","last_became_active":"2024-04-26T21:27:07.373255+0000","last_became_peered":"2024-04-26T21:27:07.373255+0000","last_unstale":"2024-04-26T21:27:56.833168+0000","last_undegraded":"2024-04-26T21:27:56.833168+0000","last_fullsized":"2024-04-26T21:27:56.833168+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:48.517197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840466+0000","last_change":"2024-04-26T21:27:02.300817+0000","last_active":"2024-04-26T21:27:56.840466+0000","last_peered":"2024-04-26T21:27:56.840466+0000","last_clean":"2024-04-26T21:27:56.840466+0000","last_became_active":"2024-04-26T21:27:02.300493+0000","last_became_peered":"2024-04-26T21:27:02.300493+0000","last_unstale":"2024-04-26T21:27:56.840466+0000","last_undegraded":"2024-04-26T21:27:56.840466+0000","last_fullsized":"2024-04-26T21:27:56.840466+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:18.783104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089662+0000","last_change":"2024-04-26T21:27:02.859982+0000","last_active":"2024-04-26T21:28:57.089662+0000","last_peered":"2024-04-26T21:28:57.089662+0000","last_clean":"2024-04-26T21:28:57.089662+0000","last_became_active":"2024-04-26T21:27:02.857204+0000","last_became_peered":"2024-04-26T21:27:02.857204+0000","last_unstale":"2024-04-26T21:28:57.089662+0000","last_undegraded":"2024-04-26T21:28:57.089662+0000","last_fullsized":"2024-04-26T21:28:57.089662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:22.423956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835920+0000","last_change":"2024-04-26T21:27:07.366039+0000","last_active":"2024-04-26T21:27:56.835920+0000","last_peered":"2024-04-26T21:27:56.835920+0000","last_clean":"2024-04-26T21:27:56.835920+0000","last_became_active":"2024-04-26T21:27:07.365560+0000","last_became_peered":"2024-04-26T21:27:07.365560+0000","last_unstale":"2024-04-26T21:27:56.835920+0000","last_undegraded":"2024-04-26T21:27:56.835920+0000","last_fullsized":"2024-04-26T21:27:56.835920+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:47.833476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":127,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089027+0000","last_change":"2024-04-26T21:27:02.306364+0000","last_active":"2024-04-26T21:28:57.089027+0000","last_peered":"2024-04-26T21:28:57.089027+0000","last_clean":"2024-04-26T21:28:57.089027+0000","last_became_active":"2024-04-26T21:27:02.306041+0000","last_became_peered":"2024-04-26T21:27:02.306041+0000","last_unstale":"2024-04-26T21:28:57.089027+0000","last_undegraded":"2024-04-26T21:28:57.089027+0000","last_fullsized":"2024-04-26T21:28:57.089027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:44.501916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840481+0000","last_change":"2024-04-26T21:27:02.859346+0000","last_active":"2024-04-26T21:27:56.840481+0000","last_peered":"2024-04-26T21:27:56.840481+0000","last_clean":"2024-04-26T21:27:56.840481+0000","last_became_active":"2024-04-26T21:27:02.859025+0000","last_became_peered":"2024-04-26T21:27:02.859025+0000","last_unstale":"2024-04-26T21:27:56.840481+0000","last_undegraded":"2024-04-26T21:27:56.840481+0000","last_fullsized":"2024-04-26T21:27:56.840481+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:06.258512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836808+0000","last_change":"2024-04-26T21:27:07.382349+0000","last_active":"2024-04-26T21:27:56.836808+0000","last_peered":"2024-04-26T21:27:56.836808+0000","last_clean":"2024-04-26T21:27:56.836808+0000","last_became_active":"2024-04-26T21:27:07.381930+0000","last_became_peered":"2024-04-26T21:27:07.381930+0000","last_unstale":"2024-04-26T21:27:56.836808+0000","last_undegraded":"2024-04-26T21:27:56.836808+0000","last_fullsized":"2024-04-26T21:27:56.836808+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:15:25.970573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.862413+0000","last_change":"2024-04-26T21:27:56.862413+0000","last_active":"2024-04-26T21:27:56.862413+0000","last_peered":"2024-04-26T21:27:56.862413+0000","last_clean":"2024-04-26T21:27:56.862413+0000","last_became_active":"2024-04-26T21:27:56.859184+0000","last_became_peered":"2024-04-26T21:27:56.859184+0000","last_unstale":"2024-04-26T21:27:56.862413+0000","last_undegraded":"2024-04-26T21:27:56.862413+0000","last_fullsized":"2024-04-26T21:27:56.862413+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:54:17.701601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089353+0000","last_change":"2024-04-26T21:27:02.865707+0000","last_active":"2024-04-26T21:28:57.089353+0000","last_peered":"2024-04-26T21:28:57.089353+0000","last_clean":"2024-04-26T21:28:57.089353+0000","last_became_active":"2024-04-26T21:27:02.865436+0000","last_became_peered":"2024-04-26T21:27:02.865436+0000","last_unstale":"2024-04-26T21:28:57.089353+0000","last_undegraded":"2024-04-26T21:28:57.089353+0000","last_fullsized":"2024-04-26T21:28:57.089353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:53.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839552+0000","last_change":"2024-04-26T21:27:07.379092+0000","last_active":"2024-04-26T21:27:56.839552+0000","last_peered":"2024-04-26T21:27:56.839552+0000","last_clean":"2024-04-26T21:27:56.839552+0000","last_became_active":"2024-04-26T21:27:07.378716+0000","last_became_peered":"2024-04-26T21:27:07.378716+0000","last_unstale":"2024-04-26T21:27:56.839552+0000","last_undegraded":"2024-04-26T21:27:56.839552+0000","last_fullsized":"2024-04-26T21:27:56.839552+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:18.906224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.852605+0000","last_change":"2024-04-26T21:27:02.307510+0000","last_active":"2024-04-26T21:27:56.852605+0000","last_peered":"2024-04-26T21:27:56.852605+0000","last_clean":"2024-04-26T21:27:56.852605+0000","last_became_active":"2024-04-26T21:27:02.306932+0000","last_became_peered":"2024-04-26T21:27:02.306932+0000","last_unstale":"2024-04-26T21:27:56.852605+0000","last_undegraded":"2024-04-26T21:27:56.852605+0000","last_fullsized":"2024-04-26T21:27:56.852605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:21:34.217767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838207+0000","last_change":"2024-04-26T21:27:02.874246+0000","last_active":"2024-04-26T21:27:56.838207+0000","last_peered":"2024-04-26T21:27:56.838207+0000","last_clean":"2024-04-26T21:27:56.838207+0000","last_became_active":"2024-04-26T21:27:02.873880+0000","last_became_peered":"2024-04-26T21:27:02.873880+0000","last_unstale":"2024-04-26T21:27:56.838207+0000","last_undegraded":"2024-04-26T21:27:56.838207+0000","last_fullsized":"2024-04-26T21:27:56.838207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:21.579832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.084718+0000","last_change":"2024-04-26T21:27:07.387201+0000","last_active":"2024-04-26T21:28:56.084718+0000","last_peered":"2024-04-26T21:28:56.084718+0000","last_clean":"2024-04-26T21:28:56.084718+0000","last_became_active":"2024-04-26T21:27:07.386327+0000","last_became_peered":"2024-04-26T21:27:07.386327+0000","last_unstale":"2024-04-26T21:28:56.084718+0000","last_undegraded":"2024-04-26T21:28:56.084718+0000","last_fullsized":"2024-04-26T21:28:56.084718+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:40.801572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088663+0000","last_change":"2024-04-26T21:27:02.304132+0000","last_active":"2024-04-26T21:28:57.088663+0000","last_peered":"2024-04-26T21:28:57.088663+0000","last_clean":"2024-04-26T21:28:57.088663+0000","last_became_active":"2024-04-26T21:27:02.303562+0000","last_became_peered":"2024-04-26T21:27:02.303562+0000","last_unstale":"2024-04-26T21:28:57.088663+0000","last_undegraded":"2024-04-26T21:28:57.088663+0000","last_fullsized":"2024-04-26T21:28:57.088663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:54.417590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.087339+0000","last_change":"2024-04-26T21:27:02.866539+0000","last_active":"2024-04-26T21:28:56.087339+0000","last_peered":"2024-04-26T21:28:56.087339+0000","last_clean":"2024-04-26T21:28:56.087339+0000","last_became_active":"2024-04-26T21:27:02.866193+0000","last_became_peered":"2024-04-26T21:27:02.866193+0000","last_unstale":"2024-04-26T21:28:56.087339+0000","last_undegraded":"2024-04-26T21:28:56.087339+0000","last_fullsized":"2024-04-26T21:28:56.087339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:44.234261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839355+0000","last_change":"2024-04-26T21:27:07.368394+0000","last_active":"2024-04-26T21:27:56.839355+0000","last_peered":"2024-04-26T21:27:56.839355+0000","last_clean":"2024-04-26T21:27:56.839355+0000","last_became_active":"2024-04-26T21:27:07.367998+0000","last_became_peered":"2024-04-26T21:27:07.367998+0000","last_unstale":"2024-04-26T21:27:56.839355+0000","last_undegraded":"2024-04-26T21:27:56.839355+0000","last_fullsized":"2024-04-26T21:27:56.839355+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:31:28.459759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":115,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:28:51.819311+0000","last_change":"2024-04-26T21:27:02.304096+0000","last_active":"2024-04-26T21:28:51.819311+0000","last_peered":"2024-04-26T21:28:51.819311+0000","last_clean":"2024-04-26T21:28:51.819311+0000","last_became_active":"2024-04-26T21:27:02.303736+0000","last_became_peered":"2024-04-26T21:27:02.303736+0000","last_unstale":"2024-04-26T21:28:51.819311+0000","last_undegraded":"2024-04-26T21:28:51.819311+0000","last_fullsized":"2024-04-26T21:28:51.819311+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:25.400913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839476+0000","last_change":"2024-04-26T21:27:02.863033+0000","last_active":"2024-04-26T21:27:56.839476+0000","last_peered":"2024-04-26T21:27:56.839476+0000","last_clean":"2024-04-26T21:27:56.839476+0000","last_became_active":"2024-04-26T21:27:02.862572+0000","last_became_peered":"2024-04-26T21:27:02.862572+0000","last_unstale":"2024-04-26T21:27:56.839476+0000","last_undegraded":"2024-04-26T21:27:56.839476+0000","last_fullsized":"2024-04-26T21:27:56.839476+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:02:55.574601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.850238+0000","last_change":"2024-04-26T21:27:07.373626+0000","last_active":"2024-04-26T21:27:56.850238+0000","last_peered":"2024-04-26T21:27:56.850238+0000","last_clean":"2024-04-26T21:27:56.850238+0000","last_became_active":"2024-04-26T21:27:07.373175+0000","last_became_peered":"2024-04-26T21:27:07.373175+0000","last_unstale":"2024-04-26T21:27:56.850238+0000","last_undegraded":"2024-04-26T21:27:56.850238+0000","last_fullsized":"2024-04-26T21:27:56.850238+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:53:09.118035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.846362+0000","last_change":"2024-04-26T21:27:56.846362+0000","last_active":"2024-04-26T21:27:56.846362+0000","last_peered":"2024-04-26T21:27:56.846362+0000","last_clean":"2024-04-26T21:27:56.846362+0000","last_became_active":"2024-04-26T21:27:56.840187+0000","last_became_peered":"2024-04-26T21:27:56.840187+0000","last_unstale":"2024-04-26T21:27:56.846362+0000","last_undegraded":"2024-04-26T21:27:56.846362+0000","last_fullsized":"2024-04-26T21:27:56.846362+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:12.064415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.086522+0000","last_change":"2024-04-26T21:27:02.858739+0000","last_active":"2024-04-26T21:28:56.086522+0000","last_peered":"2024-04-26T21:28:56.086522+0000","last_clean":"2024-04-26T21:28:56.086522+0000","last_became_active":"2024-04-26T21:27:02.857458+0000","last_became_peered":"2024-04-26T21:27:02.857458+0000","last_unstale":"2024-04-26T21:28:56.086522+0000","last_undegraded":"2024-04-26T21:28:56.086522+0000","last_fullsized":"2024-04-26T21:28:56.086522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:00.354052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833517+0000","last_change":"2024-04-26T21:27:07.377690+0000","last_active":"2024-04-26T21:27:56.833517+0000","last_peered":"2024-04-26T21:27:56.833517+0000","last_clean":"2024-04-26T21:27:56.833517+0000","last_became_active":"2024-04-26T21:27:07.377192+0000","last_became_peered":"2024-04-26T21:27:07.377192+0000","last_unstale":"2024-04-26T21:27:56.833517+0000","last_undegraded":"2024-04-26T21:27:56.833517+0000","last_fullsized":"2024-04-26T21:27:56.833517+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:48.069083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089650+0000","last_change":"2024-04-26T21:27:02.316110+0000","last_active":"2024-04-26T21:28:57.089650+0000","last_peered":"2024-04-26T21:28:57.089650+0000","last_clean":"2024-04-26T21:28:57.089650+0000","last_became_active":"2024-04-26T21:27:02.315767+0000","last_became_peered":"2024-04-26T21:27:02.315767+0000","last_unstale":"2024-04-26T21:28:57.089650+0000","last_undegraded":"2024-04-26T21:28:57.089650+0000","last_fullsized":"2024-04-26T21:28:57.089650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:00.990872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839138+0000","last_change":"2024-04-26T21:27:02.855736+0000","last_active":"2024-04-26T21:27:56.839138+0000","last_peered":"2024-04-26T21:27:56.839138+0000","last_clean":"2024-04-26T21:27:56.839138+0000","last_became_active":"2024-04-26T21:27:02.855378+0000","last_became_peered":"2024-04-26T21:27:02.855378+0000","last_unstale":"2024-04-26T21:27:56.839138+0000","last_undegraded":"2024-04-26T21:27:56.839138+0000","last_fullsized":"2024-04-26T21:27:56.839138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:55.272869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.087539+0000","last_change":"2024-04-26T21:27:07.376804+0000","last_active":"2024-04-26T21:28:56.087539+0000","last_peered":"2024-04-26T21:28:56.087539+0000","last_clean":"2024-04-26T21:28:56.087539+0000","last_became_active":"2024-04-26T21:27:07.374378+0000","last_became_peered":"2024-04-26T21:27:07.374378+0000","last_unstale":"2024-04-26T21:28:56.087539+0000","last_undegraded":"2024-04-26T21:28:56.087539+0000","last_fullsized":"2024-04-26T21:28:56.087539+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:03.930253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'33","reported_seq":238,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:28:52.820927+0000","last_change":"2024-04-26T21:27:02.297467+0000","last_active":"2024-04-26T21:28:52.820927+0000","last_peered":"2024-04-26T21:28:52.820927+0000","last_clean":"2024-04-26T21:28:52.820927+0000","last_became_active":"2024-04-26T21:27:02.297077+0000","last_became_peered":"2024-04-26T21:27:02.297077+0000","last_unstale":"2024-04-26T21:28:52.820927+0000","last_undegraded":"2024-04-26T21:28:52.820927+0000","last_fullsized":"2024-04-26T21:28:52.820927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:47.047565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55676,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":178,"num_read_kb":143,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840522+0000","last_change":"2024-04-26T21:27:02.858161+0000","last_active":"2024-04-26T21:27:56.840522+0000","last_peered":"2024-04-26T21:27:56.840522+0000","last_clean":"2024-04-26T21:27:56.840522+0000","last_became_active":"2024-04-26T21:27:02.857747+0000","last_became_peered":"2024-04-26T21:27:02.857747+0000","last_unstale":"2024-04-26T21:27:56.840522+0000","last_undegraded":"2024-04-26T21:27:56.840522+0000","last_fullsized":"2024-04-26T21:27:56.840522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:10:50.843195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.090006+0000","last_change":"2024-04-26T21:27:07.376253+0000","last_active":"2024-04-26T21:28:57.090006+0000","last_peered":"2024-04-26T21:28:57.090006+0000","last_clean":"2024-04-26T21:28:57.090006+0000","last_became_active":"2024-04-26T21:27:07.375822+0000","last_became_peered":"2024-04-26T21:27:07.375822+0000","last_unstale":"2024-04-26T21:28:57.090006+0000","last_undegraded":"2024-04-26T21:28:57.090006+0000","last_fullsized":"2024-04-26T21:28:57.090006+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:22:01.784163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835497+0000","last_change":"2024-04-26T21:27:02.300743+0000","last_active":"2024-04-26T21:27:56.835497+0000","last_peered":"2024-04-26T21:27:56.835497+0000","last_clean":"2024-04-26T21:27:56.835497+0000","last_became_active":"2024-04-26T21:27:02.300424+0000","last_became_peered":"2024-04-26T21:27:02.300424+0000","last_unstale":"2024-04-26T21:27:56.835497+0000","last_undegraded":"2024-04-26T21:27:56.835497+0000","last_fullsized":"2024-04-26T21:27:56.835497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:10.408234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.854541+0000","last_change":"2024-04-26T21:27:02.872456+0000","last_active":"2024-04-26T21:27:56.854541+0000","last_peered":"2024-04-26T21:27:56.854541+0000","last_clean":"2024-04-26T21:27:56.854541+0000","last_became_active":"2024-04-26T21:27:02.872064+0000","last_became_peered":"2024-04-26T21:27:02.872064+0000","last_unstale":"2024-04-26T21:27:56.854541+0000","last_undegraded":"2024-04-26T21:27:56.854541+0000","last_fullsized":"2024-04-26T21:27:56.854541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:33.511164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838240+0000","last_change":"2024-04-26T21:27:07.386148+0000","last_active":"2024-04-26T21:27:56.838240+0000","last_peered":"2024-04-26T21:27:56.838240+0000","last_clean":"2024-04-26T21:27:56.838240+0000","last_became_active":"2024-04-26T21:27:07.385787+0000","last_became_peered":"2024-04-26T21:27:07.385787+0000","last_unstale":"2024-04-26T21:27:56.838240+0000","last_undegraded":"2024-04-26T21:27:56.838240+0000","last_fullsized":"2024-04-26T21:27:56.838240+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:57.776660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089128+0000","last_change":"2024-04-26T21:27:02.313725+0000","last_active":"2024-04-26T21:28:57.089128+0000","last_peered":"2024-04-26T21:28:57.089128+0000","last_clean":"2024-04-26T21:28:57.089128+0000","last_became_active":"2024-04-26T21:27:02.313350+0000","last_became_peered":"2024-04-26T21:27:02.313350+0000","last_unstale":"2024-04-26T21:28:57.089128+0000","last_undegraded":"2024-04-26T21:28:57.089128+0000","last_fullsized":"2024-04-26T21:28:57.089128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:21:30.608067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.856753+0000","last_change":"2024-04-26T21:27:56.856753+0000","last_active":"2024-04-26T21:27:56.856753+0000","last_peered":"2024-04-26T21:27:56.856753+0000","last_clean":"2024-04-26T21:27:56.856753+0000","last_became_active":"2024-04-26T21:27:56.856297+0000","last_became_peered":"2024-04-26T21:27:56.856297+0000","last_unstale":"2024-04-26T21:27:56.856753+0000","last_undegraded":"2024-04-26T21:27:56.856753+0000","last_fullsized":"2024-04-26T21:27:56.856753+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:54.650077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.093819+0000","last_change":"2024-04-26T21:27:07.378091+0000","last_active":"2024-04-26T21:28:56.093819+0000","last_peered":"2024-04-26T21:28:56.093819+0000","last_clean":"2024-04-26T21:28:56.093819+0000","last_became_active":"2024-04-26T21:27:07.377671+0000","last_became_peered":"2024-04-26T21:27:07.377671+0000","last_unstale":"2024-04-26T21:28:56.093819+0000","last_undegraded":"2024-04-26T21:28:56.093819+0000","last_fullsized":"2024-04-26T21:28:56.093819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:41:36.551655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'3","reported_seq":116,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:28:50.845853+0000","last_change":"2024-04-26T21:27:02.294738+0000","last_active":"2024-04-26T21:28:50.845853+0000","last_peered":"2024-04-26T21:28:50.845853+0000","last_clean":"2024-04-26T21:28:50.845853+0000","last_became_active":"2024-04-26T21:27:02.294338+0000","last_became_peered":"2024-04-26T21:27:02.294338+0000","last_unstale":"2024-04-26T21:28:50.845853+0000","last_undegraded":"2024-04-26T21:28:50.845853+0000","last_fullsized":"2024-04-26T21:28:50.845853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:25.309107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835802+0000","last_change":"2024-04-26T21:27:02.861332+0000","last_active":"2024-04-26T21:27:56.835802+0000","last_peered":"2024-04-26T21:27:56.835802+0000","last_clean":"2024-04-26T21:27:56.835802+0000","last_became_active":"2024-04-26T21:27:02.857865+0000","last_became_peered":"2024-04-26T21:27:02.857865+0000","last_unstale":"2024-04-26T21:27:56.835802+0000","last_undegraded":"2024-04-26T21:27:56.835802+0000","last_fullsized":"2024-04-26T21:27:56.835802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:48.493086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.857930+0000","last_change":"2024-04-26T21:27:07.383219+0000","last_active":"2024-04-26T21:27:56.857930+0000","last_peered":"2024-04-26T21:27:56.857930+0000","last_clean":"2024-04-26T21:27:56.857930+0000","last_became_active":"2024-04-26T21:27:07.382820+0000","last_became_peered":"2024-04-26T21:27:07.382820+0000","last_unstale":"2024-04-26T21:27:56.857930+0000","last_undegraded":"2024-04-26T21:27:56.857930+0000","last_fullsized":"2024-04-26T21:27:56.857930+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:07.752124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839757+0000","last_change":"2024-04-26T21:27:02.307746+0000","last_active":"2024-04-26T21:27:56.839757+0000","last_peered":"2024-04-26T21:27:56.839757+0000","last_clean":"2024-04-26T21:27:56.839757+0000","last_became_active":"2024-04-26T21:27:02.307406+0000","last_became_peered":"2024-04-26T21:27:02.307406+0000","last_unstale":"2024-04-26T21:27:56.839757+0000","last_undegraded":"2024-04-26T21:27:56.839757+0000","last_fullsized":"2024-04-26T21:27:56.839757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:13.799727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088903+0000","last_change":"2024-04-26T21:27:02.861194+0000","last_active":"2024-04-26T21:28:57.088903+0000","last_peered":"2024-04-26T21:28:57.088903+0000","last_clean":"2024-04-26T21:28:57.088903+0000","last_became_active":"2024-04-26T21:27:02.860871+0000","last_became_peered":"2024-04-26T21:27:02.860871+0000","last_unstale":"2024-04-26T21:28:57.088903+0000","last_undegraded":"2024-04-26T21:28:57.088903+0000","last_fullsized":"2024-04-26T21:28:57.088903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:56.829027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.837307+0000","last_change":"2024-04-26T21:27:07.371265+0000","last_active":"2024-04-26T21:27:56.837307+0000","last_peered":"2024-04-26T21:27:56.837307+0000","last_clean":"2024-04-26T21:27:56.837307+0000","last_became_active":"2024-04-26T21:27:07.370928+0000","last_became_peered":"2024-04-26T21:27:07.370928+0000","last_unstale":"2024-04-26T21:27:56.837307+0000","last_undegraded":"2024-04-26T21:27:56.837307+0000","last_fullsized":"2024-04-26T21:27:56.837307+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:33:05.640493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840231+0000","last_change":"2024-04-26T21:27:02.301711+0000","last_active":"2024-04-26T21:27:56.840231+0000","last_peered":"2024-04-26T21:27:56.840231+0000","last_clean":"2024-04-26T21:27:56.840231+0000","last_became_active":"2024-04-26T21:27:02.301429+0000","last_became_peered":"2024-04-26T21:27:02.301429+0000","last_unstale":"2024-04-26T21:27:56.840231+0000","last_undegraded":"2024-04-26T21:27:56.840231+0000","last_fullsized":"2024-04-26T21:27:56.840231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:50.876824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839612+0000","last_change":"2024-04-26T21:27:02.854372+0000","last_active":"2024-04-26T21:27:56.839612+0000","last_peered":"2024-04-26T21:27:56.839612+0000","last_clean":"2024-04-26T21:27:56.839612+0000","last_became_active":"2024-04-26T21:27:02.853826+0000","last_became_peered":"2024-04-26T21:27:02.853826+0000","last_unstale":"2024-04-26T21:27:56.839612+0000","last_undegraded":"2024-04-26T21:27:56.839612+0000","last_fullsized":"2024-04-26T21:27:56.839612+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:08.253384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840355+0000","last_change":"2024-04-26T21:27:07.506684+0000","last_active":"2024-04-26T21:27:56.840355+0000","last_peered":"2024-04-26T21:27:56.840355+0000","last_clean":"2024-04-26T21:27:56.840355+0000","last_became_active":"2024-04-26T21:27:07.506203+0000","last_became_peered":"2024-04-26T21:27:07.506203+0000","last_unstale":"2024-04-26T21:27:56.840355+0000","last_undegraded":"2024-04-26T21:27:56.840355+0000","last_fullsized":"2024-04-26T21:27:56.840355+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:42:48.580106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.088580+0000","last_change":"2024-04-26T21:27:02.306265+0000","last_active":"2024-04-26T21:28:56.088580+0000","last_peered":"2024-04-26T21:28:56.088580+0000","last_clean":"2024-04-26T21:28:56.088580+0000","last_became_active":"2024-04-26T21:27:02.305941+0000","last_became_peered":"2024-04-26T21:27:02.305941+0000","last_unstale":"2024-04-26T21:28:56.088580+0000","last_undegraded":"2024-04-26T21:28:56.088580+0000","last_fullsized":"2024-04-26T21:28:56.088580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:29.076833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.843958+0000","last_change":"2024-04-26T21:27:02.871916+0000","last_active":"2024-04-26T21:27:56.843958+0000","last_peered":"2024-04-26T21:27:56.843958+0000","last_clean":"2024-04-26T21:27:56.843958+0000","last_became_active":"2024-04-26T21:27:02.871517+0000","last_became_peered":"2024-04-26T21:27:02.871517+0000","last_unstale":"2024-04-26T21:27:56.843958+0000","last_undegraded":"2024-04-26T21:27:56.843958+0000","last_fullsized":"2024-04-26T21:27:56.843958+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:41.361757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.854065+0000","last_change":"2024-04-26T21:27:07.383422+0000","last_active":"2024-04-26T21:27:56.854065+0000","last_peered":"2024-04-26T21:27:56.854065+0000","last_clean":"2024-04-26T21:27:56.854065+0000","last_became_active":"2024-04-26T21:27:07.382727+0000","last_became_peered":"2024-04-26T21:27:07.382727+0000","last_unstale":"2024-04-26T21:27:56.854065+0000","last_undegraded":"2024-04-26T21:27:56.854065+0000","last_fullsized":"2024-04-26T21:27:56.854065+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:22.002658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835916+0000","last_change":"2024-04-26T21:27:02.308081+0000","last_active":"2024-04-26T21:27:56.835916+0000","last_peered":"2024-04-26T21:27:56.835916+0000","last_clean":"2024-04-26T21:27:56.835916+0000","last_became_active":"2024-04-26T21:27:02.307715+0000","last_became_peered":"2024-04-26T21:27:02.307715+0000","last_unstale":"2024-04-26T21:27:56.835916+0000","last_undegraded":"2024-04-26T21:27:56.835916+0000","last_fullsized":"2024-04-26T21:27:56.835916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:28.967817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.843862+0000","last_change":"2024-04-26T21:27:02.859527+0000","last_active":"2024-04-26T21:27:56.843862+0000","last_peered":"2024-04-26T21:27:56.843862+0000","last_clean":"2024-04-26T21:27:56.843862+0000","last_became_active":"2024-04-26T21:27:02.859173+0000","last_became_peered":"2024-04-26T21:27:02.859173+0000","last_unstale":"2024-04-26T21:27:56.843862+0000","last_undegraded":"2024-04-26T21:27:56.843862+0000","last_fullsized":"2024-04-26T21:27:56.843862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:24.811630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.858266+0000","last_change":"2024-04-26T21:27:07.509778+0000","last_active":"2024-04-26T21:27:56.858266+0000","last_peered":"2024-04-26T21:27:56.858266+0000","last_clean":"2024-04-26T21:27:56.858266+0000","last_became_active":"2024-04-26T21:27:07.508344+0000","last_became_peered":"2024-04-26T21:27:07.508344+0000","last_unstale":"2024-04-26T21:27:56.858266+0000","last_undegraded":"2024-04-26T21:27:56.858266+0000","last_fullsized":"2024-04-26T21:27:56.858266+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:38:05.926717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2,"reported_epoch":94,"state":"peering","last_fresh":"2024-04-26T21:27:56.834808+0000","last_change":"2024-04-26T21:27:55.837544+0000","last_active":"2024-04-26T21:27:55.837544+0000","last_peered":"2024-04-26T21:27:08.397720+0000","last_clean":"2024-04-26T21:27:08.397720+0000","last_became_active":"2024-04-26T21:27:02.862499+0000","last_became_peered":"2024-04-26T21:27:02.862499+0000","last_unstale":"2024-04-26T21:27:56.834808+0000","last_undegraded":"2024-04-26T21:27:56.834808+0000","last_fullsized":"2024-04-26T21:27:56.834808+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":116,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:28:51.846962+0000","last_change":"2024-04-26T21:27:02.317413+0000","last_active":"2024-04-26T21:28:51.846962+0000","last_peered":"2024-04-26T21:28:51.846962+0000","last_clean":"2024-04-26T21:28:51.846962+0000","last_became_active":"2024-04-26T21:27:02.316510+0000","last_became_peered":"2024-04-26T21:27:02.316510+0000","last_unstale":"2024-04-26T21:28:51.846962+0000","last_undegraded":"2024-04-26T21:28:51.846962+0000","last_fullsized":"2024-04-26T21:28:51.846962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:49.156366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836272+0000","last_change":"2024-04-26T21:27:07.373434+0000","last_active":"2024-04-26T21:27:56.836272+0000","last_peered":"2024-04-26T21:27:56.836272+0000","last_clean":"2024-04-26T21:27:56.836272+0000","last_became_active":"2024-04-26T21:27:07.372973+0000","last_became_peered":"2024-04-26T21:27:07.372973+0000","last_unstale":"2024-04-26T21:27:56.836272+0000","last_undegraded":"2024-04-26T21:27:56.836272+0000","last_fullsized":"2024-04-26T21:27:56.836272+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:04.022651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839543+0000","last_change":"2024-04-26T21:27:02.862734+0000","last_active":"2024-04-26T21:27:56.839543+0000","last_peered":"2024-04-26T21:27:56.839543+0000","last_clean":"2024-04-26T21:27:56.839543+0000","last_became_active":"2024-04-26T21:27:02.862334+0000","last_became_peered":"2024-04-26T21:27:02.862334+0000","last_unstale":"2024-04-26T21:27:56.839543+0000","last_undegraded":"2024-04-26T21:27:56.839543+0000","last_fullsized":"2024-04-26T21:27:56.839543+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:09.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833317+0000","last_change":"2024-04-26T21:27:02.302471+0000","last_active":"2024-04-26T21:27:56.833317+0000","last_peered":"2024-04-26T21:27:56.833317+0000","last_clean":"2024-04-26T21:27:56.833317+0000","last_became_active":"2024-04-26T21:27:02.302055+0000","last_became_peered":"2024-04-26T21:27:02.302055+0000","last_unstale":"2024-04-26T21:27:56.833317+0000","last_undegraded":"2024-04-26T21:27:56.833317+0000","last_fullsized":"2024-04-26T21:27:56.833317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:29:14.649670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":363,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835725+0000","last_change":"2024-04-26T21:25:41.647633+0000","last_active":"2024-04-26T21:27:56.835725+0000","last_peered":"2024-04-26T21:27:56.835725+0000","last_clean":"2024-04-26T21:27:56.835725+0000","last_became_active":"2024-04-26T21:25:41.647091+0000","last_became_peered":"2024-04-26T21:25:41.647091+0000","last_unstale":"2024-04-26T21:27:56.835725+0000","last_undegraded":"2024-04-26T21:27:56.835725+0000","last_fullsized":"2024-04-26T21:27:56.835725+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_clean_scrub_stamp":"2024-04-26T21:21:43.547929+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:37.666065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.152820+0000","last_change":"2024-04-26T21:28:57.152820+0000","last_active":"2024-04-26T21:28:57.152820+0000","last_peered":"2024-04-26T21:28:57.152820+0000","last_clean":"2024-04-26T21:28:57.152820+0000","last_became_active":"2024-04-26T21:28:57.152453+0000","last_became_peered":"2024-04-26T21:28:57.152453+0000","last_unstale":"2024-04-26T21:28:57.152820+0000","last_undegraded":"2024-04-26T21:28:57.152820+0000","last_fullsized":"2024-04-26T21:28:57.152820+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:16:47.634651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.852207+0000","last_change":"2024-04-26T21:27:02.874116+0000","last_active":"2024-04-26T21:27:56.852207+0000","last_peered":"2024-04-26T21:27:56.852207+0000","last_clean":"2024-04-26T21:27:56.852207+0000","last_became_active":"2024-04-26T21:27:02.871334+0000","last_became_peered":"2024-04-26T21:27:02.871334+0000","last_unstale":"2024-04-26T21:27:56.852207+0000","last_undegraded":"2024-04-26T21:27:56.852207+0000","last_fullsized":"2024-04-26T21:27:56.852207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:28.835061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.848528+0000","last_change":"2024-04-26T21:27:56.847702+0000","last_active":"2024-04-26T21:27:56.848528+0000","last_peered":"2024-04-26T21:27:56.848528+0000","last_clean":"2024-04-26T21:27:56.848528+0000","last_became_active":"2024-04-26T21:27:56.844034+0000","last_became_peered":"2024-04-26T21:27:56.844034+0000","last_unstale":"2024-04-26T21:27:56.848528+0000","last_undegraded":"2024-04-26T21:27:56.848528+0000","last_fullsized":"2024-04-26T21:27:56.848528+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:54.631297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838359+0000","last_change":"2024-04-26T21:27:07.373894+0000","last_active":"2024-04-26T21:27:56.838359+0000","last_peered":"2024-04-26T21:27:56.838359+0000","last_clean":"2024-04-26T21:27:56.838359+0000","last_became_active":"2024-04-26T21:27:07.373338+0000","last_became_peered":"2024-04-26T21:27:07.373338+0000","last_unstale":"2024-04-26T21:27:56.838359+0000","last_undegraded":"2024-04-26T21:27:56.838359+0000","last_fullsized":"2024-04-26T21:27:56.838359+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:49.636493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.859353+0000","last_change":"2024-04-26T21:27:56.858975+0000","last_active":"2024-04-26T21:27:56.859353+0000","last_peered":"2024-04-26T21:27:56.859353+0000","last_clean":"2024-04-26T21:27:56.859353+0000","last_became_active":"2024-04-26T21:27:56.850022+0000","last_became_peered":"2024-04-26T21:27:56.850022+0000","last_unstale":"2024-04-26T21:27:56.859353+0000","last_undegraded":"2024-04-26T21:27:56.859353+0000","last_fullsized":"2024-04-26T21:27:56.859353+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:43:31.860906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.854497+0000","last_change":"2024-04-26T21:27:02.869831+0000","last_active":"2024-04-26T21:27:56.854497+0000","last_peered":"2024-04-26T21:27:56.854497+0000","last_clean":"2024-04-26T21:27:56.854497+0000","last_became_active":"2024-04-26T21:27:02.869375+0000","last_became_peered":"2024-04-26T21:27:02.869375+0000","last_unstale":"2024-04-26T21:27:56.854497+0000","last_undegraded":"2024-04-26T21:27:56.854497+0000","last_fullsized":"2024-04-26T21:27:56.854497+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:20.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838445+0000","last_change":"2024-04-26T21:27:07.507562+0000","last_active":"2024-04-26T21:27:56.838445+0000","last_peered":"2024-04-26T21:27:56.838445+0000","last_clean":"2024-04-26T21:27:56.838445+0000","last_became_active":"2024-04-26T21:27:07.507063+0000","last_became_peered":"2024-04-26T21:27:07.507063+0000","last_unstale":"2024-04-26T21:27:56.838445+0000","last_undegraded":"2024-04-26T21:27:56.838445+0000","last_fullsized":"2024-04-26T21:27:56.838445+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:58.192617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.096008+0000","last_change":"2024-04-26T21:27:02.315360+0000","last_active":"2024-04-26T21:28:56.096008+0000","last_peered":"2024-04-26T21:28:56.096008+0000","last_clean":"2024-04-26T21:28:56.096008+0000","last_became_active":"2024-04-26T21:27:02.314817+0000","last_became_peered":"2024-04-26T21:27:02.314817+0000","last_unstale":"2024-04-26T21:28:56.096008+0000","last_undegraded":"2024-04-26T21:28:56.096008+0000","last_fullsized":"2024-04-26T21:28:56.096008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:12.045363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":2,"reported_epoch":94,"state":"peering","last_fresh":"2024-04-26T21:27:56.841659+0000","last_change":"2024-04-26T21:28:56.105878+0000","last_active":"2024-04-26T21:28:56.105878+0000","last_peered":"2024-04-26T21:27:56.841659+0000","last_clean":"2024-04-26T21:27:56.841659+0000","last_became_active":"2024-04-26T21:27:02.856589+0000","last_became_peered":"2024-04-26T21:27:02.856589+0000","last_unstale":"2024-04-26T21:27:56.841659+0000","last_undegraded":"2024-04-26T21:27:56.841659+0000","last_fullsized":"2024-04-26T21:27:56.841659+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836750+0000","last_change":"2024-04-26T21:27:07.362016+0000","last_active":"2024-04-26T21:27:56.836750+0000","last_peered":"2024-04-26T21:27:56.836750+0000","last_clean":"2024-04-26T21:27:56.836750+0000","last_became_active":"2024-04-26T21:27:07.361360+0000","last_became_peered":"2024-04-26T21:27:07.361360+0000","last_unstale":"2024-04-26T21:27:56.836750+0000","last_undegraded":"2024-04-26T21:27:56.836750+0000","last_fullsized":"2024-04-26T21:27:56.836750+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:03.196293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833924+0000","last_change":"2024-04-26T21:27:02.308088+0000","last_active":"2024-04-26T21:27:56.833924+0000","last_peered":"2024-04-26T21:27:56.833924+0000","last_clean":"2024-04-26T21:27:56.833924+0000","last_became_active":"2024-04-26T21:27:02.307727+0000","last_became_peered":"2024-04-26T21:27:02.307727+0000","last_unstale":"2024-04-26T21:27:56.833924+0000","last_undegraded":"2024-04-26T21:27:56.833924+0000","last_fullsized":"2024-04-26T21:27:56.833924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:02.132125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"peering","last_fresh":"2024-04-26T21:27:56.845009+0000","last_change":"2024-04-26T21:28:56.086395+0000","last_active":"2024-04-26T21:28:56.086395+0000","last_peered":"2024-04-26T21:27:56.845009+0000","last_clean":"2024-04-26T21:27:56.845009+0000","last_became_active":"2024-04-26T21:27:56.840445+0000","last_became_peered":"2024-04-26T21:27:56.840445+0000","last_unstale":"2024-04-26T21:27:56.845009+0000","last_undegraded":"2024-04-26T21:27:56.845009+0000","last_fullsized":"2024-04-26T21:27:56.845009+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833696+0000","last_change":"2024-04-26T21:27:07.506609+0000","last_active":"2024-04-26T21:27:56.833696+0000","last_peered":"2024-04-26T21:27:56.833696+0000","last_clean":"2024-04-26T21:27:56.833696+0000","last_became_active":"2024-04-26T21:27:07.506100+0000","last_became_peered":"2024-04-26T21:27:07.506100+0000","last_unstale":"2024-04-26T21:27:56.833696+0000","last_undegraded":"2024-04-26T21:27:56.833696+0000","last_fullsized":"2024-04-26T21:27:56.833696+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:26.535809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838554+0000","last_change":"2024-04-26T21:27:02.297432+0000","last_active":"2024-04-26T21:27:56.838554+0000","last_peered":"2024-04-26T21:27:56.838554+0000","last_clean":"2024-04-26T21:27:56.838554+0000","last_became_active":"2024-04-26T21:27:02.297081+0000","last_became_peered":"2024-04-26T21:27:02.297081+0000","last_unstale":"2024-04-26T21:27:56.838554+0000","last_undegraded":"2024-04-26T21:27:56.838554+0000","last_fullsized":"2024-04-26T21:27:56.838554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:08.139469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839917+0000","last_change":"2024-04-26T21:27:02.866653+0000","last_active":"2024-04-26T21:27:56.839917+0000","last_peered":"2024-04-26T21:27:56.839917+0000","last_clean":"2024-04-26T21:27:56.839917+0000","last_became_active":"2024-04-26T21:27:02.866275+0000","last_became_peered":"2024-04-26T21:27:02.866275+0000","last_unstale":"2024-04-26T21:27:56.839917+0000","last_undegraded":"2024-04-26T21:27:56.839917+0000","last_fullsized":"2024-04-26T21:27:56.839917+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:39.089972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.854891+0000","last_change":"2024-04-26T21:27:07.377084+0000","last_active":"2024-04-26T21:27:56.854891+0000","last_peered":"2024-04-26T21:27:56.854891+0000","last_clean":"2024-04-26T21:27:56.854891+0000","last_became_active":"2024-04-26T21:27:07.374052+0000","last_became_peered":"2024-04-26T21:27:07.374052+0000","last_unstale":"2024-04-26T21:27:56.854891+0000","last_undegraded":"2024-04-26T21:27:56.854891+0000","last_fullsized":"2024-04-26T21:27:56.854891+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:06:20.031170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838351+0000","last_change":"2024-04-26T21:27:02.307250+0000","last_active":"2024-04-26T21:27:56.838351+0000","last_peered":"2024-04-26T21:27:56.838351+0000","last_clean":"2024-04-26T21:27:56.838351+0000","last_became_active":"2024-04-26T21:27:02.303532+0000","last_became_peered":"2024-04-26T21:27:02.303532+0000","last_unstale":"2024-04-26T21:27:56.838351+0000","last_undegraded":"2024-04-26T21:27:56.838351+0000","last_fullsized":"2024-04-26T21:27:56.838351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:36.265188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839303+0000","last_change":"2024-04-26T21:27:02.858818+0000","last_active":"2024-04-26T21:27:56.839303+0000","last_peered":"2024-04-26T21:27:56.839303+0000","last_clean":"2024-04-26T21:27:56.839303+0000","last_became_active":"2024-04-26T21:27:02.858059+0000","last_became_peered":"2024-04-26T21:27:02.858059+0000","last_unstale":"2024-04-26T21:27:56.839303+0000","last_undegraded":"2024-04-26T21:27:56.839303+0000","last_fullsized":"2024-04-26T21:27:56.839303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:02.159334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833488+0000","last_change":"2024-04-26T21:27:07.381273+0000","last_active":"2024-04-26T21:27:56.833488+0000","last_peered":"2024-04-26T21:27:56.833488+0000","last_clean":"2024-04-26T21:27:56.833488+0000","last_became_active":"2024-04-26T21:27:07.380891+0000","last_became_peered":"2024-04-26T21:27:07.380891+0000","last_unstale":"2024-04-26T21:27:56.833488+0000","last_undegraded":"2024-04-26T21:27:56.833488+0000","last_fullsized":"2024-04-26T21:27:56.833488+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:39.641577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.837799+0000","last_change":"2024-04-26T21:27:02.301363+0000","last_active":"2024-04-26T21:27:56.837799+0000","last_peered":"2024-04-26T21:27:56.837799+0000","last_clean":"2024-04-26T21:27:56.837799+0000","last_became_active":"2024-04-26T21:27:02.301059+0000","last_became_peered":"2024-04-26T21:27:02.301059+0000","last_unstale":"2024-04-26T21:27:56.837799+0000","last_undegraded":"2024-04-26T21:27:56.837799+0000","last_fullsized":"2024-04-26T21:27:56.837799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:17.586190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.841887+0000","last_change":"2024-04-26T21:27:02.860612+0000","last_active":"2024-04-26T21:27:56.841887+0000","last_peered":"2024-04-26T21:27:56.841887+0000","last_clean":"2024-04-26T21:27:56.841887+0000","last_became_active":"2024-04-26T21:27:02.857285+0000","last_became_peered":"2024-04-26T21:27:02.857285+0000","last_unstale":"2024-04-26T21:27:56.841887+0000","last_undegraded":"2024-04-26T21:27:56.841887+0000","last_fullsized":"2024-04-26T21:27:56.841887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:11.968818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.833702+0000","last_change":"2024-04-26T21:27:07.506425+0000","last_active":"2024-04-26T21:27:56.833702+0000","last_peered":"2024-04-26T21:27:56.833702+0000","last_clean":"2024-04-26T21:27:56.833702+0000","last_became_active":"2024-04-26T21:27:07.505968+0000","last_became_peered":"2024-04-26T21:27:07.505968+0000","last_unstale":"2024-04-26T21:27:56.833702+0000","last_undegraded":"2024-04-26T21:27:56.833702+0000","last_fullsized":"2024-04-26T21:27:56.833702+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:59.914560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839708+0000","last_change":"2024-04-26T21:27:02.294635+0000","last_active":"2024-04-26T21:27:56.839708+0000","last_peered":"2024-04-26T21:27:56.839708+0000","last_clean":"2024-04-26T21:27:56.839708+0000","last_became_active":"2024-04-26T21:27:02.294311+0000","last_became_peered":"2024-04-26T21:27:02.294311+0000","last_unstale":"2024-04-26T21:27:56.839708+0000","last_undegraded":"2024-04-26T21:27:56.839708+0000","last_fullsized":"2024-04-26T21:27:56.839708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:47.906546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836051+0000","last_change":"2024-04-26T21:27:02.875475+0000","last_active":"2024-04-26T21:27:56.836051+0000","last_peered":"2024-04-26T21:27:56.836051+0000","last_clean":"2024-04-26T21:27:56.836051+0000","last_became_active":"2024-04-26T21:27:02.875137+0000","last_became_peered":"2024-04-26T21:27:02.875137+0000","last_unstale":"2024-04-26T21:27:56.836051+0000","last_undegraded":"2024-04-26T21:27:56.836051+0000","last_fullsized":"2024-04-26T21:27:56.836051+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:04.988973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838642+0000","last_change":"2024-04-26T21:27:07.372927+0000","last_active":"2024-04-26T21:27:56.838642+0000","last_peered":"2024-04-26T21:27:56.838642+0000","last_clean":"2024-04-26T21:27:56.838642+0000","last_became_active":"2024-04-26T21:27:07.372371+0000","last_became_peered":"2024-04-26T21:27:07.372371+0000","last_unstale":"2024-04-26T21:27:56.838642+0000","last_undegraded":"2024-04-26T21:27:56.838642+0000","last_fullsized":"2024-04-26T21:27:56.838642+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:12:41.607167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836974+0000","last_change":"2024-04-26T21:27:02.295379+0000","last_active":"2024-04-26T21:27:56.836974+0000","last_peered":"2024-04-26T21:27:56.836974+0000","last_clean":"2024-04-26T21:27:56.836974+0000","last_became_active":"2024-04-26T21:27:02.295077+0000","last_became_peered":"2024-04-26T21:27:02.295077+0000","last_unstale":"2024-04-26T21:27:56.836974+0000","last_undegraded":"2024-04-26T21:27:56.836974+0000","last_fullsized":"2024-04-26T21:27:56.836974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:58.177960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836567+0000","last_change":"2024-04-26T21:27:02.855998+0000","last_active":"2024-04-26T21:27:56.836567+0000","last_peered":"2024-04-26T21:27:56.836567+0000","last_clean":"2024-04-26T21:27:56.836567+0000","last_became_active":"2024-04-26T21:27:02.855636+0000","last_became_peered":"2024-04-26T21:27:02.855636+0000","last_unstale":"2024-04-26T21:27:56.836567+0000","last_undegraded":"2024-04-26T21:27:56.836567+0000","last_fullsized":"2024-04-26T21:27:56.836567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:19:02.988946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.084873+0000","last_change":"2024-04-26T21:27:07.378980+0000","last_active":"2024-04-26T21:28:56.084873+0000","last_peered":"2024-04-26T21:28:56.084873+0000","last_clean":"2024-04-26T21:28:56.084873+0000","last_became_active":"2024-04-26T21:27:07.378374+0000","last_became_peered":"2024-04-26T21:27:07.378374+0000","last_unstale":"2024-04-26T21:28:56.084873+0000","last_undegraded":"2024-04-26T21:28:56.084873+0000","last_fullsized":"2024-04-26T21:28:56.084873+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:31.845297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089815+0000","last_change":"2024-04-26T21:27:02.313801+0000","last_active":"2024-04-26T21:28:57.089815+0000","last_peered":"2024-04-26T21:28:57.089815+0000","last_clean":"2024-04-26T21:28:57.089815+0000","last_became_active":"2024-04-26T21:27:02.313540+0000","last_became_peered":"2024-04-26T21:27:02.313540+0000","last_unstale":"2024-04-26T21:28:57.089815+0000","last_undegraded":"2024-04-26T21:28:57.089815+0000","last_fullsized":"2024-04-26T21:28:57.089815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:38.612261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836372+0000","last_change":"2024-04-26T21:27:02.864151+0000","last_active":"2024-04-26T21:27:56.836372+0000","last_peered":"2024-04-26T21:27:56.836372+0000","last_clean":"2024-04-26T21:27:56.836372+0000","last_became_active":"2024-04-26T21:27:02.863818+0000","last_became_peered":"2024-04-26T21:27:02.863818+0000","last_unstale":"2024-04-26T21:27:56.836372+0000","last_undegraded":"2024-04-26T21:27:56.836372+0000","last_fullsized":"2024-04-26T21:27:56.836372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:26.215370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088554+0000","last_change":"2024-04-26T21:27:07.365821+0000","last_active":"2024-04-26T21:28:57.088554+0000","last_peered":"2024-04-26T21:28:57.088554+0000","last_clean":"2024-04-26T21:28:57.088554+0000","last_became_active":"2024-04-26T21:27:07.365347+0000","last_became_peered":"2024-04-26T21:27:07.365347+0000","last_unstale":"2024-04-26T21:28:57.088554+0000","last_undegraded":"2024-04-26T21:28:57.088554+0000","last_fullsized":"2024-04-26T21:28:57.088554+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:40:11.596908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.856646+0000","last_change":"2024-04-26T21:27:02.310641+0000","last_active":"2024-04-26T21:27:56.856646+0000","last_peered":"2024-04-26T21:27:56.856646+0000","last_clean":"2024-04-26T21:27:56.856646+0000","last_became_active":"2024-04-26T21:27:02.308569+0000","last_became_peered":"2024-04-26T21:27:02.308569+0000","last_unstale":"2024-04-26T21:27:56.856646+0000","last_undegraded":"2024-04-26T21:27:56.856646+0000","last_fullsized":"2024-04-26T21:27:56.856646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:20.988098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836482+0000","last_change":"2024-04-26T21:27:02.854976+0000","last_active":"2024-04-26T21:27:56.836482+0000","last_peered":"2024-04-26T21:27:56.836482+0000","last_clean":"2024-04-26T21:27:56.836482+0000","last_became_active":"2024-04-26T21:27:02.854611+0000","last_became_peered":"2024-04-26T21:27:02.854611+0000","last_unstale":"2024-04-26T21:27:56.836482+0000","last_undegraded":"2024-04-26T21:27:56.836482+0000","last_fullsized":"2024-04-26T21:27:56.836482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:57.065657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839417+0000","last_change":"2024-04-26T21:27:02.859017+0000","last_active":"2024-04-26T21:27:56.839417+0000","last_peered":"2024-04-26T21:27:56.839417+0000","last_clean":"2024-04-26T21:27:56.839417+0000","last_became_active":"2024-04-26T21:27:02.858507+0000","last_became_peered":"2024-04-26T21:27:02.858507+0000","last_unstale":"2024-04-26T21:27:56.839417+0000","last_undegraded":"2024-04-26T21:27:56.839417+0000","last_fullsized":"2024-04-26T21:27:56.839417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:14.850891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.841251+0000","last_change":"2024-04-26T21:27:56.841251+0000","last_active":"2024-04-26T21:27:56.841251+0000","last_peered":"2024-04-26T21:27:56.841251+0000","last_clean":"2024-04-26T21:27:56.841251+0000","last_became_active":"2024-04-26T21:27:56.840862+0000","last_became_peered":"2024-04-26T21:27:56.840862+0000","last_unstale":"2024-04-26T21:27:56.841251+0000","last_undegraded":"2024-04-26T21:27:56.841251+0000","last_fullsized":"2024-04-26T21:27:56.841251+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:41.460343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.834936+0000","last_change":"2024-04-26T21:27:02.868817+0000","last_active":"2024-04-26T21:27:56.834936+0000","last_peered":"2024-04-26T21:27:56.834936+0000","last_clean":"2024-04-26T21:27:56.834936+0000","last_became_active":"2024-04-26T21:27:02.868557+0000","last_became_peered":"2024-04-26T21:27:02.868557+0000","last_unstale":"2024-04-26T21:27:56.834936+0000","last_undegraded":"2024-04-26T21:27:56.834936+0000","last_fullsized":"2024-04-26T21:27:56.834936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:55.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.085535+0000","last_change":"2024-04-26T21:27:02.876029+0000","last_active":"2024-04-26T21:28:56.085535+0000","last_peered":"2024-04-26T21:28:56.085535+0000","last_clean":"2024-04-26T21:28:56.085535+0000","last_became_active":"2024-04-26T21:27:02.875656+0000","last_became_peered":"2024-04-26T21:27:02.875656+0000","last_unstale":"2024-04-26T21:28:56.085535+0000","last_undegraded":"2024-04-26T21:28:56.085535+0000","last_fullsized":"2024-04-26T21:28:56.085535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:47.317581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.854839+0000","last_change":"2024-04-26T21:27:02.873818+0000","last_active":"2024-04-26T21:27:56.854839+0000","last_peered":"2024-04-26T21:27:56.854839+0000","last_clean":"2024-04-26T21:27:56.854839+0000","last_became_active":"2024-04-26T21:27:02.873459+0000","last_became_peered":"2024-04-26T21:27:02.873459+0000","last_unstale":"2024-04-26T21:27:56.854839+0000","last_undegraded":"2024-04-26T21:27:56.854839+0000","last_fullsized":"2024-04-26T21:27:56.854839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:07.259187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839289+0000","last_change":"2024-04-26T21:27:02.854901+0000","last_active":"2024-04-26T21:27:56.839289+0000","last_peered":"2024-04-26T21:27:56.839289+0000","last_clean":"2024-04-26T21:27:56.839289+0000","last_became_active":"2024-04-26T21:27:02.854547+0000","last_became_peered":"2024-04-26T21:27:02.854547+0000","last_unstale":"2024-04-26T21:27:56.839289+0000","last_undegraded":"2024-04-26T21:27:56.839289+0000","last_fullsized":"2024-04-26T21:27:56.839289+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:24.320868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836463+0000","last_change":"2024-04-26T21:27:02.856616+0000","last_active":"2024-04-26T21:27:56.836463+0000","last_peered":"2024-04-26T21:27:56.836463+0000","last_clean":"2024-04-26T21:27:56.836463+0000","last_became_active":"2024-04-26T21:27:02.856264+0000","last_became_peered":"2024-04-26T21:27:02.856264+0000","last_unstale":"2024-04-26T21:27:56.836463+0000","last_undegraded":"2024-04-26T21:27:56.836463+0000","last_fullsized":"2024-04-26T21:27:56.836463+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:43.374534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.842062+0000","last_change":"2024-04-26T21:27:02.859587+0000","last_active":"2024-04-26T21:27:56.842062+0000","last_peered":"2024-04-26T21:27:56.842062+0000","last_clean":"2024-04-26T21:27:56.842062+0000","last_became_active":"2024-04-26T21:27:02.859136+0000","last_became_peered":"2024-04-26T21:27:02.859136+0000","last_unstale":"2024-04-26T21:27:56.842062+0000","last_undegraded":"2024-04-26T21:27:56.842062+0000","last_fullsized":"2024-04-26T21:27:56.842062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:09.370319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088720+0000","last_change":"2024-04-26T21:27:02.864265+0000","last_active":"2024-04-26T21:28:57.088720+0000","last_peered":"2024-04-26T21:28:57.088720+0000","last_clean":"2024-04-26T21:28:57.088720+0000","last_became_active":"2024-04-26T21:27:02.863920+0000","last_became_peered":"2024-04-26T21:27:02.863920+0000","last_unstale":"2024-04-26T21:28:57.088720+0000","last_undegraded":"2024-04-26T21:28:57.088720+0000","last_fullsized":"2024-04-26T21:28:57.088720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:13.941544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838672+0000","last_change":"2024-04-26T21:27:02.860189+0000","last_active":"2024-04-26T21:27:56.838672+0000","last_peered":"2024-04-26T21:27:56.838672+0000","last_clean":"2024-04-26T21:27:56.838672+0000","last_became_active":"2024-04-26T21:27:02.859843+0000","last_became_peered":"2024-04-26T21:27:02.859843+0000","last_unstale":"2024-04-26T21:27:56.838672+0000","last_undegraded":"2024-04-26T21:27:56.838672+0000","last_fullsized":"2024-04-26T21:27:56.838672+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839012+0000","last_change":"2024-04-26T21:27:02.865459+0000","last_active":"2024-04-26T21:27:56.839012+0000","last_peered":"2024-04-26T21:27:56.839012+0000","last_clean":"2024-04-26T21:27:56.839012+0000","last_became_active":"2024-04-26T21:27:02.863602+0000","last_became_peered":"2024-04-26T21:27:02.863602+0000","last_unstale":"2024-04-26T21:27:56.839012+0000","last_undegraded":"2024-04-26T21:27:56.839012+0000","last_fullsized":"2024-04-26T21:27:56.839012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:03.733858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.855076+0000","last_change":"2024-04-26T21:27:02.873711+0000","last_active":"2024-04-26T21:27:56.855076+0000","last_peered":"2024-04-26T21:27:56.855076+0000","last_clean":"2024-04-26T21:27:56.855076+0000","last_became_active":"2024-04-26T21:27:02.873349+0000","last_became_peered":"2024-04-26T21:27:02.873349+0000","last_unstale":"2024-04-26T21:27:56.855076+0000","last_undegraded":"2024-04-26T21:27:56.855076+0000","last_fullsized":"2024-04-26T21:27:56.855076+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:10.841536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838350+0000","last_change":"2024-04-26T21:27:02.852625+0000","last_active":"2024-04-26T21:27:56.838350+0000","last_peered":"2024-04-26T21:27:56.838350+0000","last_clean":"2024-04-26T21:27:56.838350+0000","last_became_active":"2024-04-26T21:27:02.852177+0000","last_became_peered":"2024-04-26T21:27:02.852177+0000","last_unstale":"2024-04-26T21:27:56.838350+0000","last_undegraded":"2024-04-26T21:27:56.838350+0000","last_fullsized":"2024-04-26T21:27:56.838350+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:52.046678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59286,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":538,"num_read_kb":330,"num_write":161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318309,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1048576,"data_stored":509262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579946,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1728512,"data_stored":1152184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743472,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1064960,"data_stored":513572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005111,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1056768,"data_stored":509374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266748,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32800,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712352,"statfs":{"total":95995035648,"available":95961448448,"internally_reserved":0,"allocated":1654784,"data_stored":1104914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462978,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":504652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1101824,"data_stored":560238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1081344,"data_stored":515712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":561588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1052672,"data_stored":504846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888141,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":509302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724616,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":1679360,"data_stored":1110174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":55676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":57026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":57254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:28:58.467 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T21:28:58.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:58 smithi038 ceph-mon[32724]: osdmap e95: 12 total, 12 up, 12 in 2024-04-26T21:28:58.966 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:28:59.355 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:59 smithi001 ceph-mon[28054]: pgmap v386: 129 pgs: 3 peering, 126 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T21:28:59.355 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:59 smithi001 ceph-mon[28054]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:28:59.355 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:28:59 smithi001 ceph-mon[28054]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-26T21:28:59.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:59 smithi186 ceph-mon[32991]: pgmap v386: 129 pgs: 3 peering, 126 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T21:28:59.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:59 smithi186 ceph-mon[32991]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:28:59.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:28:59 smithi186 ceph-mon[32991]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-26T21:28:59.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:59 smithi038 ceph-mon[32724]: pgmap v386: 129 pgs: 3 peering, 126 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2024-04-26T21:28:59.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:59 smithi038 ceph-mon[32724]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:28:59.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:28:59 smithi038 ceph-mon[32724]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-26T21:29:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.097+0000 7f49b3c00700 1 -- 172.21.15.1:0/456759279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49ac105640 msgr2=0x7f49ac105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:01.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.097+0000 7f49b3c00700 1 --2- 172.21.15.1:0/456759279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49ac105640 0x7f49ac105ac0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4994009990 tx=0x7f4994009ca0 comp rx=0 tx=0).stop 2024-04-26T21:29:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.097+0000 7f49b3c00700 1 -- 172.21.15.1:0/456759279 shutdown_connections 2024-04-26T21:29:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.097+0000 7f49b3c00700 1 --2- 172.21.15.1:0/456759279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49ac106000 0x7f49ac112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.097+0000 7f49b3c00700 1 --2- 172.21.15.1:0/456759279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49ac10b640 0x7f49ac10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.097+0000 7f49b3c00700 1 --2- 172.21.15.1:0/456759279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49ac105640 0x7f49ac105ac0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.097+0000 7f49b3c00700 1 -- 172.21.15.1:0/456759279 >> 172.21.15.1:0/456759279 conn(0x7f49ac101130 msgr2=0x7f49ac103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.097+0000 7f49b3c00700 1 -- 172.21.15.1:0/456759279 shutdown_connections 2024-04-26T21:29:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.097+0000 7f49b3c00700 1 -- 172.21.15.1:0/456759279 wait complete. 2024-04-26T21:29:01.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.098+0000 7f49b3c00700 1 Processor -- start 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.098+0000 7f49b3c00700 1 -- start start 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49b3c00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49ac105640 0x7f49ac19bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49b3c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49ac106000 0x7f49ac19c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49b3c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49ac10b640 0x7f49ac1a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49b3c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac1158d0 con 0x7f49ac106000 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49b3c00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac115750 con 0x7f49ac105640 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49b3c00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac115a50 con 0x7f49ac10b640 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49a3e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49ac106000 0x7f49ac19c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49b1200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49ac105640 0x7f49ac19bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49b1200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49ac105640 0x7f49ac19bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42886/0 (socket says 172.21.15.1:42886) 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49b1200700 1 -- 172.21.15.1:0/4096432666 learned_addr learned my addr 172.21.15.1:0/4096432666 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.099+0000 7f49b1c00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49ac10b640 0x7f49ac1a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:01.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.100+0000 7f49b1200700 1 -- 172.21.15.1:0/4096432666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49ac10b640 msgr2=0x7f49ac1a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.100+0000 7f49b1200700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49ac10b640 0x7f49ac1a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.100+0000 7f49b1200700 1 -- 172.21.15.1:0/4096432666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49ac106000 msgr2=0x7f49ac19c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.100+0000 7f49b1200700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49ac106000 0x7f49ac19c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.100+0000 7f49b1200700 1 -- 172.21.15.1:0/4096432666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ac1a0d70 con 0x7f49ac105640 2024-04-26T21:29:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.100+0000 7f49a3e00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49ac106000 0x7f49ac19c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.100+0000 7f49b1c00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49ac10b640 0x7f49ac1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:01.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.100+0000 7f49b1200700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49ac105640 0x7f49ac19bd60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f499c00eb30 tx=0x7f499c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.113+0000 7f49aa000700 1 -- 172.21.15.1:0/4096432666 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c00cbd0 con 0x7f49ac105640 2024-04-26T21:29:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.113+0000 7f49aa000700 1 -- 172.21.15.1:0/4096432666 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f499c0044d0 con 0x7f49ac105640 2024-04-26T21:29:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.113+0000 7f49aa000700 1 -- 172.21.15.1:0/4096432666 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499c003e30 con 0x7f49ac105640 2024-04-26T21:29:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.113+0000 7f49b3c00700 1 -- 172.21.15.1:0/4096432666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ac1a1060 con 0x7f49ac105640 2024-04-26T21:29:01.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.113+0000 7f49b3c00700 1 -- 172.21.15.1:0/4096432666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ac1a88e0 con 0x7f49ac105640 2024-04-26T21:29:01.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.114+0000 7f49b3c00700 1 -- 172.21.15.1:0/4096432666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ac06a110 con 0x7f49ac105640 2024-04-26T21:29:01.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.116+0000 7f49aa000700 1 -- 172.21.15.1:0/4096432666 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f499c004000 con 0x7f49ac105640 2024-04-26T21:29:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.118+0000 7f49aa000700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49900a3e20 0x7f49900a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.118+0000 7f49aa000700 1 -- 172.21.15.1:0/4096432666 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f499c0d0df0 con 0x7f49ac105640 2024-04-26T21:29:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.118+0000 7f49a3e00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49900a3e20 0x7f49900a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:01.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.119+0000 7f49a3e00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49900a3e20 0x7f49900a62e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f49ac1037b0 tx=0x7f499400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:01.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.120+0000 7f49aa000700 1 -- 172.21.15.1:0/4096432666 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f499c054060 con 0x7f49ac105640 2024-04-26T21:29:01.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:01 smithi001 ceph-mon[28054]: pgmap v387: 129 pgs: 3 peering, 126 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:01.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.346+0000 7f49b3c00700 1 -- 172.21.15.1:0/4096432666 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49ac063860 con 0x7f49900a3e20 2024-04-26T21:29:01.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.373+0000 7f49aa000700 1 -- 172.21.15.1:0/4096432666 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340296 (secure 0 0 0) 0x7f49ac063860 con 0x7f49900a3e20 2024-04-26T21:29:01.376 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.380+0000 7f49b3c00700 1 -- 172.21.15.1:0/4096432666 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49900a3e20 msgr2=0x7f49900a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.380+0000 7f49b3c00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49900a3e20 0x7f49900a62e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f49ac1037b0 tx=0x7f499400b260 comp rx=0 tx=0).stop 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.380+0000 7f49b3c00700 1 -- 172.21.15.1:0/4096432666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49ac105640 msgr2=0x7f49ac19bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.380+0000 7f49b3c00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49ac105640 0x7f49ac19bd60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f499c00eb30 tx=0x7f499c00ee40 comp rx=0 tx=0).stop 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.380+0000 7f49b3c00700 1 -- 172.21.15.1:0/4096432666 shutdown_connections 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.380+0000 7f49b3c00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f49900a3e20 0x7f49900a62e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.380+0000 7f49b3c00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49ac10b640 0x7f49ac1a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.380+0000 7f49b3c00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49ac105640 0x7f49ac19bd60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.380+0000 7f49b3c00700 1 --2- 172.21.15.1:0/4096432666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49ac106000 0x7f49ac19c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.380+0000 7f49b3c00700 1 -- 172.21.15.1:0/4096432666 >> 172.21.15.1:0/4096432666 conn(0x7f49ac101130 msgr2=0x7f49ac101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.381+0000 7f49b3c00700 1 -- 172.21.15.1:0/4096432666 shutdown_connections 2024-04-26T21:29:01.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:01.381+0000 7f49b3c00700 1 -- 172.21.15.1:0/4096432666 wait complete. 2024-04-26T21:29:01.383 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T21:29:01.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:01 smithi186 ceph-mon[32991]: pgmap v387: 129 pgs: 3 peering, 126 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:01.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:01 smithi038 ceph-mon[32724]: pgmap v387: 129 pgs: 3 peering, 126 active+clean; 635 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:02.227 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-04-26T21:28:59.353891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649888,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":699,"num_read_kb":464,"num_write":344,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180758,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":421,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380364,"kb_used_data":14504,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561460,"statfs":{"total":1151940427776,"available":1151550935040,"internally_reserved":0,"allocated":14852096,"data_stored":8151822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003950"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120762+0000","last_change":"2024-04-26T21:27:02.854724+0000","last_active":"2024-04-26T21:28:57.120762+0000","last_peered":"2024-04-26T21:28:57.120762+0000","last_clean":"2024-04-26T21:28:57.120762+0000","last_became_active":"2024-04-26T21:27:02.854125+0000","last_became_peered":"2024-04-26T21:27:02.854125+0000","last_unstale":"2024-04-26T21:28:57.120762+0000","last_undegraded":"2024-04-26T21:28:57.120762+0000","last_fullsized":"2024-04-26T21:28:57.120762+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:16.500220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.085070+0000","last_change":"2024-04-26T21:27:02.852846+0000","last_active":"2024-04-26T21:28:56.085070+0000","last_peered":"2024-04-26T21:28:56.085070+0000","last_clean":"2024-04-26T21:28:56.085070+0000","last_became_active":"2024-04-26T21:27:02.851243+0000","last_became_peered":"2024-04-26T21:27:02.851243+0000","last_unstale":"2024-04-26T21:28:56.085070+0000","last_undegraded":"2024-04-26T21:28:56.085070+0000","last_fullsized":"2024-04-26T21:28:56.085070+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:09:11.240955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.086854+0000","last_change":"2024-04-26T21:27:02.855856+0000","last_active":"2024-04-26T21:28:56.086854+0000","last_peered":"2024-04-26T21:28:56.086854+0000","last_clean":"2024-04-26T21:28:56.086854+0000","last_became_active":"2024-04-26T21:27:02.855446+0000","last_became_peered":"2024-04-26T21:27:02.855446+0000","last_unstale":"2024-04-26T21:28:56.086854+0000","last_undegraded":"2024-04-26T21:28:56.086854+0000","last_fullsized":"2024-04-26T21:28:56.086854+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:35.183449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089053+0000","last_change":"2024-04-26T21:27:02.860949+0000","last_active":"2024-04-26T21:28:57.089053+0000","last_peered":"2024-04-26T21:28:57.089053+0000","last_clean":"2024-04-26T21:28:57.089053+0000","last_became_active":"2024-04-26T21:27:02.860567+0000","last_became_peered":"2024-04-26T21:27:02.860567+0000","last_unstale":"2024-04-26T21:28:57.089053+0000","last_undegraded":"2024-04-26T21:28:57.089053+0000","last_fullsized":"2024-04-26T21:28:57.089053+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:09.851885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839655+0000","last_change":"2024-04-26T21:27:02.876019+0000","last_active":"2024-04-26T21:27:56.839655+0000","last_peered":"2024-04-26T21:27:56.839655+0000","last_clean":"2024-04-26T21:27:56.839655+0000","last_became_active":"2024-04-26T21:27:02.875652+0000","last_became_peered":"2024-04-26T21:27:02.875652+0000","last_unstale":"2024-04-26T21:27:56.839655+0000","last_undegraded":"2024-04-26T21:27:56.839655+0000","last_fullsized":"2024-04-26T21:27:56.839655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:35.017787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088087+0000","last_change":"2024-04-26T21:27:02.860229+0000","last_active":"2024-04-26T21:28:57.088087+0000","last_peered":"2024-04-26T21:28:57.088087+0000","last_clean":"2024-04-26T21:28:57.088087+0000","last_became_active":"2024-04-26T21:27:02.859863+0000","last_became_peered":"2024-04-26T21:27:02.859863+0000","last_unstale":"2024-04-26T21:28:57.088087+0000","last_undegraded":"2024-04-26T21:28:57.088087+0000","last_fullsized":"2024-04-26T21:28:57.088087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:47.337073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835778+0000","last_change":"2024-04-26T21:27:02.863253+0000","last_active":"2024-04-26T21:27:56.835778+0000","last_peered":"2024-04-26T21:27:56.835778+0000","last_clean":"2024-04-26T21:27:56.835778+0000","last_became_active":"2024-04-26T21:27:02.861623+0000","last_became_peered":"2024-04-26T21:27:02.861623+0000","last_unstale":"2024-04-26T21:27:56.835778+0000","last_undegraded":"2024-04-26T21:27:56.835778+0000","last_fullsized":"2024-04-26T21:27:56.835778+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:57.642504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838500+0000","last_change":"2024-04-26T21:27:02.860822+0000","last_active":"2024-04-26T21:27:56.838500+0000","last_peered":"2024-04-26T21:27:56.838500+0000","last_clean":"2024-04-26T21:27:56.838500+0000","last_became_active":"2024-04-26T21:27:02.860482+0000","last_became_peered":"2024-04-26T21:27:02.860482+0000","last_unstale":"2024-04-26T21:27:56.838500+0000","last_undegraded":"2024-04-26T21:27:56.838500+0000","last_fullsized":"2024-04-26T21:27:56.838500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:59.224624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141309+0000","last_change":"2024-04-26T21:27:56.856389+0000","last_active":"2024-04-26T21:28:57.141309+0000","last_peered":"2024-04-26T21:28:57.141309+0000","last_clean":"2024-04-26T21:28:57.141309+0000","last_became_active":"2024-04-26T21:27:56.856008+0000","last_became_peered":"2024-04-26T21:27:56.856008+0000","last_unstale":"2024-04-26T21:28:57.141309+0000","last_undegraded":"2024-04-26T21:28:57.141309+0000","last_fullsized":"2024-04-26T21:28:57.141309+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:04.770535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088910+0000","last_change":"2024-04-26T21:27:02.860545+0000","last_active":"2024-04-26T21:28:57.088910+0000","last_peered":"2024-04-26T21:28:57.088910+0000","last_clean":"2024-04-26T21:28:57.088910+0000","last_became_active":"2024-04-26T21:27:02.858577+0000","last_became_peered":"2024-04-26T21:27:02.858577+0000","last_unstale":"2024-04-26T21:28:57.088910+0000","last_undegraded":"2024-04-26T21:28:57.088910+0000","last_fullsized":"2024-04-26T21:28:57.088910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:43.390592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836456+0000","last_change":"2024-04-26T21:27:02.858685+0000","last_active":"2024-04-26T21:27:56.836456+0000","last_peered":"2024-04-26T21:27:56.836456+0000","last_clean":"2024-04-26T21:27:56.836456+0000","last_became_active":"2024-04-26T21:27:02.857054+0000","last_became_peered":"2024-04-26T21:27:02.857054+0000","last_unstale":"2024-04-26T21:27:56.836456+0000","last_undegraded":"2024-04-26T21:27:56.836456+0000","last_fullsized":"2024-04-26T21:27:56.836456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:29.406132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088929+0000","last_change":"2024-04-26T21:27:02.852457+0000","last_active":"2024-04-26T21:28:57.088929+0000","last_peered":"2024-04-26T21:28:57.088929+0000","last_clean":"2024-04-26T21:28:57.088929+0000","last_became_active":"2024-04-26T21:27:02.852064+0000","last_became_peered":"2024-04-26T21:27:02.852064+0000","last_unstale":"2024-04-26T21:28:57.088929+0000","last_undegraded":"2024-04-26T21:28:57.088929+0000","last_fullsized":"2024-04-26T21:28:57.088929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:16.798413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.149646+0000","last_change":"2024-04-26T21:27:02.872618+0000","last_active":"2024-04-26T21:28:57.149646+0000","last_peered":"2024-04-26T21:28:57.149646+0000","last_clean":"2024-04-26T21:28:57.149646+0000","last_became_active":"2024-04-26T21:27:02.872270+0000","last_became_peered":"2024-04-26T21:27:02.872270+0000","last_unstale":"2024-04-26T21:28:57.149646+0000","last_undegraded":"2024-04-26T21:28:57.149646+0000","last_fullsized":"2024-04-26T21:28:57.149646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:45.828419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.084143+0000","last_change":"2024-04-26T21:27:02.868590+0000","last_active":"2024-04-26T21:28:56.084143+0000","last_peered":"2024-04-26T21:28:56.084143+0000","last_clean":"2024-04-26T21:28:56.084143+0000","last_became_active":"2024-04-26T21:27:02.866641+0000","last_became_peered":"2024-04-26T21:27:02.866641+0000","last_unstale":"2024-04-26T21:28:56.084143+0000","last_undegraded":"2024-04-26T21:28:56.084143+0000","last_fullsized":"2024-04-26T21:28:56.084143+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:17.313420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.084420+0000","last_change":"2024-04-26T21:27:02.867735+0000","last_active":"2024-04-26T21:28:56.084420+0000","last_peered":"2024-04-26T21:28:56.084420+0000","last_clean":"2024-04-26T21:28:56.084420+0000","last_became_active":"2024-04-26T21:27:02.867217+0000","last_became_peered":"2024-04-26T21:27:02.867217+0000","last_unstale":"2024-04-26T21:28:56.084420+0000","last_undegraded":"2024-04-26T21:28:56.084420+0000","last_fullsized":"2024-04-26T21:28:56.084420+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:04:45.533220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140625+0000","last_change":"2024-04-26T21:27:02.857541+0000","last_active":"2024-04-26T21:28:57.140625+0000","last_peered":"2024-04-26T21:28:57.140625+0000","last_clean":"2024-04-26T21:28:57.140625+0000","last_became_active":"2024-04-26T21:27:02.855563+0000","last_became_peered":"2024-04-26T21:27:02.855563+0000","last_unstale":"2024-04-26T21:28:57.140625+0000","last_undegraded":"2024-04-26T21:28:57.140625+0000","last_fullsized":"2024-04-26T21:28:57.140625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:39.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120658+0000","last_change":"2024-04-26T21:27:02.854461+0000","last_active":"2024-04-26T21:28:57.120658+0000","last_peered":"2024-04-26T21:28:57.120658+0000","last_clean":"2024-04-26T21:28:57.120658+0000","last_became_active":"2024-04-26T21:27:02.853691+0000","last_became_peered":"2024-04-26T21:27:02.853691+0000","last_unstale":"2024-04-26T21:28:57.120658+0000","last_undegraded":"2024-04-26T21:28:57.120658+0000","last_fullsized":"2024-04-26T21:28:57.120658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:09:48.284180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.096142+0000","last_change":"2024-04-26T21:27:02.864308+0000","last_active":"2024-04-26T21:28:56.096142+0000","last_peered":"2024-04-26T21:28:56.096142+0000","last_clean":"2024-04-26T21:28:56.096142+0000","last_became_active":"2024-04-26T21:27:02.863928+0000","last_became_peered":"2024-04-26T21:27:02.863928+0000","last_unstale":"2024-04-26T21:28:56.096142+0000","last_undegraded":"2024-04-26T21:28:56.096142+0000","last_fullsized":"2024-04-26T21:28:56.096142+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:12.169394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140846+0000","last_change":"2024-04-26T21:27:02.876044+0000","last_active":"2024-04-26T21:28:57.140846+0000","last_peered":"2024-04-26T21:28:57.140846+0000","last_clean":"2024-04-26T21:28:57.140846+0000","last_became_active":"2024-04-26T21:27:02.875681+0000","last_became_peered":"2024-04-26T21:27:02.875681+0000","last_unstale":"2024-04-26T21:28:57.140846+0000","last_undegraded":"2024-04-26T21:28:57.140846+0000","last_fullsized":"2024-04-26T21:28:57.140846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:26.246916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140001+0000","last_change":"2024-04-26T21:27:07.374736+0000","last_active":"2024-04-26T21:28:57.140001+0000","last_peered":"2024-04-26T21:28:57.140001+0000","last_clean":"2024-04-26T21:28:57.140001+0000","last_became_active":"2024-04-26T21:27:07.374116+0000","last_became_peered":"2024-04-26T21:27:07.374116+0000","last_unstale":"2024-04-26T21:28:57.140001+0000","last_undegraded":"2024-04-26T21:28:57.140001+0000","last_fullsized":"2024-04-26T21:28:57.140001+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:11.450926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.093657+0000","last_change":"2024-04-26T21:27:02.302510+0000","last_active":"2024-04-26T21:28:56.093657+0000","last_peered":"2024-04-26T21:28:56.093657+0000","last_clean":"2024-04-26T21:28:56.093657+0000","last_became_active":"2024-04-26T21:27:02.302088+0000","last_became_peered":"2024-04-26T21:27:02.302088+0000","last_unstale":"2024-04-26T21:28:56.093657+0000","last_undegraded":"2024-04-26T21:28:56.093657+0000","last_fullsized":"2024-04-26T21:28:56.093657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:43.007722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840502+0000","last_change":"2024-04-26T21:27:02.861455+0000","last_active":"2024-04-26T21:27:56.840502+0000","last_peered":"2024-04-26T21:27:56.840502+0000","last_clean":"2024-04-26T21:27:56.840502+0000","last_became_active":"2024-04-26T21:27:02.861107+0000","last_became_peered":"2024-04-26T21:27:02.861107+0000","last_unstale":"2024-04-26T21:27:56.840502+0000","last_undegraded":"2024-04-26T21:27:56.840502+0000","last_fullsized":"2024-04-26T21:27:56.840502+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:10:18.382881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088435+0000","last_change":"2024-04-26T21:27:07.377797+0000","last_active":"2024-04-26T21:28:57.088435+0000","last_peered":"2024-04-26T21:28:57.088435+0000","last_clean":"2024-04-26T21:28:57.088435+0000","last_became_active":"2024-04-26T21:27:07.377314+0000","last_became_peered":"2024-04-26T21:27:07.377314+0000","last_unstale":"2024-04-26T21:28:57.088435+0000","last_undegraded":"2024-04-26T21:28:57.088435+0000","last_fullsized":"2024-04-26T21:28:57.088435+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:02:34.761034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840568+0000","last_change":"2024-04-26T21:27:02.309488+0000","last_active":"2024-04-26T21:27:56.840568+0000","last_peered":"2024-04-26T21:27:56.840568+0000","last_clean":"2024-04-26T21:27:56.840568+0000","last_became_active":"2024-04-26T21:27:02.308592+0000","last_became_peered":"2024-04-26T21:27:02.308592+0000","last_unstale":"2024-04-26T21:27:56.840568+0000","last_undegraded":"2024-04-26T21:27:56.840568+0000","last_fullsized":"2024-04-26T21:27:56.840568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:36.563847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.087039+0000","last_change":"2024-04-26T21:27:02.863204+0000","last_active":"2024-04-26T21:28:56.087039+0000","last_peered":"2024-04-26T21:28:56.087039+0000","last_clean":"2024-04-26T21:28:56.087039+0000","last_became_active":"2024-04-26T21:27:02.862466+0000","last_became_peered":"2024-04-26T21:27:02.862466+0000","last_unstale":"2024-04-26T21:28:56.087039+0000","last_undegraded":"2024-04-26T21:28:56.087039+0000","last_fullsized":"2024-04-26T21:28:56.087039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:29.360830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088593+0000","last_change":"2024-04-26T21:27:07.378032+0000","last_active":"2024-04-26T21:28:57.088593+0000","last_peered":"2024-04-26T21:28:57.088593+0000","last_clean":"2024-04-26T21:28:57.088593+0000","last_became_active":"2024-04-26T21:27:07.377255+0000","last_became_peered":"2024-04-26T21:27:07.377255+0000","last_unstale":"2024-04-26T21:28:57.088593+0000","last_undegraded":"2024-04-26T21:28:57.088593+0000","last_fullsized":"2024-04-26T21:28:57.088593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:18.989977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.850816+0000","last_change":"2024-04-26T21:27:56.850816+0000","last_active":"2024-04-26T21:27:56.850816+0000","last_peered":"2024-04-26T21:27:56.850816+0000","last_clean":"2024-04-26T21:27:56.850816+0000","last_became_active":"2024-04-26T21:27:56.843299+0000","last_became_peered":"2024-04-26T21:27:56.843299+0000","last_unstale":"2024-04-26T21:27:56.850816+0000","last_undegraded":"2024-04-26T21:27:56.850816+0000","last_fullsized":"2024-04-26T21:27:56.850816+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:21:38.864693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141842+0000","last_change":"2024-04-26T21:27:02.876447+0000","last_active":"2024-04-26T21:28:57.141842+0000","last_peered":"2024-04-26T21:28:57.141842+0000","last_clean":"2024-04-26T21:28:57.141842+0000","last_became_active":"2024-04-26T21:27:02.876017+0000","last_became_peered":"2024-04-26T21:27:02.876017+0000","last_unstale":"2024-04-26T21:28:57.141842+0000","last_undegraded":"2024-04-26T21:28:57.141842+0000","last_fullsized":"2024-04-26T21:28:57.141842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:57:05.453787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088379+0000","last_change":"2024-04-26T21:27:07.373700+0000","last_active":"2024-04-26T21:28:57.088379+0000","last_peered":"2024-04-26T21:28:57.088379+0000","last_clean":"2024-04-26T21:28:57.088379+0000","last_became_active":"2024-04-26T21:27:07.373255+0000","last_became_peered":"2024-04-26T21:27:07.373255+0000","last_unstale":"2024-04-26T21:28:57.088379+0000","last_undegraded":"2024-04-26T21:28:57.088379+0000","last_fullsized":"2024-04-26T21:28:57.088379+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:48.517197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840466+0000","last_change":"2024-04-26T21:27:02.300817+0000","last_active":"2024-04-26T21:27:56.840466+0000","last_peered":"2024-04-26T21:27:56.840466+0000","last_clean":"2024-04-26T21:27:56.840466+0000","last_became_active":"2024-04-26T21:27:02.300493+0000","last_became_peered":"2024-04-26T21:27:02.300493+0000","last_unstale":"2024-04-26T21:27:56.840466+0000","last_undegraded":"2024-04-26T21:27:56.840466+0000","last_fullsized":"2024-04-26T21:27:56.840466+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:18.783104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089662+0000","last_change":"2024-04-26T21:27:02.859982+0000","last_active":"2024-04-26T21:28:57.089662+0000","last_peered":"2024-04-26T21:28:57.089662+0000","last_clean":"2024-04-26T21:28:57.089662+0000","last_became_active":"2024-04-26T21:27:02.857204+0000","last_became_peered":"2024-04-26T21:27:02.857204+0000","last_unstale":"2024-04-26T21:28:57.089662+0000","last_undegraded":"2024-04-26T21:28:57.089662+0000","last_fullsized":"2024-04-26T21:28:57.089662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:22.423956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140395+0000","last_change":"2024-04-26T21:27:07.366039+0000","last_active":"2024-04-26T21:28:57.140395+0000","last_peered":"2024-04-26T21:28:57.140395+0000","last_clean":"2024-04-26T21:28:57.140395+0000","last_became_active":"2024-04-26T21:27:07.365560+0000","last_became_peered":"2024-04-26T21:27:07.365560+0000","last_unstale":"2024-04-26T21:28:57.140395+0000","last_undegraded":"2024-04-26T21:28:57.140395+0000","last_fullsized":"2024-04-26T21:28:57.140395+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:47.833476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":127,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089027+0000","last_change":"2024-04-26T21:27:02.306364+0000","last_active":"2024-04-26T21:28:57.089027+0000","last_peered":"2024-04-26T21:28:57.089027+0000","last_clean":"2024-04-26T21:28:57.089027+0000","last_became_active":"2024-04-26T21:27:02.306041+0000","last_became_peered":"2024-04-26T21:27:02.306041+0000","last_unstale":"2024-04-26T21:28:57.089027+0000","last_undegraded":"2024-04-26T21:28:57.089027+0000","last_fullsized":"2024-04-26T21:28:57.089027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:44.501916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840481+0000","last_change":"2024-04-26T21:27:02.859346+0000","last_active":"2024-04-26T21:27:56.840481+0000","last_peered":"2024-04-26T21:27:56.840481+0000","last_clean":"2024-04-26T21:27:56.840481+0000","last_became_active":"2024-04-26T21:27:02.859025+0000","last_became_peered":"2024-04-26T21:27:02.859025+0000","last_unstale":"2024-04-26T21:27:56.840481+0000","last_undegraded":"2024-04-26T21:27:56.840481+0000","last_fullsized":"2024-04-26T21:27:56.840481+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:06.258512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140349+0000","last_change":"2024-04-26T21:27:07.382349+0000","last_active":"2024-04-26T21:28:57.140349+0000","last_peered":"2024-04-26T21:28:57.140349+0000","last_clean":"2024-04-26T21:28:57.140349+0000","last_became_active":"2024-04-26T21:27:07.381930+0000","last_became_peered":"2024-04-26T21:27:07.381930+0000","last_unstale":"2024-04-26T21:28:57.140349+0000","last_undegraded":"2024-04-26T21:28:57.140349+0000","last_fullsized":"2024-04-26T21:28:57.140349+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:15:25.970573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.087951+0000","last_change":"2024-04-26T21:27:56.862413+0000","last_active":"2024-04-26T21:28:57.087951+0000","last_peered":"2024-04-26T21:28:57.087951+0000","last_clean":"2024-04-26T21:28:57.087951+0000","last_became_active":"2024-04-26T21:27:56.859184+0000","last_became_peered":"2024-04-26T21:27:56.859184+0000","last_unstale":"2024-04-26T21:28:57.087951+0000","last_undegraded":"2024-04-26T21:28:57.087951+0000","last_fullsized":"2024-04-26T21:28:57.087951+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:54:17.701601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089353+0000","last_change":"2024-04-26T21:27:02.865707+0000","last_active":"2024-04-26T21:28:57.089353+0000","last_peered":"2024-04-26T21:28:57.089353+0000","last_clean":"2024-04-26T21:28:57.089353+0000","last_became_active":"2024-04-26T21:27:02.865436+0000","last_became_peered":"2024-04-26T21:27:02.865436+0000","last_unstale":"2024-04-26T21:28:57.089353+0000","last_undegraded":"2024-04-26T21:28:57.089353+0000","last_fullsized":"2024-04-26T21:28:57.089353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:53.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120620+0000","last_change":"2024-04-26T21:27:07.379092+0000","last_active":"2024-04-26T21:28:57.120620+0000","last_peered":"2024-04-26T21:28:57.120620+0000","last_clean":"2024-04-26T21:28:57.120620+0000","last_became_active":"2024-04-26T21:27:07.378716+0000","last_became_peered":"2024-04-26T21:27:07.378716+0000","last_unstale":"2024-04-26T21:28:57.120620+0000","last_undegraded":"2024-04-26T21:28:57.120620+0000","last_fullsized":"2024-04-26T21:28:57.120620+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:18.906224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147874+0000","last_change":"2024-04-26T21:27:02.307510+0000","last_active":"2024-04-26T21:28:57.147874+0000","last_peered":"2024-04-26T21:28:57.147874+0000","last_clean":"2024-04-26T21:28:57.147874+0000","last_became_active":"2024-04-26T21:27:02.306932+0000","last_became_peered":"2024-04-26T21:27:02.306932+0000","last_unstale":"2024-04-26T21:28:57.147874+0000","last_undegraded":"2024-04-26T21:28:57.147874+0000","last_fullsized":"2024-04-26T21:28:57.147874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:21:34.217767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838207+0000","last_change":"2024-04-26T21:27:02.874246+0000","last_active":"2024-04-26T21:27:56.838207+0000","last_peered":"2024-04-26T21:27:56.838207+0000","last_clean":"2024-04-26T21:27:56.838207+0000","last_became_active":"2024-04-26T21:27:02.873880+0000","last_became_peered":"2024-04-26T21:27:02.873880+0000","last_unstale":"2024-04-26T21:27:56.838207+0000","last_undegraded":"2024-04-26T21:27:56.838207+0000","last_fullsized":"2024-04-26T21:27:56.838207+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:21.579832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.084718+0000","last_change":"2024-04-26T21:27:07.387201+0000","last_active":"2024-04-26T21:28:56.084718+0000","last_peered":"2024-04-26T21:28:56.084718+0000","last_clean":"2024-04-26T21:28:56.084718+0000","last_became_active":"2024-04-26T21:27:07.386327+0000","last_became_peered":"2024-04-26T21:27:07.386327+0000","last_unstale":"2024-04-26T21:28:56.084718+0000","last_undegraded":"2024-04-26T21:28:56.084718+0000","last_fullsized":"2024-04-26T21:28:56.084718+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:40.801572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088663+0000","last_change":"2024-04-26T21:27:02.304132+0000","last_active":"2024-04-26T21:28:57.088663+0000","last_peered":"2024-04-26T21:28:57.088663+0000","last_clean":"2024-04-26T21:28:57.088663+0000","last_became_active":"2024-04-26T21:27:02.303562+0000","last_became_peered":"2024-04-26T21:27:02.303562+0000","last_unstale":"2024-04-26T21:28:57.088663+0000","last_undegraded":"2024-04-26T21:28:57.088663+0000","last_fullsized":"2024-04-26T21:28:57.088663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:54.417590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.087339+0000","last_change":"2024-04-26T21:27:02.866539+0000","last_active":"2024-04-26T21:28:56.087339+0000","last_peered":"2024-04-26T21:28:56.087339+0000","last_clean":"2024-04-26T21:28:56.087339+0000","last_became_active":"2024-04-26T21:27:02.866193+0000","last_became_peered":"2024-04-26T21:27:02.866193+0000","last_unstale":"2024-04-26T21:28:56.087339+0000","last_undegraded":"2024-04-26T21:28:56.087339+0000","last_fullsized":"2024-04-26T21:28:56.087339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:44.234261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839355+0000","last_change":"2024-04-26T21:27:07.368394+0000","last_active":"2024-04-26T21:27:56.839355+0000","last_peered":"2024-04-26T21:27:56.839355+0000","last_clean":"2024-04-26T21:27:56.839355+0000","last_became_active":"2024-04-26T21:27:07.367998+0000","last_became_peered":"2024-04-26T21:27:07.367998+0000","last_unstale":"2024-04-26T21:27:56.839355+0000","last_undegraded":"2024-04-26T21:27:56.839355+0000","last_fullsized":"2024-04-26T21:27:56.839355+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:31:28.459759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":124,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088548+0000","last_change":"2024-04-26T21:27:02.304096+0000","last_active":"2024-04-26T21:28:57.088548+0000","last_peered":"2024-04-26T21:28:57.088548+0000","last_clean":"2024-04-26T21:28:57.088548+0000","last_became_active":"2024-04-26T21:27:02.303736+0000","last_became_peered":"2024-04-26T21:27:02.303736+0000","last_unstale":"2024-04-26T21:28:57.088548+0000","last_undegraded":"2024-04-26T21:28:57.088548+0000","last_fullsized":"2024-04-26T21:28:57.088548+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:25.400913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142465+0000","last_change":"2024-04-26T21:27:02.863033+0000","last_active":"2024-04-26T21:28:57.142465+0000","last_peered":"2024-04-26T21:28:57.142465+0000","last_clean":"2024-04-26T21:28:57.142465+0000","last_became_active":"2024-04-26T21:27:02.862572+0000","last_became_peered":"2024-04-26T21:27:02.862572+0000","last_unstale":"2024-04-26T21:28:57.142465+0000","last_undegraded":"2024-04-26T21:28:57.142465+0000","last_fullsized":"2024-04-26T21:28:57.142465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:02:55.574601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.850238+0000","last_change":"2024-04-26T21:27:07.373626+0000","last_active":"2024-04-26T21:27:56.850238+0000","last_peered":"2024-04-26T21:27:56.850238+0000","last_clean":"2024-04-26T21:27:56.850238+0000","last_became_active":"2024-04-26T21:27:07.373175+0000","last_became_peered":"2024-04-26T21:27:07.373175+0000","last_unstale":"2024-04-26T21:27:56.850238+0000","last_undegraded":"2024-04-26T21:27:56.850238+0000","last_fullsized":"2024-04-26T21:27:56.850238+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:53:09.118035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.846362+0000","last_change":"2024-04-26T21:27:56.846362+0000","last_active":"2024-04-26T21:27:56.846362+0000","last_peered":"2024-04-26T21:27:56.846362+0000","last_clean":"2024-04-26T21:27:56.846362+0000","last_became_active":"2024-04-26T21:27:56.840187+0000","last_became_peered":"2024-04-26T21:27:56.840187+0000","last_unstale":"2024-04-26T21:27:56.846362+0000","last_undegraded":"2024-04-26T21:27:56.846362+0000","last_fullsized":"2024-04-26T21:27:56.846362+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:12.064415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.086522+0000","last_change":"2024-04-26T21:27:02.858739+0000","last_active":"2024-04-26T21:28:56.086522+0000","last_peered":"2024-04-26T21:28:56.086522+0000","last_clean":"2024-04-26T21:28:56.086522+0000","last_became_active":"2024-04-26T21:27:02.857458+0000","last_became_peered":"2024-04-26T21:27:02.857458+0000","last_unstale":"2024-04-26T21:28:56.086522+0000","last_undegraded":"2024-04-26T21:28:56.086522+0000","last_fullsized":"2024-04-26T21:28:56.086522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:00.354052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089405+0000","last_change":"2024-04-26T21:27:07.377690+0000","last_active":"2024-04-26T21:28:57.089405+0000","last_peered":"2024-04-26T21:28:57.089405+0000","last_clean":"2024-04-26T21:28:57.089405+0000","last_became_active":"2024-04-26T21:27:07.377192+0000","last_became_peered":"2024-04-26T21:27:07.377192+0000","last_unstale":"2024-04-26T21:28:57.089405+0000","last_undegraded":"2024-04-26T21:28:57.089405+0000","last_fullsized":"2024-04-26T21:28:57.089405+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:48.069083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089650+0000","last_change":"2024-04-26T21:27:02.316110+0000","last_active":"2024-04-26T21:28:57.089650+0000","last_peered":"2024-04-26T21:28:57.089650+0000","last_clean":"2024-04-26T21:28:57.089650+0000","last_became_active":"2024-04-26T21:27:02.315767+0000","last_became_peered":"2024-04-26T21:27:02.315767+0000","last_unstale":"2024-04-26T21:28:57.089650+0000","last_undegraded":"2024-04-26T21:28:57.089650+0000","last_fullsized":"2024-04-26T21:28:57.089650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:00.990872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142071+0000","last_change":"2024-04-26T21:27:02.855736+0000","last_active":"2024-04-26T21:28:57.142071+0000","last_peered":"2024-04-26T21:28:57.142071+0000","last_clean":"2024-04-26T21:28:57.142071+0000","last_became_active":"2024-04-26T21:27:02.855378+0000","last_became_peered":"2024-04-26T21:27:02.855378+0000","last_unstale":"2024-04-26T21:28:57.142071+0000","last_undegraded":"2024-04-26T21:28:57.142071+0000","last_fullsized":"2024-04-26T21:28:57.142071+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:55.272869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.087539+0000","last_change":"2024-04-26T21:27:07.376804+0000","last_active":"2024-04-26T21:28:56.087539+0000","last_peered":"2024-04-26T21:28:56.087539+0000","last_clean":"2024-04-26T21:28:56.087539+0000","last_became_active":"2024-04-26T21:27:07.374378+0000","last_became_peered":"2024-04-26T21:27:07.374378+0000","last_unstale":"2024-04-26T21:28:56.087539+0000","last_undegraded":"2024-04-26T21:28:56.087539+0000","last_fullsized":"2024-04-26T21:28:56.087539+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:03.930253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'33","reported_seq":252,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.828967+0000","last_change":"2024-04-26T21:27:02.297467+0000","last_active":"2024-04-26T21:28:57.828967+0000","last_peered":"2024-04-26T21:28:57.828967+0000","last_clean":"2024-04-26T21:28:57.828967+0000","last_became_active":"2024-04-26T21:27:02.297077+0000","last_became_peered":"2024-04-26T21:27:02.297077+0000","last_unstale":"2024-04-26T21:28:57.828967+0000","last_undegraded":"2024-04-26T21:28:57.828967+0000","last_fullsized":"2024-04-26T21:28:57.828967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:47.047565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55676,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":188,"num_read_kb":148,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840522+0000","last_change":"2024-04-26T21:27:02.858161+0000","last_active":"2024-04-26T21:27:56.840522+0000","last_peered":"2024-04-26T21:27:56.840522+0000","last_clean":"2024-04-26T21:27:56.840522+0000","last_became_active":"2024-04-26T21:27:02.857747+0000","last_became_peered":"2024-04-26T21:27:02.857747+0000","last_unstale":"2024-04-26T21:27:56.840522+0000","last_undegraded":"2024-04-26T21:27:56.840522+0000","last_fullsized":"2024-04-26T21:27:56.840522+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:10:50.843195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.090006+0000","last_change":"2024-04-26T21:27:07.376253+0000","last_active":"2024-04-26T21:28:57.090006+0000","last_peered":"2024-04-26T21:28:57.090006+0000","last_clean":"2024-04-26T21:28:57.090006+0000","last_became_active":"2024-04-26T21:27:07.375822+0000","last_became_peered":"2024-04-26T21:27:07.375822+0000","last_unstale":"2024-04-26T21:28:57.090006+0000","last_undegraded":"2024-04-26T21:28:57.090006+0000","last_fullsized":"2024-04-26T21:28:57.090006+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:22:01.784163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140286+0000","last_change":"2024-04-26T21:27:02.300743+0000","last_active":"2024-04-26T21:28:57.140286+0000","last_peered":"2024-04-26T21:28:57.140286+0000","last_clean":"2024-04-26T21:28:57.140286+0000","last_became_active":"2024-04-26T21:27:02.300424+0000","last_became_peered":"2024-04-26T21:27:02.300424+0000","last_unstale":"2024-04-26T21:28:57.140286+0000","last_undegraded":"2024-04-26T21:28:57.140286+0000","last_fullsized":"2024-04-26T21:28:57.140286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:10.408234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148845+0000","last_change":"2024-04-26T21:27:02.872456+0000","last_active":"2024-04-26T21:28:57.148845+0000","last_peered":"2024-04-26T21:28:57.148845+0000","last_clean":"2024-04-26T21:28:57.148845+0000","last_became_active":"2024-04-26T21:27:02.872064+0000","last_became_peered":"2024-04-26T21:27:02.872064+0000","last_unstale":"2024-04-26T21:28:57.148845+0000","last_undegraded":"2024-04-26T21:28:57.148845+0000","last_fullsized":"2024-04-26T21:28:57.148845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:33.511164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120361+0000","last_change":"2024-04-26T21:27:07.386148+0000","last_active":"2024-04-26T21:28:57.120361+0000","last_peered":"2024-04-26T21:28:57.120361+0000","last_clean":"2024-04-26T21:28:57.120361+0000","last_became_active":"2024-04-26T21:27:07.385787+0000","last_became_peered":"2024-04-26T21:27:07.385787+0000","last_unstale":"2024-04-26T21:28:57.120361+0000","last_undegraded":"2024-04-26T21:28:57.120361+0000","last_fullsized":"2024-04-26T21:28:57.120361+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:57.776660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089128+0000","last_change":"2024-04-26T21:27:02.313725+0000","last_active":"2024-04-26T21:28:57.089128+0000","last_peered":"2024-04-26T21:28:57.089128+0000","last_clean":"2024-04-26T21:28:57.089128+0000","last_became_active":"2024-04-26T21:27:02.313350+0000","last_became_peered":"2024-04-26T21:27:02.313350+0000","last_unstale":"2024-04-26T21:28:57.089128+0000","last_undegraded":"2024-04-26T21:28:57.089128+0000","last_fullsized":"2024-04-26T21:28:57.089128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:21:30.608067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.856753+0000","last_change":"2024-04-26T21:27:56.856753+0000","last_active":"2024-04-26T21:27:56.856753+0000","last_peered":"2024-04-26T21:27:56.856753+0000","last_clean":"2024-04-26T21:27:56.856753+0000","last_became_active":"2024-04-26T21:27:56.856297+0000","last_became_peered":"2024-04-26T21:27:56.856297+0000","last_unstale":"2024-04-26T21:27:56.856753+0000","last_undegraded":"2024-04-26T21:27:56.856753+0000","last_fullsized":"2024-04-26T21:27:56.856753+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:54.650077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.093819+0000","last_change":"2024-04-26T21:27:07.378091+0000","last_active":"2024-04-26T21:28:56.093819+0000","last_peered":"2024-04-26T21:28:56.093819+0000","last_clean":"2024-04-26T21:28:56.093819+0000","last_became_active":"2024-04-26T21:27:07.377671+0000","last_became_peered":"2024-04-26T21:27:07.377671+0000","last_unstale":"2024-04-26T21:28:56.093819+0000","last_undegraded":"2024-04-26T21:28:56.093819+0000","last_fullsized":"2024-04-26T21:28:56.093819+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:41:36.551655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'3","reported_seq":125,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089026+0000","last_change":"2024-04-26T21:27:02.294738+0000","last_active":"2024-04-26T21:28:57.089026+0000","last_peered":"2024-04-26T21:28:57.089026+0000","last_clean":"2024-04-26T21:28:57.089026+0000","last_became_active":"2024-04-26T21:27:02.294338+0000","last_became_peered":"2024-04-26T21:27:02.294338+0000","last_unstale":"2024-04-26T21:28:57.089026+0000","last_undegraded":"2024-04-26T21:28:57.089026+0000","last_fullsized":"2024-04-26T21:28:57.089026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:25.309107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835802+0000","last_change":"2024-04-26T21:27:02.861332+0000","last_active":"2024-04-26T21:27:56.835802+0000","last_peered":"2024-04-26T21:27:56.835802+0000","last_clean":"2024-04-26T21:27:56.835802+0000","last_became_active":"2024-04-26T21:27:02.857865+0000","last_became_peered":"2024-04-26T21:27:02.857865+0000","last_unstale":"2024-04-26T21:27:56.835802+0000","last_undegraded":"2024-04-26T21:27:56.835802+0000","last_fullsized":"2024-04-26T21:27:56.835802+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:48.493086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150359+0000","last_change":"2024-04-26T21:27:07.383219+0000","last_active":"2024-04-26T21:28:57.150359+0000","last_peered":"2024-04-26T21:28:57.150359+0000","last_clean":"2024-04-26T21:28:57.150359+0000","last_became_active":"2024-04-26T21:27:07.382820+0000","last_became_peered":"2024-04-26T21:27:07.382820+0000","last_unstale":"2024-04-26T21:28:57.150359+0000","last_undegraded":"2024-04-26T21:28:57.150359+0000","last_fullsized":"2024-04-26T21:28:57.150359+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:07.752124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839757+0000","last_change":"2024-04-26T21:27:02.307746+0000","last_active":"2024-04-26T21:27:56.839757+0000","last_peered":"2024-04-26T21:27:56.839757+0000","last_clean":"2024-04-26T21:27:56.839757+0000","last_became_active":"2024-04-26T21:27:02.307406+0000","last_became_peered":"2024-04-26T21:27:02.307406+0000","last_unstale":"2024-04-26T21:27:56.839757+0000","last_undegraded":"2024-04-26T21:27:56.839757+0000","last_fullsized":"2024-04-26T21:27:56.839757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:13.799727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088903+0000","last_change":"2024-04-26T21:27:02.861194+0000","last_active":"2024-04-26T21:28:57.088903+0000","last_peered":"2024-04-26T21:28:57.088903+0000","last_clean":"2024-04-26T21:28:57.088903+0000","last_became_active":"2024-04-26T21:27:02.860871+0000","last_became_peered":"2024-04-26T21:27:02.860871+0000","last_unstale":"2024-04-26T21:28:57.088903+0000","last_undegraded":"2024-04-26T21:28:57.088903+0000","last_fullsized":"2024-04-26T21:28:57.088903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:56.829027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.144000+0000","last_change":"2024-04-26T21:28:57.143451+0000","last_active":"2024-04-26T21:28:57.144000+0000","last_peered":"2024-04-26T21:28:57.144000+0000","last_clean":"2024-04-26T21:28:57.144000+0000","last_became_active":"2024-04-26T21:28:57.142896+0000","last_became_peered":"2024-04-26T21:28:57.142896+0000","last_unstale":"2024-04-26T21:28:57.144000+0000","last_undegraded":"2024-04-26T21:28:57.144000+0000","last_fullsized":"2024-04-26T21:28:57.144000+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:40.800757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840231+0000","last_change":"2024-04-26T21:27:02.301711+0000","last_active":"2024-04-26T21:27:56.840231+0000","last_peered":"2024-04-26T21:27:56.840231+0000","last_clean":"2024-04-26T21:27:56.840231+0000","last_became_active":"2024-04-26T21:27:02.301429+0000","last_became_peered":"2024-04-26T21:27:02.301429+0000","last_unstale":"2024-04-26T21:27:56.840231+0000","last_undegraded":"2024-04-26T21:27:56.840231+0000","last_fullsized":"2024-04-26T21:27:56.840231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:50.876824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120572+0000","last_change":"2024-04-26T21:27:02.854372+0000","last_active":"2024-04-26T21:28:57.120572+0000","last_peered":"2024-04-26T21:28:57.120572+0000","last_clean":"2024-04-26T21:28:57.120572+0000","last_became_active":"2024-04-26T21:27:02.853826+0000","last_became_peered":"2024-04-26T21:27:02.853826+0000","last_unstale":"2024-04-26T21:28:57.120572+0000","last_undegraded":"2024-04-26T21:28:57.120572+0000","last_fullsized":"2024-04-26T21:28:57.120572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:08.253384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.840355+0000","last_change":"2024-04-26T21:27:07.506684+0000","last_active":"2024-04-26T21:27:56.840355+0000","last_peered":"2024-04-26T21:27:56.840355+0000","last_clean":"2024-04-26T21:27:56.840355+0000","last_became_active":"2024-04-26T21:27:07.506203+0000","last_became_peered":"2024-04-26T21:27:07.506203+0000","last_unstale":"2024-04-26T21:27:56.840355+0000","last_undegraded":"2024-04-26T21:27:56.840355+0000","last_fullsized":"2024-04-26T21:27:56.840355+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:42:48.580106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.088580+0000","last_change":"2024-04-26T21:27:02.306265+0000","last_active":"2024-04-26T21:28:56.088580+0000","last_peered":"2024-04-26T21:28:56.088580+0000","last_clean":"2024-04-26T21:28:56.088580+0000","last_became_active":"2024-04-26T21:27:02.305941+0000","last_became_peered":"2024-04-26T21:27:02.305941+0000","last_unstale":"2024-04-26T21:28:56.088580+0000","last_undegraded":"2024-04-26T21:28:56.088580+0000","last_fullsized":"2024-04-26T21:28:56.088580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:29.076833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088569+0000","last_change":"2024-04-26T21:27:02.871916+0000","last_active":"2024-04-26T21:28:57.088569+0000","last_peered":"2024-04-26T21:28:57.088569+0000","last_clean":"2024-04-26T21:28:57.088569+0000","last_became_active":"2024-04-26T21:27:02.871517+0000","last_became_peered":"2024-04-26T21:27:02.871517+0000","last_unstale":"2024-04-26T21:28:57.088569+0000","last_undegraded":"2024-04-26T21:28:57.088569+0000","last_fullsized":"2024-04-26T21:28:57.088569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:41.361757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148111+0000","last_change":"2024-04-26T21:27:07.383422+0000","last_active":"2024-04-26T21:28:57.148111+0000","last_peered":"2024-04-26T21:28:57.148111+0000","last_clean":"2024-04-26T21:28:57.148111+0000","last_became_active":"2024-04-26T21:27:07.382727+0000","last_became_peered":"2024-04-26T21:27:07.382727+0000","last_unstale":"2024-04-26T21:28:57.148111+0000","last_undegraded":"2024-04-26T21:28:57.148111+0000","last_fullsized":"2024-04-26T21:28:57.148111+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:22.002658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835916+0000","last_change":"2024-04-26T21:27:02.308081+0000","last_active":"2024-04-26T21:27:56.835916+0000","last_peered":"2024-04-26T21:27:56.835916+0000","last_clean":"2024-04-26T21:27:56.835916+0000","last_became_active":"2024-04-26T21:27:02.307715+0000","last_became_peered":"2024-04-26T21:27:02.307715+0000","last_unstale":"2024-04-26T21:27:56.835916+0000","last_undegraded":"2024-04-26T21:27:56.835916+0000","last_fullsized":"2024-04-26T21:27:56.835916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:28.967817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088446+0000","last_change":"2024-04-26T21:27:02.859527+0000","last_active":"2024-04-26T21:28:57.088446+0000","last_peered":"2024-04-26T21:28:57.088446+0000","last_clean":"2024-04-26T21:28:57.088446+0000","last_became_active":"2024-04-26T21:27:02.859173+0000","last_became_peered":"2024-04-26T21:27:02.859173+0000","last_unstale":"2024-04-26T21:28:57.088446+0000","last_undegraded":"2024-04-26T21:28:57.088446+0000","last_fullsized":"2024-04-26T21:28:57.088446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:24.811630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147035+0000","last_change":"2024-04-26T21:27:07.509778+0000","last_active":"2024-04-26T21:28:57.147035+0000","last_peered":"2024-04-26T21:28:57.147035+0000","last_clean":"2024-04-26T21:28:57.147035+0000","last_became_active":"2024-04-26T21:27:07.508344+0000","last_became_peered":"2024-04-26T21:27:07.508344+0000","last_unstale":"2024-04-26T21:28:57.147035+0000","last_undegraded":"2024-04-26T21:28:57.147035+0000","last_fullsized":"2024-04-26T21:28:57.147035+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:38:05.926717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2,"reported_epoch":94,"state":"peering","last_fresh":"2024-04-26T21:27:56.834808+0000","last_change":"2024-04-26T21:27:55.837544+0000","last_active":"2024-04-26T21:27:55.837544+0000","last_peered":"2024-04-26T21:27:08.397720+0000","last_clean":"2024-04-26T21:27:08.397720+0000","last_became_active":"2024-04-26T21:27:02.862499+0000","last_became_peered":"2024-04-26T21:27:02.862499+0000","last_unstale":"2024-04-26T21:27:56.834808+0000","last_undegraded":"2024-04-26T21:27:56.834808+0000","last_fullsized":"2024-04-26T21:27:56.834808+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":125,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147854+0000","last_change":"2024-04-26T21:27:02.317413+0000","last_active":"2024-04-26T21:28:57.147854+0000","last_peered":"2024-04-26T21:28:57.147854+0000","last_clean":"2024-04-26T21:28:57.147854+0000","last_became_active":"2024-04-26T21:27:02.316510+0000","last_became_peered":"2024-04-26T21:27:02.316510+0000","last_unstale":"2024-04-26T21:28:57.147854+0000","last_undegraded":"2024-04-26T21:28:57.147854+0000","last_fullsized":"2024-04-26T21:28:57.147854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:49.156366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836272+0000","last_change":"2024-04-26T21:27:07.373434+0000","last_active":"2024-04-26T21:27:56.836272+0000","last_peered":"2024-04-26T21:27:56.836272+0000","last_clean":"2024-04-26T21:27:56.836272+0000","last_became_active":"2024-04-26T21:27:07.372973+0000","last_became_peered":"2024-04-26T21:27:07.372973+0000","last_unstale":"2024-04-26T21:27:56.836272+0000","last_undegraded":"2024-04-26T21:27:56.836272+0000","last_fullsized":"2024-04-26T21:27:56.836272+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:04.022651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839543+0000","last_change":"2024-04-26T21:27:02.862734+0000","last_active":"2024-04-26T21:27:56.839543+0000","last_peered":"2024-04-26T21:27:56.839543+0000","last_clean":"2024-04-26T21:27:56.839543+0000","last_became_active":"2024-04-26T21:27:02.862334+0000","last_became_peered":"2024-04-26T21:27:02.862334+0000","last_unstale":"2024-04-26T21:27:56.839543+0000","last_undegraded":"2024-04-26T21:27:56.839543+0000","last_fullsized":"2024-04-26T21:27:56.839543+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:09.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089403+0000","last_change":"2024-04-26T21:27:02.302471+0000","last_active":"2024-04-26T21:28:57.089403+0000","last_peered":"2024-04-26T21:28:57.089403+0000","last_clean":"2024-04-26T21:28:57.089403+0000","last_became_active":"2024-04-26T21:27:02.302055+0000","last_became_peered":"2024-04-26T21:27:02.302055+0000","last_unstale":"2024-04-26T21:28:57.089403+0000","last_undegraded":"2024-04-26T21:28:57.089403+0000","last_fullsized":"2024-04-26T21:28:57.089403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:29:14.649670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":363,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.835725+0000","last_change":"2024-04-26T21:25:41.647633+0000","last_active":"2024-04-26T21:27:56.835725+0000","last_peered":"2024-04-26T21:27:56.835725+0000","last_clean":"2024-04-26T21:27:56.835725+0000","last_became_active":"2024-04-26T21:25:41.647091+0000","last_became_peered":"2024-04-26T21:25:41.647091+0000","last_unstale":"2024-04-26T21:27:56.835725+0000","last_undegraded":"2024-04-26T21:27:56.835725+0000","last_fullsized":"2024-04-26T21:27:56.835725+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_clean_scrub_stamp":"2024-04-26T21:21:43.547929+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:37.666065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.152820+0000","last_change":"2024-04-26T21:28:57.152820+0000","last_active":"2024-04-26T21:28:57.152820+0000","last_peered":"2024-04-26T21:28:57.152820+0000","last_clean":"2024-04-26T21:28:57.152820+0000","last_became_active":"2024-04-26T21:28:57.152453+0000","last_became_peered":"2024-04-26T21:28:57.152453+0000","last_unstale":"2024-04-26T21:28:57.152820+0000","last_undegraded":"2024-04-26T21:28:57.152820+0000","last_fullsized":"2024-04-26T21:28:57.152820+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:16:47.634651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147552+0000","last_change":"2024-04-26T21:27:02.874116+0000","last_active":"2024-04-26T21:28:57.147552+0000","last_peered":"2024-04-26T21:28:57.147552+0000","last_clean":"2024-04-26T21:28:57.147552+0000","last_became_active":"2024-04-26T21:27:02.871334+0000","last_became_peered":"2024-04-26T21:27:02.871334+0000","last_unstale":"2024-04-26T21:28:57.147552+0000","last_undegraded":"2024-04-26T21:28:57.147552+0000","last_fullsized":"2024-04-26T21:28:57.147552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:28.835061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120084+0000","last_change":"2024-04-26T21:27:56.847702+0000","last_active":"2024-04-26T21:28:57.120084+0000","last_peered":"2024-04-26T21:28:57.120084+0000","last_clean":"2024-04-26T21:28:57.120084+0000","last_became_active":"2024-04-26T21:27:56.844034+0000","last_became_peered":"2024-04-26T21:27:56.844034+0000","last_unstale":"2024-04-26T21:28:57.120084+0000","last_undegraded":"2024-04-26T21:28:57.120084+0000","last_fullsized":"2024-04-26T21:28:57.120084+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:54.631297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120208+0000","last_change":"2024-04-26T21:27:07.373894+0000","last_active":"2024-04-26T21:28:57.120208+0000","last_peered":"2024-04-26T21:28:57.120208+0000","last_clean":"2024-04-26T21:28:57.120208+0000","last_became_active":"2024-04-26T21:27:07.373338+0000","last_became_peered":"2024-04-26T21:27:07.373338+0000","last_unstale":"2024-04-26T21:28:57.120208+0000","last_undegraded":"2024-04-26T21:28:57.120208+0000","last_fullsized":"2024-04-26T21:28:57.120208+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:49.636493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120527+0000","last_change":"2024-04-26T21:27:56.858975+0000","last_active":"2024-04-26T21:28:57.120527+0000","last_peered":"2024-04-26T21:28:57.120527+0000","last_clean":"2024-04-26T21:28:57.120527+0000","last_became_active":"2024-04-26T21:27:56.850022+0000","last_became_peered":"2024-04-26T21:27:56.850022+0000","last_unstale":"2024-04-26T21:28:57.120527+0000","last_undegraded":"2024-04-26T21:28:57.120527+0000","last_fullsized":"2024-04-26T21:28:57.120527+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:43:31.860906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148536+0000","last_change":"2024-04-26T21:27:02.869831+0000","last_active":"2024-04-26T21:28:57.148536+0000","last_peered":"2024-04-26T21:28:57.148536+0000","last_clean":"2024-04-26T21:28:57.148536+0000","last_became_active":"2024-04-26T21:27:02.869375+0000","last_became_peered":"2024-04-26T21:27:02.869375+0000","last_unstale":"2024-04-26T21:28:57.148536+0000","last_undegraded":"2024-04-26T21:28:57.148536+0000","last_fullsized":"2024-04-26T21:28:57.148536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:20.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838445+0000","last_change":"2024-04-26T21:27:07.507562+0000","last_active":"2024-04-26T21:27:56.838445+0000","last_peered":"2024-04-26T21:27:56.838445+0000","last_clean":"2024-04-26T21:27:56.838445+0000","last_became_active":"2024-04-26T21:27:07.507063+0000","last_became_peered":"2024-04-26T21:27:07.507063+0000","last_unstale":"2024-04-26T21:27:56.838445+0000","last_undegraded":"2024-04-26T21:27:56.838445+0000","last_fullsized":"2024-04-26T21:27:56.838445+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:58.192617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.096008+0000","last_change":"2024-04-26T21:27:02.315360+0000","last_active":"2024-04-26T21:28:56.096008+0000","last_peered":"2024-04-26T21:28:56.096008+0000","last_clean":"2024-04-26T21:28:56.096008+0000","last_became_active":"2024-04-26T21:27:02.314817+0000","last_became_peered":"2024-04-26T21:27:02.314817+0000","last_unstale":"2024-04-26T21:28:56.096008+0000","last_undegraded":"2024-04-26T21:28:56.096008+0000","last_fullsized":"2024-04-26T21:28:56.096008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:12.045363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":2,"reported_epoch":94,"state":"peering","last_fresh":"2024-04-26T21:27:56.841659+0000","last_change":"2024-04-26T21:28:56.105878+0000","last_active":"2024-04-26T21:28:56.105878+0000","last_peered":"2024-04-26T21:27:56.841659+0000","last_clean":"2024-04-26T21:27:56.841659+0000","last_became_active":"2024-04-26T21:27:02.856589+0000","last_became_peered":"2024-04-26T21:27:02.856589+0000","last_unstale":"2024-04-26T21:27:56.841659+0000","last_undegraded":"2024-04-26T21:27:56.841659+0000","last_fullsized":"2024-04-26T21:27:56.841659+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.139990+0000","last_change":"2024-04-26T21:27:07.362016+0000","last_active":"2024-04-26T21:28:57.139990+0000","last_peered":"2024-04-26T21:28:57.139990+0000","last_clean":"2024-04-26T21:28:57.139990+0000","last_became_active":"2024-04-26T21:27:07.361360+0000","last_became_peered":"2024-04-26T21:27:07.361360+0000","last_unstale":"2024-04-26T21:28:57.139990+0000","last_undegraded":"2024-04-26T21:28:57.139990+0000","last_fullsized":"2024-04-26T21:28:57.139990+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:03.196293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089562+0000","last_change":"2024-04-26T21:27:02.308088+0000","last_active":"2024-04-26T21:28:57.089562+0000","last_peered":"2024-04-26T21:28:57.089562+0000","last_clean":"2024-04-26T21:28:57.089562+0000","last_became_active":"2024-04-26T21:27:02.307727+0000","last_became_peered":"2024-04-26T21:27:02.307727+0000","last_unstale":"2024-04-26T21:28:57.089562+0000","last_undegraded":"2024-04-26T21:28:57.089562+0000","last_fullsized":"2024-04-26T21:28:57.089562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:02.132125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"peering","last_fresh":"2024-04-26T21:27:56.845009+0000","last_change":"2024-04-26T21:28:56.086395+0000","last_active":"2024-04-26T21:28:56.086395+0000","last_peered":"2024-04-26T21:27:56.845009+0000","last_clean":"2024-04-26T21:27:56.845009+0000","last_became_active":"2024-04-26T21:27:56.840445+0000","last_became_peered":"2024-04-26T21:27:56.840445+0000","last_unstale":"2024-04-26T21:27:56.845009+0000","last_undegraded":"2024-04-26T21:27:56.845009+0000","last_fullsized":"2024-04-26T21:27:56.845009+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089207+0000","last_change":"2024-04-26T21:27:07.506609+0000","last_active":"2024-04-26T21:28:57.089207+0000","last_peered":"2024-04-26T21:28:57.089207+0000","last_clean":"2024-04-26T21:28:57.089207+0000","last_became_active":"2024-04-26T21:27:07.506100+0000","last_became_peered":"2024-04-26T21:27:07.506100+0000","last_unstale":"2024-04-26T21:28:57.089207+0000","last_undegraded":"2024-04-26T21:28:57.089207+0000","last_fullsized":"2024-04-26T21:28:57.089207+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:26.535809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142097+0000","last_change":"2024-04-26T21:27:02.297432+0000","last_active":"2024-04-26T21:28:57.142097+0000","last_peered":"2024-04-26T21:28:57.142097+0000","last_clean":"2024-04-26T21:28:57.142097+0000","last_became_active":"2024-04-26T21:27:02.297081+0000","last_became_peered":"2024-04-26T21:27:02.297081+0000","last_unstale":"2024-04-26T21:28:57.142097+0000","last_undegraded":"2024-04-26T21:28:57.142097+0000","last_fullsized":"2024-04-26T21:28:57.142097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:08.139469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839917+0000","last_change":"2024-04-26T21:27:02.866653+0000","last_active":"2024-04-26T21:27:56.839917+0000","last_peered":"2024-04-26T21:27:56.839917+0000","last_clean":"2024-04-26T21:27:56.839917+0000","last_became_active":"2024-04-26T21:27:02.866275+0000","last_became_peered":"2024-04-26T21:27:02.866275+0000","last_unstale":"2024-04-26T21:27:56.839917+0000","last_undegraded":"2024-04-26T21:27:56.839917+0000","last_fullsized":"2024-04-26T21:27:56.839917+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:39.089972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.854891+0000","last_change":"2024-04-26T21:27:07.377084+0000","last_active":"2024-04-26T21:27:56.854891+0000","last_peered":"2024-04-26T21:27:56.854891+0000","last_clean":"2024-04-26T21:27:56.854891+0000","last_became_active":"2024-04-26T21:27:07.374052+0000","last_became_peered":"2024-04-26T21:27:07.374052+0000","last_unstale":"2024-04-26T21:27:56.854891+0000","last_undegraded":"2024-04-26T21:27:56.854891+0000","last_fullsized":"2024-04-26T21:27:56.854891+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:06:20.031170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838351+0000","last_change":"2024-04-26T21:27:02.307250+0000","last_active":"2024-04-26T21:27:56.838351+0000","last_peered":"2024-04-26T21:27:56.838351+0000","last_clean":"2024-04-26T21:27:56.838351+0000","last_became_active":"2024-04-26T21:27:02.303532+0000","last_became_peered":"2024-04-26T21:27:02.303532+0000","last_unstale":"2024-04-26T21:27:56.838351+0000","last_undegraded":"2024-04-26T21:27:56.838351+0000","last_fullsized":"2024-04-26T21:27:56.838351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:36.265188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120172+0000","last_change":"2024-04-26T21:27:02.858818+0000","last_active":"2024-04-26T21:28:57.120172+0000","last_peered":"2024-04-26T21:28:57.120172+0000","last_clean":"2024-04-26T21:28:57.120172+0000","last_became_active":"2024-04-26T21:27:02.858059+0000","last_became_peered":"2024-04-26T21:27:02.858059+0000","last_unstale":"2024-04-26T21:28:57.120172+0000","last_undegraded":"2024-04-26T21:28:57.120172+0000","last_fullsized":"2024-04-26T21:28:57.120172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:02.159334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088650+0000","last_change":"2024-04-26T21:27:07.381273+0000","last_active":"2024-04-26T21:28:57.088650+0000","last_peered":"2024-04-26T21:28:57.088650+0000","last_clean":"2024-04-26T21:28:57.088650+0000","last_became_active":"2024-04-26T21:27:07.380891+0000","last_became_peered":"2024-04-26T21:27:07.380891+0000","last_unstale":"2024-04-26T21:28:57.088650+0000","last_undegraded":"2024-04-26T21:28:57.088650+0000","last_fullsized":"2024-04-26T21:28:57.088650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:39.641577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140340+0000","last_change":"2024-04-26T21:27:02.301363+0000","last_active":"2024-04-26T21:28:57.140340+0000","last_peered":"2024-04-26T21:28:57.140340+0000","last_clean":"2024-04-26T21:28:57.140340+0000","last_became_active":"2024-04-26T21:27:02.301059+0000","last_became_peered":"2024-04-26T21:27:02.301059+0000","last_unstale":"2024-04-26T21:28:57.140340+0000","last_undegraded":"2024-04-26T21:28:57.140340+0000","last_fullsized":"2024-04-26T21:28:57.140340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:17.586190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.841887+0000","last_change":"2024-04-26T21:27:02.860612+0000","last_active":"2024-04-26T21:27:56.841887+0000","last_peered":"2024-04-26T21:27:56.841887+0000","last_clean":"2024-04-26T21:27:56.841887+0000","last_became_active":"2024-04-26T21:27:02.857285+0000","last_became_peered":"2024-04-26T21:27:02.857285+0000","last_unstale":"2024-04-26T21:27:56.841887+0000","last_undegraded":"2024-04-26T21:27:56.841887+0000","last_fullsized":"2024-04-26T21:27:56.841887+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:11.968818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089342+0000","last_change":"2024-04-26T21:27:07.506425+0000","last_active":"2024-04-26T21:28:57.089342+0000","last_peered":"2024-04-26T21:28:57.089342+0000","last_clean":"2024-04-26T21:28:57.089342+0000","last_became_active":"2024-04-26T21:27:07.505968+0000","last_became_peered":"2024-04-26T21:27:07.505968+0000","last_unstale":"2024-04-26T21:28:57.089342+0000","last_undegraded":"2024-04-26T21:28:57.089342+0000","last_fullsized":"2024-04-26T21:28:57.089342+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:59.914560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.839708+0000","last_change":"2024-04-26T21:27:02.294635+0000","last_active":"2024-04-26T21:27:56.839708+0000","last_peered":"2024-04-26T21:27:56.839708+0000","last_clean":"2024-04-26T21:27:56.839708+0000","last_became_active":"2024-04-26T21:27:02.294311+0000","last_became_peered":"2024-04-26T21:27:02.294311+0000","last_unstale":"2024-04-26T21:27:56.839708+0000","last_undegraded":"2024-04-26T21:27:56.839708+0000","last_fullsized":"2024-04-26T21:27:56.839708+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:47.906546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836051+0000","last_change":"2024-04-26T21:27:02.875475+0000","last_active":"2024-04-26T21:27:56.836051+0000","last_peered":"2024-04-26T21:27:56.836051+0000","last_clean":"2024-04-26T21:27:56.836051+0000","last_became_active":"2024-04-26T21:27:02.875137+0000","last_became_peered":"2024-04-26T21:27:02.875137+0000","last_unstale":"2024-04-26T21:27:56.836051+0000","last_undegraded":"2024-04-26T21:27:56.836051+0000","last_fullsized":"2024-04-26T21:27:56.836051+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:04.988973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141589+0000","last_change":"2024-04-26T21:27:07.372927+0000","last_active":"2024-04-26T21:28:57.141589+0000","last_peered":"2024-04-26T21:28:57.141589+0000","last_clean":"2024-04-26T21:28:57.141589+0000","last_became_active":"2024-04-26T21:27:07.372371+0000","last_became_peered":"2024-04-26T21:27:07.372371+0000","last_unstale":"2024-04-26T21:28:57.141589+0000","last_undegraded":"2024-04-26T21:28:57.141589+0000","last_fullsized":"2024-04-26T21:28:57.141589+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:12:41.607167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140654+0000","last_change":"2024-04-26T21:27:02.295379+0000","last_active":"2024-04-26T21:28:57.140654+0000","last_peered":"2024-04-26T21:28:57.140654+0000","last_clean":"2024-04-26T21:28:57.140654+0000","last_became_active":"2024-04-26T21:27:02.295077+0000","last_became_peered":"2024-04-26T21:27:02.295077+0000","last_unstale":"2024-04-26T21:28:57.140654+0000","last_undegraded":"2024-04-26T21:28:57.140654+0000","last_fullsized":"2024-04-26T21:28:57.140654+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:58.177960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836567+0000","last_change":"2024-04-26T21:27:02.855998+0000","last_active":"2024-04-26T21:27:56.836567+0000","last_peered":"2024-04-26T21:27:56.836567+0000","last_clean":"2024-04-26T21:27:56.836567+0000","last_became_active":"2024-04-26T21:27:02.855636+0000","last_became_peered":"2024-04-26T21:27:02.855636+0000","last_unstale":"2024-04-26T21:27:56.836567+0000","last_undegraded":"2024-04-26T21:27:56.836567+0000","last_fullsized":"2024-04-26T21:27:56.836567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:19:02.988946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.084873+0000","last_change":"2024-04-26T21:27:07.378980+0000","last_active":"2024-04-26T21:28:56.084873+0000","last_peered":"2024-04-26T21:28:56.084873+0000","last_clean":"2024-04-26T21:28:56.084873+0000","last_became_active":"2024-04-26T21:27:07.378374+0000","last_became_peered":"2024-04-26T21:27:07.378374+0000","last_unstale":"2024-04-26T21:28:56.084873+0000","last_undegraded":"2024-04-26T21:28:56.084873+0000","last_fullsized":"2024-04-26T21:28:56.084873+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:31.845297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089815+0000","last_change":"2024-04-26T21:27:02.313801+0000","last_active":"2024-04-26T21:28:57.089815+0000","last_peered":"2024-04-26T21:28:57.089815+0000","last_clean":"2024-04-26T21:28:57.089815+0000","last_became_active":"2024-04-26T21:27:02.313540+0000","last_became_peered":"2024-04-26T21:27:02.313540+0000","last_unstale":"2024-04-26T21:28:57.089815+0000","last_undegraded":"2024-04-26T21:28:57.089815+0000","last_fullsized":"2024-04-26T21:28:57.089815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:38.612261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836372+0000","last_change":"2024-04-26T21:27:02.864151+0000","last_active":"2024-04-26T21:27:56.836372+0000","last_peered":"2024-04-26T21:27:56.836372+0000","last_clean":"2024-04-26T21:27:56.836372+0000","last_became_active":"2024-04-26T21:27:02.863818+0000","last_became_peered":"2024-04-26T21:27:02.863818+0000","last_unstale":"2024-04-26T21:27:56.836372+0000","last_undegraded":"2024-04-26T21:27:56.836372+0000","last_fullsized":"2024-04-26T21:27:56.836372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:26.215370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088554+0000","last_change":"2024-04-26T21:27:07.365821+0000","last_active":"2024-04-26T21:28:57.088554+0000","last_peered":"2024-04-26T21:28:57.088554+0000","last_clean":"2024-04-26T21:28:57.088554+0000","last_became_active":"2024-04-26T21:27:07.365347+0000","last_became_peered":"2024-04-26T21:27:07.365347+0000","last_unstale":"2024-04-26T21:28:57.088554+0000","last_undegraded":"2024-04-26T21:28:57.088554+0000","last_fullsized":"2024-04-26T21:28:57.088554+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:40:11.596908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150070+0000","last_change":"2024-04-26T21:27:02.310641+0000","last_active":"2024-04-26T21:28:57.150070+0000","last_peered":"2024-04-26T21:28:57.150070+0000","last_clean":"2024-04-26T21:28:57.150070+0000","last_became_active":"2024-04-26T21:27:02.308569+0000","last_became_peered":"2024-04-26T21:27:02.308569+0000","last_unstale":"2024-04-26T21:28:57.150070+0000","last_undegraded":"2024-04-26T21:28:57.150070+0000","last_fullsized":"2024-04-26T21:28:57.150070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:20.988098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.836482+0000","last_change":"2024-04-26T21:27:02.854976+0000","last_active":"2024-04-26T21:27:56.836482+0000","last_peered":"2024-04-26T21:27:56.836482+0000","last_clean":"2024-04-26T21:27:56.836482+0000","last_became_active":"2024-04-26T21:27:02.854611+0000","last_became_peered":"2024-04-26T21:27:02.854611+0000","last_unstale":"2024-04-26T21:27:56.836482+0000","last_undegraded":"2024-04-26T21:27:56.836482+0000","last_fullsized":"2024-04-26T21:27:56.836482+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:57.065657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120248+0000","last_change":"2024-04-26T21:27:02.859017+0000","last_active":"2024-04-26T21:28:57.120248+0000","last_peered":"2024-04-26T21:28:57.120248+0000","last_clean":"2024-04-26T21:28:57.120248+0000","last_became_active":"2024-04-26T21:27:02.858507+0000","last_became_peered":"2024-04-26T21:27:02.858507+0000","last_unstale":"2024-04-26T21:28:57.120248+0000","last_undegraded":"2024-04-26T21:28:57.120248+0000","last_fullsized":"2024-04-26T21:28:57.120248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:14.850891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140387+0000","last_change":"2024-04-26T21:27:56.841251+0000","last_active":"2024-04-26T21:28:57.140387+0000","last_peered":"2024-04-26T21:28:57.140387+0000","last_clean":"2024-04-26T21:28:57.140387+0000","last_became_active":"2024-04-26T21:27:56.840862+0000","last_became_peered":"2024-04-26T21:27:56.840862+0000","last_unstale":"2024-04-26T21:28:57.140387+0000","last_undegraded":"2024-04-26T21:28:57.140387+0000","last_fullsized":"2024-04-26T21:28:57.140387+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:41.460343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140100+0000","last_change":"2024-04-26T21:27:02.868817+0000","last_active":"2024-04-26T21:28:57.140100+0000","last_peered":"2024-04-26T21:28:57.140100+0000","last_clean":"2024-04-26T21:28:57.140100+0000","last_became_active":"2024-04-26T21:27:02.868557+0000","last_became_peered":"2024-04-26T21:27:02.868557+0000","last_unstale":"2024-04-26T21:28:57.140100+0000","last_undegraded":"2024-04-26T21:28:57.140100+0000","last_fullsized":"2024-04-26T21:28:57.140100+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:55.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":27,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-04-26T21:28:56.085535+0000","last_change":"2024-04-26T21:27:02.876029+0000","last_active":"2024-04-26T21:28:56.085535+0000","last_peered":"2024-04-26T21:28:56.085535+0000","last_clean":"2024-04-26T21:28:56.085535+0000","last_became_active":"2024-04-26T21:27:02.875656+0000","last_became_peered":"2024-04-26T21:27:02.875656+0000","last_unstale":"2024-04-26T21:28:56.085535+0000","last_undegraded":"2024-04-26T21:28:56.085535+0000","last_fullsized":"2024-04-26T21:28:56.085535+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:47.317581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150613+0000","last_change":"2024-04-26T21:27:02.873818+0000","last_active":"2024-04-26T21:28:57.150613+0000","last_peered":"2024-04-26T21:28:57.150613+0000","last_clean":"2024-04-26T21:28:57.150613+0000","last_became_active":"2024-04-26T21:27:02.873459+0000","last_became_peered":"2024-04-26T21:27:02.873459+0000","last_unstale":"2024-04-26T21:28:57.150613+0000","last_undegraded":"2024-04-26T21:28:57.150613+0000","last_fullsized":"2024-04-26T21:28:57.150613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:07.259187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120678+0000","last_change":"2024-04-26T21:27:02.854901+0000","last_active":"2024-04-26T21:28:57.120678+0000","last_peered":"2024-04-26T21:28:57.120678+0000","last_clean":"2024-04-26T21:28:57.120678+0000","last_became_active":"2024-04-26T21:27:02.854547+0000","last_became_peered":"2024-04-26T21:27:02.854547+0000","last_unstale":"2024-04-26T21:28:57.120678+0000","last_undegraded":"2024-04-26T21:28:57.120678+0000","last_fullsized":"2024-04-26T21:28:57.120678+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:24.320868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141003+0000","last_change":"2024-04-26T21:27:02.856616+0000","last_active":"2024-04-26T21:28:57.141003+0000","last_peered":"2024-04-26T21:28:57.141003+0000","last_clean":"2024-04-26T21:28:57.141003+0000","last_became_active":"2024-04-26T21:27:02.856264+0000","last_became_peered":"2024-04-26T21:27:02.856264+0000","last_unstale":"2024-04-26T21:28:57.141003+0000","last_undegraded":"2024-04-26T21:28:57.141003+0000","last_fullsized":"2024-04-26T21:28:57.141003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:43.374534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.842062+0000","last_change":"2024-04-26T21:27:02.859587+0000","last_active":"2024-04-26T21:27:56.842062+0000","last_peered":"2024-04-26T21:27:56.842062+0000","last_clean":"2024-04-26T21:27:56.842062+0000","last_became_active":"2024-04-26T21:27:02.859136+0000","last_became_peered":"2024-04-26T21:27:02.859136+0000","last_unstale":"2024-04-26T21:27:56.842062+0000","last_undegraded":"2024-04-26T21:27:56.842062+0000","last_fullsized":"2024-04-26T21:27:56.842062+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:09.370319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088720+0000","last_change":"2024-04-26T21:27:02.864265+0000","last_active":"2024-04-26T21:28:57.088720+0000","last_peered":"2024-04-26T21:28:57.088720+0000","last_clean":"2024-04-26T21:28:57.088720+0000","last_became_active":"2024-04-26T21:27:02.863920+0000","last_became_peered":"2024-04-26T21:27:02.863920+0000","last_unstale":"2024-04-26T21:28:57.088720+0000","last_undegraded":"2024-04-26T21:28:57.088720+0000","last_fullsized":"2024-04-26T21:28:57.088720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:13.941544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-26T21:27:56.838672+0000","last_change":"2024-04-26T21:27:02.860189+0000","last_active":"2024-04-26T21:27:56.838672+0000","last_peered":"2024-04-26T21:27:56.838672+0000","last_clean":"2024-04-26T21:27:56.838672+0000","last_became_active":"2024-04-26T21:27:02.859843+0000","last_became_peered":"2024-04-26T21:27:02.859843+0000","last_unstale":"2024-04-26T21:27:56.838672+0000","last_undegraded":"2024-04-26T21:27:56.838672+0000","last_fullsized":"2024-04-26T21:27:56.838672+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141963+0000","last_change":"2024-04-26T21:27:02.865459+0000","last_active":"2024-04-26T21:28:57.141963+0000","last_peered":"2024-04-26T21:28:57.141963+0000","last_clean":"2024-04-26T21:28:57.141963+0000","last_became_active":"2024-04-26T21:27:02.863602+0000","last_became_peered":"2024-04-26T21:27:02.863602+0000","last_unstale":"2024-04-26T21:28:57.141963+0000","last_undegraded":"2024-04-26T21:28:57.141963+0000","last_fullsized":"2024-04-26T21:28:57.141963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:03.733858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.149411+0000","last_change":"2024-04-26T21:27:02.873711+0000","last_active":"2024-04-26T21:28:57.149411+0000","last_peered":"2024-04-26T21:28:57.149411+0000","last_clean":"2024-04-26T21:28:57.149411+0000","last_became_active":"2024-04-26T21:27:02.873349+0000","last_became_peered":"2024-04-26T21:27:02.873349+0000","last_unstale":"2024-04-26T21:28:57.149411+0000","last_undegraded":"2024-04-26T21:28:57.149411+0000","last_fullsized":"2024-04-26T21:28:57.149411+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:10.841536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120403+0000","last_change":"2024-04-26T21:27:02.852625+0000","last_active":"2024-04-26T21:28:57.120403+0000","last_peered":"2024-04-26T21:28:57.120403+0000","last_clean":"2024-04-26T21:28:57.120403+0000","last_became_active":"2024-04-26T21:27:02.852177+0000","last_became_peered":"2024-04-26T21:27:02.852177+0000","last_unstale":"2024-04-26T21:28:57.120403+0000","last_undegraded":"2024-04-26T21:28:57.120403+0000","last_fullsized":"2024-04-26T21:28:57.120403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:52.046678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59286,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":563,"num_read_kb":345,"num_write":161,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318310,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":528427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579947,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28740,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716412,"statfs":{"total":95995035648,"available":95965605888,"internally_reserved":0,"allocated":1761280,"data_stored":1171349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743472,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1064960,"data_stored":513572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005111,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1056768,"data_stored":509374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266748,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32800,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712352,"statfs":{"total":95995035648,"available":95961448448,"internally_reserved":0,"allocated":1654784,"data_stored":1104914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":523817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1134592,"data_stored":579403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1081344,"data_stored":515712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28136,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717016,"statfs":{"total":95995035648,"available":95966224384,"internally_reserved":0,"allocated":1142784,"data_stored":571260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1052672,"data_stored":504846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1069056,"data_stored":518974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724616,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":1679360,"data_stored":1110174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":55676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":57026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":57254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:29:02.232 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T21:29:02.727 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:03.355 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:03 smithi001 ceph-mon[28054]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:03.356 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:03 smithi001 ceph-mon[28054]: pgmap v388: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-26T21:29:03.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:03 smithi186 ceph-mon[32991]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:03.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:03 smithi186 ceph-mon[32991]: pgmap v388: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-26T21:29:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:03 smithi038 ceph-mon[32724]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:03.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:03 smithi038 ceph-mon[32724]: pgmap v388: 129 pgs: 1 peering, 128 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 127 B/s wr, 6 op/s 2024-04-26T21:29:04.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:04 smithi186 ceph-mon[32991]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T21:29:04.559 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:04 smithi001 ceph-mon[28054]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T21:29:04.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:04 smithi038 ceph-mon[32724]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T21:29:05.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.115+0000 7f429ea00700 1 -- 172.21.15.1:0/3224834406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42981098e0 msgr2=0x7f4298110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:05.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.115+0000 7f429ea00700 1 --2- 172.21.15.1:0/3224834406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42981098e0 0x7f4298110310 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f429000b0a0 tx=0x7f429000b3b0 comp rx=0 tx=0).stop 2024-04-26T21:29:05.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.115+0000 7f429ea00700 1 -- 172.21.15.1:0/3224834406 shutdown_connections 2024-04-26T21:29:05.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.115+0000 7f429ea00700 1 --2- 172.21.15.1:0/3224834406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4298101800 0x7f4298101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:05.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.115+0000 7f429ea00700 1 --2- 172.21.15.1:0/3224834406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42981098e0 0x7f4298110310 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:05.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.115+0000 7f429ea00700 1 --2- 172.21.15.1:0/3224834406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f42981021b0 0x7f4298109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:05.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.115+0000 7f429ea00700 1 -- 172.21.15.1:0/3224834406 >> 172.21.15.1:0/3224834406 conn(0x7f4298078520 msgr2=0x7f42980ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:05.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.115+0000 7f429ea00700 1 -- 172.21.15.1:0/3224834406 shutdown_connections 2024-04-26T21:29:05.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.115+0000 7f429ea00700 1 -- 172.21.15.1:0/3224834406 wait complete. 2024-04-26T21:29:05.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.117+0000 7f429ea00700 1 Processor -- start 2024-04-26T21:29:05.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.117+0000 7f429ea00700 1 -- start start 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.117+0000 7f429ea00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4298101800 0x7f429819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.117+0000 7f429ea00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42981021b0 0x7f429819fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.117+0000 7f429ea00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42981098e0 0x7f4298199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.117+0000 7f429ea00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f42981135f0 con 0x7f4298101800 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.117+0000 7f429ea00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4298113470 con 0x7f42981098e0 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.117+0000 7f429ea00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4298113770 con 0x7f42981021b0 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f4297e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4298101800 0x7f429819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f4297e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4298101800 0x7f429819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44880/0 (socket says 172.21.15.1:44880) 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f4297e00700 1 -- 172.21.15.1:0/2442961047 learned_addr learned my addr 172.21.15.1:0/2442961047 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f429ca00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42981098e0 0x7f4298199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:05.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f4297400700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42981021b0 0x7f429819fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:05.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f429ca00700 1 -- 172.21.15.1:0/2442961047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42981021b0 msgr2=0x7f429819fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:05.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f429ca00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42981021b0 0x7f429819fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:05.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f429ca00700 1 -- 172.21.15.1:0/2442961047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4298101800 msgr2=0x7f429819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:05.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f429ca00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4298101800 0x7f429819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:05.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f429ca00700 1 -- 172.21.15.1:0/2442961047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429819a070 con 0x7f42981098e0 2024-04-26T21:29:05.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f4297e00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4298101800 0x7f429819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:05.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.118+0000 7f4297400700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42981021b0 0x7f429819fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:29:05.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.119+0000 7f429ca00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42981098e0 0x7f4298199890 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f429000b0a0 tx=0x7f4290004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:05.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.119+0000 7f4294c00700 1 -- 172.21.15.1:0/2442961047 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4290021070 con 0x7f42981098e0 2024-04-26T21:29:05.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.119+0000 7f4294c00700 1 -- 172.21.15.1:0/2442961047 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4290007b80 con 0x7f42981098e0 2024-04-26T21:29:05.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.119+0000 7f429ea00700 1 -- 172.21.15.1:0/2442961047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f429819a300 con 0x7f42981098e0 2024-04-26T21:29:05.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.119+0000 7f4294c00700 1 -- 172.21.15.1:0/2442961047 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4290005040 con 0x7f42981098e0 2024-04-26T21:29:05.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.119+0000 7f429ea00700 1 -- 172.21.15.1:0/2442961047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42981a6520 con 0x7f42981098e0 2024-04-26T21:29:05.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.120+0000 7f429ea00700 1 -- 172.21.15.1:0/2442961047 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4298102930 con 0x7f42981098e0 2024-04-26T21:29:05.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.123+0000 7f4294c00700 1 -- 172.21.15.1:0/2442961047 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f4290024070 con 0x7f42981098e0 2024-04-26T21:29:05.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.124+0000 7f4294c00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f427c0a3d50 0x7f427c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.124+0000 7f4294c00700 1 -- 172.21.15.1:0/2442961047 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f429000f020 con 0x7f42981098e0 2024-04-26T21:29:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.124+0000 7f4297e00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f427c0a3d50 0x7f427c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:05.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.125+0000 7f4297e00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f427c0a3d50 0x7f427c0a6210 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4288011550 tx=0x7f4288009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:05.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.127+0000 7f4294c00700 1 -- 172.21.15.1:0/2442961047 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f429002b020 con 0x7f42981098e0 2024-04-26T21:29:05.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.358+0000 7f429ea00700 1 -- 172.21.15.1:0/2442961047 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f429819ad70 con 0x7f427c0a3d50 2024-04-26T21:29:05.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.387+0000 7f4294c00700 1 -- 172.21.15.1:0/2442961047 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340430 (secure 0 0 0) 0x7f429819ad70 con 0x7f427c0a3d50 2024-04-26T21:29:05.389 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:05 smithi001 ceph-mon[28054]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 5 op/s 2024-04-26T21:29:05.389 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:05 smithi001 ceph-mon[28054]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T21:29:05.389 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:05 smithi001 ceph-mon[28054]: Cluster is now healthy 2024-04-26T21:29:05.390 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:05.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.394+0000 7f429ea00700 1 -- 172.21.15.1:0/2442961047 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f427c0a3d50 msgr2=0x7f427c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.394+0000 7f429ea00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f427c0a3d50 0x7f427c0a6210 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4288011550 tx=0x7f4288009260 comp rx=0 tx=0).stop 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.394+0000 7f429ea00700 1 -- 172.21.15.1:0/2442961047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42981098e0 msgr2=0x7f4298199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.394+0000 7f429ea00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42981098e0 0x7f4298199890 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f429000b0a0 tx=0x7f4290004d00 comp rx=0 tx=0).stop 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.395+0000 7f429ea00700 1 -- 172.21.15.1:0/2442961047 shutdown_connections 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.395+0000 7f429ea00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f427c0a3d50 0x7f427c0a6210 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.395+0000 7f429ea00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42981098e0 0x7f4298199890 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.395+0000 7f429ea00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4298101800 0x7f429819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.395+0000 7f429ea00700 1 --2- 172.21.15.1:0/2442961047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42981021b0 0x7f429819fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.395+0000 7f429ea00700 1 -- 172.21.15.1:0/2442961047 >> 172.21.15.1:0/2442961047 conn(0x7f4298078520 msgr2=0x7f42981000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.395+0000 7f429ea00700 1 -- 172.21.15.1:0/2442961047 shutdown_connections 2024-04-26T21:29:05.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:05.395+0000 7f429ea00700 1 -- 172.21.15.1:0/2442961047 wait complete. 2024-04-26T21:29:05.398 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T21:29:05.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:05 smithi186 ceph-mon[32991]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 5 op/s 2024-04-26T21:29:05.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:05 smithi186 ceph-mon[32991]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T21:29:05.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:05 smithi186 ceph-mon[32991]: Cluster is now healthy 2024-04-26T21:29:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:05 smithi038 ceph-mon[32724]: pgmap v389: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 127 B/s wr, 5 op/s 2024-04-26T21:29:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:05 smithi038 ceph-mon[32724]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T21:29:05.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:05 smithi038 ceph-mon[32724]: Cluster is now healthy 2024-04-26T21:29:06.247 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-04-26T21:29:03.372467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649888,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":709,"num_read_kb":464,"num_write":347,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180758,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":418,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380572,"kb_used_data":14696,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561252,"statfs":{"total":1151940427776,"available":1151550722048,"internally_reserved":0,"allocated":15048704,"data_stored":8266454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":40,"num_read_kb":15,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.020628"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120762+0000","last_change":"2024-04-26T21:27:02.854724+0000","last_active":"2024-04-26T21:28:57.120762+0000","last_peered":"2024-04-26T21:28:57.120762+0000","last_clean":"2024-04-26T21:28:57.120762+0000","last_became_active":"2024-04-26T21:27:02.854125+0000","last_became_peered":"2024-04-26T21:27:02.854125+0000","last_unstale":"2024-04-26T21:28:57.120762+0000","last_undegraded":"2024-04-26T21:28:57.120762+0000","last_fullsized":"2024-04-26T21:28:57.120762+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:16.500220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089008+0000","last_change":"2024-04-26T21:27:02.852846+0000","last_active":"2024-04-26T21:28:57.089008+0000","last_peered":"2024-04-26T21:28:57.089008+0000","last_clean":"2024-04-26T21:28:57.089008+0000","last_became_active":"2024-04-26T21:27:02.851243+0000","last_became_peered":"2024-04-26T21:27:02.851243+0000","last_unstale":"2024-04-26T21:28:57.089008+0000","last_undegraded":"2024-04-26T21:28:57.089008+0000","last_fullsized":"2024-04-26T21:28:57.089008+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:09:11.240955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089984+0000","last_change":"2024-04-26T21:27:02.855856+0000","last_active":"2024-04-26T21:28:57.089984+0000","last_peered":"2024-04-26T21:28:57.089984+0000","last_clean":"2024-04-26T21:28:57.089984+0000","last_became_active":"2024-04-26T21:27:02.855446+0000","last_became_peered":"2024-04-26T21:27:02.855446+0000","last_unstale":"2024-04-26T21:28:57.089984+0000","last_undegraded":"2024-04-26T21:28:57.089984+0000","last_fullsized":"2024-04-26T21:28:57.089984+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:35.183449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089111+0000","last_change":"2024-04-26T21:27:02.860949+0000","last_active":"2024-04-26T21:28:57.089111+0000","last_peered":"2024-04-26T21:28:57.089111+0000","last_clean":"2024-04-26T21:28:57.089111+0000","last_became_active":"2024-04-26T21:27:02.860567+0000","last_became_peered":"2024-04-26T21:27:02.860567+0000","last_unstale":"2024-04-26T21:28:57.089111+0000","last_undegraded":"2024-04-26T21:28:57.089111+0000","last_fullsized":"2024-04-26T21:28:57.089111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:09.851885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083417+0000","last_change":"2024-04-26T21:27:02.876019+0000","last_active":"2024-04-26T21:28:57.083417+0000","last_peered":"2024-04-26T21:28:57.083417+0000","last_clean":"2024-04-26T21:28:57.083417+0000","last_became_active":"2024-04-26T21:27:02.875652+0000","last_became_peered":"2024-04-26T21:27:02.875652+0000","last_unstale":"2024-04-26T21:28:57.083417+0000","last_undegraded":"2024-04-26T21:28:57.083417+0000","last_fullsized":"2024-04-26T21:28:57.083417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:35.017787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088087+0000","last_change":"2024-04-26T21:27:02.860229+0000","last_active":"2024-04-26T21:28:57.088087+0000","last_peered":"2024-04-26T21:28:57.088087+0000","last_clean":"2024-04-26T21:28:57.088087+0000","last_became_active":"2024-04-26T21:27:02.859863+0000","last_became_peered":"2024-04-26T21:27:02.859863+0000","last_unstale":"2024-04-26T21:28:57.088087+0000","last_undegraded":"2024-04-26T21:28:57.088087+0000","last_fullsized":"2024-04-26T21:28:57.088087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:47.337073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140980+0000","last_change":"2024-04-26T21:27:02.863253+0000","last_active":"2024-04-26T21:28:57.140980+0000","last_peered":"2024-04-26T21:28:57.140980+0000","last_clean":"2024-04-26T21:28:57.140980+0000","last_became_active":"2024-04-26T21:27:02.861623+0000","last_became_peered":"2024-04-26T21:27:02.861623+0000","last_unstale":"2024-04-26T21:28:57.140980+0000","last_undegraded":"2024-04-26T21:28:57.140980+0000","last_fullsized":"2024-04-26T21:28:57.140980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:57.642504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.133344+0000","last_change":"2024-04-26T21:27:02.860822+0000","last_active":"2024-04-26T21:28:57.133344+0000","last_peered":"2024-04-26T21:28:57.133344+0000","last_clean":"2024-04-26T21:28:57.133344+0000","last_became_active":"2024-04-26T21:27:02.860482+0000","last_became_peered":"2024-04-26T21:27:02.860482+0000","last_unstale":"2024-04-26T21:28:57.133344+0000","last_undegraded":"2024-04-26T21:28:57.133344+0000","last_fullsized":"2024-04-26T21:28:57.133344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:59.224624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141309+0000","last_change":"2024-04-26T21:27:56.856389+0000","last_active":"2024-04-26T21:28:57.141309+0000","last_peered":"2024-04-26T21:28:57.141309+0000","last_clean":"2024-04-26T21:28:57.141309+0000","last_became_active":"2024-04-26T21:27:56.856008+0000","last_became_peered":"2024-04-26T21:27:56.856008+0000","last_unstale":"2024-04-26T21:28:57.141309+0000","last_undegraded":"2024-04-26T21:28:57.141309+0000","last_fullsized":"2024-04-26T21:28:57.141309+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:04.770535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088910+0000","last_change":"2024-04-26T21:27:02.860545+0000","last_active":"2024-04-26T21:28:57.088910+0000","last_peered":"2024-04-26T21:28:57.088910+0000","last_clean":"2024-04-26T21:28:57.088910+0000","last_became_active":"2024-04-26T21:27:02.858577+0000","last_became_peered":"2024-04-26T21:27:02.858577+0000","last_unstale":"2024-04-26T21:28:57.088910+0000","last_undegraded":"2024-04-26T21:28:57.088910+0000","last_fullsized":"2024-04-26T21:28:57.088910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:43.390592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141372+0000","last_change":"2024-04-26T21:27:02.858685+0000","last_active":"2024-04-26T21:28:57.141372+0000","last_peered":"2024-04-26T21:28:57.141372+0000","last_clean":"2024-04-26T21:28:57.141372+0000","last_became_active":"2024-04-26T21:27:02.857054+0000","last_became_peered":"2024-04-26T21:27:02.857054+0000","last_unstale":"2024-04-26T21:28:57.141372+0000","last_undegraded":"2024-04-26T21:28:57.141372+0000","last_fullsized":"2024-04-26T21:28:57.141372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:29.406132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088929+0000","last_change":"2024-04-26T21:27:02.852457+0000","last_active":"2024-04-26T21:28:57.088929+0000","last_peered":"2024-04-26T21:28:57.088929+0000","last_clean":"2024-04-26T21:28:57.088929+0000","last_became_active":"2024-04-26T21:27:02.852064+0000","last_became_peered":"2024-04-26T21:27:02.852064+0000","last_unstale":"2024-04-26T21:28:57.088929+0000","last_undegraded":"2024-04-26T21:28:57.088929+0000","last_fullsized":"2024-04-26T21:28:57.088929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:16.798413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.149646+0000","last_change":"2024-04-26T21:27:02.872618+0000","last_active":"2024-04-26T21:28:57.149646+0000","last_peered":"2024-04-26T21:28:57.149646+0000","last_clean":"2024-04-26T21:28:57.149646+0000","last_became_active":"2024-04-26T21:27:02.872270+0000","last_became_peered":"2024-04-26T21:27:02.872270+0000","last_unstale":"2024-04-26T21:28:57.149646+0000","last_undegraded":"2024-04-26T21:28:57.149646+0000","last_fullsized":"2024-04-26T21:28:57.149646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:45.828419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084860+0000","last_change":"2024-04-26T21:27:02.868590+0000","last_active":"2024-04-26T21:28:57.084860+0000","last_peered":"2024-04-26T21:28:57.084860+0000","last_clean":"2024-04-26T21:28:57.084860+0000","last_became_active":"2024-04-26T21:27:02.866641+0000","last_became_peered":"2024-04-26T21:27:02.866641+0000","last_unstale":"2024-04-26T21:28:57.084860+0000","last_undegraded":"2024-04-26T21:28:57.084860+0000","last_fullsized":"2024-04-26T21:28:57.084860+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:17.313420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084439+0000","last_change":"2024-04-26T21:27:02.867735+0000","last_active":"2024-04-26T21:28:57.084439+0000","last_peered":"2024-04-26T21:28:57.084439+0000","last_clean":"2024-04-26T21:28:57.084439+0000","last_became_active":"2024-04-26T21:27:02.867217+0000","last_became_peered":"2024-04-26T21:27:02.867217+0000","last_unstale":"2024-04-26T21:28:57.084439+0000","last_undegraded":"2024-04-26T21:28:57.084439+0000","last_fullsized":"2024-04-26T21:28:57.084439+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:04:45.533220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140625+0000","last_change":"2024-04-26T21:27:02.857541+0000","last_active":"2024-04-26T21:28:57.140625+0000","last_peered":"2024-04-26T21:28:57.140625+0000","last_clean":"2024-04-26T21:28:57.140625+0000","last_became_active":"2024-04-26T21:27:02.855563+0000","last_became_peered":"2024-04-26T21:27:02.855563+0000","last_unstale":"2024-04-26T21:28:57.140625+0000","last_undegraded":"2024-04-26T21:28:57.140625+0000","last_fullsized":"2024-04-26T21:28:57.140625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:39.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120658+0000","last_change":"2024-04-26T21:27:02.854461+0000","last_active":"2024-04-26T21:28:57.120658+0000","last_peered":"2024-04-26T21:28:57.120658+0000","last_clean":"2024-04-26T21:28:57.120658+0000","last_became_active":"2024-04-26T21:27:02.853691+0000","last_became_peered":"2024-04-26T21:27:02.853691+0000","last_unstale":"2024-04-26T21:28:57.120658+0000","last_undegraded":"2024-04-26T21:28:57.120658+0000","last_fullsized":"2024-04-26T21:28:57.120658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:09:48.284180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.094647+0000","last_change":"2024-04-26T21:27:02.864308+0000","last_active":"2024-04-26T21:28:57.094647+0000","last_peered":"2024-04-26T21:28:57.094647+0000","last_clean":"2024-04-26T21:28:57.094647+0000","last_became_active":"2024-04-26T21:27:02.863928+0000","last_became_peered":"2024-04-26T21:27:02.863928+0000","last_unstale":"2024-04-26T21:28:57.094647+0000","last_undegraded":"2024-04-26T21:28:57.094647+0000","last_fullsized":"2024-04-26T21:28:57.094647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:12.169394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140846+0000","last_change":"2024-04-26T21:27:02.876044+0000","last_active":"2024-04-26T21:28:57.140846+0000","last_peered":"2024-04-26T21:28:57.140846+0000","last_clean":"2024-04-26T21:28:57.140846+0000","last_became_active":"2024-04-26T21:27:02.875681+0000","last_became_peered":"2024-04-26T21:27:02.875681+0000","last_unstale":"2024-04-26T21:28:57.140846+0000","last_undegraded":"2024-04-26T21:28:57.140846+0000","last_fullsized":"2024-04-26T21:28:57.140846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:26.246916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140001+0000","last_change":"2024-04-26T21:27:07.374736+0000","last_active":"2024-04-26T21:28:57.140001+0000","last_peered":"2024-04-26T21:28:57.140001+0000","last_clean":"2024-04-26T21:28:57.140001+0000","last_became_active":"2024-04-26T21:27:07.374116+0000","last_became_peered":"2024-04-26T21:27:07.374116+0000","last_unstale":"2024-04-26T21:28:57.140001+0000","last_undegraded":"2024-04-26T21:28:57.140001+0000","last_fullsized":"2024-04-26T21:28:57.140001+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:11.450926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.095069+0000","last_change":"2024-04-26T21:27:02.302510+0000","last_active":"2024-04-26T21:28:57.095069+0000","last_peered":"2024-04-26T21:28:57.095069+0000","last_clean":"2024-04-26T21:28:57.095069+0000","last_became_active":"2024-04-26T21:27:02.302088+0000","last_became_peered":"2024-04-26T21:27:02.302088+0000","last_unstale":"2024-04-26T21:28:57.095069+0000","last_undegraded":"2024-04-26T21:28:57.095069+0000","last_fullsized":"2024-04-26T21:28:57.095069+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:43.007722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.133973+0000","last_change":"2024-04-26T21:27:02.861455+0000","last_active":"2024-04-26T21:28:57.133973+0000","last_peered":"2024-04-26T21:28:57.133973+0000","last_clean":"2024-04-26T21:28:57.133973+0000","last_became_active":"2024-04-26T21:27:02.861107+0000","last_became_peered":"2024-04-26T21:27:02.861107+0000","last_unstale":"2024-04-26T21:28:57.133973+0000","last_undegraded":"2024-04-26T21:28:57.133973+0000","last_fullsized":"2024-04-26T21:28:57.133973+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:10:18.382881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088435+0000","last_change":"2024-04-26T21:27:07.377797+0000","last_active":"2024-04-26T21:28:57.088435+0000","last_peered":"2024-04-26T21:28:57.088435+0000","last_clean":"2024-04-26T21:28:57.088435+0000","last_became_active":"2024-04-26T21:27:07.377314+0000","last_became_peered":"2024-04-26T21:27:07.377314+0000","last_unstale":"2024-04-26T21:28:57.088435+0000","last_undegraded":"2024-04-26T21:28:57.088435+0000","last_fullsized":"2024-04-26T21:28:57.088435+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:02:34.761034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134016+0000","last_change":"2024-04-26T21:27:02.309488+0000","last_active":"2024-04-26T21:28:57.134016+0000","last_peered":"2024-04-26T21:28:57.134016+0000","last_clean":"2024-04-26T21:28:57.134016+0000","last_became_active":"2024-04-26T21:27:02.308592+0000","last_became_peered":"2024-04-26T21:27:02.308592+0000","last_unstale":"2024-04-26T21:28:57.134016+0000","last_undegraded":"2024-04-26T21:28:57.134016+0000","last_fullsized":"2024-04-26T21:28:57.134016+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:36.563847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089012+0000","last_change":"2024-04-26T21:27:02.863204+0000","last_active":"2024-04-26T21:28:57.089012+0000","last_peered":"2024-04-26T21:28:57.089012+0000","last_clean":"2024-04-26T21:28:57.089012+0000","last_became_active":"2024-04-26T21:27:02.862466+0000","last_became_peered":"2024-04-26T21:27:02.862466+0000","last_unstale":"2024-04-26T21:28:57.089012+0000","last_undegraded":"2024-04-26T21:28:57.089012+0000","last_fullsized":"2024-04-26T21:28:57.089012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:29.360830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088593+0000","last_change":"2024-04-26T21:27:07.378032+0000","last_active":"2024-04-26T21:28:57.088593+0000","last_peered":"2024-04-26T21:28:57.088593+0000","last_clean":"2024-04-26T21:28:57.088593+0000","last_became_active":"2024-04-26T21:27:07.377255+0000","last_became_peered":"2024-04-26T21:27:07.377255+0000","last_unstale":"2024-04-26T21:28:57.088593+0000","last_undegraded":"2024-04-26T21:28:57.088593+0000","last_fullsized":"2024-04-26T21:28:57.088593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:18.989977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084022+0000","last_change":"2024-04-26T21:27:56.850816+0000","last_active":"2024-04-26T21:28:57.084022+0000","last_peered":"2024-04-26T21:28:57.084022+0000","last_clean":"2024-04-26T21:28:57.084022+0000","last_became_active":"2024-04-26T21:27:56.843299+0000","last_became_peered":"2024-04-26T21:27:56.843299+0000","last_unstale":"2024-04-26T21:28:57.084022+0000","last_undegraded":"2024-04-26T21:28:57.084022+0000","last_fullsized":"2024-04-26T21:28:57.084022+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:21:38.864693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141842+0000","last_change":"2024-04-26T21:27:02.876447+0000","last_active":"2024-04-26T21:28:57.141842+0000","last_peered":"2024-04-26T21:28:57.141842+0000","last_clean":"2024-04-26T21:28:57.141842+0000","last_became_active":"2024-04-26T21:27:02.876017+0000","last_became_peered":"2024-04-26T21:27:02.876017+0000","last_unstale":"2024-04-26T21:28:57.141842+0000","last_undegraded":"2024-04-26T21:28:57.141842+0000","last_fullsized":"2024-04-26T21:28:57.141842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:57:05.453787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088379+0000","last_change":"2024-04-26T21:27:07.373700+0000","last_active":"2024-04-26T21:28:57.088379+0000","last_peered":"2024-04-26T21:28:57.088379+0000","last_clean":"2024-04-26T21:28:57.088379+0000","last_became_active":"2024-04-26T21:27:07.373255+0000","last_became_peered":"2024-04-26T21:27:07.373255+0000","last_unstale":"2024-04-26T21:28:57.088379+0000","last_undegraded":"2024-04-26T21:28:57.088379+0000","last_fullsized":"2024-04-26T21:28:57.088379+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:48.517197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083901+0000","last_change":"2024-04-26T21:27:02.300817+0000","last_active":"2024-04-26T21:28:57.083901+0000","last_peered":"2024-04-26T21:28:57.083901+0000","last_clean":"2024-04-26T21:28:57.083901+0000","last_became_active":"2024-04-26T21:27:02.300493+0000","last_became_peered":"2024-04-26T21:27:02.300493+0000","last_unstale":"2024-04-26T21:28:57.083901+0000","last_undegraded":"2024-04-26T21:28:57.083901+0000","last_fullsized":"2024-04-26T21:28:57.083901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:18.783104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089662+0000","last_change":"2024-04-26T21:27:02.859982+0000","last_active":"2024-04-26T21:28:57.089662+0000","last_peered":"2024-04-26T21:28:57.089662+0000","last_clean":"2024-04-26T21:28:57.089662+0000","last_became_active":"2024-04-26T21:27:02.857204+0000","last_became_peered":"2024-04-26T21:27:02.857204+0000","last_unstale":"2024-04-26T21:28:57.089662+0000","last_undegraded":"2024-04-26T21:28:57.089662+0000","last_fullsized":"2024-04-26T21:28:57.089662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:22.423956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140395+0000","last_change":"2024-04-26T21:27:07.366039+0000","last_active":"2024-04-26T21:28:57.140395+0000","last_peered":"2024-04-26T21:28:57.140395+0000","last_clean":"2024-04-26T21:28:57.140395+0000","last_became_active":"2024-04-26T21:27:07.365560+0000","last_became_peered":"2024-04-26T21:27:07.365560+0000","last_unstale":"2024-04-26T21:28:57.140395+0000","last_undegraded":"2024-04-26T21:28:57.140395+0000","last_fullsized":"2024-04-26T21:28:57.140395+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:47.833476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":132,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:00.834393+0000","last_change":"2024-04-26T21:27:02.306364+0000","last_active":"2024-04-26T21:29:00.834393+0000","last_peered":"2024-04-26T21:29:00.834393+0000","last_clean":"2024-04-26T21:29:00.834393+0000","last_became_active":"2024-04-26T21:27:02.306041+0000","last_became_peered":"2024-04-26T21:27:02.306041+0000","last_unstale":"2024-04-26T21:29:00.834393+0000","last_undegraded":"2024-04-26T21:29:00.834393+0000","last_fullsized":"2024-04-26T21:29:00.834393+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:44.501916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083650+0000","last_change":"2024-04-26T21:27:02.859346+0000","last_active":"2024-04-26T21:28:57.083650+0000","last_peered":"2024-04-26T21:28:57.083650+0000","last_clean":"2024-04-26T21:28:57.083650+0000","last_became_active":"2024-04-26T21:27:02.859025+0000","last_became_peered":"2024-04-26T21:27:02.859025+0000","last_unstale":"2024-04-26T21:28:57.083650+0000","last_undegraded":"2024-04-26T21:28:57.083650+0000","last_fullsized":"2024-04-26T21:28:57.083650+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:06.258512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140349+0000","last_change":"2024-04-26T21:27:07.382349+0000","last_active":"2024-04-26T21:28:57.140349+0000","last_peered":"2024-04-26T21:28:57.140349+0000","last_clean":"2024-04-26T21:28:57.140349+0000","last_became_active":"2024-04-26T21:27:07.381930+0000","last_became_peered":"2024-04-26T21:27:07.381930+0000","last_unstale":"2024-04-26T21:28:57.140349+0000","last_undegraded":"2024-04-26T21:28:57.140349+0000","last_fullsized":"2024-04-26T21:28:57.140349+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:15:25.970573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.087951+0000","last_change":"2024-04-26T21:27:56.862413+0000","last_active":"2024-04-26T21:28:57.087951+0000","last_peered":"2024-04-26T21:28:57.087951+0000","last_clean":"2024-04-26T21:28:57.087951+0000","last_became_active":"2024-04-26T21:27:56.859184+0000","last_became_peered":"2024-04-26T21:27:56.859184+0000","last_unstale":"2024-04-26T21:28:57.087951+0000","last_undegraded":"2024-04-26T21:28:57.087951+0000","last_fullsized":"2024-04-26T21:28:57.087951+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:54:17.701601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089353+0000","last_change":"2024-04-26T21:27:02.865707+0000","last_active":"2024-04-26T21:28:57.089353+0000","last_peered":"2024-04-26T21:28:57.089353+0000","last_clean":"2024-04-26T21:28:57.089353+0000","last_became_active":"2024-04-26T21:27:02.865436+0000","last_became_peered":"2024-04-26T21:27:02.865436+0000","last_unstale":"2024-04-26T21:28:57.089353+0000","last_undegraded":"2024-04-26T21:28:57.089353+0000","last_fullsized":"2024-04-26T21:28:57.089353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:53.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120620+0000","last_change":"2024-04-26T21:27:07.379092+0000","last_active":"2024-04-26T21:28:57.120620+0000","last_peered":"2024-04-26T21:28:57.120620+0000","last_clean":"2024-04-26T21:28:57.120620+0000","last_became_active":"2024-04-26T21:27:07.378716+0000","last_became_peered":"2024-04-26T21:27:07.378716+0000","last_unstale":"2024-04-26T21:28:57.120620+0000","last_undegraded":"2024-04-26T21:28:57.120620+0000","last_fullsized":"2024-04-26T21:28:57.120620+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:18.906224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147874+0000","last_change":"2024-04-26T21:27:02.307510+0000","last_active":"2024-04-26T21:28:57.147874+0000","last_peered":"2024-04-26T21:28:57.147874+0000","last_clean":"2024-04-26T21:28:57.147874+0000","last_became_active":"2024-04-26T21:27:02.306932+0000","last_became_peered":"2024-04-26T21:27:02.306932+0000","last_unstale":"2024-04-26T21:28:57.147874+0000","last_undegraded":"2024-04-26T21:28:57.147874+0000","last_fullsized":"2024-04-26T21:28:57.147874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:21:34.217767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.132628+0000","last_change":"2024-04-26T21:27:02.874246+0000","last_active":"2024-04-26T21:28:57.132628+0000","last_peered":"2024-04-26T21:28:57.132628+0000","last_clean":"2024-04-26T21:28:57.132628+0000","last_became_active":"2024-04-26T21:27:02.873880+0000","last_became_peered":"2024-04-26T21:27:02.873880+0000","last_unstale":"2024-04-26T21:28:57.132628+0000","last_undegraded":"2024-04-26T21:28:57.132628+0000","last_fullsized":"2024-04-26T21:28:57.132628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:21.579832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088828+0000","last_change":"2024-04-26T21:27:07.387201+0000","last_active":"2024-04-26T21:28:57.088828+0000","last_peered":"2024-04-26T21:28:57.088828+0000","last_clean":"2024-04-26T21:28:57.088828+0000","last_became_active":"2024-04-26T21:27:07.386327+0000","last_became_peered":"2024-04-26T21:27:07.386327+0000","last_unstale":"2024-04-26T21:28:57.088828+0000","last_undegraded":"2024-04-26T21:28:57.088828+0000","last_fullsized":"2024-04-26T21:28:57.088828+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:40.801572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088663+0000","last_change":"2024-04-26T21:27:02.304132+0000","last_active":"2024-04-26T21:28:57.088663+0000","last_peered":"2024-04-26T21:28:57.088663+0000","last_clean":"2024-04-26T21:28:57.088663+0000","last_became_active":"2024-04-26T21:27:02.303562+0000","last_became_peered":"2024-04-26T21:27:02.303562+0000","last_unstale":"2024-04-26T21:28:57.088663+0000","last_undegraded":"2024-04-26T21:28:57.088663+0000","last_fullsized":"2024-04-26T21:28:57.088663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:54.417590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.092781+0000","last_change":"2024-04-26T21:27:02.866539+0000","last_active":"2024-04-26T21:28:57.092781+0000","last_peered":"2024-04-26T21:28:57.092781+0000","last_clean":"2024-04-26T21:28:57.092781+0000","last_became_active":"2024-04-26T21:27:02.866193+0000","last_became_peered":"2024-04-26T21:27:02.866193+0000","last_unstale":"2024-04-26T21:28:57.092781+0000","last_undegraded":"2024-04-26T21:28:57.092781+0000","last_fullsized":"2024-04-26T21:28:57.092781+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:44.234261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134645+0000","last_change":"2024-04-26T21:27:07.368394+0000","last_active":"2024-04-26T21:28:57.134645+0000","last_peered":"2024-04-26T21:28:57.134645+0000","last_clean":"2024-04-26T21:28:57.134645+0000","last_became_active":"2024-04-26T21:27:07.367998+0000","last_became_peered":"2024-04-26T21:27:07.367998+0000","last_unstale":"2024-04-26T21:28:57.134645+0000","last_undegraded":"2024-04-26T21:28:57.134645+0000","last_fullsized":"2024-04-26T21:28:57.134645+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:31:28.459759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":124,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088548+0000","last_change":"2024-04-26T21:27:02.304096+0000","last_active":"2024-04-26T21:28:57.088548+0000","last_peered":"2024-04-26T21:28:57.088548+0000","last_clean":"2024-04-26T21:28:57.088548+0000","last_became_active":"2024-04-26T21:27:02.303736+0000","last_became_peered":"2024-04-26T21:27:02.303736+0000","last_unstale":"2024-04-26T21:28:57.088548+0000","last_undegraded":"2024-04-26T21:28:57.088548+0000","last_fullsized":"2024-04-26T21:28:57.088548+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:25.400913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142465+0000","last_change":"2024-04-26T21:27:02.863033+0000","last_active":"2024-04-26T21:28:57.142465+0000","last_peered":"2024-04-26T21:28:57.142465+0000","last_clean":"2024-04-26T21:28:57.142465+0000","last_became_active":"2024-04-26T21:27:02.862572+0000","last_became_peered":"2024-04-26T21:27:02.862572+0000","last_unstale":"2024-04-26T21:28:57.142465+0000","last_undegraded":"2024-04-26T21:28:57.142465+0000","last_fullsized":"2024-04-26T21:28:57.142465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:02:55.574601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142841+0000","last_change":"2024-04-26T21:28:57.142094+0000","last_active":"2024-04-26T21:28:57.142841+0000","last_peered":"2024-04-26T21:28:57.142841+0000","last_clean":"2024-04-26T21:28:57.142841+0000","last_became_active":"2024-04-26T21:28:57.141572+0000","last_became_peered":"2024-04-26T21:28:57.141572+0000","last_unstale":"2024-04-26T21:28:57.142841+0000","last_undegraded":"2024-04-26T21:28:57.142841+0000","last_fullsized":"2024-04-26T21:28:57.142841+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:35.087842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141281+0000","last_change":"2024-04-26T21:27:56.846362+0000","last_active":"2024-04-26T21:28:57.141281+0000","last_peered":"2024-04-26T21:28:57.141281+0000","last_clean":"2024-04-26T21:28:57.141281+0000","last_became_active":"2024-04-26T21:27:56.840187+0000","last_became_peered":"2024-04-26T21:27:56.840187+0000","last_unstale":"2024-04-26T21:28:57.141281+0000","last_undegraded":"2024-04-26T21:28:57.141281+0000","last_fullsized":"2024-04-26T21:28:57.141281+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:12.064415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088889+0000","last_change":"2024-04-26T21:27:02.858739+0000","last_active":"2024-04-26T21:28:57.088889+0000","last_peered":"2024-04-26T21:28:57.088889+0000","last_clean":"2024-04-26T21:28:57.088889+0000","last_became_active":"2024-04-26T21:27:02.857458+0000","last_became_peered":"2024-04-26T21:27:02.857458+0000","last_unstale":"2024-04-26T21:28:57.088889+0000","last_undegraded":"2024-04-26T21:28:57.088889+0000","last_fullsized":"2024-04-26T21:28:57.088889+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:00.354052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089405+0000","last_change":"2024-04-26T21:27:07.377690+0000","last_active":"2024-04-26T21:28:57.089405+0000","last_peered":"2024-04-26T21:28:57.089405+0000","last_clean":"2024-04-26T21:28:57.089405+0000","last_became_active":"2024-04-26T21:27:07.377192+0000","last_became_peered":"2024-04-26T21:27:07.377192+0000","last_unstale":"2024-04-26T21:28:57.089405+0000","last_undegraded":"2024-04-26T21:28:57.089405+0000","last_fullsized":"2024-04-26T21:28:57.089405+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:48.069083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089650+0000","last_change":"2024-04-26T21:27:02.316110+0000","last_active":"2024-04-26T21:28:57.089650+0000","last_peered":"2024-04-26T21:28:57.089650+0000","last_clean":"2024-04-26T21:28:57.089650+0000","last_became_active":"2024-04-26T21:27:02.315767+0000","last_became_peered":"2024-04-26T21:27:02.315767+0000","last_unstale":"2024-04-26T21:28:57.089650+0000","last_undegraded":"2024-04-26T21:28:57.089650+0000","last_fullsized":"2024-04-26T21:28:57.089650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:00.990872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142071+0000","last_change":"2024-04-26T21:27:02.855736+0000","last_active":"2024-04-26T21:28:57.142071+0000","last_peered":"2024-04-26T21:28:57.142071+0000","last_clean":"2024-04-26T21:28:57.142071+0000","last_became_active":"2024-04-26T21:27:02.855378+0000","last_became_peered":"2024-04-26T21:27:02.855378+0000","last_unstale":"2024-04-26T21:28:57.142071+0000","last_undegraded":"2024-04-26T21:28:57.142071+0000","last_fullsized":"2024-04-26T21:28:57.142071+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:55.272869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.091733+0000","last_change":"2024-04-26T21:27:07.376804+0000","last_active":"2024-04-26T21:28:57.091733+0000","last_peered":"2024-04-26T21:28:57.091733+0000","last_clean":"2024-04-26T21:28:57.091733+0000","last_became_active":"2024-04-26T21:27:07.374378+0000","last_became_peered":"2024-04-26T21:27:07.374378+0000","last_unstale":"2024-04-26T21:28:57.091733+0000","last_undegraded":"2024-04-26T21:28:57.091733+0000","last_fullsized":"2024-04-26T21:28:57.091733+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:03.930253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'33","reported_seq":252,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.828967+0000","last_change":"2024-04-26T21:27:02.297467+0000","last_active":"2024-04-26T21:28:57.828967+0000","last_peered":"2024-04-26T21:28:57.828967+0000","last_clean":"2024-04-26T21:28:57.828967+0000","last_became_active":"2024-04-26T21:27:02.297077+0000","last_became_peered":"2024-04-26T21:27:02.297077+0000","last_unstale":"2024-04-26T21:28:57.828967+0000","last_undegraded":"2024-04-26T21:28:57.828967+0000","last_fullsized":"2024-04-26T21:28:57.828967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:47.047565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55676,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":188,"num_read_kb":148,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083876+0000","last_change":"2024-04-26T21:27:02.858161+0000","last_active":"2024-04-26T21:28:57.083876+0000","last_peered":"2024-04-26T21:28:57.083876+0000","last_clean":"2024-04-26T21:28:57.083876+0000","last_became_active":"2024-04-26T21:27:02.857747+0000","last_became_peered":"2024-04-26T21:27:02.857747+0000","last_unstale":"2024-04-26T21:28:57.083876+0000","last_undegraded":"2024-04-26T21:28:57.083876+0000","last_fullsized":"2024-04-26T21:28:57.083876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:10:50.843195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.090006+0000","last_change":"2024-04-26T21:27:07.376253+0000","last_active":"2024-04-26T21:28:57.090006+0000","last_peered":"2024-04-26T21:28:57.090006+0000","last_clean":"2024-04-26T21:28:57.090006+0000","last_became_active":"2024-04-26T21:27:07.375822+0000","last_became_peered":"2024-04-26T21:27:07.375822+0000","last_unstale":"2024-04-26T21:28:57.090006+0000","last_undegraded":"2024-04-26T21:28:57.090006+0000","last_fullsized":"2024-04-26T21:28:57.090006+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:22:01.784163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140286+0000","last_change":"2024-04-26T21:27:02.300743+0000","last_active":"2024-04-26T21:28:57.140286+0000","last_peered":"2024-04-26T21:28:57.140286+0000","last_clean":"2024-04-26T21:28:57.140286+0000","last_became_active":"2024-04-26T21:27:02.300424+0000","last_became_peered":"2024-04-26T21:27:02.300424+0000","last_unstale":"2024-04-26T21:28:57.140286+0000","last_undegraded":"2024-04-26T21:28:57.140286+0000","last_fullsized":"2024-04-26T21:28:57.140286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:10.408234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148845+0000","last_change":"2024-04-26T21:27:02.872456+0000","last_active":"2024-04-26T21:28:57.148845+0000","last_peered":"2024-04-26T21:28:57.148845+0000","last_clean":"2024-04-26T21:28:57.148845+0000","last_became_active":"2024-04-26T21:27:02.872064+0000","last_became_peered":"2024-04-26T21:27:02.872064+0000","last_unstale":"2024-04-26T21:28:57.148845+0000","last_undegraded":"2024-04-26T21:28:57.148845+0000","last_fullsized":"2024-04-26T21:28:57.148845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:33.511164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120361+0000","last_change":"2024-04-26T21:27:07.386148+0000","last_active":"2024-04-26T21:28:57.120361+0000","last_peered":"2024-04-26T21:28:57.120361+0000","last_clean":"2024-04-26T21:28:57.120361+0000","last_became_active":"2024-04-26T21:27:07.385787+0000","last_became_peered":"2024-04-26T21:27:07.385787+0000","last_unstale":"2024-04-26T21:28:57.120361+0000","last_undegraded":"2024-04-26T21:28:57.120361+0000","last_fullsized":"2024-04-26T21:28:57.120361+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:57.776660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089128+0000","last_change":"2024-04-26T21:27:02.313725+0000","last_active":"2024-04-26T21:28:57.089128+0000","last_peered":"2024-04-26T21:28:57.089128+0000","last_clean":"2024-04-26T21:28:57.089128+0000","last_became_active":"2024-04-26T21:27:02.313350+0000","last_became_peered":"2024-04-26T21:27:02.313350+0000","last_unstale":"2024-04-26T21:28:57.089128+0000","last_undegraded":"2024-04-26T21:28:57.089128+0000","last_fullsized":"2024-04-26T21:28:57.089128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:21:30.608067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141462+0000","last_change":"2024-04-26T21:27:56.856753+0000","last_active":"2024-04-26T21:28:57.141462+0000","last_peered":"2024-04-26T21:28:57.141462+0000","last_clean":"2024-04-26T21:28:57.141462+0000","last_became_active":"2024-04-26T21:27:56.856297+0000","last_became_peered":"2024-04-26T21:27:56.856297+0000","last_unstale":"2024-04-26T21:28:57.141462+0000","last_undegraded":"2024-04-26T21:28:57.141462+0000","last_fullsized":"2024-04-26T21:28:57.141462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:54.650077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.095669+0000","last_change":"2024-04-26T21:27:07.378091+0000","last_active":"2024-04-26T21:28:57.095669+0000","last_peered":"2024-04-26T21:28:57.095669+0000","last_clean":"2024-04-26T21:28:57.095669+0000","last_became_active":"2024-04-26T21:27:07.377671+0000","last_became_peered":"2024-04-26T21:27:07.377671+0000","last_unstale":"2024-04-26T21:28:57.095669+0000","last_undegraded":"2024-04-26T21:28:57.095669+0000","last_fullsized":"2024-04-26T21:28:57.095669+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:41:36.551655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'3","reported_seq":130,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:00.865786+0000","last_change":"2024-04-26T21:27:02.294738+0000","last_active":"2024-04-26T21:29:00.865786+0000","last_peered":"2024-04-26T21:29:00.865786+0000","last_clean":"2024-04-26T21:29:00.865786+0000","last_became_active":"2024-04-26T21:27:02.294338+0000","last_became_peered":"2024-04-26T21:27:02.294338+0000","last_unstale":"2024-04-26T21:29:00.865786+0000","last_undegraded":"2024-04-26T21:29:00.865786+0000","last_fullsized":"2024-04-26T21:29:00.865786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:25.309107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141783+0000","last_change":"2024-04-26T21:27:02.861332+0000","last_active":"2024-04-26T21:28:57.141783+0000","last_peered":"2024-04-26T21:28:57.141783+0000","last_clean":"2024-04-26T21:28:57.141783+0000","last_became_active":"2024-04-26T21:27:02.857865+0000","last_became_peered":"2024-04-26T21:27:02.857865+0000","last_unstale":"2024-04-26T21:28:57.141783+0000","last_undegraded":"2024-04-26T21:28:57.141783+0000","last_fullsized":"2024-04-26T21:28:57.141783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:48.493086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150359+0000","last_change":"2024-04-26T21:27:07.383219+0000","last_active":"2024-04-26T21:28:57.150359+0000","last_peered":"2024-04-26T21:28:57.150359+0000","last_clean":"2024-04-26T21:28:57.150359+0000","last_became_active":"2024-04-26T21:27:07.382820+0000","last_became_peered":"2024-04-26T21:27:07.382820+0000","last_unstale":"2024-04-26T21:28:57.150359+0000","last_undegraded":"2024-04-26T21:28:57.150359+0000","last_fullsized":"2024-04-26T21:28:57.150359+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:07.752124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083444+0000","last_change":"2024-04-26T21:27:02.307746+0000","last_active":"2024-04-26T21:28:57.083444+0000","last_peered":"2024-04-26T21:28:57.083444+0000","last_clean":"2024-04-26T21:28:57.083444+0000","last_became_active":"2024-04-26T21:27:02.307406+0000","last_became_peered":"2024-04-26T21:27:02.307406+0000","last_unstale":"2024-04-26T21:28:57.083444+0000","last_undegraded":"2024-04-26T21:28:57.083444+0000","last_fullsized":"2024-04-26T21:28:57.083444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:13.799727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088903+0000","last_change":"2024-04-26T21:27:02.861194+0000","last_active":"2024-04-26T21:28:57.088903+0000","last_peered":"2024-04-26T21:28:57.088903+0000","last_clean":"2024-04-26T21:28:57.088903+0000","last_became_active":"2024-04-26T21:27:02.860871+0000","last_became_peered":"2024-04-26T21:27:02.860871+0000","last_unstale":"2024-04-26T21:28:57.088903+0000","last_undegraded":"2024-04-26T21:28:57.088903+0000","last_fullsized":"2024-04-26T21:28:57.088903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:56.829027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.144000+0000","last_change":"2024-04-26T21:28:57.143451+0000","last_active":"2024-04-26T21:28:57.144000+0000","last_peered":"2024-04-26T21:28:57.144000+0000","last_clean":"2024-04-26T21:28:57.144000+0000","last_became_active":"2024-04-26T21:28:57.142896+0000","last_became_peered":"2024-04-26T21:28:57.142896+0000","last_unstale":"2024-04-26T21:28:57.144000+0000","last_undegraded":"2024-04-26T21:28:57.144000+0000","last_fullsized":"2024-04-26T21:28:57.144000+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:40.800757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084171+0000","last_change":"2024-04-26T21:27:02.301711+0000","last_active":"2024-04-26T21:28:57.084171+0000","last_peered":"2024-04-26T21:28:57.084171+0000","last_clean":"2024-04-26T21:28:57.084171+0000","last_became_active":"2024-04-26T21:27:02.301429+0000","last_became_peered":"2024-04-26T21:27:02.301429+0000","last_unstale":"2024-04-26T21:28:57.084171+0000","last_undegraded":"2024-04-26T21:28:57.084171+0000","last_fullsized":"2024-04-26T21:28:57.084171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:50.876824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120572+0000","last_change":"2024-04-26T21:27:02.854372+0000","last_active":"2024-04-26T21:28:57.120572+0000","last_peered":"2024-04-26T21:28:57.120572+0000","last_clean":"2024-04-26T21:28:57.120572+0000","last_became_active":"2024-04-26T21:27:02.853826+0000","last_became_peered":"2024-04-26T21:27:02.853826+0000","last_unstale":"2024-04-26T21:28:57.120572+0000","last_undegraded":"2024-04-26T21:28:57.120572+0000","last_fullsized":"2024-04-26T21:28:57.120572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:08.253384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083532+0000","last_change":"2024-04-26T21:27:07.506684+0000","last_active":"2024-04-26T21:28:57.083532+0000","last_peered":"2024-04-26T21:28:57.083532+0000","last_clean":"2024-04-26T21:28:57.083532+0000","last_became_active":"2024-04-26T21:27:07.506203+0000","last_became_peered":"2024-04-26T21:27:07.506203+0000","last_unstale":"2024-04-26T21:28:57.083532+0000","last_undegraded":"2024-04-26T21:28:57.083532+0000","last_fullsized":"2024-04-26T21:28:57.083532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:42:48.580106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.094138+0000","last_change":"2024-04-26T21:27:02.306265+0000","last_active":"2024-04-26T21:28:57.094138+0000","last_peered":"2024-04-26T21:28:57.094138+0000","last_clean":"2024-04-26T21:28:57.094138+0000","last_became_active":"2024-04-26T21:27:02.305941+0000","last_became_peered":"2024-04-26T21:27:02.305941+0000","last_unstale":"2024-04-26T21:28:57.094138+0000","last_undegraded":"2024-04-26T21:28:57.094138+0000","last_fullsized":"2024-04-26T21:28:57.094138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:29.076833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088569+0000","last_change":"2024-04-26T21:27:02.871916+0000","last_active":"2024-04-26T21:28:57.088569+0000","last_peered":"2024-04-26T21:28:57.088569+0000","last_clean":"2024-04-26T21:28:57.088569+0000","last_became_active":"2024-04-26T21:27:02.871517+0000","last_became_peered":"2024-04-26T21:27:02.871517+0000","last_unstale":"2024-04-26T21:28:57.088569+0000","last_undegraded":"2024-04-26T21:28:57.088569+0000","last_fullsized":"2024-04-26T21:28:57.088569+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:41.361757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148111+0000","last_change":"2024-04-26T21:27:07.383422+0000","last_active":"2024-04-26T21:28:57.148111+0000","last_peered":"2024-04-26T21:28:57.148111+0000","last_clean":"2024-04-26T21:28:57.148111+0000","last_became_active":"2024-04-26T21:27:07.382727+0000","last_became_peered":"2024-04-26T21:27:07.382727+0000","last_unstale":"2024-04-26T21:28:57.148111+0000","last_undegraded":"2024-04-26T21:28:57.148111+0000","last_fullsized":"2024-04-26T21:28:57.148111+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:22.002658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141284+0000","last_change":"2024-04-26T21:27:02.308081+0000","last_active":"2024-04-26T21:28:57.141284+0000","last_peered":"2024-04-26T21:28:57.141284+0000","last_clean":"2024-04-26T21:28:57.141284+0000","last_became_active":"2024-04-26T21:27:02.307715+0000","last_became_peered":"2024-04-26T21:27:02.307715+0000","last_unstale":"2024-04-26T21:28:57.141284+0000","last_undegraded":"2024-04-26T21:28:57.141284+0000","last_fullsized":"2024-04-26T21:28:57.141284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:28.967817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088446+0000","last_change":"2024-04-26T21:27:02.859527+0000","last_active":"2024-04-26T21:28:57.088446+0000","last_peered":"2024-04-26T21:28:57.088446+0000","last_clean":"2024-04-26T21:28:57.088446+0000","last_became_active":"2024-04-26T21:27:02.859173+0000","last_became_peered":"2024-04-26T21:27:02.859173+0000","last_unstale":"2024-04-26T21:28:57.088446+0000","last_undegraded":"2024-04-26T21:28:57.088446+0000","last_fullsized":"2024-04-26T21:28:57.088446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:24.811630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147035+0000","last_change":"2024-04-26T21:27:07.509778+0000","last_active":"2024-04-26T21:28:57.147035+0000","last_peered":"2024-04-26T21:28:57.147035+0000","last_clean":"2024-04-26T21:28:57.147035+0000","last_became_active":"2024-04-26T21:27:07.508344+0000","last_became_peered":"2024-04-26T21:27:07.508344+0000","last_unstale":"2024-04-26T21:28:57.147035+0000","last_undegraded":"2024-04-26T21:28:57.147035+0000","last_fullsized":"2024-04-26T21:28:57.147035+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:38:05.926717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.143833+0000","last_change":"2024-04-26T21:28:57.143833+0000","last_active":"2024-04-26T21:28:57.143833+0000","last_peered":"2024-04-26T21:28:57.143833+0000","last_clean":"2024-04-26T21:28:57.143833+0000","last_became_active":"2024-04-26T21:28:57.142433+0000","last_became_peered":"2024-04-26T21:28:57.142433+0000","last_unstale":"2024-04-26T21:28:57.143833+0000","last_undegraded":"2024-04-26T21:28:57.143833+0000","last_fullsized":"2024-04-26T21:28:57.143833+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:41:45.191169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":125,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147854+0000","last_change":"2024-04-26T21:27:02.317413+0000","last_active":"2024-04-26T21:28:57.147854+0000","last_peered":"2024-04-26T21:28:57.147854+0000","last_clean":"2024-04-26T21:28:57.147854+0000","last_became_active":"2024-04-26T21:27:02.316510+0000","last_became_peered":"2024-04-26T21:27:02.316510+0000","last_unstale":"2024-04-26T21:28:57.147854+0000","last_undegraded":"2024-04-26T21:28:57.147854+0000","last_fullsized":"2024-04-26T21:28:57.147854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:49.156366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141068+0000","last_change":"2024-04-26T21:27:07.373434+0000","last_active":"2024-04-26T21:28:57.141068+0000","last_peered":"2024-04-26T21:28:57.141068+0000","last_clean":"2024-04-26T21:28:57.141068+0000","last_became_active":"2024-04-26T21:27:07.372973+0000","last_became_peered":"2024-04-26T21:27:07.372973+0000","last_unstale":"2024-04-26T21:28:57.141068+0000","last_undegraded":"2024-04-26T21:28:57.141068+0000","last_fullsized":"2024-04-26T21:28:57.141068+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:04.022651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.133068+0000","last_change":"2024-04-26T21:27:02.862734+0000","last_active":"2024-04-26T21:28:57.133068+0000","last_peered":"2024-04-26T21:28:57.133068+0000","last_clean":"2024-04-26T21:28:57.133068+0000","last_became_active":"2024-04-26T21:27:02.862334+0000","last_became_peered":"2024-04-26T21:27:02.862334+0000","last_unstale":"2024-04-26T21:28:57.133068+0000","last_undegraded":"2024-04-26T21:28:57.133068+0000","last_fullsized":"2024-04-26T21:28:57.133068+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:09.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089403+0000","last_change":"2024-04-26T21:27:02.302471+0000","last_active":"2024-04-26T21:28:57.089403+0000","last_peered":"2024-04-26T21:28:57.089403+0000","last_clean":"2024-04-26T21:28:57.089403+0000","last_became_active":"2024-04-26T21:27:02.302055+0000","last_became_peered":"2024-04-26T21:27:02.302055+0000","last_unstale":"2024-04-26T21:28:57.089403+0000","last_undegraded":"2024-04-26T21:28:57.089403+0000","last_fullsized":"2024-04-26T21:28:57.089403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:29:14.649670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":367,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141165+0000","last_change":"2024-04-26T21:25:41.647633+0000","last_active":"2024-04-26T21:28:57.141165+0000","last_peered":"2024-04-26T21:28:57.141165+0000","last_clean":"2024-04-26T21:28:57.141165+0000","last_became_active":"2024-04-26T21:25:41.647091+0000","last_became_peered":"2024-04-26T21:25:41.647091+0000","last_unstale":"2024-04-26T21:28:57.141165+0000","last_undegraded":"2024-04-26T21:28:57.141165+0000","last_fullsized":"2024-04-26T21:28:57.141165+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_clean_scrub_stamp":"2024-04-26T21:21:43.547929+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:37.666065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.152820+0000","last_change":"2024-04-26T21:28:57.152820+0000","last_active":"2024-04-26T21:28:57.152820+0000","last_peered":"2024-04-26T21:28:57.152820+0000","last_clean":"2024-04-26T21:28:57.152820+0000","last_became_active":"2024-04-26T21:28:57.152453+0000","last_became_peered":"2024-04-26T21:28:57.152453+0000","last_unstale":"2024-04-26T21:28:57.152820+0000","last_undegraded":"2024-04-26T21:28:57.152820+0000","last_fullsized":"2024-04-26T21:28:57.152820+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:16:47.634651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147552+0000","last_change":"2024-04-26T21:27:02.874116+0000","last_active":"2024-04-26T21:28:57.147552+0000","last_peered":"2024-04-26T21:28:57.147552+0000","last_clean":"2024-04-26T21:28:57.147552+0000","last_became_active":"2024-04-26T21:27:02.871334+0000","last_became_peered":"2024-04-26T21:27:02.871334+0000","last_unstale":"2024-04-26T21:28:57.147552+0000","last_undegraded":"2024-04-26T21:28:57.147552+0000","last_fullsized":"2024-04-26T21:28:57.147552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:28.835061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120084+0000","last_change":"2024-04-26T21:27:56.847702+0000","last_active":"2024-04-26T21:28:57.120084+0000","last_peered":"2024-04-26T21:28:57.120084+0000","last_clean":"2024-04-26T21:28:57.120084+0000","last_became_active":"2024-04-26T21:27:56.844034+0000","last_became_peered":"2024-04-26T21:27:56.844034+0000","last_unstale":"2024-04-26T21:28:57.120084+0000","last_undegraded":"2024-04-26T21:28:57.120084+0000","last_fullsized":"2024-04-26T21:28:57.120084+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:54.631297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120208+0000","last_change":"2024-04-26T21:27:07.373894+0000","last_active":"2024-04-26T21:28:57.120208+0000","last_peered":"2024-04-26T21:28:57.120208+0000","last_clean":"2024-04-26T21:28:57.120208+0000","last_became_active":"2024-04-26T21:27:07.373338+0000","last_became_peered":"2024-04-26T21:27:07.373338+0000","last_unstale":"2024-04-26T21:28:57.120208+0000","last_undegraded":"2024-04-26T21:28:57.120208+0000","last_fullsized":"2024-04-26T21:28:57.120208+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:49.636493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120527+0000","last_change":"2024-04-26T21:27:56.858975+0000","last_active":"2024-04-26T21:28:57.120527+0000","last_peered":"2024-04-26T21:28:57.120527+0000","last_clean":"2024-04-26T21:28:57.120527+0000","last_became_active":"2024-04-26T21:27:56.850022+0000","last_became_peered":"2024-04-26T21:27:56.850022+0000","last_unstale":"2024-04-26T21:28:57.120527+0000","last_undegraded":"2024-04-26T21:28:57.120527+0000","last_fullsized":"2024-04-26T21:28:57.120527+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:43:31.860906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148536+0000","last_change":"2024-04-26T21:27:02.869831+0000","last_active":"2024-04-26T21:28:57.148536+0000","last_peered":"2024-04-26T21:28:57.148536+0000","last_clean":"2024-04-26T21:28:57.148536+0000","last_became_active":"2024-04-26T21:27:02.869375+0000","last_became_peered":"2024-04-26T21:27:02.869375+0000","last_unstale":"2024-04-26T21:28:57.148536+0000","last_undegraded":"2024-04-26T21:28:57.148536+0000","last_fullsized":"2024-04-26T21:28:57.148536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:20.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.132623+0000","last_change":"2024-04-26T21:27:07.507562+0000","last_active":"2024-04-26T21:28:57.132623+0000","last_peered":"2024-04-26T21:28:57.132623+0000","last_clean":"2024-04-26T21:28:57.132623+0000","last_became_active":"2024-04-26T21:27:07.507063+0000","last_became_peered":"2024-04-26T21:27:07.507063+0000","last_unstale":"2024-04-26T21:28:57.132623+0000","last_undegraded":"2024-04-26T21:28:57.132623+0000","last_fullsized":"2024-04-26T21:28:57.132623+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:58.192617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.094611+0000","last_change":"2024-04-26T21:27:02.315360+0000","last_active":"2024-04-26T21:28:57.094611+0000","last_peered":"2024-04-26T21:28:57.094611+0000","last_clean":"2024-04-26T21:28:57.094611+0000","last_became_active":"2024-04-26T21:27:02.314817+0000","last_became_peered":"2024-04-26T21:27:02.314817+0000","last_unstale":"2024-04-26T21:28:57.094611+0000","last_undegraded":"2024-04-26T21:28:57.094611+0000","last_fullsized":"2024-04-26T21:28:57.094611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:12.045363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.145131+0000","last_change":"2024-04-26T21:28:57.145131+0000","last_active":"2024-04-26T21:28:57.145131+0000","last_peered":"2024-04-26T21:28:57.145131+0000","last_clean":"2024-04-26T21:28:57.145131+0000","last_became_active":"2024-04-26T21:28:57.144059+0000","last_became_peered":"2024-04-26T21:28:57.144059+0000","last_unstale":"2024-04-26T21:28:57.145131+0000","last_undegraded":"2024-04-26T21:28:57.145131+0000","last_fullsized":"2024-04-26T21:28:57.145131+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:11:12.442030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.139990+0000","last_change":"2024-04-26T21:27:07.362016+0000","last_active":"2024-04-26T21:28:57.139990+0000","last_peered":"2024-04-26T21:28:57.139990+0000","last_clean":"2024-04-26T21:28:57.139990+0000","last_became_active":"2024-04-26T21:27:07.361360+0000","last_became_peered":"2024-04-26T21:27:07.361360+0000","last_unstale":"2024-04-26T21:28:57.139990+0000","last_undegraded":"2024-04-26T21:28:57.139990+0000","last_fullsized":"2024-04-26T21:28:57.139990+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:03.196293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089562+0000","last_change":"2024-04-26T21:27:02.308088+0000","last_active":"2024-04-26T21:28:57.089562+0000","last_peered":"2024-04-26T21:28:57.089562+0000","last_clean":"2024-04-26T21:28:57.089562+0000","last_became_active":"2024-04-26T21:27:02.307727+0000","last_became_peered":"2024-04-26T21:27:02.307727+0000","last_unstale":"2024-04-26T21:28:57.089562+0000","last_undegraded":"2024-04-26T21:28:57.089562+0000","last_fullsized":"2024-04-26T21:28:57.089562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:02.132125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.153650+0000","last_change":"2024-04-26T21:28:57.153650+0000","last_active":"2024-04-26T21:28:57.153650+0000","last_peered":"2024-04-26T21:28:57.153650+0000","last_clean":"2024-04-26T21:28:57.153650+0000","last_became_active":"2024-04-26T21:28:57.152541+0000","last_became_peered":"2024-04-26T21:28:57.152541+0000","last_unstale":"2024-04-26T21:28:57.153650+0000","last_undegraded":"2024-04-26T21:28:57.153650+0000","last_fullsized":"2024-04-26T21:28:57.153650+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:53.236806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089207+0000","last_change":"2024-04-26T21:27:07.506609+0000","last_active":"2024-04-26T21:28:57.089207+0000","last_peered":"2024-04-26T21:28:57.089207+0000","last_clean":"2024-04-26T21:28:57.089207+0000","last_became_active":"2024-04-26T21:27:07.506100+0000","last_became_peered":"2024-04-26T21:27:07.506100+0000","last_unstale":"2024-04-26T21:28:57.089207+0000","last_undegraded":"2024-04-26T21:28:57.089207+0000","last_fullsized":"2024-04-26T21:28:57.089207+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:26.535809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142097+0000","last_change":"2024-04-26T21:27:02.297432+0000","last_active":"2024-04-26T21:28:57.142097+0000","last_peered":"2024-04-26T21:28:57.142097+0000","last_clean":"2024-04-26T21:28:57.142097+0000","last_became_active":"2024-04-26T21:27:02.297081+0000","last_became_peered":"2024-04-26T21:27:02.297081+0000","last_unstale":"2024-04-26T21:28:57.142097+0000","last_undegraded":"2024-04-26T21:28:57.142097+0000","last_fullsized":"2024-04-26T21:28:57.142097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:08.139469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083343+0000","last_change":"2024-04-26T21:27:02.866653+0000","last_active":"2024-04-26T21:28:57.083343+0000","last_peered":"2024-04-26T21:28:57.083343+0000","last_clean":"2024-04-26T21:28:57.083343+0000","last_became_active":"2024-04-26T21:27:02.866275+0000","last_became_peered":"2024-04-26T21:27:02.866275+0000","last_unstale":"2024-04-26T21:28:57.083343+0000","last_undegraded":"2024-04-26T21:28:57.083343+0000","last_fullsized":"2024-04-26T21:28:57.083343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:39.089972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.143381+0000","last_change":"2024-04-26T21:28:57.142863+0000","last_active":"2024-04-26T21:28:57.143381+0000","last_peered":"2024-04-26T21:28:57.143381+0000","last_clean":"2024-04-26T21:28:57.143381+0000","last_became_active":"2024-04-26T21:28:57.141745+0000","last_became_peered":"2024-04-26T21:28:57.141745+0000","last_unstale":"2024-04-26T21:28:57.143381+0000","last_undegraded":"2024-04-26T21:28:57.143381+0000","last_fullsized":"2024-04-26T21:28:57.143381+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:07.261644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134213+0000","last_change":"2024-04-26T21:27:02.307250+0000","last_active":"2024-04-26T21:28:57.134213+0000","last_peered":"2024-04-26T21:28:57.134213+0000","last_clean":"2024-04-26T21:28:57.134213+0000","last_became_active":"2024-04-26T21:27:02.303532+0000","last_became_peered":"2024-04-26T21:27:02.303532+0000","last_unstale":"2024-04-26T21:28:57.134213+0000","last_undegraded":"2024-04-26T21:28:57.134213+0000","last_fullsized":"2024-04-26T21:28:57.134213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:36.265188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120172+0000","last_change":"2024-04-26T21:27:02.858818+0000","last_active":"2024-04-26T21:28:57.120172+0000","last_peered":"2024-04-26T21:28:57.120172+0000","last_clean":"2024-04-26T21:28:57.120172+0000","last_became_active":"2024-04-26T21:27:02.858059+0000","last_became_peered":"2024-04-26T21:27:02.858059+0000","last_unstale":"2024-04-26T21:28:57.120172+0000","last_undegraded":"2024-04-26T21:28:57.120172+0000","last_fullsized":"2024-04-26T21:28:57.120172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:02.159334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088650+0000","last_change":"2024-04-26T21:27:07.381273+0000","last_active":"2024-04-26T21:28:57.088650+0000","last_peered":"2024-04-26T21:28:57.088650+0000","last_clean":"2024-04-26T21:28:57.088650+0000","last_became_active":"2024-04-26T21:27:07.380891+0000","last_became_peered":"2024-04-26T21:27:07.380891+0000","last_unstale":"2024-04-26T21:28:57.088650+0000","last_undegraded":"2024-04-26T21:28:57.088650+0000","last_fullsized":"2024-04-26T21:28:57.088650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:39.641577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140340+0000","last_change":"2024-04-26T21:27:02.301363+0000","last_active":"2024-04-26T21:28:57.140340+0000","last_peered":"2024-04-26T21:28:57.140340+0000","last_clean":"2024-04-26T21:28:57.140340+0000","last_became_active":"2024-04-26T21:27:02.301059+0000","last_became_peered":"2024-04-26T21:27:02.301059+0000","last_unstale":"2024-04-26T21:28:57.140340+0000","last_undegraded":"2024-04-26T21:28:57.140340+0000","last_fullsized":"2024-04-26T21:28:57.140340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:17.586190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134340+0000","last_change":"2024-04-26T21:27:02.860612+0000","last_active":"2024-04-26T21:28:57.134340+0000","last_peered":"2024-04-26T21:28:57.134340+0000","last_clean":"2024-04-26T21:28:57.134340+0000","last_became_active":"2024-04-26T21:27:02.857285+0000","last_became_peered":"2024-04-26T21:27:02.857285+0000","last_unstale":"2024-04-26T21:28:57.134340+0000","last_undegraded":"2024-04-26T21:28:57.134340+0000","last_fullsized":"2024-04-26T21:28:57.134340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:11.968818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089342+0000","last_change":"2024-04-26T21:27:07.506425+0000","last_active":"2024-04-26T21:28:57.089342+0000","last_peered":"2024-04-26T21:28:57.089342+0000","last_clean":"2024-04-26T21:28:57.089342+0000","last_became_active":"2024-04-26T21:27:07.505968+0000","last_became_peered":"2024-04-26T21:27:07.505968+0000","last_unstale":"2024-04-26T21:28:57.089342+0000","last_undegraded":"2024-04-26T21:28:57.089342+0000","last_fullsized":"2024-04-26T21:28:57.089342+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:59.914560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083503+0000","last_change":"2024-04-26T21:27:02.294635+0000","last_active":"2024-04-26T21:28:57.083503+0000","last_peered":"2024-04-26T21:28:57.083503+0000","last_clean":"2024-04-26T21:28:57.083503+0000","last_became_active":"2024-04-26T21:27:02.294311+0000","last_became_peered":"2024-04-26T21:27:02.294311+0000","last_unstale":"2024-04-26T21:28:57.083503+0000","last_undegraded":"2024-04-26T21:28:57.083503+0000","last_fullsized":"2024-04-26T21:28:57.083503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:47.906546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141554+0000","last_change":"2024-04-26T21:27:02.875475+0000","last_active":"2024-04-26T21:28:57.141554+0000","last_peered":"2024-04-26T21:28:57.141554+0000","last_clean":"2024-04-26T21:28:57.141554+0000","last_became_active":"2024-04-26T21:27:02.875137+0000","last_became_peered":"2024-04-26T21:27:02.875137+0000","last_unstale":"2024-04-26T21:28:57.141554+0000","last_undegraded":"2024-04-26T21:28:57.141554+0000","last_fullsized":"2024-04-26T21:28:57.141554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:04.988973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141589+0000","last_change":"2024-04-26T21:27:07.372927+0000","last_active":"2024-04-26T21:28:57.141589+0000","last_peered":"2024-04-26T21:28:57.141589+0000","last_clean":"2024-04-26T21:28:57.141589+0000","last_became_active":"2024-04-26T21:27:07.372371+0000","last_became_peered":"2024-04-26T21:27:07.372371+0000","last_unstale":"2024-04-26T21:28:57.141589+0000","last_undegraded":"2024-04-26T21:28:57.141589+0000","last_fullsized":"2024-04-26T21:28:57.141589+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:12:41.607167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140654+0000","last_change":"2024-04-26T21:27:02.295379+0000","last_active":"2024-04-26T21:28:57.140654+0000","last_peered":"2024-04-26T21:28:57.140654+0000","last_clean":"2024-04-26T21:28:57.140654+0000","last_became_active":"2024-04-26T21:27:02.295077+0000","last_became_peered":"2024-04-26T21:27:02.295077+0000","last_unstale":"2024-04-26T21:28:57.140654+0000","last_undegraded":"2024-04-26T21:28:57.140654+0000","last_fullsized":"2024-04-26T21:28:57.140654+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:58.177960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141479+0000","last_change":"2024-04-26T21:27:02.855998+0000","last_active":"2024-04-26T21:28:57.141479+0000","last_peered":"2024-04-26T21:28:57.141479+0000","last_clean":"2024-04-26T21:28:57.141479+0000","last_became_active":"2024-04-26T21:27:02.855636+0000","last_became_peered":"2024-04-26T21:27:02.855636+0000","last_unstale":"2024-04-26T21:28:57.141479+0000","last_undegraded":"2024-04-26T21:28:57.141479+0000","last_fullsized":"2024-04-26T21:28:57.141479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:19:02.988946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084647+0000","last_change":"2024-04-26T21:27:07.378980+0000","last_active":"2024-04-26T21:28:57.084647+0000","last_peered":"2024-04-26T21:28:57.084647+0000","last_clean":"2024-04-26T21:28:57.084647+0000","last_became_active":"2024-04-26T21:27:07.378374+0000","last_became_peered":"2024-04-26T21:27:07.378374+0000","last_unstale":"2024-04-26T21:28:57.084647+0000","last_undegraded":"2024-04-26T21:28:57.084647+0000","last_fullsized":"2024-04-26T21:28:57.084647+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:31.845297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089815+0000","last_change":"2024-04-26T21:27:02.313801+0000","last_active":"2024-04-26T21:28:57.089815+0000","last_peered":"2024-04-26T21:28:57.089815+0000","last_clean":"2024-04-26T21:28:57.089815+0000","last_became_active":"2024-04-26T21:27:02.313540+0000","last_became_peered":"2024-04-26T21:27:02.313540+0000","last_unstale":"2024-04-26T21:28:57.089815+0000","last_undegraded":"2024-04-26T21:28:57.089815+0000","last_fullsized":"2024-04-26T21:28:57.089815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:38.612261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140890+0000","last_change":"2024-04-26T21:27:02.864151+0000","last_active":"2024-04-26T21:28:57.140890+0000","last_peered":"2024-04-26T21:28:57.140890+0000","last_clean":"2024-04-26T21:28:57.140890+0000","last_became_active":"2024-04-26T21:27:02.863818+0000","last_became_peered":"2024-04-26T21:27:02.863818+0000","last_unstale":"2024-04-26T21:28:57.140890+0000","last_undegraded":"2024-04-26T21:28:57.140890+0000","last_fullsized":"2024-04-26T21:28:57.140890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:26.215370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088554+0000","last_change":"2024-04-26T21:27:07.365821+0000","last_active":"2024-04-26T21:28:57.088554+0000","last_peered":"2024-04-26T21:28:57.088554+0000","last_clean":"2024-04-26T21:28:57.088554+0000","last_became_active":"2024-04-26T21:27:07.365347+0000","last_became_peered":"2024-04-26T21:27:07.365347+0000","last_unstale":"2024-04-26T21:28:57.088554+0000","last_undegraded":"2024-04-26T21:28:57.088554+0000","last_fullsized":"2024-04-26T21:28:57.088554+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:40:11.596908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150070+0000","last_change":"2024-04-26T21:27:02.310641+0000","last_active":"2024-04-26T21:28:57.150070+0000","last_peered":"2024-04-26T21:28:57.150070+0000","last_clean":"2024-04-26T21:28:57.150070+0000","last_became_active":"2024-04-26T21:27:02.308569+0000","last_became_peered":"2024-04-26T21:27:02.308569+0000","last_unstale":"2024-04-26T21:28:57.150070+0000","last_undegraded":"2024-04-26T21:28:57.150070+0000","last_fullsized":"2024-04-26T21:28:57.150070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:20.988098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141041+0000","last_change":"2024-04-26T21:27:02.854976+0000","last_active":"2024-04-26T21:28:57.141041+0000","last_peered":"2024-04-26T21:28:57.141041+0000","last_clean":"2024-04-26T21:28:57.141041+0000","last_became_active":"2024-04-26T21:27:02.854611+0000","last_became_peered":"2024-04-26T21:27:02.854611+0000","last_unstale":"2024-04-26T21:28:57.141041+0000","last_undegraded":"2024-04-26T21:28:57.141041+0000","last_fullsized":"2024-04-26T21:28:57.141041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:57.065657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120248+0000","last_change":"2024-04-26T21:27:02.859017+0000","last_active":"2024-04-26T21:28:57.120248+0000","last_peered":"2024-04-26T21:28:57.120248+0000","last_clean":"2024-04-26T21:28:57.120248+0000","last_became_active":"2024-04-26T21:27:02.858507+0000","last_became_peered":"2024-04-26T21:27:02.858507+0000","last_unstale":"2024-04-26T21:28:57.120248+0000","last_undegraded":"2024-04-26T21:28:57.120248+0000","last_fullsized":"2024-04-26T21:28:57.120248+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:14.850891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140387+0000","last_change":"2024-04-26T21:27:56.841251+0000","last_active":"2024-04-26T21:28:57.140387+0000","last_peered":"2024-04-26T21:28:57.140387+0000","last_clean":"2024-04-26T21:28:57.140387+0000","last_became_active":"2024-04-26T21:27:56.840862+0000","last_became_peered":"2024-04-26T21:27:56.840862+0000","last_unstale":"2024-04-26T21:28:57.140387+0000","last_undegraded":"2024-04-26T21:28:57.140387+0000","last_fullsized":"2024-04-26T21:28:57.140387+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:41.460343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140100+0000","last_change":"2024-04-26T21:27:02.868817+0000","last_active":"2024-04-26T21:28:57.140100+0000","last_peered":"2024-04-26T21:28:57.140100+0000","last_clean":"2024-04-26T21:28:57.140100+0000","last_became_active":"2024-04-26T21:27:02.868557+0000","last_became_peered":"2024-04-26T21:27:02.868557+0000","last_unstale":"2024-04-26T21:28:57.140100+0000","last_undegraded":"2024-04-26T21:28:57.140100+0000","last_fullsized":"2024-04-26T21:28:57.140100+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:55.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.085674+0000","last_change":"2024-04-26T21:27:02.876029+0000","last_active":"2024-04-26T21:28:57.085674+0000","last_peered":"2024-04-26T21:28:57.085674+0000","last_clean":"2024-04-26T21:28:57.085674+0000","last_became_active":"2024-04-26T21:27:02.875656+0000","last_became_peered":"2024-04-26T21:27:02.875656+0000","last_unstale":"2024-04-26T21:28:57.085674+0000","last_undegraded":"2024-04-26T21:28:57.085674+0000","last_fullsized":"2024-04-26T21:28:57.085674+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:47.317581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150613+0000","last_change":"2024-04-26T21:27:02.873818+0000","last_active":"2024-04-26T21:28:57.150613+0000","last_peered":"2024-04-26T21:28:57.150613+0000","last_clean":"2024-04-26T21:28:57.150613+0000","last_became_active":"2024-04-26T21:27:02.873459+0000","last_became_peered":"2024-04-26T21:27:02.873459+0000","last_unstale":"2024-04-26T21:28:57.150613+0000","last_undegraded":"2024-04-26T21:28:57.150613+0000","last_fullsized":"2024-04-26T21:28:57.150613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:07.259187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120678+0000","last_change":"2024-04-26T21:27:02.854901+0000","last_active":"2024-04-26T21:28:57.120678+0000","last_peered":"2024-04-26T21:28:57.120678+0000","last_clean":"2024-04-26T21:28:57.120678+0000","last_became_active":"2024-04-26T21:27:02.854547+0000","last_became_peered":"2024-04-26T21:27:02.854547+0000","last_unstale":"2024-04-26T21:28:57.120678+0000","last_undegraded":"2024-04-26T21:28:57.120678+0000","last_fullsized":"2024-04-26T21:28:57.120678+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:24.320868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141003+0000","last_change":"2024-04-26T21:27:02.856616+0000","last_active":"2024-04-26T21:28:57.141003+0000","last_peered":"2024-04-26T21:28:57.141003+0000","last_clean":"2024-04-26T21:28:57.141003+0000","last_became_active":"2024-04-26T21:27:02.856264+0000","last_became_peered":"2024-04-26T21:27:02.856264+0000","last_unstale":"2024-04-26T21:28:57.141003+0000","last_undegraded":"2024-04-26T21:28:57.141003+0000","last_fullsized":"2024-04-26T21:28:57.141003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:43.374534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.133975+0000","last_change":"2024-04-26T21:27:02.859587+0000","last_active":"2024-04-26T21:28:57.133975+0000","last_peered":"2024-04-26T21:28:57.133975+0000","last_clean":"2024-04-26T21:28:57.133975+0000","last_became_active":"2024-04-26T21:27:02.859136+0000","last_became_peered":"2024-04-26T21:27:02.859136+0000","last_unstale":"2024-04-26T21:28:57.133975+0000","last_undegraded":"2024-04-26T21:28:57.133975+0000","last_fullsized":"2024-04-26T21:28:57.133975+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:09.370319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088720+0000","last_change":"2024-04-26T21:27:02.864265+0000","last_active":"2024-04-26T21:28:57.088720+0000","last_peered":"2024-04-26T21:28:57.088720+0000","last_clean":"2024-04-26T21:28:57.088720+0000","last_became_active":"2024-04-26T21:27:02.863920+0000","last_became_peered":"2024-04-26T21:27:02.863920+0000","last_unstale":"2024-04-26T21:28:57.088720+0000","last_undegraded":"2024-04-26T21:28:57.088720+0000","last_fullsized":"2024-04-26T21:28:57.088720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:13.941544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134222+0000","last_change":"2024-04-26T21:27:02.860189+0000","last_active":"2024-04-26T21:28:57.134222+0000","last_peered":"2024-04-26T21:28:57.134222+0000","last_clean":"2024-04-26T21:28:57.134222+0000","last_became_active":"2024-04-26T21:27:02.859843+0000","last_became_peered":"2024-04-26T21:27:02.859843+0000","last_unstale":"2024-04-26T21:28:57.134222+0000","last_undegraded":"2024-04-26T21:28:57.134222+0000","last_fullsized":"2024-04-26T21:28:57.134222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141963+0000","last_change":"2024-04-26T21:27:02.865459+0000","last_active":"2024-04-26T21:28:57.141963+0000","last_peered":"2024-04-26T21:28:57.141963+0000","last_clean":"2024-04-26T21:28:57.141963+0000","last_became_active":"2024-04-26T21:27:02.863602+0000","last_became_peered":"2024-04-26T21:27:02.863602+0000","last_unstale":"2024-04-26T21:28:57.141963+0000","last_undegraded":"2024-04-26T21:28:57.141963+0000","last_fullsized":"2024-04-26T21:28:57.141963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:03.733858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.149411+0000","last_change":"2024-04-26T21:27:02.873711+0000","last_active":"2024-04-26T21:28:57.149411+0000","last_peered":"2024-04-26T21:28:57.149411+0000","last_clean":"2024-04-26T21:28:57.149411+0000","last_became_active":"2024-04-26T21:27:02.873349+0000","last_became_peered":"2024-04-26T21:27:02.873349+0000","last_unstale":"2024-04-26T21:28:57.149411+0000","last_undegraded":"2024-04-26T21:28:57.149411+0000","last_fullsized":"2024-04-26T21:28:57.149411+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:10.841536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120403+0000","last_change":"2024-04-26T21:27:02.852625+0000","last_active":"2024-04-26T21:28:57.120403+0000","last_peered":"2024-04-26T21:28:57.120403+0000","last_clean":"2024-04-26T21:28:57.120403+0000","last_became_active":"2024-04-26T21:27:02.852177+0000","last_became_peered":"2024-04-26T21:27:02.852177+0000","last_unstale":"2024-04-26T21:28:57.120403+0000","last_undegraded":"2024-04-26T21:28:57.120403+0000","last_fullsized":"2024-04-26T21:28:57.120403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:52.046678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59286,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":573,"num_read_kb":345,"num_write":164,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318310,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":528427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579947,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28740,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716412,"statfs":{"total":95995035648,"available":95965605888,"internally_reserved":0,"allocated":1761280,"data_stored":1171349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743473,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1097728,"data_stored":532737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005112,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":528539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266749,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32832,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712320,"statfs":{"total":95995035648,"available":95961415680,"internally_reserved":0,"allocated":1687552,"data_stored":1124079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":523817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1134592,"data_stored":579403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1097728,"data_stored":525205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411420,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1159168,"data_stored":580753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1085440,"data_stored":524011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":528467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37000,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708152,"statfs":{"total":95995035648,"available":95957147648,"internally_reserved":0,"allocated":1695744,"data_stored":1119667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":55676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":57026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":57254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:29:06.253 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-26T21:29:06.729 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:07.232 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:07 smithi001 ceph-mon[28054]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:07.232 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:07 smithi001 ceph-mon[28054]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 110 B/s wr, 6 op/s 2024-04-26T21:29:07.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:07 smithi186 ceph-mon[32991]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:07.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:07 smithi186 ceph-mon[32991]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 110 B/s wr, 6 op/s 2024-04-26T21:29:07.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:07 smithi038 ceph-mon[32724]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:07.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:07 smithi038 ceph-mon[32724]: pgmap v390: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 110 B/s wr, 6 op/s 2024-04-26T21:29:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.210+0000 7f463bc00700 1 -- 172.21.15.1:0/481059006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4634103ea0 msgr2=0x7f46341102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.210+0000 7f463bc00700 1 --2- 172.21.15.1:0/481059006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4634103ea0 0x7f46341102c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f462c009f90 tx=0x7f462c00b330 comp rx=0 tx=0).stop 2024-04-26T21:29:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.210+0000 7f463bc00700 1 -- 172.21.15.1:0/481059006 shutdown_connections 2024-04-26T21:29:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.210+0000 7f463bc00700 1 --2- 172.21.15.1:0/481059006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4634109480 0x7f4634109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.210+0000 7f463bc00700 1 --2- 172.21.15.1:0/481059006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4634103ea0 0x7f46341102c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.210+0000 7f463bc00700 1 --2- 172.21.15.1:0/481059006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46341034e0 0x7f4634103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.210+0000 7f463bc00700 1 -- 172.21.15.1:0/481059006 >> 172.21.15.1:0/481059006 conn(0x7f463407a5e0 msgr2=0x7f463407a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.210+0000 7f463bc00700 1 -- 172.21.15.1:0/481059006 shutdown_connections 2024-04-26T21:29:09.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.210+0000 7f463bc00700 1 -- 172.21.15.1:0/481059006 wait complete. 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.211+0000 7f463bc00700 1 Processor -- start 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.212+0000 7f463bc00700 1 -- start start 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.212+0000 7f463bc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46341034e0 0x7f463419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.212+0000 7f463bc00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4634103ea0 0x7f463419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.212+0000 7f463bc00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4634109480 0x7f46341a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.212+0000 7f463bc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4634113600 con 0x7f46341034e0 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.212+0000 7f463bc00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4634113480 con 0x7f4634109480 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.212+0000 7f463bc00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4634113780 con 0x7f4634103ea0 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.212+0000 7f4639200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46341034e0 0x7f463419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4639200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46341034e0 0x7f463419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33540/0 (socket says 172.21.15.1:33540) 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4639200700 1 -- 172.21.15.1:0/2418274693 learned_addr learned my addr 172.21.15.1:0/2418274693 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4639c00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4634109480 0x7f46341a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4633e00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4634103ea0 0x7f463419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4633e00700 1 -- 172.21.15.1:0/2418274693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4634109480 msgr2=0x7f46341a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4633e00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4634109480 0x7f46341a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4633e00700 1 -- 172.21.15.1:0/2418274693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46341034e0 msgr2=0x7f463419bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4633e00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46341034e0 0x7f463419bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4633e00700 1 -- 172.21.15.1:0/2418274693 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46341a0d70 con 0x7f4634103ea0 2024-04-26T21:29:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4639200700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46341034e0 0x7f463419bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:09.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4639c00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4634109480 0x7f46341a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:29:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.213+0000 7f4633e00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4634103ea0 0x7f463419c2a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f462000eb90 tx=0x7f462000eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.214+0000 7f4631600700 1 -- 172.21.15.1:0/2418274693 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462000ccf0 con 0x7f4634103ea0 2024-04-26T21:29:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.214+0000 7f463bc00700 1 -- 172.21.15.1:0/2418274693 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46341a1060 con 0x7f4634103ea0 2024-04-26T21:29:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.214+0000 7f4631600700 1 -- 172.21.15.1:0/2418274693 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46200044d0 con 0x7f4634103ea0 2024-04-26T21:29:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.214+0000 7f4631600700 1 -- 172.21.15.1:0/2418274693 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620010430 con 0x7f4634103ea0 2024-04-26T21:29:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.214+0000 7f463bc00700 1 -- 172.21.15.1:0/2418274693 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46341a88e0 con 0x7f4634103ea0 2024-04-26T21:29:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.215+0000 7f463bc00700 1 -- 172.21.15.1:0/2418274693 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f463406a110 con 0x7f4634103ea0 2024-04-26T21:29:09.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.217+0000 7f4631600700 1 -- 172.21.15.1:0/2418274693 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f4620003710 con 0x7f4634103ea0 2024-04-26T21:29:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.218+0000 7f4631600700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f461c0a3d50 0x7f461c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.219+0000 7f4631600700 1 -- 172.21.15.1:0/2418274693 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f46200d0cb0 con 0x7f4634103ea0 2024-04-26T21:29:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.219+0000 7f4639200700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f461c0a3d50 0x7f461c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.219+0000 7f4639200700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f461c0a3d50 0x7f461c0a6210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4628011550 tx=0x7f4628009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:09.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.223+0000 7f4631600700 1 -- 172.21.15.1:0/2418274693 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f462009d0d0 con 0x7f4634103ea0 2024-04-26T21:29:09.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:09 smithi001 ceph-mon[28054]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 99 B/s wr, 6 op/s 2024-04-26T21:29:09.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:09 smithi186 ceph-mon[32991]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 99 B/s wr, 6 op/s 2024-04-26T21:29:09.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.548+0000 7f463bc00700 1 -- 172.21.15.1:0/2418274693 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f46341a8b70 con 0x7f4634103ea0 2024-04-26T21:29:09.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.549+0000 7f4631600700 1 -- 172.21.15.1:0/2418274693 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1493 (secure 0 0 0) 0x7f462009c9d0 con 0x7f4634103ea0 2024-04-26T21:29:09.550 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:09.550 INFO:teuthology.orchestra.run.smithi001.stdout:{"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":543,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":95,"num_osds":12,"num_up_osds":12,"osd_up_since":1714166763,"num_in_osds":12,"osd_in_since":1714166743,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129}],"num_pgs":129,"num_pools":4,"num_objects":64,"data_bytes":649888,"bytes_used":389705728,"bytes_avail":1151550722048,"bytes_total":1151940427776,"read_bytes_sec":2980,"write_bytes_sec":99,"read_op_per_sec":6,"write_op_per_sec":0},"fsmap":{"epoch":23,"id":1,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":1,"rank":2,"name":"e","status":"up:standby-replay","gid":24515},{"filesystem_id":1,"rank":1,"name":"c","status":"up:standby-replay","gid":24523},{"filesystem_id":1,"rank":0,"name":"i","status":"up:standby-replay","gid":24547},{"filesystem_id":1,"rank":2,"name":"h","status":"up:active","gid":24548},{"filesystem_id":1,"rank":1,"name":"g","status":"up:active","gid":24572},{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24602}],"up:standby":6},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.1:8443/"}},"servicemap":{"epoch":27,"modified":"2024-04-26T21:28:37.343502+0000","services":{}},"progress_events":{}} 2024-04-26T21:29:09.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 -- 172.21.15.1:0/2418274693 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f461c0a3d50 msgr2=0x7f461c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:09.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f461c0a3d50 0x7f461c0a6210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4628011550 tx=0x7f4628009260 comp rx=0 tx=0).stop 2024-04-26T21:29:09.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 -- 172.21.15.1:0/2418274693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4634103ea0 msgr2=0x7f463419c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:09.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4634103ea0 0x7f463419c2a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f462000eb90 tx=0x7f462000eea0 comp rx=0 tx=0).stop 2024-04-26T21:29:09.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 -- 172.21.15.1:0/2418274693 shutdown_connections 2024-04-26T21:29:09.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f461c0a3d50 0x7f461c0a6210 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:09.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4634109480 0x7f46341a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:09.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46341034e0 0x7f463419bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:09.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 --2- 172.21.15.1:0/2418274693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4634103ea0 0x7f463419c2a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:09.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 -- 172.21.15.1:0/2418274693 >> 172.21.15.1:0/2418274693 conn(0x7f463407a5e0 msgr2=0x7f4634101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:09.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.552+0000 7f463bc00700 1 -- 172.21.15.1:0/2418274693 shutdown_connections 2024-04-26T21:29:09.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:09.553+0000 7f463bc00700 1 -- 172.21.15.1:0/2418274693 wait complete. 2024-04-26T21:29:09.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:09 smithi038 ceph-mon[32724]: pgmap v391: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 99 B/s wr, 6 op/s 2024-04-26T21:29:10.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:10 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2418274693' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T21:29:10.417 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-26T21:29:10.417 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T21:29:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:10 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2418274693' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T21:29:10.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:10 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2418274693' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T21:29:10.921 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:11.338 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:11 smithi001 ceph-mon[28054]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:29:11.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:11 smithi186 ceph-mon[32991]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:29:11.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:11 smithi038 ceph-mon[32724]: pgmap v392: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-04-26T21:29:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.146+0000 7f5546000700 1 -- 172.21.15.1:0/1398282742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55401094a0 msgr2=0x7f5540109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.146+0000 7f5546000700 1 --2- 172.21.15.1:0/1398282742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55401094a0 0x7f5540109880 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5530009990 tx=0x7f5530009ca0 comp rx=0 tx=0).stop 2024-04-26T21:29:13.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.146+0000 7f5546000700 1 -- 172.21.15.1:0/1398282742 shutdown_connections 2024-04-26T21:29:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.146+0000 7f5546000700 1 --2- 172.21.15.1:0/1398282742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55401094a0 0x7f5540109880 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.146+0000 7f5546000700 1 --2- 172.21.15.1:0/1398282742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f554006bad0 0x7f55401102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.146+0000 7f5546000700 1 --2- 172.21.15.1:0/1398282742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f554006b130 0x7f554006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.146+0000 7f5546000700 1 -- 172.21.15.1:0/1398282742 >> 172.21.15.1:0/1398282742 conn(0x7f55400792d0 msgr2=0x7f55400796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.146+0000 7f5546000700 1 -- 172.21.15.1:0/1398282742 shutdown_connections 2024-04-26T21:29:13.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.146+0000 7f5546000700 1 -- 172.21.15.1:0/1398282742 wait complete. 2024-04-26T21:29:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.147+0000 7f5546000700 1 Processor -- start 2024-04-26T21:29:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.147+0000 7f5546000700 1 -- start start 2024-04-26T21:29:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f5546000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f554006b130 0x7f554019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f5546000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f554006bad0 0x7f554019c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:13.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f5546000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55401094a0 0x7f55401a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f5546000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f55401135e0 con 0x7f554006b130 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f5546000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5540113460 con 0x7f554006bad0 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f5546000700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5540113760 con 0x7f55401094a0 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f553f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f554006b130 0x7f554019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f553f400700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f554006b130 0x7f554019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33554/0 (socket says 172.21.15.1:33554) 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f553f400700 1 -- 172.21.15.1:0/4221513335 learned_addr learned my addr 172.21.15.1:0/4221513335 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f553fe00700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55401094a0 0x7f55401a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.148+0000 7f553ea00700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f554006bad0 0x7f554019c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.149+0000 7f553f400700 1 -- 172.21.15.1:0/4221513335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55401094a0 msgr2=0x7f55401a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.149+0000 7f553f400700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55401094a0 0x7f55401a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.149+0000 7f553f400700 1 -- 172.21.15.1:0/4221513335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f554006bad0 msgr2=0x7f554019c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.149+0000 7f553f400700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f554006bad0 0x7f554019c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.149+0000 7f553f400700 1 -- 172.21.15.1:0/4221513335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55401a0d40 con 0x7f554006b130 2024-04-26T21:29:13.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.149+0000 7f553f400700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f554006b130 0x7f554019bd30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5530005910 tx=0x7f553000b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:13.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.149+0000 7f5537e00700 1 -- 172.21.15.1:0/4221513335 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530027070 con 0x7f554006b130 2024-04-26T21:29:13.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.149+0000 7f5537e00700 1 -- 172.21.15.1:0/4221513335 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5530022470 con 0x7f554006b130 2024-04-26T21:29:13.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.149+0000 7f5546000700 1 -- 172.21.15.1:0/4221513335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55401a0fd0 con 0x7f554006b130 2024-04-26T21:29:13.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.149+0000 7f5537e00700 1 -- 172.21.15.1:0/4221513335 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5530017650 con 0x7f554006b130 2024-04-26T21:29:13.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.150+0000 7f5546000700 1 -- 172.21.15.1:0/4221513335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55401a88b0 con 0x7f554006b130 2024-04-26T21:29:13.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.151+0000 7f5546000700 1 -- 172.21.15.1:0/4221513335 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f554006a110 con 0x7f554006b130 2024-04-26T21:29:13.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.153+0000 7f5537e00700 1 -- 172.21.15.1:0/4221513335 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f553002a070 con 0x7f554006b130 2024-04-26T21:29:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.154+0000 7f5537e00700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f55240a3d50 0x7f55240a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.154+0000 7f5537e00700 1 -- 172.21.15.1:0/4221513335 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f55300d1630 con 0x7f554006b130 2024-04-26T21:29:13.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.154+0000 7f553ea00700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f55240a3d50 0x7f55240a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:13.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.155+0000 7f553ea00700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f55240a3d50 0x7f55240a6210 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5540079a40 tx=0x7f552c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:13.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.159+0000 7f5537e00700 1 -- 172.21.15.1:0/4221513335 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5530013070 con 0x7f554006b130 2024-04-26T21:29:13.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:13 smithi001 ceph-mon[28054]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:29:13.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.385+0000 7f5546000700 1 -- 172.21.15.1:0/4221513335 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f554019d090 con 0x7f55240a3d50 2024-04-26T21:29:13.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.414+0000 7f5537e00700 1 -- 172.21.15.1:0/4221513335 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340377 (secure 0 0 0) 0x7f554019d090 con 0x7f55240a3d50 2024-04-26T21:29:13.423 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:13.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.423+0000 7f5546000700 1 -- 172.21.15.1:0/4221513335 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f55240a3d50 msgr2=0x7f55240a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:13.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.423+0000 7f5546000700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f55240a3d50 0x7f55240a6210 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5540079a40 tx=0x7f552c009260 comp rx=0 tx=0).stop 2024-04-26T21:29:13.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.423+0000 7f5546000700 1 -- 172.21.15.1:0/4221513335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f554006b130 msgr2=0x7f554019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:13.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.423+0000 7f5546000700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f554006b130 0x7f554019bd30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5530005910 tx=0x7f553000b670 comp rx=0 tx=0).stop 2024-04-26T21:29:13.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.423+0000 7f5546000700 1 -- 172.21.15.1:0/4221513335 shutdown_connections 2024-04-26T21:29:13.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.423+0000 7f5546000700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f55240a3d50 0x7f55240a6210 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:13.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.423+0000 7f5546000700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55401094a0 0x7f55401a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:13.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.423+0000 7f5546000700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f554006b130 0x7f554019bd30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:13.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.423+0000 7f5546000700 1 --2- 172.21.15.1:0/4221513335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f554006bad0 0x7f554019c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:13.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.423+0000 7f5546000700 1 -- 172.21.15.1:0/4221513335 >> 172.21.15.1:0/4221513335 conn(0x7f55400792d0 msgr2=0x7f554010e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:13.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.424+0000 7f5546000700 1 -- 172.21.15.1:0/4221513335 shutdown_connections 2024-04-26T21:29:13.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:13.424+0000 7f5546000700 1 -- 172.21.15.1:0/4221513335 wait complete. 2024-04-26T21:29:13.426 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T21:29:13.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:13 smithi186 ceph-mon[32991]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:29:13.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:13 smithi038 ceph-mon[32724]: pgmap v393: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-26T21:29:14.282 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-04-26T21:29:11.390190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649888,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":759,"num_read_kb":494,"num_write":347,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180758,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380572,"kb_used_data":14696,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561252,"statfs":{"total":1151940427776,"available":1151550722048,"internally_reserved":0,"allocated":15048704,"data_stored":8266454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.036298"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120793+0000","last_change":"2024-04-26T21:27:02.854724+0000","last_active":"2024-04-26T21:28:57.120793+0000","last_peered":"2024-04-26T21:28:57.120793+0000","last_clean":"2024-04-26T21:28:57.120793+0000","last_became_active":"2024-04-26T21:27:02.854125+0000","last_became_peered":"2024-04-26T21:27:02.854125+0000","last_unstale":"2024-04-26T21:28:57.120793+0000","last_undegraded":"2024-04-26T21:28:57.120793+0000","last_fullsized":"2024-04-26T21:28:57.120793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:16.500220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089008+0000","last_change":"2024-04-26T21:27:02.852846+0000","last_active":"2024-04-26T21:28:57.089008+0000","last_peered":"2024-04-26T21:28:57.089008+0000","last_clean":"2024-04-26T21:28:57.089008+0000","last_became_active":"2024-04-26T21:27:02.851243+0000","last_became_peered":"2024-04-26T21:27:02.851243+0000","last_unstale":"2024-04-26T21:28:57.089008+0000","last_undegraded":"2024-04-26T21:28:57.089008+0000","last_fullsized":"2024-04-26T21:28:57.089008+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:09:11.240955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089984+0000","last_change":"2024-04-26T21:27:02.855856+0000","last_active":"2024-04-26T21:28:57.089984+0000","last_peered":"2024-04-26T21:28:57.089984+0000","last_clean":"2024-04-26T21:28:57.089984+0000","last_became_active":"2024-04-26T21:27:02.855446+0000","last_became_peered":"2024-04-26T21:27:02.855446+0000","last_unstale":"2024-04-26T21:28:57.089984+0000","last_undegraded":"2024-04-26T21:28:57.089984+0000","last_fullsized":"2024-04-26T21:28:57.089984+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:35.183449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089111+0000","last_change":"2024-04-26T21:27:02.860949+0000","last_active":"2024-04-26T21:28:57.089111+0000","last_peered":"2024-04-26T21:28:57.089111+0000","last_clean":"2024-04-26T21:28:57.089111+0000","last_became_active":"2024-04-26T21:27:02.860567+0000","last_became_peered":"2024-04-26T21:27:02.860567+0000","last_unstale":"2024-04-26T21:28:57.089111+0000","last_undegraded":"2024-04-26T21:28:57.089111+0000","last_fullsized":"2024-04-26T21:28:57.089111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:09.851885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083417+0000","last_change":"2024-04-26T21:27:02.876019+0000","last_active":"2024-04-26T21:28:57.083417+0000","last_peered":"2024-04-26T21:28:57.083417+0000","last_clean":"2024-04-26T21:28:57.083417+0000","last_became_active":"2024-04-26T21:27:02.875652+0000","last_became_peered":"2024-04-26T21:27:02.875652+0000","last_unstale":"2024-04-26T21:28:57.083417+0000","last_undegraded":"2024-04-26T21:28:57.083417+0000","last_fullsized":"2024-04-26T21:28:57.083417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:35.017787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088087+0000","last_change":"2024-04-26T21:27:02.860229+0000","last_active":"2024-04-26T21:28:57.088087+0000","last_peered":"2024-04-26T21:28:57.088087+0000","last_clean":"2024-04-26T21:28:57.088087+0000","last_became_active":"2024-04-26T21:27:02.859863+0000","last_became_peered":"2024-04-26T21:27:02.859863+0000","last_unstale":"2024-04-26T21:28:57.088087+0000","last_undegraded":"2024-04-26T21:28:57.088087+0000","last_fullsized":"2024-04-26T21:28:57.088087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:47.337073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140980+0000","last_change":"2024-04-26T21:27:02.863253+0000","last_active":"2024-04-26T21:28:57.140980+0000","last_peered":"2024-04-26T21:28:57.140980+0000","last_clean":"2024-04-26T21:28:57.140980+0000","last_became_active":"2024-04-26T21:27:02.861623+0000","last_became_peered":"2024-04-26T21:27:02.861623+0000","last_unstale":"2024-04-26T21:28:57.140980+0000","last_undegraded":"2024-04-26T21:28:57.140980+0000","last_fullsized":"2024-04-26T21:28:57.140980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:57.642504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.133344+0000","last_change":"2024-04-26T21:27:02.860822+0000","last_active":"2024-04-26T21:28:57.133344+0000","last_peered":"2024-04-26T21:28:57.133344+0000","last_clean":"2024-04-26T21:28:57.133344+0000","last_became_active":"2024-04-26T21:27:02.860482+0000","last_became_peered":"2024-04-26T21:27:02.860482+0000","last_unstale":"2024-04-26T21:28:57.133344+0000","last_undegraded":"2024-04-26T21:28:57.133344+0000","last_fullsized":"2024-04-26T21:28:57.133344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:59.224624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141309+0000","last_change":"2024-04-26T21:27:56.856389+0000","last_active":"2024-04-26T21:28:57.141309+0000","last_peered":"2024-04-26T21:28:57.141309+0000","last_clean":"2024-04-26T21:28:57.141309+0000","last_became_active":"2024-04-26T21:27:56.856008+0000","last_became_peered":"2024-04-26T21:27:56.856008+0000","last_unstale":"2024-04-26T21:28:57.141309+0000","last_undegraded":"2024-04-26T21:28:57.141309+0000","last_fullsized":"2024-04-26T21:28:57.141309+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:04.770535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088910+0000","last_change":"2024-04-26T21:27:02.860545+0000","last_active":"2024-04-26T21:28:57.088910+0000","last_peered":"2024-04-26T21:28:57.088910+0000","last_clean":"2024-04-26T21:28:57.088910+0000","last_became_active":"2024-04-26T21:27:02.858577+0000","last_became_peered":"2024-04-26T21:27:02.858577+0000","last_unstale":"2024-04-26T21:28:57.088910+0000","last_undegraded":"2024-04-26T21:28:57.088910+0000","last_fullsized":"2024-04-26T21:28:57.088910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:43.390592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141372+0000","last_change":"2024-04-26T21:27:02.858685+0000","last_active":"2024-04-26T21:28:57.141372+0000","last_peered":"2024-04-26T21:28:57.141372+0000","last_clean":"2024-04-26T21:28:57.141372+0000","last_became_active":"2024-04-26T21:27:02.857054+0000","last_became_peered":"2024-04-26T21:27:02.857054+0000","last_unstale":"2024-04-26T21:28:57.141372+0000","last_undegraded":"2024-04-26T21:28:57.141372+0000","last_fullsized":"2024-04-26T21:28:57.141372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:29.406132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088929+0000","last_change":"2024-04-26T21:27:02.852457+0000","last_active":"2024-04-26T21:28:57.088929+0000","last_peered":"2024-04-26T21:28:57.088929+0000","last_clean":"2024-04-26T21:28:57.088929+0000","last_became_active":"2024-04-26T21:27:02.852064+0000","last_became_peered":"2024-04-26T21:27:02.852064+0000","last_unstale":"2024-04-26T21:28:57.088929+0000","last_undegraded":"2024-04-26T21:28:57.088929+0000","last_fullsized":"2024-04-26T21:28:57.088929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:16.798413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.149646+0000","last_change":"2024-04-26T21:27:02.872618+0000","last_active":"2024-04-26T21:28:57.149646+0000","last_peered":"2024-04-26T21:28:57.149646+0000","last_clean":"2024-04-26T21:28:57.149646+0000","last_became_active":"2024-04-26T21:27:02.872270+0000","last_became_peered":"2024-04-26T21:27:02.872270+0000","last_unstale":"2024-04-26T21:28:57.149646+0000","last_undegraded":"2024-04-26T21:28:57.149646+0000","last_fullsized":"2024-04-26T21:28:57.149646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:45.828419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084860+0000","last_change":"2024-04-26T21:27:02.868590+0000","last_active":"2024-04-26T21:28:57.084860+0000","last_peered":"2024-04-26T21:28:57.084860+0000","last_clean":"2024-04-26T21:28:57.084860+0000","last_became_active":"2024-04-26T21:27:02.866641+0000","last_became_peered":"2024-04-26T21:27:02.866641+0000","last_unstale":"2024-04-26T21:28:57.084860+0000","last_undegraded":"2024-04-26T21:28:57.084860+0000","last_fullsized":"2024-04-26T21:28:57.084860+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:17.313420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084439+0000","last_change":"2024-04-26T21:27:02.867735+0000","last_active":"2024-04-26T21:28:57.084439+0000","last_peered":"2024-04-26T21:28:57.084439+0000","last_clean":"2024-04-26T21:28:57.084439+0000","last_became_active":"2024-04-26T21:27:02.867217+0000","last_became_peered":"2024-04-26T21:27:02.867217+0000","last_unstale":"2024-04-26T21:28:57.084439+0000","last_undegraded":"2024-04-26T21:28:57.084439+0000","last_fullsized":"2024-04-26T21:28:57.084439+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:04:45.533220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140625+0000","last_change":"2024-04-26T21:27:02.857541+0000","last_active":"2024-04-26T21:28:57.140625+0000","last_peered":"2024-04-26T21:28:57.140625+0000","last_clean":"2024-04-26T21:28:57.140625+0000","last_became_active":"2024-04-26T21:27:02.855563+0000","last_became_peered":"2024-04-26T21:27:02.855563+0000","last_unstale":"2024-04-26T21:28:57.140625+0000","last_undegraded":"2024-04-26T21:28:57.140625+0000","last_fullsized":"2024-04-26T21:28:57.140625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:39.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120658+0000","last_change":"2024-04-26T21:27:02.854461+0000","last_active":"2024-04-26T21:28:57.120658+0000","last_peered":"2024-04-26T21:28:57.120658+0000","last_clean":"2024-04-26T21:28:57.120658+0000","last_became_active":"2024-04-26T21:27:02.853691+0000","last_became_peered":"2024-04-26T21:27:02.853691+0000","last_unstale":"2024-04-26T21:28:57.120658+0000","last_undegraded":"2024-04-26T21:28:57.120658+0000","last_fullsized":"2024-04-26T21:28:57.120658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:09:48.284180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.094647+0000","last_change":"2024-04-26T21:27:02.864308+0000","last_active":"2024-04-26T21:28:57.094647+0000","last_peered":"2024-04-26T21:28:57.094647+0000","last_clean":"2024-04-26T21:28:57.094647+0000","last_became_active":"2024-04-26T21:27:02.863928+0000","last_became_peered":"2024-04-26T21:27:02.863928+0000","last_unstale":"2024-04-26T21:28:57.094647+0000","last_undegraded":"2024-04-26T21:28:57.094647+0000","last_fullsized":"2024-04-26T21:28:57.094647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:12.169394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140846+0000","last_change":"2024-04-26T21:27:02.876044+0000","last_active":"2024-04-26T21:28:57.140846+0000","last_peered":"2024-04-26T21:28:57.140846+0000","last_clean":"2024-04-26T21:28:57.140846+0000","last_became_active":"2024-04-26T21:27:02.875681+0000","last_became_peered":"2024-04-26T21:27:02.875681+0000","last_unstale":"2024-04-26T21:28:57.140846+0000","last_undegraded":"2024-04-26T21:28:57.140846+0000","last_fullsized":"2024-04-26T21:28:57.140846+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:26.246916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140001+0000","last_change":"2024-04-26T21:27:07.374736+0000","last_active":"2024-04-26T21:28:57.140001+0000","last_peered":"2024-04-26T21:28:57.140001+0000","last_clean":"2024-04-26T21:28:57.140001+0000","last_became_active":"2024-04-26T21:27:07.374116+0000","last_became_peered":"2024-04-26T21:27:07.374116+0000","last_unstale":"2024-04-26T21:28:57.140001+0000","last_undegraded":"2024-04-26T21:28:57.140001+0000","last_fullsized":"2024-04-26T21:28:57.140001+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:11.450926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.095218+0000","last_change":"2024-04-26T21:27:02.302510+0000","last_active":"2024-04-26T21:28:57.095218+0000","last_peered":"2024-04-26T21:28:57.095218+0000","last_clean":"2024-04-26T21:28:57.095218+0000","last_became_active":"2024-04-26T21:27:02.302088+0000","last_became_peered":"2024-04-26T21:27:02.302088+0000","last_unstale":"2024-04-26T21:28:57.095218+0000","last_undegraded":"2024-04-26T21:28:57.095218+0000","last_fullsized":"2024-04-26T21:28:57.095218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:43.007722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134536+0000","last_change":"2024-04-26T21:27:02.861455+0000","last_active":"2024-04-26T21:28:57.134536+0000","last_peered":"2024-04-26T21:28:57.134536+0000","last_clean":"2024-04-26T21:28:57.134536+0000","last_became_active":"2024-04-26T21:27:02.861107+0000","last_became_peered":"2024-04-26T21:27:02.861107+0000","last_unstale":"2024-04-26T21:28:57.134536+0000","last_undegraded":"2024-04-26T21:28:57.134536+0000","last_fullsized":"2024-04-26T21:28:57.134536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:10:18.382881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088435+0000","last_change":"2024-04-26T21:27:07.377797+0000","last_active":"2024-04-26T21:28:57.088435+0000","last_peered":"2024-04-26T21:28:57.088435+0000","last_clean":"2024-04-26T21:28:57.088435+0000","last_became_active":"2024-04-26T21:27:07.377314+0000","last_became_peered":"2024-04-26T21:27:07.377314+0000","last_unstale":"2024-04-26T21:28:57.088435+0000","last_undegraded":"2024-04-26T21:28:57.088435+0000","last_fullsized":"2024-04-26T21:28:57.088435+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:02:34.761034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134833+0000","last_change":"2024-04-26T21:27:02.309488+0000","last_active":"2024-04-26T21:28:57.134833+0000","last_peered":"2024-04-26T21:28:57.134833+0000","last_clean":"2024-04-26T21:28:57.134833+0000","last_became_active":"2024-04-26T21:27:02.308592+0000","last_became_peered":"2024-04-26T21:27:02.308592+0000","last_unstale":"2024-04-26T21:28:57.134833+0000","last_undegraded":"2024-04-26T21:28:57.134833+0000","last_fullsized":"2024-04-26T21:28:57.134833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:36.563847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089012+0000","last_change":"2024-04-26T21:27:02.863204+0000","last_active":"2024-04-26T21:28:57.089012+0000","last_peered":"2024-04-26T21:28:57.089012+0000","last_clean":"2024-04-26T21:28:57.089012+0000","last_became_active":"2024-04-26T21:27:02.862466+0000","last_became_peered":"2024-04-26T21:27:02.862466+0000","last_unstale":"2024-04-26T21:28:57.089012+0000","last_undegraded":"2024-04-26T21:28:57.089012+0000","last_fullsized":"2024-04-26T21:28:57.089012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:29.360830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088593+0000","last_change":"2024-04-26T21:27:07.378032+0000","last_active":"2024-04-26T21:28:57.088593+0000","last_peered":"2024-04-26T21:28:57.088593+0000","last_clean":"2024-04-26T21:28:57.088593+0000","last_became_active":"2024-04-26T21:27:07.377255+0000","last_became_peered":"2024-04-26T21:27:07.377255+0000","last_unstale":"2024-04-26T21:28:57.088593+0000","last_undegraded":"2024-04-26T21:28:57.088593+0000","last_fullsized":"2024-04-26T21:28:57.088593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:18.989977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084022+0000","last_change":"2024-04-26T21:27:56.850816+0000","last_active":"2024-04-26T21:28:57.084022+0000","last_peered":"2024-04-26T21:28:57.084022+0000","last_clean":"2024-04-26T21:28:57.084022+0000","last_became_active":"2024-04-26T21:27:56.843299+0000","last_became_peered":"2024-04-26T21:27:56.843299+0000","last_unstale":"2024-04-26T21:28:57.084022+0000","last_undegraded":"2024-04-26T21:28:57.084022+0000","last_fullsized":"2024-04-26T21:28:57.084022+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:21:38.864693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141842+0000","last_change":"2024-04-26T21:27:02.876447+0000","last_active":"2024-04-26T21:28:57.141842+0000","last_peered":"2024-04-26T21:28:57.141842+0000","last_clean":"2024-04-26T21:28:57.141842+0000","last_became_active":"2024-04-26T21:27:02.876017+0000","last_became_peered":"2024-04-26T21:27:02.876017+0000","last_unstale":"2024-04-26T21:28:57.141842+0000","last_undegraded":"2024-04-26T21:28:57.141842+0000","last_fullsized":"2024-04-26T21:28:57.141842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:57:05.453787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088379+0000","last_change":"2024-04-26T21:27:07.373700+0000","last_active":"2024-04-26T21:28:57.088379+0000","last_peered":"2024-04-26T21:28:57.088379+0000","last_clean":"2024-04-26T21:28:57.088379+0000","last_became_active":"2024-04-26T21:27:07.373255+0000","last_became_peered":"2024-04-26T21:27:07.373255+0000","last_unstale":"2024-04-26T21:28:57.088379+0000","last_undegraded":"2024-04-26T21:28:57.088379+0000","last_fullsized":"2024-04-26T21:28:57.088379+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:48.517197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083901+0000","last_change":"2024-04-26T21:27:02.300817+0000","last_active":"2024-04-26T21:28:57.083901+0000","last_peered":"2024-04-26T21:28:57.083901+0000","last_clean":"2024-04-26T21:28:57.083901+0000","last_became_active":"2024-04-26T21:27:02.300493+0000","last_became_peered":"2024-04-26T21:27:02.300493+0000","last_unstale":"2024-04-26T21:28:57.083901+0000","last_undegraded":"2024-04-26T21:28:57.083901+0000","last_fullsized":"2024-04-26T21:28:57.083901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:18.783104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089662+0000","last_change":"2024-04-26T21:27:02.859982+0000","last_active":"2024-04-26T21:28:57.089662+0000","last_peered":"2024-04-26T21:28:57.089662+0000","last_clean":"2024-04-26T21:28:57.089662+0000","last_became_active":"2024-04-26T21:27:02.857204+0000","last_became_peered":"2024-04-26T21:27:02.857204+0000","last_unstale":"2024-04-26T21:28:57.089662+0000","last_undegraded":"2024-04-26T21:28:57.089662+0000","last_fullsized":"2024-04-26T21:28:57.089662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:22.423956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140395+0000","last_change":"2024-04-26T21:27:07.366039+0000","last_active":"2024-04-26T21:28:57.140395+0000","last_peered":"2024-04-26T21:28:57.140395+0000","last_clean":"2024-04-26T21:28:57.140395+0000","last_became_active":"2024-04-26T21:27:07.365560+0000","last_became_peered":"2024-04-26T21:27:07.365560+0000","last_unstale":"2024-04-26T21:28:57.140395+0000","last_undegraded":"2024-04-26T21:28:57.140395+0000","last_fullsized":"2024-04-26T21:28:57.140395+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:47.833476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":137,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:05.842256+0000","last_change":"2024-04-26T21:27:02.306364+0000","last_active":"2024-04-26T21:29:05.842256+0000","last_peered":"2024-04-26T21:29:05.842256+0000","last_clean":"2024-04-26T21:29:05.842256+0000","last_became_active":"2024-04-26T21:27:02.306041+0000","last_became_peered":"2024-04-26T21:27:02.306041+0000","last_unstale":"2024-04-26T21:29:05.842256+0000","last_undegraded":"2024-04-26T21:29:05.842256+0000","last_fullsized":"2024-04-26T21:29:05.842256+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:44.501916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083650+0000","last_change":"2024-04-26T21:27:02.859346+0000","last_active":"2024-04-26T21:28:57.083650+0000","last_peered":"2024-04-26T21:28:57.083650+0000","last_clean":"2024-04-26T21:28:57.083650+0000","last_became_active":"2024-04-26T21:27:02.859025+0000","last_became_peered":"2024-04-26T21:27:02.859025+0000","last_unstale":"2024-04-26T21:28:57.083650+0000","last_undegraded":"2024-04-26T21:28:57.083650+0000","last_fullsized":"2024-04-26T21:28:57.083650+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:06.258512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140349+0000","last_change":"2024-04-26T21:27:07.382349+0000","last_active":"2024-04-26T21:28:57.140349+0000","last_peered":"2024-04-26T21:28:57.140349+0000","last_clean":"2024-04-26T21:28:57.140349+0000","last_became_active":"2024-04-26T21:27:07.381930+0000","last_became_peered":"2024-04-26T21:27:07.381930+0000","last_unstale":"2024-04-26T21:28:57.140349+0000","last_undegraded":"2024-04-26T21:28:57.140349+0000","last_fullsized":"2024-04-26T21:28:57.140349+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:15:25.970573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.087951+0000","last_change":"2024-04-26T21:27:56.862413+0000","last_active":"2024-04-26T21:28:57.087951+0000","last_peered":"2024-04-26T21:28:57.087951+0000","last_clean":"2024-04-26T21:28:57.087951+0000","last_became_active":"2024-04-26T21:27:56.859184+0000","last_became_peered":"2024-04-26T21:27:56.859184+0000","last_unstale":"2024-04-26T21:28:57.087951+0000","last_undegraded":"2024-04-26T21:28:57.087951+0000","last_fullsized":"2024-04-26T21:28:57.087951+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:54:17.701601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089353+0000","last_change":"2024-04-26T21:27:02.865707+0000","last_active":"2024-04-26T21:28:57.089353+0000","last_peered":"2024-04-26T21:28:57.089353+0000","last_clean":"2024-04-26T21:28:57.089353+0000","last_became_active":"2024-04-26T21:27:02.865436+0000","last_became_peered":"2024-04-26T21:27:02.865436+0000","last_unstale":"2024-04-26T21:28:57.089353+0000","last_undegraded":"2024-04-26T21:28:57.089353+0000","last_fullsized":"2024-04-26T21:28:57.089353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:53.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120620+0000","last_change":"2024-04-26T21:27:07.379092+0000","last_active":"2024-04-26T21:28:57.120620+0000","last_peered":"2024-04-26T21:28:57.120620+0000","last_clean":"2024-04-26T21:28:57.120620+0000","last_became_active":"2024-04-26T21:27:07.378716+0000","last_became_peered":"2024-04-26T21:27:07.378716+0000","last_unstale":"2024-04-26T21:28:57.120620+0000","last_undegraded":"2024-04-26T21:28:57.120620+0000","last_fullsized":"2024-04-26T21:28:57.120620+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:18.906224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147874+0000","last_change":"2024-04-26T21:27:02.307510+0000","last_active":"2024-04-26T21:28:57.147874+0000","last_peered":"2024-04-26T21:28:57.147874+0000","last_clean":"2024-04-26T21:28:57.147874+0000","last_became_active":"2024-04-26T21:27:02.306932+0000","last_became_peered":"2024-04-26T21:27:02.306932+0000","last_unstale":"2024-04-26T21:28:57.147874+0000","last_undegraded":"2024-04-26T21:28:57.147874+0000","last_fullsized":"2024-04-26T21:28:57.147874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:21:34.217767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.132628+0000","last_change":"2024-04-26T21:27:02.874246+0000","last_active":"2024-04-26T21:28:57.132628+0000","last_peered":"2024-04-26T21:28:57.132628+0000","last_clean":"2024-04-26T21:28:57.132628+0000","last_became_active":"2024-04-26T21:27:02.873880+0000","last_became_peered":"2024-04-26T21:27:02.873880+0000","last_unstale":"2024-04-26T21:28:57.132628+0000","last_undegraded":"2024-04-26T21:28:57.132628+0000","last_fullsized":"2024-04-26T21:28:57.132628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:21.579832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088920+0000","last_change":"2024-04-26T21:27:07.387201+0000","last_active":"2024-04-26T21:28:57.088920+0000","last_peered":"2024-04-26T21:28:57.088920+0000","last_clean":"2024-04-26T21:28:57.088920+0000","last_became_active":"2024-04-26T21:27:07.386327+0000","last_became_peered":"2024-04-26T21:27:07.386327+0000","last_unstale":"2024-04-26T21:28:57.088920+0000","last_undegraded":"2024-04-26T21:28:57.088920+0000","last_fullsized":"2024-04-26T21:28:57.088920+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:40.801572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088663+0000","last_change":"2024-04-26T21:27:02.304132+0000","last_active":"2024-04-26T21:28:57.088663+0000","last_peered":"2024-04-26T21:28:57.088663+0000","last_clean":"2024-04-26T21:28:57.088663+0000","last_became_active":"2024-04-26T21:27:02.303562+0000","last_became_peered":"2024-04-26T21:27:02.303562+0000","last_unstale":"2024-04-26T21:28:57.088663+0000","last_undegraded":"2024-04-26T21:28:57.088663+0000","last_fullsized":"2024-04-26T21:28:57.088663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:54.417590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.092781+0000","last_change":"2024-04-26T21:27:02.866539+0000","last_active":"2024-04-26T21:28:57.092781+0000","last_peered":"2024-04-26T21:28:57.092781+0000","last_clean":"2024-04-26T21:28:57.092781+0000","last_became_active":"2024-04-26T21:27:02.866193+0000","last_became_peered":"2024-04-26T21:27:02.866193+0000","last_unstale":"2024-04-26T21:28:57.092781+0000","last_undegraded":"2024-04-26T21:28:57.092781+0000","last_fullsized":"2024-04-26T21:28:57.092781+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:44.234261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134645+0000","last_change":"2024-04-26T21:27:07.368394+0000","last_active":"2024-04-26T21:28:57.134645+0000","last_peered":"2024-04-26T21:28:57.134645+0000","last_clean":"2024-04-26T21:28:57.134645+0000","last_became_active":"2024-04-26T21:27:07.367998+0000","last_became_peered":"2024-04-26T21:27:07.367998+0000","last_unstale":"2024-04-26T21:28:57.134645+0000","last_undegraded":"2024-04-26T21:28:57.134645+0000","last_fullsized":"2024-04-26T21:28:57.134645+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:31:28.459759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":134,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:06.843286+0000","last_change":"2024-04-26T21:27:02.304096+0000","last_active":"2024-04-26T21:29:06.843286+0000","last_peered":"2024-04-26T21:29:06.843286+0000","last_clean":"2024-04-26T21:29:06.843286+0000","last_became_active":"2024-04-26T21:27:02.303736+0000","last_became_peered":"2024-04-26T21:27:02.303736+0000","last_unstale":"2024-04-26T21:29:06.843286+0000","last_undegraded":"2024-04-26T21:29:06.843286+0000","last_fullsized":"2024-04-26T21:29:06.843286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:25.400913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142465+0000","last_change":"2024-04-26T21:27:02.863033+0000","last_active":"2024-04-26T21:28:57.142465+0000","last_peered":"2024-04-26T21:28:57.142465+0000","last_clean":"2024-04-26T21:28:57.142465+0000","last_became_active":"2024-04-26T21:27:02.862572+0000","last_became_peered":"2024-04-26T21:27:02.862572+0000","last_unstale":"2024-04-26T21:28:57.142465+0000","last_undegraded":"2024-04-26T21:28:57.142465+0000","last_fullsized":"2024-04-26T21:28:57.142465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:02:55.574601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142841+0000","last_change":"2024-04-26T21:28:57.142094+0000","last_active":"2024-04-26T21:28:57.142841+0000","last_peered":"2024-04-26T21:28:57.142841+0000","last_clean":"2024-04-26T21:28:57.142841+0000","last_became_active":"2024-04-26T21:28:57.141572+0000","last_became_peered":"2024-04-26T21:28:57.141572+0000","last_unstale":"2024-04-26T21:28:57.142841+0000","last_undegraded":"2024-04-26T21:28:57.142841+0000","last_fullsized":"2024-04-26T21:28:57.142841+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:35.087842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141281+0000","last_change":"2024-04-26T21:27:56.846362+0000","last_active":"2024-04-26T21:28:57.141281+0000","last_peered":"2024-04-26T21:28:57.141281+0000","last_clean":"2024-04-26T21:28:57.141281+0000","last_became_active":"2024-04-26T21:27:56.840187+0000","last_became_peered":"2024-04-26T21:27:56.840187+0000","last_unstale":"2024-04-26T21:28:57.141281+0000","last_undegraded":"2024-04-26T21:28:57.141281+0000","last_fullsized":"2024-04-26T21:28:57.141281+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:12.064415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088889+0000","last_change":"2024-04-26T21:27:02.858739+0000","last_active":"2024-04-26T21:28:57.088889+0000","last_peered":"2024-04-26T21:28:57.088889+0000","last_clean":"2024-04-26T21:28:57.088889+0000","last_became_active":"2024-04-26T21:27:02.857458+0000","last_became_peered":"2024-04-26T21:27:02.857458+0000","last_unstale":"2024-04-26T21:28:57.088889+0000","last_undegraded":"2024-04-26T21:28:57.088889+0000","last_fullsized":"2024-04-26T21:28:57.088889+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:00.354052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089405+0000","last_change":"2024-04-26T21:27:07.377690+0000","last_active":"2024-04-26T21:28:57.089405+0000","last_peered":"2024-04-26T21:28:57.089405+0000","last_clean":"2024-04-26T21:28:57.089405+0000","last_became_active":"2024-04-26T21:27:07.377192+0000","last_became_peered":"2024-04-26T21:27:07.377192+0000","last_unstale":"2024-04-26T21:28:57.089405+0000","last_undegraded":"2024-04-26T21:28:57.089405+0000","last_fullsized":"2024-04-26T21:28:57.089405+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:48.069083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089650+0000","last_change":"2024-04-26T21:27:02.316110+0000","last_active":"2024-04-26T21:28:57.089650+0000","last_peered":"2024-04-26T21:28:57.089650+0000","last_clean":"2024-04-26T21:28:57.089650+0000","last_became_active":"2024-04-26T21:27:02.315767+0000","last_became_peered":"2024-04-26T21:27:02.315767+0000","last_unstale":"2024-04-26T21:28:57.089650+0000","last_undegraded":"2024-04-26T21:28:57.089650+0000","last_fullsized":"2024-04-26T21:28:57.089650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:00.990872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142071+0000","last_change":"2024-04-26T21:27:02.855736+0000","last_active":"2024-04-26T21:28:57.142071+0000","last_peered":"2024-04-26T21:28:57.142071+0000","last_clean":"2024-04-26T21:28:57.142071+0000","last_became_active":"2024-04-26T21:27:02.855378+0000","last_became_peered":"2024-04-26T21:27:02.855378+0000","last_unstale":"2024-04-26T21:28:57.142071+0000","last_undegraded":"2024-04-26T21:28:57.142071+0000","last_fullsized":"2024-04-26T21:28:57.142071+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:55.272869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.091733+0000","last_change":"2024-04-26T21:27:07.376804+0000","last_active":"2024-04-26T21:28:57.091733+0000","last_peered":"2024-04-26T21:28:57.091733+0000","last_clean":"2024-04-26T21:28:57.091733+0000","last_became_active":"2024-04-26T21:27:07.374378+0000","last_became_peered":"2024-04-26T21:27:07.374378+0000","last_unstale":"2024-04-26T21:28:57.091733+0000","last_undegraded":"2024-04-26T21:28:57.091733+0000","last_fullsized":"2024-04-26T21:28:57.091733+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:03.930253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'33","reported_seq":272,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:07.844914+0000","last_change":"2024-04-26T21:27:02.297467+0000","last_active":"2024-04-26T21:29:07.844914+0000","last_peered":"2024-04-26T21:29:07.844914+0000","last_clean":"2024-04-26T21:29:07.844914+0000","last_became_active":"2024-04-26T21:27:02.297077+0000","last_became_peered":"2024-04-26T21:27:02.297077+0000","last_unstale":"2024-04-26T21:29:07.844914+0000","last_undegraded":"2024-04-26T21:29:07.844914+0000","last_fullsized":"2024-04-26T21:29:07.844914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:47.047565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55676,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":208,"num_read_kb":158,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083876+0000","last_change":"2024-04-26T21:27:02.858161+0000","last_active":"2024-04-26T21:28:57.083876+0000","last_peered":"2024-04-26T21:28:57.083876+0000","last_clean":"2024-04-26T21:28:57.083876+0000","last_became_active":"2024-04-26T21:27:02.857747+0000","last_became_peered":"2024-04-26T21:27:02.857747+0000","last_unstale":"2024-04-26T21:28:57.083876+0000","last_undegraded":"2024-04-26T21:28:57.083876+0000","last_fullsized":"2024-04-26T21:28:57.083876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:10:50.843195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.090006+0000","last_change":"2024-04-26T21:27:07.376253+0000","last_active":"2024-04-26T21:28:57.090006+0000","last_peered":"2024-04-26T21:28:57.090006+0000","last_clean":"2024-04-26T21:28:57.090006+0000","last_became_active":"2024-04-26T21:27:07.375822+0000","last_became_peered":"2024-04-26T21:27:07.375822+0000","last_unstale":"2024-04-26T21:28:57.090006+0000","last_undegraded":"2024-04-26T21:28:57.090006+0000","last_fullsized":"2024-04-26T21:28:57.090006+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:22:01.784163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140286+0000","last_change":"2024-04-26T21:27:02.300743+0000","last_active":"2024-04-26T21:28:57.140286+0000","last_peered":"2024-04-26T21:28:57.140286+0000","last_clean":"2024-04-26T21:28:57.140286+0000","last_became_active":"2024-04-26T21:27:02.300424+0000","last_became_peered":"2024-04-26T21:27:02.300424+0000","last_unstale":"2024-04-26T21:28:57.140286+0000","last_undegraded":"2024-04-26T21:28:57.140286+0000","last_fullsized":"2024-04-26T21:28:57.140286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:10.408234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148845+0000","last_change":"2024-04-26T21:27:02.872456+0000","last_active":"2024-04-26T21:28:57.148845+0000","last_peered":"2024-04-26T21:28:57.148845+0000","last_clean":"2024-04-26T21:28:57.148845+0000","last_became_active":"2024-04-26T21:27:02.872064+0000","last_became_peered":"2024-04-26T21:27:02.872064+0000","last_unstale":"2024-04-26T21:28:57.148845+0000","last_undegraded":"2024-04-26T21:28:57.148845+0000","last_fullsized":"2024-04-26T21:28:57.148845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:33.511164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120361+0000","last_change":"2024-04-26T21:27:07.386148+0000","last_active":"2024-04-26T21:28:57.120361+0000","last_peered":"2024-04-26T21:28:57.120361+0000","last_clean":"2024-04-26T21:28:57.120361+0000","last_became_active":"2024-04-26T21:27:07.385787+0000","last_became_peered":"2024-04-26T21:27:07.385787+0000","last_unstale":"2024-04-26T21:28:57.120361+0000","last_undegraded":"2024-04-26T21:28:57.120361+0000","last_fullsized":"2024-04-26T21:28:57.120361+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:57.776660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089128+0000","last_change":"2024-04-26T21:27:02.313725+0000","last_active":"2024-04-26T21:28:57.089128+0000","last_peered":"2024-04-26T21:28:57.089128+0000","last_clean":"2024-04-26T21:28:57.089128+0000","last_became_active":"2024-04-26T21:27:02.313350+0000","last_became_peered":"2024-04-26T21:27:02.313350+0000","last_unstale":"2024-04-26T21:28:57.089128+0000","last_undegraded":"2024-04-26T21:28:57.089128+0000","last_fullsized":"2024-04-26T21:28:57.089128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:21:30.608067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141462+0000","last_change":"2024-04-26T21:27:56.856753+0000","last_active":"2024-04-26T21:28:57.141462+0000","last_peered":"2024-04-26T21:28:57.141462+0000","last_clean":"2024-04-26T21:28:57.141462+0000","last_became_active":"2024-04-26T21:27:56.856297+0000","last_became_peered":"2024-04-26T21:27:56.856297+0000","last_unstale":"2024-04-26T21:28:57.141462+0000","last_undegraded":"2024-04-26T21:28:57.141462+0000","last_fullsized":"2024-04-26T21:28:57.141462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:54.650077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.095669+0000","last_change":"2024-04-26T21:27:07.378091+0000","last_active":"2024-04-26T21:28:57.095669+0000","last_peered":"2024-04-26T21:28:57.095669+0000","last_clean":"2024-04-26T21:28:57.095669+0000","last_became_active":"2024-04-26T21:27:07.377671+0000","last_became_peered":"2024-04-26T21:27:07.377671+0000","last_unstale":"2024-04-26T21:28:57.095669+0000","last_undegraded":"2024-04-26T21:28:57.095669+0000","last_fullsized":"2024-04-26T21:28:57.095669+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:41:36.551655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'3","reported_seq":135,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:05.875691+0000","last_change":"2024-04-26T21:27:02.294738+0000","last_active":"2024-04-26T21:29:05.875691+0000","last_peered":"2024-04-26T21:29:05.875691+0000","last_clean":"2024-04-26T21:29:05.875691+0000","last_became_active":"2024-04-26T21:27:02.294338+0000","last_became_peered":"2024-04-26T21:27:02.294338+0000","last_unstale":"2024-04-26T21:29:05.875691+0000","last_undegraded":"2024-04-26T21:29:05.875691+0000","last_fullsized":"2024-04-26T21:29:05.875691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:25.309107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141783+0000","last_change":"2024-04-26T21:27:02.861332+0000","last_active":"2024-04-26T21:28:57.141783+0000","last_peered":"2024-04-26T21:28:57.141783+0000","last_clean":"2024-04-26T21:28:57.141783+0000","last_became_active":"2024-04-26T21:27:02.857865+0000","last_became_peered":"2024-04-26T21:27:02.857865+0000","last_unstale":"2024-04-26T21:28:57.141783+0000","last_undegraded":"2024-04-26T21:28:57.141783+0000","last_fullsized":"2024-04-26T21:28:57.141783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:48.493086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150359+0000","last_change":"2024-04-26T21:27:07.383219+0000","last_active":"2024-04-26T21:28:57.150359+0000","last_peered":"2024-04-26T21:28:57.150359+0000","last_clean":"2024-04-26T21:28:57.150359+0000","last_became_active":"2024-04-26T21:27:07.382820+0000","last_became_peered":"2024-04-26T21:27:07.382820+0000","last_unstale":"2024-04-26T21:28:57.150359+0000","last_undegraded":"2024-04-26T21:28:57.150359+0000","last_fullsized":"2024-04-26T21:28:57.150359+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:07.752124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083444+0000","last_change":"2024-04-26T21:27:02.307746+0000","last_active":"2024-04-26T21:28:57.083444+0000","last_peered":"2024-04-26T21:28:57.083444+0000","last_clean":"2024-04-26T21:28:57.083444+0000","last_became_active":"2024-04-26T21:27:02.307406+0000","last_became_peered":"2024-04-26T21:27:02.307406+0000","last_unstale":"2024-04-26T21:28:57.083444+0000","last_undegraded":"2024-04-26T21:28:57.083444+0000","last_fullsized":"2024-04-26T21:28:57.083444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:13.799727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088903+0000","last_change":"2024-04-26T21:27:02.861194+0000","last_active":"2024-04-26T21:28:57.088903+0000","last_peered":"2024-04-26T21:28:57.088903+0000","last_clean":"2024-04-26T21:28:57.088903+0000","last_became_active":"2024-04-26T21:27:02.860871+0000","last_became_peered":"2024-04-26T21:27:02.860871+0000","last_unstale":"2024-04-26T21:28:57.088903+0000","last_undegraded":"2024-04-26T21:28:57.088903+0000","last_fullsized":"2024-04-26T21:28:57.088903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:56.829027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.144000+0000","last_change":"2024-04-26T21:28:57.143451+0000","last_active":"2024-04-26T21:28:57.144000+0000","last_peered":"2024-04-26T21:28:57.144000+0000","last_clean":"2024-04-26T21:28:57.144000+0000","last_became_active":"2024-04-26T21:28:57.142896+0000","last_became_peered":"2024-04-26T21:28:57.142896+0000","last_unstale":"2024-04-26T21:28:57.144000+0000","last_undegraded":"2024-04-26T21:28:57.144000+0000","last_fullsized":"2024-04-26T21:28:57.144000+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:40.800757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084171+0000","last_change":"2024-04-26T21:27:02.301711+0000","last_active":"2024-04-26T21:28:57.084171+0000","last_peered":"2024-04-26T21:28:57.084171+0000","last_clean":"2024-04-26T21:28:57.084171+0000","last_became_active":"2024-04-26T21:27:02.301429+0000","last_became_peered":"2024-04-26T21:27:02.301429+0000","last_unstale":"2024-04-26T21:28:57.084171+0000","last_undegraded":"2024-04-26T21:28:57.084171+0000","last_fullsized":"2024-04-26T21:28:57.084171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:50.876824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120572+0000","last_change":"2024-04-26T21:27:02.854372+0000","last_active":"2024-04-26T21:28:57.120572+0000","last_peered":"2024-04-26T21:28:57.120572+0000","last_clean":"2024-04-26T21:28:57.120572+0000","last_became_active":"2024-04-26T21:27:02.853826+0000","last_became_peered":"2024-04-26T21:27:02.853826+0000","last_unstale":"2024-04-26T21:28:57.120572+0000","last_undegraded":"2024-04-26T21:28:57.120572+0000","last_fullsized":"2024-04-26T21:28:57.120572+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:08.253384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083532+0000","last_change":"2024-04-26T21:27:07.506684+0000","last_active":"2024-04-26T21:28:57.083532+0000","last_peered":"2024-04-26T21:28:57.083532+0000","last_clean":"2024-04-26T21:28:57.083532+0000","last_became_active":"2024-04-26T21:27:07.506203+0000","last_became_peered":"2024-04-26T21:27:07.506203+0000","last_unstale":"2024-04-26T21:28:57.083532+0000","last_undegraded":"2024-04-26T21:28:57.083532+0000","last_fullsized":"2024-04-26T21:28:57.083532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:42:48.580106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.094179+0000","last_change":"2024-04-26T21:27:02.306265+0000","last_active":"2024-04-26T21:28:57.094179+0000","last_peered":"2024-04-26T21:28:57.094179+0000","last_clean":"2024-04-26T21:28:57.094179+0000","last_became_active":"2024-04-26T21:27:02.305941+0000","last_became_peered":"2024-04-26T21:27:02.305941+0000","last_unstale":"2024-04-26T21:28:57.094179+0000","last_undegraded":"2024-04-26T21:28:57.094179+0000","last_fullsized":"2024-04-26T21:28:57.094179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:29.076833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088626+0000","last_change":"2024-04-26T21:27:02.871916+0000","last_active":"2024-04-26T21:28:57.088626+0000","last_peered":"2024-04-26T21:28:57.088626+0000","last_clean":"2024-04-26T21:28:57.088626+0000","last_became_active":"2024-04-26T21:27:02.871517+0000","last_became_peered":"2024-04-26T21:27:02.871517+0000","last_unstale":"2024-04-26T21:28:57.088626+0000","last_undegraded":"2024-04-26T21:28:57.088626+0000","last_fullsized":"2024-04-26T21:28:57.088626+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:41.361757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148111+0000","last_change":"2024-04-26T21:27:07.383422+0000","last_active":"2024-04-26T21:28:57.148111+0000","last_peered":"2024-04-26T21:28:57.148111+0000","last_clean":"2024-04-26T21:28:57.148111+0000","last_became_active":"2024-04-26T21:27:07.382727+0000","last_became_peered":"2024-04-26T21:27:07.382727+0000","last_unstale":"2024-04-26T21:28:57.148111+0000","last_undegraded":"2024-04-26T21:28:57.148111+0000","last_fullsized":"2024-04-26T21:28:57.148111+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:22.002658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141798+0000","last_change":"2024-04-26T21:27:02.308081+0000","last_active":"2024-04-26T21:28:57.141798+0000","last_peered":"2024-04-26T21:28:57.141798+0000","last_clean":"2024-04-26T21:28:57.141798+0000","last_became_active":"2024-04-26T21:27:02.307715+0000","last_became_peered":"2024-04-26T21:27:02.307715+0000","last_unstale":"2024-04-26T21:28:57.141798+0000","last_undegraded":"2024-04-26T21:28:57.141798+0000","last_fullsized":"2024-04-26T21:28:57.141798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:28.967817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088500+0000","last_change":"2024-04-26T21:27:02.859527+0000","last_active":"2024-04-26T21:28:57.088500+0000","last_peered":"2024-04-26T21:28:57.088500+0000","last_clean":"2024-04-26T21:28:57.088500+0000","last_became_active":"2024-04-26T21:27:02.859173+0000","last_became_peered":"2024-04-26T21:27:02.859173+0000","last_unstale":"2024-04-26T21:28:57.088500+0000","last_undegraded":"2024-04-26T21:28:57.088500+0000","last_fullsized":"2024-04-26T21:28:57.088500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:24.811630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147035+0000","last_change":"2024-04-26T21:27:07.509778+0000","last_active":"2024-04-26T21:28:57.147035+0000","last_peered":"2024-04-26T21:28:57.147035+0000","last_clean":"2024-04-26T21:28:57.147035+0000","last_became_active":"2024-04-26T21:27:07.508344+0000","last_became_peered":"2024-04-26T21:27:07.508344+0000","last_unstale":"2024-04-26T21:28:57.147035+0000","last_undegraded":"2024-04-26T21:28:57.147035+0000","last_fullsized":"2024-04-26T21:28:57.147035+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:38:05.926717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.143833+0000","last_change":"2024-04-26T21:28:57.143833+0000","last_active":"2024-04-26T21:28:57.143833+0000","last_peered":"2024-04-26T21:28:57.143833+0000","last_clean":"2024-04-26T21:28:57.143833+0000","last_became_active":"2024-04-26T21:28:57.142433+0000","last_became_peered":"2024-04-26T21:28:57.142433+0000","last_unstale":"2024-04-26T21:28:57.143833+0000","last_undegraded":"2024-04-26T21:28:57.143833+0000","last_fullsized":"2024-04-26T21:28:57.143833+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:41:45.191169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":136,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:07.878479+0000","last_change":"2024-04-26T21:27:02.317413+0000","last_active":"2024-04-26T21:29:07.878479+0000","last_peered":"2024-04-26T21:29:07.878479+0000","last_clean":"2024-04-26T21:29:07.878479+0000","last_became_active":"2024-04-26T21:27:02.316510+0000","last_became_peered":"2024-04-26T21:27:02.316510+0000","last_unstale":"2024-04-26T21:29:07.878479+0000","last_undegraded":"2024-04-26T21:29:07.878479+0000","last_fullsized":"2024-04-26T21:29:07.878479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:49.156366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141068+0000","last_change":"2024-04-26T21:27:07.373434+0000","last_active":"2024-04-26T21:28:57.141068+0000","last_peered":"2024-04-26T21:28:57.141068+0000","last_clean":"2024-04-26T21:28:57.141068+0000","last_became_active":"2024-04-26T21:27:07.372973+0000","last_became_peered":"2024-04-26T21:27:07.372973+0000","last_unstale":"2024-04-26T21:28:57.141068+0000","last_undegraded":"2024-04-26T21:28:57.141068+0000","last_fullsized":"2024-04-26T21:28:57.141068+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:04.022651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.133271+0000","last_change":"2024-04-26T21:27:02.862734+0000","last_active":"2024-04-26T21:28:57.133271+0000","last_peered":"2024-04-26T21:28:57.133271+0000","last_clean":"2024-04-26T21:28:57.133271+0000","last_became_active":"2024-04-26T21:27:02.862334+0000","last_became_peered":"2024-04-26T21:27:02.862334+0000","last_unstale":"2024-04-26T21:28:57.133271+0000","last_undegraded":"2024-04-26T21:28:57.133271+0000","last_fullsized":"2024-04-26T21:28:57.133271+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:09.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089528+0000","last_change":"2024-04-26T21:27:02.302471+0000","last_active":"2024-04-26T21:28:57.089528+0000","last_peered":"2024-04-26T21:28:57.089528+0000","last_clean":"2024-04-26T21:28:57.089528+0000","last_became_active":"2024-04-26T21:27:02.302055+0000","last_became_peered":"2024-04-26T21:27:02.302055+0000","last_unstale":"2024-04-26T21:28:57.089528+0000","last_undegraded":"2024-04-26T21:28:57.089528+0000","last_fullsized":"2024-04-26T21:28:57.089528+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:29:14.649670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":367,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141165+0000","last_change":"2024-04-26T21:25:41.647633+0000","last_active":"2024-04-26T21:28:57.141165+0000","last_peered":"2024-04-26T21:28:57.141165+0000","last_clean":"2024-04-26T21:28:57.141165+0000","last_became_active":"2024-04-26T21:25:41.647091+0000","last_became_peered":"2024-04-26T21:25:41.647091+0000","last_unstale":"2024-04-26T21:28:57.141165+0000","last_undegraded":"2024-04-26T21:28:57.141165+0000","last_fullsized":"2024-04-26T21:28:57.141165+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_clean_scrub_stamp":"2024-04-26T21:21:43.547929+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:37.666065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.152820+0000","last_change":"2024-04-26T21:28:57.152820+0000","last_active":"2024-04-26T21:28:57.152820+0000","last_peered":"2024-04-26T21:28:57.152820+0000","last_clean":"2024-04-26T21:28:57.152820+0000","last_became_active":"2024-04-26T21:28:57.152453+0000","last_became_peered":"2024-04-26T21:28:57.152453+0000","last_unstale":"2024-04-26T21:28:57.152820+0000","last_undegraded":"2024-04-26T21:28:57.152820+0000","last_fullsized":"2024-04-26T21:28:57.152820+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:16:47.634651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147632+0000","last_change":"2024-04-26T21:27:02.874116+0000","last_active":"2024-04-26T21:28:57.147632+0000","last_peered":"2024-04-26T21:28:57.147632+0000","last_clean":"2024-04-26T21:28:57.147632+0000","last_became_active":"2024-04-26T21:27:02.871334+0000","last_became_peered":"2024-04-26T21:27:02.871334+0000","last_unstale":"2024-04-26T21:28:57.147632+0000","last_undegraded":"2024-04-26T21:28:57.147632+0000","last_fullsized":"2024-04-26T21:28:57.147632+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:28.835061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120084+0000","last_change":"2024-04-26T21:27:56.847702+0000","last_active":"2024-04-26T21:28:57.120084+0000","last_peered":"2024-04-26T21:28:57.120084+0000","last_clean":"2024-04-26T21:28:57.120084+0000","last_became_active":"2024-04-26T21:27:56.844034+0000","last_became_peered":"2024-04-26T21:27:56.844034+0000","last_unstale":"2024-04-26T21:28:57.120084+0000","last_undegraded":"2024-04-26T21:28:57.120084+0000","last_fullsized":"2024-04-26T21:28:57.120084+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:54.631297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120208+0000","last_change":"2024-04-26T21:27:07.373894+0000","last_active":"2024-04-26T21:28:57.120208+0000","last_peered":"2024-04-26T21:28:57.120208+0000","last_clean":"2024-04-26T21:28:57.120208+0000","last_became_active":"2024-04-26T21:27:07.373338+0000","last_became_peered":"2024-04-26T21:27:07.373338+0000","last_unstale":"2024-04-26T21:28:57.120208+0000","last_undegraded":"2024-04-26T21:28:57.120208+0000","last_fullsized":"2024-04-26T21:28:57.120208+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:49.636493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120527+0000","last_change":"2024-04-26T21:27:56.858975+0000","last_active":"2024-04-26T21:28:57.120527+0000","last_peered":"2024-04-26T21:28:57.120527+0000","last_clean":"2024-04-26T21:28:57.120527+0000","last_became_active":"2024-04-26T21:27:56.850022+0000","last_became_peered":"2024-04-26T21:27:56.850022+0000","last_unstale":"2024-04-26T21:28:57.120527+0000","last_undegraded":"2024-04-26T21:28:57.120527+0000","last_fullsized":"2024-04-26T21:28:57.120527+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:43:31.860906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148649+0000","last_change":"2024-04-26T21:27:02.869831+0000","last_active":"2024-04-26T21:28:57.148649+0000","last_peered":"2024-04-26T21:28:57.148649+0000","last_clean":"2024-04-26T21:28:57.148649+0000","last_became_active":"2024-04-26T21:27:02.869375+0000","last_became_peered":"2024-04-26T21:27:02.869375+0000","last_unstale":"2024-04-26T21:28:57.148649+0000","last_undegraded":"2024-04-26T21:28:57.148649+0000","last_fullsized":"2024-04-26T21:28:57.148649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:20.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.132837+0000","last_change":"2024-04-26T21:27:07.507562+0000","last_active":"2024-04-26T21:28:57.132837+0000","last_peered":"2024-04-26T21:28:57.132837+0000","last_clean":"2024-04-26T21:28:57.132837+0000","last_became_active":"2024-04-26T21:27:07.507063+0000","last_became_peered":"2024-04-26T21:27:07.507063+0000","last_unstale":"2024-04-26T21:28:57.132837+0000","last_undegraded":"2024-04-26T21:28:57.132837+0000","last_fullsized":"2024-04-26T21:28:57.132837+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:58.192617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'9","reported_seq":40,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.094611+0000","last_change":"2024-04-26T21:27:02.315360+0000","last_active":"2024-04-26T21:28:57.094611+0000","last_peered":"2024-04-26T21:28:57.094611+0000","last_clean":"2024-04-26T21:28:57.094611+0000","last_became_active":"2024-04-26T21:27:02.314817+0000","last_became_peered":"2024-04-26T21:27:02.314817+0000","last_unstale":"2024-04-26T21:28:57.094611+0000","last_undegraded":"2024-04-26T21:28:57.094611+0000","last_fullsized":"2024-04-26T21:28:57.094611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:12.045363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.145131+0000","last_change":"2024-04-26T21:28:57.145131+0000","last_active":"2024-04-26T21:28:57.145131+0000","last_peered":"2024-04-26T21:28:57.145131+0000","last_clean":"2024-04-26T21:28:57.145131+0000","last_became_active":"2024-04-26T21:28:57.144059+0000","last_became_peered":"2024-04-26T21:28:57.144059+0000","last_unstale":"2024-04-26T21:28:57.145131+0000","last_undegraded":"2024-04-26T21:28:57.145131+0000","last_fullsized":"2024-04-26T21:28:57.145131+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:11:12.442030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.139990+0000","last_change":"2024-04-26T21:27:07.362016+0000","last_active":"2024-04-26T21:28:57.139990+0000","last_peered":"2024-04-26T21:28:57.139990+0000","last_clean":"2024-04-26T21:28:57.139990+0000","last_became_active":"2024-04-26T21:27:07.361360+0000","last_became_peered":"2024-04-26T21:27:07.361360+0000","last_unstale":"2024-04-26T21:28:57.139990+0000","last_undegraded":"2024-04-26T21:28:57.139990+0000","last_fullsized":"2024-04-26T21:28:57.139990+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:03.196293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089562+0000","last_change":"2024-04-26T21:27:02.308088+0000","last_active":"2024-04-26T21:28:57.089562+0000","last_peered":"2024-04-26T21:28:57.089562+0000","last_clean":"2024-04-26T21:28:57.089562+0000","last_became_active":"2024-04-26T21:27:02.307727+0000","last_became_peered":"2024-04-26T21:27:02.307727+0000","last_unstale":"2024-04-26T21:28:57.089562+0000","last_undegraded":"2024-04-26T21:28:57.089562+0000","last_fullsized":"2024-04-26T21:28:57.089562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:02.132125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.153650+0000","last_change":"2024-04-26T21:28:57.153650+0000","last_active":"2024-04-26T21:28:57.153650+0000","last_peered":"2024-04-26T21:28:57.153650+0000","last_clean":"2024-04-26T21:28:57.153650+0000","last_became_active":"2024-04-26T21:28:57.152541+0000","last_became_peered":"2024-04-26T21:28:57.152541+0000","last_unstale":"2024-04-26T21:28:57.153650+0000","last_undegraded":"2024-04-26T21:28:57.153650+0000","last_fullsized":"2024-04-26T21:28:57.153650+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:53.236806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089207+0000","last_change":"2024-04-26T21:27:07.506609+0000","last_active":"2024-04-26T21:28:57.089207+0000","last_peered":"2024-04-26T21:28:57.089207+0000","last_clean":"2024-04-26T21:28:57.089207+0000","last_became_active":"2024-04-26T21:27:07.506100+0000","last_became_peered":"2024-04-26T21:27:07.506100+0000","last_unstale":"2024-04-26T21:28:57.089207+0000","last_undegraded":"2024-04-26T21:28:57.089207+0000","last_fullsized":"2024-04-26T21:28:57.089207+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:26.535809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142097+0000","last_change":"2024-04-26T21:27:02.297432+0000","last_active":"2024-04-26T21:28:57.142097+0000","last_peered":"2024-04-26T21:28:57.142097+0000","last_clean":"2024-04-26T21:28:57.142097+0000","last_became_active":"2024-04-26T21:27:02.297081+0000","last_became_peered":"2024-04-26T21:27:02.297081+0000","last_unstale":"2024-04-26T21:28:57.142097+0000","last_undegraded":"2024-04-26T21:28:57.142097+0000","last_fullsized":"2024-04-26T21:28:57.142097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:08.139469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083343+0000","last_change":"2024-04-26T21:27:02.866653+0000","last_active":"2024-04-26T21:28:57.083343+0000","last_peered":"2024-04-26T21:28:57.083343+0000","last_clean":"2024-04-26T21:28:57.083343+0000","last_became_active":"2024-04-26T21:27:02.866275+0000","last_became_peered":"2024-04-26T21:27:02.866275+0000","last_unstale":"2024-04-26T21:28:57.083343+0000","last_undegraded":"2024-04-26T21:28:57.083343+0000","last_fullsized":"2024-04-26T21:28:57.083343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:39.089972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.143381+0000","last_change":"2024-04-26T21:28:57.142863+0000","last_active":"2024-04-26T21:28:57.143381+0000","last_peered":"2024-04-26T21:28:57.143381+0000","last_clean":"2024-04-26T21:28:57.143381+0000","last_became_active":"2024-04-26T21:28:57.141745+0000","last_became_peered":"2024-04-26T21:28:57.141745+0000","last_unstale":"2024-04-26T21:28:57.143381+0000","last_undegraded":"2024-04-26T21:28:57.143381+0000","last_fullsized":"2024-04-26T21:28:57.143381+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:07.261644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134213+0000","last_change":"2024-04-26T21:27:02.307250+0000","last_active":"2024-04-26T21:28:57.134213+0000","last_peered":"2024-04-26T21:28:57.134213+0000","last_clean":"2024-04-26T21:28:57.134213+0000","last_became_active":"2024-04-26T21:27:02.303532+0000","last_became_peered":"2024-04-26T21:27:02.303532+0000","last_unstale":"2024-04-26T21:28:57.134213+0000","last_undegraded":"2024-04-26T21:28:57.134213+0000","last_fullsized":"2024-04-26T21:28:57.134213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:36.265188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120172+0000","last_change":"2024-04-26T21:27:02.858818+0000","last_active":"2024-04-26T21:28:57.120172+0000","last_peered":"2024-04-26T21:28:57.120172+0000","last_clean":"2024-04-26T21:28:57.120172+0000","last_became_active":"2024-04-26T21:27:02.858059+0000","last_became_peered":"2024-04-26T21:27:02.858059+0000","last_unstale":"2024-04-26T21:28:57.120172+0000","last_undegraded":"2024-04-26T21:28:57.120172+0000","last_fullsized":"2024-04-26T21:28:57.120172+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:02.159334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088650+0000","last_change":"2024-04-26T21:27:07.381273+0000","last_active":"2024-04-26T21:28:57.088650+0000","last_peered":"2024-04-26T21:28:57.088650+0000","last_clean":"2024-04-26T21:28:57.088650+0000","last_became_active":"2024-04-26T21:27:07.380891+0000","last_became_peered":"2024-04-26T21:27:07.380891+0000","last_unstale":"2024-04-26T21:28:57.088650+0000","last_undegraded":"2024-04-26T21:28:57.088650+0000","last_fullsized":"2024-04-26T21:28:57.088650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:39.641577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140340+0000","last_change":"2024-04-26T21:27:02.301363+0000","last_active":"2024-04-26T21:28:57.140340+0000","last_peered":"2024-04-26T21:28:57.140340+0000","last_clean":"2024-04-26T21:28:57.140340+0000","last_became_active":"2024-04-26T21:27:02.301059+0000","last_became_peered":"2024-04-26T21:27:02.301059+0000","last_unstale":"2024-04-26T21:28:57.140340+0000","last_undegraded":"2024-04-26T21:28:57.140340+0000","last_fullsized":"2024-04-26T21:28:57.140340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:17.586190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134340+0000","last_change":"2024-04-26T21:27:02.860612+0000","last_active":"2024-04-26T21:28:57.134340+0000","last_peered":"2024-04-26T21:28:57.134340+0000","last_clean":"2024-04-26T21:28:57.134340+0000","last_became_active":"2024-04-26T21:27:02.857285+0000","last_became_peered":"2024-04-26T21:27:02.857285+0000","last_unstale":"2024-04-26T21:28:57.134340+0000","last_undegraded":"2024-04-26T21:28:57.134340+0000","last_fullsized":"2024-04-26T21:28:57.134340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:11.968818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089342+0000","last_change":"2024-04-26T21:27:07.506425+0000","last_active":"2024-04-26T21:28:57.089342+0000","last_peered":"2024-04-26T21:28:57.089342+0000","last_clean":"2024-04-26T21:28:57.089342+0000","last_became_active":"2024-04-26T21:27:07.505968+0000","last_became_peered":"2024-04-26T21:27:07.505968+0000","last_unstale":"2024-04-26T21:28:57.089342+0000","last_undegraded":"2024-04-26T21:28:57.089342+0000","last_fullsized":"2024-04-26T21:28:57.089342+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:59.914560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083503+0000","last_change":"2024-04-26T21:27:02.294635+0000","last_active":"2024-04-26T21:28:57.083503+0000","last_peered":"2024-04-26T21:28:57.083503+0000","last_clean":"2024-04-26T21:28:57.083503+0000","last_became_active":"2024-04-26T21:27:02.294311+0000","last_became_peered":"2024-04-26T21:27:02.294311+0000","last_unstale":"2024-04-26T21:28:57.083503+0000","last_undegraded":"2024-04-26T21:28:57.083503+0000","last_fullsized":"2024-04-26T21:28:57.083503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:47.906546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141554+0000","last_change":"2024-04-26T21:27:02.875475+0000","last_active":"2024-04-26T21:28:57.141554+0000","last_peered":"2024-04-26T21:28:57.141554+0000","last_clean":"2024-04-26T21:28:57.141554+0000","last_became_active":"2024-04-26T21:27:02.875137+0000","last_became_peered":"2024-04-26T21:27:02.875137+0000","last_unstale":"2024-04-26T21:28:57.141554+0000","last_undegraded":"2024-04-26T21:28:57.141554+0000","last_fullsized":"2024-04-26T21:28:57.141554+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:04.988973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141589+0000","last_change":"2024-04-26T21:27:07.372927+0000","last_active":"2024-04-26T21:28:57.141589+0000","last_peered":"2024-04-26T21:28:57.141589+0000","last_clean":"2024-04-26T21:28:57.141589+0000","last_became_active":"2024-04-26T21:27:07.372371+0000","last_became_peered":"2024-04-26T21:27:07.372371+0000","last_unstale":"2024-04-26T21:28:57.141589+0000","last_undegraded":"2024-04-26T21:28:57.141589+0000","last_fullsized":"2024-04-26T21:28:57.141589+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:12:41.607167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140654+0000","last_change":"2024-04-26T21:27:02.295379+0000","last_active":"2024-04-26T21:28:57.140654+0000","last_peered":"2024-04-26T21:28:57.140654+0000","last_clean":"2024-04-26T21:28:57.140654+0000","last_became_active":"2024-04-26T21:27:02.295077+0000","last_became_peered":"2024-04-26T21:27:02.295077+0000","last_unstale":"2024-04-26T21:28:57.140654+0000","last_undegraded":"2024-04-26T21:28:57.140654+0000","last_fullsized":"2024-04-26T21:28:57.140654+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:58.177960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141479+0000","last_change":"2024-04-26T21:27:02.855998+0000","last_active":"2024-04-26T21:28:57.141479+0000","last_peered":"2024-04-26T21:28:57.141479+0000","last_clean":"2024-04-26T21:28:57.141479+0000","last_became_active":"2024-04-26T21:27:02.855636+0000","last_became_peered":"2024-04-26T21:27:02.855636+0000","last_unstale":"2024-04-26T21:28:57.141479+0000","last_undegraded":"2024-04-26T21:28:57.141479+0000","last_fullsized":"2024-04-26T21:28:57.141479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:19:02.988946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084647+0000","last_change":"2024-04-26T21:27:07.378980+0000","last_active":"2024-04-26T21:28:57.084647+0000","last_peered":"2024-04-26T21:28:57.084647+0000","last_clean":"2024-04-26T21:28:57.084647+0000","last_became_active":"2024-04-26T21:27:07.378374+0000","last_became_peered":"2024-04-26T21:27:07.378374+0000","last_unstale":"2024-04-26T21:28:57.084647+0000","last_undegraded":"2024-04-26T21:28:57.084647+0000","last_fullsized":"2024-04-26T21:28:57.084647+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:31.845297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089815+0000","last_change":"2024-04-26T21:27:02.313801+0000","last_active":"2024-04-26T21:28:57.089815+0000","last_peered":"2024-04-26T21:28:57.089815+0000","last_clean":"2024-04-26T21:28:57.089815+0000","last_became_active":"2024-04-26T21:27:02.313540+0000","last_became_peered":"2024-04-26T21:27:02.313540+0000","last_unstale":"2024-04-26T21:28:57.089815+0000","last_undegraded":"2024-04-26T21:28:57.089815+0000","last_fullsized":"2024-04-26T21:28:57.089815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:38.612261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140890+0000","last_change":"2024-04-26T21:27:02.864151+0000","last_active":"2024-04-26T21:28:57.140890+0000","last_peered":"2024-04-26T21:28:57.140890+0000","last_clean":"2024-04-26T21:28:57.140890+0000","last_became_active":"2024-04-26T21:27:02.863818+0000","last_became_peered":"2024-04-26T21:27:02.863818+0000","last_unstale":"2024-04-26T21:28:57.140890+0000","last_undegraded":"2024-04-26T21:28:57.140890+0000","last_fullsized":"2024-04-26T21:28:57.140890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:26.215370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088554+0000","last_change":"2024-04-26T21:27:07.365821+0000","last_active":"2024-04-26T21:28:57.088554+0000","last_peered":"2024-04-26T21:28:57.088554+0000","last_clean":"2024-04-26T21:28:57.088554+0000","last_became_active":"2024-04-26T21:27:07.365347+0000","last_became_peered":"2024-04-26T21:27:07.365347+0000","last_unstale":"2024-04-26T21:28:57.088554+0000","last_undegraded":"2024-04-26T21:28:57.088554+0000","last_fullsized":"2024-04-26T21:28:57.088554+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:40:11.596908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150070+0000","last_change":"2024-04-26T21:27:02.310641+0000","last_active":"2024-04-26T21:28:57.150070+0000","last_peered":"2024-04-26T21:28:57.150070+0000","last_clean":"2024-04-26T21:28:57.150070+0000","last_became_active":"2024-04-26T21:27:02.308569+0000","last_became_peered":"2024-04-26T21:27:02.308569+0000","last_unstale":"2024-04-26T21:28:57.150070+0000","last_undegraded":"2024-04-26T21:28:57.150070+0000","last_fullsized":"2024-04-26T21:28:57.150070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:20.988098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141041+0000","last_change":"2024-04-26T21:27:02.854976+0000","last_active":"2024-04-26T21:28:57.141041+0000","last_peered":"2024-04-26T21:28:57.141041+0000","last_clean":"2024-04-26T21:28:57.141041+0000","last_became_active":"2024-04-26T21:27:02.854611+0000","last_became_peered":"2024-04-26T21:27:02.854611+0000","last_unstale":"2024-04-26T21:28:57.141041+0000","last_undegraded":"2024-04-26T21:28:57.141041+0000","last_fullsized":"2024-04-26T21:28:57.141041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:57.065657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120299+0000","last_change":"2024-04-26T21:27:02.859017+0000","last_active":"2024-04-26T21:28:57.120299+0000","last_peered":"2024-04-26T21:28:57.120299+0000","last_clean":"2024-04-26T21:28:57.120299+0000","last_became_active":"2024-04-26T21:27:02.858507+0000","last_became_peered":"2024-04-26T21:27:02.858507+0000","last_unstale":"2024-04-26T21:28:57.120299+0000","last_undegraded":"2024-04-26T21:28:57.120299+0000","last_fullsized":"2024-04-26T21:28:57.120299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:14.850891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140440+0000","last_change":"2024-04-26T21:27:56.841251+0000","last_active":"2024-04-26T21:28:57.140440+0000","last_peered":"2024-04-26T21:28:57.140440+0000","last_clean":"2024-04-26T21:28:57.140440+0000","last_became_active":"2024-04-26T21:27:56.840862+0000","last_became_peered":"2024-04-26T21:27:56.840862+0000","last_unstale":"2024-04-26T21:28:57.140440+0000","last_undegraded":"2024-04-26T21:28:57.140440+0000","last_fullsized":"2024-04-26T21:28:57.140440+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:41.460343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140223+0000","last_change":"2024-04-26T21:27:02.868817+0000","last_active":"2024-04-26T21:28:57.140223+0000","last_peered":"2024-04-26T21:28:57.140223+0000","last_clean":"2024-04-26T21:28:57.140223+0000","last_became_active":"2024-04-26T21:27:02.868557+0000","last_became_peered":"2024-04-26T21:27:02.868557+0000","last_unstale":"2024-04-26T21:28:57.140223+0000","last_undegraded":"2024-04-26T21:28:57.140223+0000","last_fullsized":"2024-04-26T21:28:57.140223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:55.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.085674+0000","last_change":"2024-04-26T21:27:02.876029+0000","last_active":"2024-04-26T21:28:57.085674+0000","last_peered":"2024-04-26T21:28:57.085674+0000","last_clean":"2024-04-26T21:28:57.085674+0000","last_became_active":"2024-04-26T21:27:02.875656+0000","last_became_peered":"2024-04-26T21:27:02.875656+0000","last_unstale":"2024-04-26T21:28:57.085674+0000","last_undegraded":"2024-04-26T21:28:57.085674+0000","last_fullsized":"2024-04-26T21:28:57.085674+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:47.317581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150613+0000","last_change":"2024-04-26T21:27:02.873818+0000","last_active":"2024-04-26T21:28:57.150613+0000","last_peered":"2024-04-26T21:28:57.150613+0000","last_clean":"2024-04-26T21:28:57.150613+0000","last_became_active":"2024-04-26T21:27:02.873459+0000","last_became_peered":"2024-04-26T21:27:02.873459+0000","last_unstale":"2024-04-26T21:28:57.150613+0000","last_undegraded":"2024-04-26T21:28:57.150613+0000","last_fullsized":"2024-04-26T21:28:57.150613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:07.259187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120678+0000","last_change":"2024-04-26T21:27:02.854901+0000","last_active":"2024-04-26T21:28:57.120678+0000","last_peered":"2024-04-26T21:28:57.120678+0000","last_clean":"2024-04-26T21:28:57.120678+0000","last_became_active":"2024-04-26T21:27:02.854547+0000","last_became_peered":"2024-04-26T21:27:02.854547+0000","last_unstale":"2024-04-26T21:28:57.120678+0000","last_undegraded":"2024-04-26T21:28:57.120678+0000","last_fullsized":"2024-04-26T21:28:57.120678+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:24.320868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141003+0000","last_change":"2024-04-26T21:27:02.856616+0000","last_active":"2024-04-26T21:28:57.141003+0000","last_peered":"2024-04-26T21:28:57.141003+0000","last_clean":"2024-04-26T21:28:57.141003+0000","last_became_active":"2024-04-26T21:27:02.856264+0000","last_became_peered":"2024-04-26T21:27:02.856264+0000","last_unstale":"2024-04-26T21:28:57.141003+0000","last_undegraded":"2024-04-26T21:28:57.141003+0000","last_fullsized":"2024-04-26T21:28:57.141003+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:43.374534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.133975+0000","last_change":"2024-04-26T21:27:02.859587+0000","last_active":"2024-04-26T21:28:57.133975+0000","last_peered":"2024-04-26T21:28:57.133975+0000","last_clean":"2024-04-26T21:28:57.133975+0000","last_became_active":"2024-04-26T21:27:02.859136+0000","last_became_peered":"2024-04-26T21:27:02.859136+0000","last_unstale":"2024-04-26T21:28:57.133975+0000","last_undegraded":"2024-04-26T21:28:57.133975+0000","last_fullsized":"2024-04-26T21:28:57.133975+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:09.370319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088720+0000","last_change":"2024-04-26T21:27:02.864265+0000","last_active":"2024-04-26T21:28:57.088720+0000","last_peered":"2024-04-26T21:28:57.088720+0000","last_clean":"2024-04-26T21:28:57.088720+0000","last_became_active":"2024-04-26T21:27:02.863920+0000","last_became_peered":"2024-04-26T21:27:02.863920+0000","last_unstale":"2024-04-26T21:28:57.088720+0000","last_undegraded":"2024-04-26T21:28:57.088720+0000","last_fullsized":"2024-04-26T21:28:57.088720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:13.941544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134222+0000","last_change":"2024-04-26T21:27:02.860189+0000","last_active":"2024-04-26T21:28:57.134222+0000","last_peered":"2024-04-26T21:28:57.134222+0000","last_clean":"2024-04-26T21:28:57.134222+0000","last_became_active":"2024-04-26T21:27:02.859843+0000","last_became_peered":"2024-04-26T21:27:02.859843+0000","last_unstale":"2024-04-26T21:28:57.134222+0000","last_undegraded":"2024-04-26T21:28:57.134222+0000","last_fullsized":"2024-04-26T21:28:57.134222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141963+0000","last_change":"2024-04-26T21:27:02.865459+0000","last_active":"2024-04-26T21:28:57.141963+0000","last_peered":"2024-04-26T21:28:57.141963+0000","last_clean":"2024-04-26T21:28:57.141963+0000","last_became_active":"2024-04-26T21:27:02.863602+0000","last_became_peered":"2024-04-26T21:27:02.863602+0000","last_unstale":"2024-04-26T21:28:57.141963+0000","last_undegraded":"2024-04-26T21:28:57.141963+0000","last_fullsized":"2024-04-26T21:28:57.141963+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:03.733858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.149411+0000","last_change":"2024-04-26T21:27:02.873711+0000","last_active":"2024-04-26T21:28:57.149411+0000","last_peered":"2024-04-26T21:28:57.149411+0000","last_clean":"2024-04-26T21:28:57.149411+0000","last_became_active":"2024-04-26T21:27:02.873349+0000","last_became_peered":"2024-04-26T21:27:02.873349+0000","last_unstale":"2024-04-26T21:28:57.149411+0000","last_undegraded":"2024-04-26T21:28:57.149411+0000","last_fullsized":"2024-04-26T21:28:57.149411+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:10.841536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120403+0000","last_change":"2024-04-26T21:27:02.852625+0000","last_active":"2024-04-26T21:28:57.120403+0000","last_peered":"2024-04-26T21:28:57.120403+0000","last_clean":"2024-04-26T21:28:57.120403+0000","last_became_active":"2024-04-26T21:27:02.852177+0000","last_became_peered":"2024-04-26T21:27:02.852177+0000","last_unstale":"2024-04-26T21:28:57.120403+0000","last_undegraded":"2024-04-26T21:28:57.120403+0000","last_fullsized":"2024-04-26T21:28:57.120403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:52.046678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59286,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":623,"num_read_kb":375,"num_write":164,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318312,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":528427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579949,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28740,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716412,"statfs":{"total":95995035648,"available":95965605888,"internally_reserved":0,"allocated":1761280,"data_stored":1171349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743475,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1097728,"data_stored":532737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005114,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":528539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266751,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32832,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712320,"statfs":{"total":95995035648,"available":95961415680,"internally_reserved":0,"allocated":1687552,"data_stored":1124079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462981,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":523817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1134592,"data_stored":579403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1097728,"data_stored":525205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1159168,"data_stored":580753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1085440,"data_stored":524011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888144,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":528467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724618,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37000,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708152,"statfs":{"total":95995035648,"available":95957147648,"internally_reserved":0,"allocated":1695744,"data_stored":1119667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":55676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":57026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":57254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:29:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:15 smithi186 ceph-mon[32991]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:15.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:15 smithi186 ceph-mon[32991]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:15.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:15 smithi001 ceph-mon[28054]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:15.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:15 smithi001 ceph-mon[28054]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:15.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:15 smithi038 ceph-mon[32724]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:15.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:15 smithi038 ceph-mon[32724]: pgmap v394: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:17.288 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T21:29:17.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:17 smithi186 ceph-mon[32991]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:17.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:17 smithi001 ceph-mon[28054]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:17.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:17 smithi038 ceph-mon[32724]: pgmap v395: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:17.758 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:19.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:19 smithi186 ceph-mon[32991]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:19.485 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:19 smithi001 ceph-mon[28054]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:19.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:19 smithi038 ceph-mon[32724]: pgmap v396: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:20.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.003+0000 7ff663000700 1 -- 172.21.15.1:0/478459549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff65c105640 msgr2=0x7ff65c105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.003+0000 7ff663000700 1 --2- 172.21.15.1:0/478459549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff65c105640 0x7ff65c105ac0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff644009990 tx=0x7ff644009ca0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.003+0000 7ff663000700 1 -- 172.21.15.1:0/478459549 shutdown_connections 2024-04-26T21:29:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.003+0000 7ff663000700 1 --2- 172.21.15.1:0/478459549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff65c106000 0x7ff65c112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.003+0000 7ff663000700 1 --2- 172.21.15.1:0/478459549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff65c10b640 0x7ff65c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.003+0000 7ff663000700 1 --2- 172.21.15.1:0/478459549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff65c105640 0x7ff65c105ac0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.004+0000 7ff663000700 1 -- 172.21.15.1:0/478459549 >> 172.21.15.1:0/478459549 conn(0x7ff65c101130 msgr2=0x7ff65c103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.004+0000 7ff663000700 1 -- 172.21.15.1:0/478459549 shutdown_connections 2024-04-26T21:29:20.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.004+0000 7ff663000700 1 -- 172.21.15.1:0/478459549 wait complete. 2024-04-26T21:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.005+0000 7ff663000700 1 Processor -- start 2024-04-26T21:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.005+0000 7ff663000700 1 -- start start 2024-04-26T21:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.006+0000 7ff663000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff65c105640 0x7ff65c19bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.006+0000 7ff663000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff65c106000 0x7ff65c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.006+0000 7ff663000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff65c10b640 0x7ff65c1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.006+0000 7ff663000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c115890 con 0x7ff65c10b640 2024-04-26T21:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.006+0000 7ff663000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c115710 con 0x7ff65c105640 2024-04-26T21:29:20.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.006+0000 7ff663000700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c115a10 con 0x7ff65c106000 2024-04-26T21:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.006+0000 7ff65be00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff65c105640 0x7ff65c19bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.006+0000 7ff65be00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff65c105640 0x7ff65c19bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37406/0 (socket says 172.21.15.1:37406) 2024-04-26T21:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.006+0000 7ff65be00700 1 -- 172.21.15.1:0/869369573 learned_addr learned my addr 172.21.15.1:0/869369573 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.006+0000 7ff661000700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff65c10b640 0x7ff65c1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:20.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.007+0000 7ff653e00700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff65c106000 0x7ff65c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.007+0000 7ff65be00700 1 -- 172.21.15.1:0/869369573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff65c106000 msgr2=0x7ff65c19c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.007+0000 7ff65be00700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff65c106000 0x7ff65c19c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.007+0000 7ff65be00700 1 -- 172.21.15.1:0/869369573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff65c10b640 msgr2=0x7ff65c1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.007+0000 7ff65be00700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff65c10b640 0x7ff65c1a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.007+0000 7ff65be00700 1 -- 172.21.15.1:0/869369573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c1a0d50 con 0x7ff65c105640 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.007+0000 7ff65be00700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff65c105640 0x7ff65c19bd40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00e9a0 tx=0x7ff64c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.008+0000 7ff659600700 1 -- 172.21.15.1:0/869369573 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c00ca10 con 0x7ff65c105640 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.008+0000 7ff659600700 1 -- 172.21.15.1:0/869369573 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff64c00cbb0 con 0x7ff65c105640 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.008+0000 7ff663000700 1 -- 172.21.15.1:0/869369573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65c1a1040 con 0x7ff65c105640 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.008+0000 7ff659600700 1 -- 172.21.15.1:0/869369573 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64c018680 con 0x7ff65c105640 2024-04-26T21:29:20.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.008+0000 7ff663000700 1 -- 172.21.15.1:0/869369573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65c1a88c0 con 0x7ff65c105640 2024-04-26T21:29:20.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.009+0000 7ff663000700 1 -- 172.21.15.1:0/869369573 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff65c06a110 con 0x7ff65c105640 2024-04-26T21:29:20.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.011+0000 7ff659600700 1 -- 172.21.15.1:0/869369573 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7ff64c010430 con 0x7ff65c105640 2024-04-26T21:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.012+0000 7ff659600700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff6400a3d50 0x7ff6400a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.013+0000 7ff659600700 1 -- 172.21.15.1:0/869369573 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7ff64c0d0d70 con 0x7ff65c105640 2024-04-26T21:29:20.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.013+0000 7ff653e00700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff6400a3d50 0x7ff6400a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:20.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.013+0000 7ff653e00700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff6400a3d50 0x7ff6400a6210 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff65c103780 tx=0x7ff644005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:20.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.015+0000 7ff659600700 1 -- 172.21.15.1:0/869369573 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff64c09d190 con 0x7ff65c105640 2024-04-26T21:29:20.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.241+0000 7ff663000700 1 -- 172.21.15.1:0/869369573 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff65c063860 con 0x7ff6400a3d50 2024-04-26T21:29:20.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.259+0000 7ff659600700 1 -- 172.21.15.1:0/869369573 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340377 (secure 0 0 0) 0x7ff64407b1b0 con 0x7ff6400a3d50 2024-04-26T21:29:20.261 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.267+0000 7ff663000700 1 -- 172.21.15.1:0/869369573 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff6400a3d50 msgr2=0x7ff6400a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.267+0000 7ff663000700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff6400a3d50 0x7ff6400a6210 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff65c103780 tx=0x7ff644005fd0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.267+0000 7ff663000700 1 -- 172.21.15.1:0/869369573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff65c105640 msgr2=0x7ff65c19bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:20.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.267+0000 7ff663000700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff65c105640 0x7ff65c19bd40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff64c00e9a0 tx=0x7ff64c00ecb0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.268+0000 7ff663000700 1 -- 172.21.15.1:0/869369573 shutdown_connections 2024-04-26T21:29:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.268+0000 7ff663000700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff6400a3d50 0x7ff6400a6210 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.268+0000 7ff663000700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff65c10b640 0x7ff65c1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.268+0000 7ff663000700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff65c105640 0x7ff65c19bd40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.268+0000 7ff663000700 1 --2- 172.21.15.1:0/869369573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff65c106000 0x7ff65c19c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.268+0000 7ff663000700 1 -- 172.21.15.1:0/869369573 >> 172.21.15.1:0/869369573 conn(0x7ff65c101130 msgr2=0x7ff65c101860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.268+0000 7ff663000700 1 -- 172.21.15.1:0/869369573 shutdown_connections 2024-04-26T21:29:20.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:20.268+0000 7ff663000700 1 -- 172.21.15.1:0/869369573 wait complete. 2024-04-26T21:29:20.272 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-04-26T21:29:21.207 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-04-26T21:29:19.414823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649888,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":819,"num_read_kb":524,"num_write":347,"num_write_kb":3049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1180758,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":279,"ondisk_log_size":279,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":380572,"kb_used_data":14696,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124561252,"statfs":{"total":1151940427776,"available":1151550722048,"internally_reserved":0,"allocated":15048704,"data_stored":8266454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.027042"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120793+0000","last_change":"2024-04-26T21:27:02.854724+0000","last_active":"2024-04-26T21:28:57.120793+0000","last_peered":"2024-04-26T21:28:57.120793+0000","last_clean":"2024-04-26T21:28:57.120793+0000","last_became_active":"2024-04-26T21:27:02.854125+0000","last_became_peered":"2024-04-26T21:27:02.854125+0000","last_unstale":"2024-04-26T21:28:57.120793+0000","last_undegraded":"2024-04-26T21:28:57.120793+0000","last_fullsized":"2024-04-26T21:28:57.120793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:16.500220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089088+0000","last_change":"2024-04-26T21:27:02.852846+0000","last_active":"2024-04-26T21:28:57.089088+0000","last_peered":"2024-04-26T21:28:57.089088+0000","last_clean":"2024-04-26T21:28:57.089088+0000","last_became_active":"2024-04-26T21:27:02.851243+0000","last_became_peered":"2024-04-26T21:27:02.851243+0000","last_unstale":"2024-04-26T21:28:57.089088+0000","last_undegraded":"2024-04-26T21:28:57.089088+0000","last_fullsized":"2024-04-26T21:28:57.089088+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:09:11.240955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.090139+0000","last_change":"2024-04-26T21:27:02.855856+0000","last_active":"2024-04-26T21:28:57.090139+0000","last_peered":"2024-04-26T21:28:57.090139+0000","last_clean":"2024-04-26T21:28:57.090139+0000","last_became_active":"2024-04-26T21:27:02.855446+0000","last_became_peered":"2024-04-26T21:27:02.855446+0000","last_unstale":"2024-04-26T21:28:57.090139+0000","last_undegraded":"2024-04-26T21:28:57.090139+0000","last_fullsized":"2024-04-26T21:28:57.090139+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:19:35.183449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089111+0000","last_change":"2024-04-26T21:27:02.860949+0000","last_active":"2024-04-26T21:28:57.089111+0000","last_peered":"2024-04-26T21:28:57.089111+0000","last_clean":"2024-04-26T21:28:57.089111+0000","last_became_active":"2024-04-26T21:27:02.860567+0000","last_became_peered":"2024-04-26T21:27:02.860567+0000","last_unstale":"2024-04-26T21:28:57.089111+0000","last_undegraded":"2024-04-26T21:28:57.089111+0000","last_fullsized":"2024-04-26T21:28:57.089111+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:59:09.851885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083417+0000","last_change":"2024-04-26T21:27:02.876019+0000","last_active":"2024-04-26T21:28:57.083417+0000","last_peered":"2024-04-26T21:28:57.083417+0000","last_clean":"2024-04-26T21:28:57.083417+0000","last_became_active":"2024-04-26T21:27:02.875652+0000","last_became_peered":"2024-04-26T21:27:02.875652+0000","last_unstale":"2024-04-26T21:28:57.083417+0000","last_undegraded":"2024-04-26T21:28:57.083417+0000","last_fullsized":"2024-04-26T21:28:57.083417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:31:35.017787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088087+0000","last_change":"2024-04-26T21:27:02.860229+0000","last_active":"2024-04-26T21:28:57.088087+0000","last_peered":"2024-04-26T21:28:57.088087+0000","last_clean":"2024-04-26T21:28:57.088087+0000","last_became_active":"2024-04-26T21:27:02.859863+0000","last_became_peered":"2024-04-26T21:27:02.859863+0000","last_unstale":"2024-04-26T21:28:57.088087+0000","last_undegraded":"2024-04-26T21:28:57.088087+0000","last_fullsized":"2024-04-26T21:28:57.088087+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:29:47.337073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140980+0000","last_change":"2024-04-26T21:27:02.863253+0000","last_active":"2024-04-26T21:28:57.140980+0000","last_peered":"2024-04-26T21:28:57.140980+0000","last_clean":"2024-04-26T21:28:57.140980+0000","last_became_active":"2024-04-26T21:27:02.861623+0000","last_became_peered":"2024-04-26T21:27:02.861623+0000","last_unstale":"2024-04-26T21:28:57.140980+0000","last_undegraded":"2024-04-26T21:28:57.140980+0000","last_fullsized":"2024-04-26T21:28:57.140980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:01:57.642504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.133344+0000","last_change":"2024-04-26T21:27:02.860822+0000","last_active":"2024-04-26T21:28:57.133344+0000","last_peered":"2024-04-26T21:28:57.133344+0000","last_clean":"2024-04-26T21:28:57.133344+0000","last_became_active":"2024-04-26T21:27:02.860482+0000","last_became_peered":"2024-04-26T21:27:02.860482+0000","last_unstale":"2024-04-26T21:28:57.133344+0000","last_undegraded":"2024-04-26T21:28:57.133344+0000","last_fullsized":"2024-04-26T21:28:57.133344+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:24:59.224624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141309+0000","last_change":"2024-04-26T21:27:56.856389+0000","last_active":"2024-04-26T21:28:57.141309+0000","last_peered":"2024-04-26T21:28:57.141309+0000","last_clean":"2024-04-26T21:28:57.141309+0000","last_became_active":"2024-04-26T21:27:56.856008+0000","last_became_peered":"2024-04-26T21:27:56.856008+0000","last_unstale":"2024-04-26T21:28:57.141309+0000","last_undegraded":"2024-04-26T21:28:57.141309+0000","last_fullsized":"2024-04-26T21:28:57.141309+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:00:04.770535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,11],"acting":[1,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088910+0000","last_change":"2024-04-26T21:27:02.860545+0000","last_active":"2024-04-26T21:28:57.088910+0000","last_peered":"2024-04-26T21:28:57.088910+0000","last_clean":"2024-04-26T21:28:57.088910+0000","last_became_active":"2024-04-26T21:27:02.858577+0000","last_became_peered":"2024-04-26T21:27:02.858577+0000","last_unstale":"2024-04-26T21:28:57.088910+0000","last_undegraded":"2024-04-26T21:28:57.088910+0000","last_fullsized":"2024-04-26T21:28:57.088910+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:31:43.390592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141372+0000","last_change":"2024-04-26T21:27:02.858685+0000","last_active":"2024-04-26T21:28:57.141372+0000","last_peered":"2024-04-26T21:28:57.141372+0000","last_clean":"2024-04-26T21:28:57.141372+0000","last_became_active":"2024-04-26T21:27:02.857054+0000","last_became_peered":"2024-04-26T21:27:02.857054+0000","last_unstale":"2024-04-26T21:28:57.141372+0000","last_undegraded":"2024-04-26T21:28:57.141372+0000","last_fullsized":"2024-04-26T21:28:57.141372+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:24:29.406132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088929+0000","last_change":"2024-04-26T21:27:02.852457+0000","last_active":"2024-04-26T21:28:57.088929+0000","last_peered":"2024-04-26T21:28:57.088929+0000","last_clean":"2024-04-26T21:28:57.088929+0000","last_became_active":"2024-04-26T21:27:02.852064+0000","last_became_peered":"2024-04-26T21:27:02.852064+0000","last_unstale":"2024-04-26T21:28:57.088929+0000","last_undegraded":"2024-04-26T21:28:57.088929+0000","last_fullsized":"2024-04-26T21:28:57.088929+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:21:16.798413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.149646+0000","last_change":"2024-04-26T21:27:02.872618+0000","last_active":"2024-04-26T21:28:57.149646+0000","last_peered":"2024-04-26T21:28:57.149646+0000","last_clean":"2024-04-26T21:28:57.149646+0000","last_became_active":"2024-04-26T21:27:02.872270+0000","last_became_peered":"2024-04-26T21:27:02.872270+0000","last_unstale":"2024-04-26T21:28:57.149646+0000","last_undegraded":"2024-04-26T21:28:57.149646+0000","last_fullsized":"2024-04-26T21:28:57.149646+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:45.828419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084860+0000","last_change":"2024-04-26T21:27:02.868590+0000","last_active":"2024-04-26T21:28:57.084860+0000","last_peered":"2024-04-26T21:28:57.084860+0000","last_clean":"2024-04-26T21:28:57.084860+0000","last_became_active":"2024-04-26T21:27:02.866641+0000","last_became_peered":"2024-04-26T21:27:02.866641+0000","last_unstale":"2024-04-26T21:28:57.084860+0000","last_undegraded":"2024-04-26T21:28:57.084860+0000","last_fullsized":"2024-04-26T21:28:57.084860+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:31:17.313420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084439+0000","last_change":"2024-04-26T21:27:02.867735+0000","last_active":"2024-04-26T21:28:57.084439+0000","last_peered":"2024-04-26T21:28:57.084439+0000","last_clean":"2024-04-26T21:28:57.084439+0000","last_became_active":"2024-04-26T21:27:02.867217+0000","last_became_peered":"2024-04-26T21:27:02.867217+0000","last_unstale":"2024-04-26T21:28:57.084439+0000","last_undegraded":"2024-04-26T21:28:57.084439+0000","last_fullsized":"2024-04-26T21:28:57.084439+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:04:45.533220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140625+0000","last_change":"2024-04-26T21:27:02.857541+0000","last_active":"2024-04-26T21:28:57.140625+0000","last_peered":"2024-04-26T21:28:57.140625+0000","last_clean":"2024-04-26T21:28:57.140625+0000","last_became_active":"2024-04-26T21:27:02.855563+0000","last_became_peered":"2024-04-26T21:27:02.855563+0000","last_unstale":"2024-04-26T21:28:57.140625+0000","last_undegraded":"2024-04-26T21:28:57.140625+0000","last_fullsized":"2024-04-26T21:28:57.140625+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:23:39.954692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120658+0000","last_change":"2024-04-26T21:27:02.854461+0000","last_active":"2024-04-26T21:28:57.120658+0000","last_peered":"2024-04-26T21:28:57.120658+0000","last_clean":"2024-04-26T21:28:57.120658+0000","last_became_active":"2024-04-26T21:27:02.853691+0000","last_became_peered":"2024-04-26T21:27:02.853691+0000","last_unstale":"2024-04-26T21:28:57.120658+0000","last_undegraded":"2024-04-26T21:28:57.120658+0000","last_fullsized":"2024-04-26T21:28:57.120658+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:09:48.284180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.094647+0000","last_change":"2024-04-26T21:27:02.864308+0000","last_active":"2024-04-26T21:28:57.094647+0000","last_peered":"2024-04-26T21:28:57.094647+0000","last_clean":"2024-04-26T21:28:57.094647+0000","last_became_active":"2024-04-26T21:27:02.863928+0000","last_became_peered":"2024-04-26T21:27:02.863928+0000","last_unstale":"2024-04-26T21:28:57.094647+0000","last_undegraded":"2024-04-26T21:28:57.094647+0000","last_fullsized":"2024-04-26T21:28:57.094647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:57:12.169394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140901+0000","last_change":"2024-04-26T21:27:02.876044+0000","last_active":"2024-04-26T21:28:57.140901+0000","last_peered":"2024-04-26T21:28:57.140901+0000","last_clean":"2024-04-26T21:28:57.140901+0000","last_became_active":"2024-04-26T21:27:02.875681+0000","last_became_peered":"2024-04-26T21:27:02.875681+0000","last_unstale":"2024-04-26T21:28:57.140901+0000","last_undegraded":"2024-04-26T21:28:57.140901+0000","last_fullsized":"2024-04-26T21:28:57.140901+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:48:26.246916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140001+0000","last_change":"2024-04-26T21:27:07.374736+0000","last_active":"2024-04-26T21:28:57.140001+0000","last_peered":"2024-04-26T21:28:57.140001+0000","last_clean":"2024-04-26T21:28:57.140001+0000","last_became_active":"2024-04-26T21:27:07.374116+0000","last_became_peered":"2024-04-26T21:27:07.374116+0000","last_unstale":"2024-04-26T21:28:57.140001+0000","last_undegraded":"2024-04-26T21:28:57.140001+0000","last_fullsized":"2024-04-26T21:28:57.140001+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:05:11.450926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.095218+0000","last_change":"2024-04-26T21:27:02.302510+0000","last_active":"2024-04-26T21:28:57.095218+0000","last_peered":"2024-04-26T21:28:57.095218+0000","last_clean":"2024-04-26T21:28:57.095218+0000","last_became_active":"2024-04-26T21:27:02.302088+0000","last_became_peered":"2024-04-26T21:27:02.302088+0000","last_unstale":"2024-04-26T21:28:57.095218+0000","last_undegraded":"2024-04-26T21:28:57.095218+0000","last_fullsized":"2024-04-26T21:28:57.095218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:39:43.007722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134536+0000","last_change":"2024-04-26T21:27:02.861455+0000","last_active":"2024-04-26T21:28:57.134536+0000","last_peered":"2024-04-26T21:28:57.134536+0000","last_clean":"2024-04-26T21:28:57.134536+0000","last_became_active":"2024-04-26T21:27:02.861107+0000","last_became_peered":"2024-04-26T21:27:02.861107+0000","last_unstale":"2024-04-26T21:28:57.134536+0000","last_undegraded":"2024-04-26T21:28:57.134536+0000","last_fullsized":"2024-04-26T21:28:57.134536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:10:18.382881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088435+0000","last_change":"2024-04-26T21:27:07.377797+0000","last_active":"2024-04-26T21:28:57.088435+0000","last_peered":"2024-04-26T21:28:57.088435+0000","last_clean":"2024-04-26T21:28:57.088435+0000","last_became_active":"2024-04-26T21:27:07.377314+0000","last_became_peered":"2024-04-26T21:27:07.377314+0000","last_unstale":"2024-04-26T21:28:57.088435+0000","last_undegraded":"2024-04-26T21:28:57.088435+0000","last_fullsized":"2024-04-26T21:28:57.088435+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:02:34.761034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134833+0000","last_change":"2024-04-26T21:27:02.309488+0000","last_active":"2024-04-26T21:28:57.134833+0000","last_peered":"2024-04-26T21:28:57.134833+0000","last_clean":"2024-04-26T21:28:57.134833+0000","last_became_active":"2024-04-26T21:27:02.308592+0000","last_became_peered":"2024-04-26T21:27:02.308592+0000","last_unstale":"2024-04-26T21:28:57.134833+0000","last_undegraded":"2024-04-26T21:28:57.134833+0000","last_fullsized":"2024-04-26T21:28:57.134833+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:32:36.563847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089064+0000","last_change":"2024-04-26T21:27:02.863204+0000","last_active":"2024-04-26T21:28:57.089064+0000","last_peered":"2024-04-26T21:28:57.089064+0000","last_clean":"2024-04-26T21:28:57.089064+0000","last_became_active":"2024-04-26T21:27:02.862466+0000","last_became_peered":"2024-04-26T21:27:02.862466+0000","last_unstale":"2024-04-26T21:28:57.089064+0000","last_undegraded":"2024-04-26T21:28:57.089064+0000","last_fullsized":"2024-04-26T21:28:57.089064+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:26:29.360830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088593+0000","last_change":"2024-04-26T21:27:07.378032+0000","last_active":"2024-04-26T21:28:57.088593+0000","last_peered":"2024-04-26T21:28:57.088593+0000","last_clean":"2024-04-26T21:28:57.088593+0000","last_became_active":"2024-04-26T21:27:07.377255+0000","last_became_peered":"2024-04-26T21:27:07.377255+0000","last_unstale":"2024-04-26T21:28:57.088593+0000","last_undegraded":"2024-04-26T21:28:57.088593+0000","last_fullsized":"2024-04-26T21:28:57.088593+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:40:18.989977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084022+0000","last_change":"2024-04-26T21:27:56.850816+0000","last_active":"2024-04-26T21:28:57.084022+0000","last_peered":"2024-04-26T21:28:57.084022+0000","last_clean":"2024-04-26T21:28:57.084022+0000","last_became_active":"2024-04-26T21:27:56.843299+0000","last_became_peered":"2024-04-26T21:27:56.843299+0000","last_unstale":"2024-04-26T21:28:57.084022+0000","last_undegraded":"2024-04-26T21:28:57.084022+0000","last_fullsized":"2024-04-26T21:28:57.084022+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:21:38.864693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141842+0000","last_change":"2024-04-26T21:27:02.876447+0000","last_active":"2024-04-26T21:28:57.141842+0000","last_peered":"2024-04-26T21:28:57.141842+0000","last_clean":"2024-04-26T21:28:57.141842+0000","last_became_active":"2024-04-26T21:27:02.876017+0000","last_became_peered":"2024-04-26T21:27:02.876017+0000","last_unstale":"2024-04-26T21:28:57.141842+0000","last_undegraded":"2024-04-26T21:28:57.141842+0000","last_fullsized":"2024-04-26T21:28:57.141842+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:57:05.453787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088379+0000","last_change":"2024-04-26T21:27:07.373700+0000","last_active":"2024-04-26T21:28:57.088379+0000","last_peered":"2024-04-26T21:28:57.088379+0000","last_clean":"2024-04-26T21:28:57.088379+0000","last_became_active":"2024-04-26T21:27:07.373255+0000","last_became_peered":"2024-04-26T21:27:07.373255+0000","last_unstale":"2024-04-26T21:28:57.088379+0000","last_undegraded":"2024-04-26T21:28:57.088379+0000","last_fullsized":"2024-04-26T21:28:57.088379+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:48.517197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083901+0000","last_change":"2024-04-26T21:27:02.300817+0000","last_active":"2024-04-26T21:28:57.083901+0000","last_peered":"2024-04-26T21:28:57.083901+0000","last_clean":"2024-04-26T21:28:57.083901+0000","last_became_active":"2024-04-26T21:27:02.300493+0000","last_became_peered":"2024-04-26T21:27:02.300493+0000","last_unstale":"2024-04-26T21:28:57.083901+0000","last_undegraded":"2024-04-26T21:28:57.083901+0000","last_fullsized":"2024-04-26T21:28:57.083901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:13:18.783104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089662+0000","last_change":"2024-04-26T21:27:02.859982+0000","last_active":"2024-04-26T21:28:57.089662+0000","last_peered":"2024-04-26T21:28:57.089662+0000","last_clean":"2024-04-26T21:28:57.089662+0000","last_became_active":"2024-04-26T21:27:02.857204+0000","last_became_peered":"2024-04-26T21:27:02.857204+0000","last_unstale":"2024-04-26T21:28:57.089662+0000","last_undegraded":"2024-04-26T21:28:57.089662+0000","last_fullsized":"2024-04-26T21:28:57.089662+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:00:22.423956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140395+0000","last_change":"2024-04-26T21:27:07.366039+0000","last_active":"2024-04-26T21:28:57.140395+0000","last_peered":"2024-04-26T21:28:57.140395+0000","last_clean":"2024-04-26T21:28:57.140395+0000","last_became_active":"2024-04-26T21:27:07.365560+0000","last_became_peered":"2024-04-26T21:27:07.365560+0000","last_unstale":"2024-04-26T21:28:57.140395+0000","last_undegraded":"2024-04-26T21:28:57.140395+0000","last_fullsized":"2024-04-26T21:28:57.140395+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:45:47.833476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":147,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:15.858003+0000","last_change":"2024-04-26T21:27:02.306364+0000","last_active":"2024-04-26T21:29:15.858003+0000","last_peered":"2024-04-26T21:29:15.858003+0000","last_clean":"2024-04-26T21:29:15.858003+0000","last_became_active":"2024-04-26T21:27:02.306041+0000","last_became_peered":"2024-04-26T21:27:02.306041+0000","last_unstale":"2024-04-26T21:29:15.858003+0000","last_undegraded":"2024-04-26T21:29:15.858003+0000","last_fullsized":"2024-04-26T21:29:15.858003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:15:44.501916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083650+0000","last_change":"2024-04-26T21:27:02.859346+0000","last_active":"2024-04-26T21:28:57.083650+0000","last_peered":"2024-04-26T21:28:57.083650+0000","last_clean":"2024-04-26T21:28:57.083650+0000","last_became_active":"2024-04-26T21:27:02.859025+0000","last_became_peered":"2024-04-26T21:27:02.859025+0000","last_unstale":"2024-04-26T21:28:57.083650+0000","last_undegraded":"2024-04-26T21:28:57.083650+0000","last_fullsized":"2024-04-26T21:28:57.083650+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:54:06.258512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140349+0000","last_change":"2024-04-26T21:27:07.382349+0000","last_active":"2024-04-26T21:28:57.140349+0000","last_peered":"2024-04-26T21:28:57.140349+0000","last_clean":"2024-04-26T21:28:57.140349+0000","last_became_active":"2024-04-26T21:27:07.381930+0000","last_became_peered":"2024-04-26T21:27:07.381930+0000","last_unstale":"2024-04-26T21:28:57.140349+0000","last_undegraded":"2024-04-26T21:28:57.140349+0000","last_fullsized":"2024-04-26T21:28:57.140349+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:15:25.970573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":44,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.087951+0000","last_change":"2024-04-26T21:27:56.862413+0000","last_active":"2024-04-26T21:28:57.087951+0000","last_peered":"2024-04-26T21:28:57.087951+0000","last_clean":"2024-04-26T21:28:57.087951+0000","last_became_active":"2024-04-26T21:27:56.859184+0000","last_became_peered":"2024-04-26T21:27:56.859184+0000","last_unstale":"2024-04-26T21:28:57.087951+0000","last_undegraded":"2024-04-26T21:28:57.087951+0000","last_fullsized":"2024-04-26T21:28:57.087951+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:54:17.701601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089353+0000","last_change":"2024-04-26T21:27:02.865707+0000","last_active":"2024-04-26T21:28:57.089353+0000","last_peered":"2024-04-26T21:28:57.089353+0000","last_clean":"2024-04-26T21:28:57.089353+0000","last_became_active":"2024-04-26T21:27:02.865436+0000","last_became_peered":"2024-04-26T21:27:02.865436+0000","last_unstale":"2024-04-26T21:28:57.089353+0000","last_undegraded":"2024-04-26T21:28:57.089353+0000","last_fullsized":"2024-04-26T21:28:57.089353+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:50:53.406735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120688+0000","last_change":"2024-04-26T21:27:07.379092+0000","last_active":"2024-04-26T21:28:57.120688+0000","last_peered":"2024-04-26T21:28:57.120688+0000","last_clean":"2024-04-26T21:28:57.120688+0000","last_became_active":"2024-04-26T21:27:07.378716+0000","last_became_peered":"2024-04-26T21:27:07.378716+0000","last_unstale":"2024-04-26T21:28:57.120688+0000","last_undegraded":"2024-04-26T21:28:57.120688+0000","last_fullsized":"2024-04-26T21:28:57.120688+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:45:18.906224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147874+0000","last_change":"2024-04-26T21:27:02.307510+0000","last_active":"2024-04-26T21:28:57.147874+0000","last_peered":"2024-04-26T21:28:57.147874+0000","last_clean":"2024-04-26T21:28:57.147874+0000","last_became_active":"2024-04-26T21:27:02.306932+0000","last_became_peered":"2024-04-26T21:27:02.306932+0000","last_unstale":"2024-04-26T21:28:57.147874+0000","last_undegraded":"2024-04-26T21:28:57.147874+0000","last_fullsized":"2024-04-26T21:28:57.147874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:21:34.217767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.132628+0000","last_change":"2024-04-26T21:27:02.874246+0000","last_active":"2024-04-26T21:28:57.132628+0000","last_peered":"2024-04-26T21:28:57.132628+0000","last_clean":"2024-04-26T21:28:57.132628+0000","last_became_active":"2024-04-26T21:27:02.873880+0000","last_became_peered":"2024-04-26T21:27:02.873880+0000","last_unstale":"2024-04-26T21:28:57.132628+0000","last_undegraded":"2024-04-26T21:28:57.132628+0000","last_fullsized":"2024-04-26T21:28:57.132628+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:00:21.579832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088920+0000","last_change":"2024-04-26T21:27:07.387201+0000","last_active":"2024-04-26T21:28:57.088920+0000","last_peered":"2024-04-26T21:28:57.088920+0000","last_clean":"2024-04-26T21:28:57.088920+0000","last_became_active":"2024-04-26T21:27:07.386327+0000","last_became_peered":"2024-04-26T21:27:07.386327+0000","last_unstale":"2024-04-26T21:28:57.088920+0000","last_undegraded":"2024-04-26T21:28:57.088920+0000","last_fullsized":"2024-04-26T21:28:57.088920+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:13:40.801572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088663+0000","last_change":"2024-04-26T21:27:02.304132+0000","last_active":"2024-04-26T21:28:57.088663+0000","last_peered":"2024-04-26T21:28:57.088663+0000","last_clean":"2024-04-26T21:28:57.088663+0000","last_became_active":"2024-04-26T21:27:02.303562+0000","last_became_peered":"2024-04-26T21:27:02.303562+0000","last_unstale":"2024-04-26T21:28:57.088663+0000","last_undegraded":"2024-04-26T21:28:57.088663+0000","last_fullsized":"2024-04-26T21:28:57.088663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:15:54.417590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.092781+0000","last_change":"2024-04-26T21:27:02.866539+0000","last_active":"2024-04-26T21:28:57.092781+0000","last_peered":"2024-04-26T21:28:57.092781+0000","last_clean":"2024-04-26T21:28:57.092781+0000","last_became_active":"2024-04-26T21:27:02.866193+0000","last_became_peered":"2024-04-26T21:27:02.866193+0000","last_unstale":"2024-04-26T21:28:57.092781+0000","last_undegraded":"2024-04-26T21:28:57.092781+0000","last_fullsized":"2024-04-26T21:28:57.092781+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:22:44.234261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134645+0000","last_change":"2024-04-26T21:27:07.368394+0000","last_active":"2024-04-26T21:28:57.134645+0000","last_peered":"2024-04-26T21:28:57.134645+0000","last_clean":"2024-04-26T21:28:57.134645+0000","last_became_active":"2024-04-26T21:27:07.367998+0000","last_became_peered":"2024-04-26T21:27:07.367998+0000","last_unstale":"2024-04-26T21:28:57.134645+0000","last_undegraded":"2024-04-26T21:28:57.134645+0000","last_fullsized":"2024-04-26T21:28:57.134645+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:31:28.459759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":144,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:16.858887+0000","last_change":"2024-04-26T21:27:02.304096+0000","last_active":"2024-04-26T21:29:16.858887+0000","last_peered":"2024-04-26T21:29:16.858887+0000","last_clean":"2024-04-26T21:29:16.858887+0000","last_became_active":"2024-04-26T21:27:02.303736+0000","last_became_peered":"2024-04-26T21:27:02.303736+0000","last_unstale":"2024-04-26T21:29:16.858887+0000","last_undegraded":"2024-04-26T21:29:16.858887+0000","last_fullsized":"2024-04-26T21:29:16.858887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:25.400913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142465+0000","last_change":"2024-04-26T21:27:02.863033+0000","last_active":"2024-04-26T21:28:57.142465+0000","last_peered":"2024-04-26T21:28:57.142465+0000","last_clean":"2024-04-26T21:28:57.142465+0000","last_became_active":"2024-04-26T21:27:02.862572+0000","last_became_peered":"2024-04-26T21:27:02.862572+0000","last_unstale":"2024-04-26T21:28:57.142465+0000","last_undegraded":"2024-04-26T21:28:57.142465+0000","last_fullsized":"2024-04-26T21:28:57.142465+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:02:55.574601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142841+0000","last_change":"2024-04-26T21:28:57.142094+0000","last_active":"2024-04-26T21:28:57.142841+0000","last_peered":"2024-04-26T21:28:57.142841+0000","last_clean":"2024-04-26T21:28:57.142841+0000","last_became_active":"2024-04-26T21:28:57.141572+0000","last_became_peered":"2024-04-26T21:28:57.141572+0000","last_unstale":"2024-04-26T21:28:57.142841+0000","last_undegraded":"2024-04-26T21:28:57.142841+0000","last_fullsized":"2024-04-26T21:28:57.142841+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:45:35.087842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":43,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141281+0000","last_change":"2024-04-26T21:27:56.846362+0000","last_active":"2024-04-26T21:28:57.141281+0000","last_peered":"2024-04-26T21:28:57.141281+0000","last_clean":"2024-04-26T21:28:57.141281+0000","last_became_active":"2024-04-26T21:27:56.840187+0000","last_became_peered":"2024-04-26T21:27:56.840187+0000","last_unstale":"2024-04-26T21:28:57.141281+0000","last_undegraded":"2024-04-26T21:28:57.141281+0000","last_fullsized":"2024-04-26T21:28:57.141281+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:17:12.064415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088889+0000","last_change":"2024-04-26T21:27:02.858739+0000","last_active":"2024-04-26T21:28:57.088889+0000","last_peered":"2024-04-26T21:28:57.088889+0000","last_clean":"2024-04-26T21:28:57.088889+0000","last_became_active":"2024-04-26T21:27:02.857458+0000","last_became_peered":"2024-04-26T21:27:02.857458+0000","last_unstale":"2024-04-26T21:28:57.088889+0000","last_undegraded":"2024-04-26T21:28:57.088889+0000","last_fullsized":"2024-04-26T21:28:57.088889+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:18:00.354052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089405+0000","last_change":"2024-04-26T21:27:07.377690+0000","last_active":"2024-04-26T21:28:57.089405+0000","last_peered":"2024-04-26T21:28:57.089405+0000","last_clean":"2024-04-26T21:28:57.089405+0000","last_became_active":"2024-04-26T21:27:07.377192+0000","last_became_peered":"2024-04-26T21:27:07.377192+0000","last_unstale":"2024-04-26T21:28:57.089405+0000","last_undegraded":"2024-04-26T21:28:57.089405+0000","last_fullsized":"2024-04-26T21:28:57.089405+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:36:48.069083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089650+0000","last_change":"2024-04-26T21:27:02.316110+0000","last_active":"2024-04-26T21:28:57.089650+0000","last_peered":"2024-04-26T21:28:57.089650+0000","last_clean":"2024-04-26T21:28:57.089650+0000","last_became_active":"2024-04-26T21:27:02.315767+0000","last_became_peered":"2024-04-26T21:27:02.315767+0000","last_unstale":"2024-04-26T21:28:57.089650+0000","last_undegraded":"2024-04-26T21:28:57.089650+0000","last_fullsized":"2024-04-26T21:28:57.089650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:24:00.990872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142071+0000","last_change":"2024-04-26T21:27:02.855736+0000","last_active":"2024-04-26T21:28:57.142071+0000","last_peered":"2024-04-26T21:28:57.142071+0000","last_clean":"2024-04-26T21:28:57.142071+0000","last_became_active":"2024-04-26T21:27:02.855378+0000","last_became_peered":"2024-04-26T21:27:02.855378+0000","last_unstale":"2024-04-26T21:28:57.142071+0000","last_undegraded":"2024-04-26T21:28:57.142071+0000","last_fullsized":"2024-04-26T21:28:57.142071+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:11:55.272869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.091733+0000","last_change":"2024-04-26T21:27:07.376804+0000","last_active":"2024-04-26T21:28:57.091733+0000","last_peered":"2024-04-26T21:28:57.091733+0000","last_clean":"2024-04-26T21:28:57.091733+0000","last_became_active":"2024-04-26T21:27:07.374378+0000","last_became_peered":"2024-04-26T21:27:07.374378+0000","last_unstale":"2024-04-26T21:28:57.091733+0000","last_undegraded":"2024-04-26T21:28:57.091733+0000","last_fullsized":"2024-04-26T21:28:57.091733+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:48:03.930253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'33","reported_seq":292,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:17.860400+0000","last_change":"2024-04-26T21:27:02.297467+0000","last_active":"2024-04-26T21:29:17.860400+0000","last_peered":"2024-04-26T21:29:17.860400+0000","last_clean":"2024-04-26T21:29:17.860400+0000","last_became_active":"2024-04-26T21:27:02.297077+0000","last_became_peered":"2024-04-26T21:27:02.297077+0000","last_unstale":"2024-04-26T21:29:17.860400+0000","last_undegraded":"2024-04-26T21:29:17.860400+0000","last_fullsized":"2024-04-26T21:29:17.860400+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:41:47.047565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55676,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":228,"num_read_kb":168,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083876+0000","last_change":"2024-04-26T21:27:02.858161+0000","last_active":"2024-04-26T21:28:57.083876+0000","last_peered":"2024-04-26T21:28:57.083876+0000","last_clean":"2024-04-26T21:28:57.083876+0000","last_became_active":"2024-04-26T21:27:02.857747+0000","last_became_peered":"2024-04-26T21:27:02.857747+0000","last_unstale":"2024-04-26T21:28:57.083876+0000","last_undegraded":"2024-04-26T21:28:57.083876+0000","last_fullsized":"2024-04-26T21:28:57.083876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:10:50.843195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.090006+0000","last_change":"2024-04-26T21:27:07.376253+0000","last_active":"2024-04-26T21:28:57.090006+0000","last_peered":"2024-04-26T21:28:57.090006+0000","last_clean":"2024-04-26T21:28:57.090006+0000","last_became_active":"2024-04-26T21:27:07.375822+0000","last_became_peered":"2024-04-26T21:27:07.375822+0000","last_unstale":"2024-04-26T21:28:57.090006+0000","last_undegraded":"2024-04-26T21:28:57.090006+0000","last_fullsized":"2024-04-26T21:28:57.090006+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:22:01.784163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":38,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140286+0000","last_change":"2024-04-26T21:27:02.300743+0000","last_active":"2024-04-26T21:28:57.140286+0000","last_peered":"2024-04-26T21:28:57.140286+0000","last_clean":"2024-04-26T21:28:57.140286+0000","last_became_active":"2024-04-26T21:27:02.300424+0000","last_became_peered":"2024-04-26T21:27:02.300424+0000","last_unstale":"2024-04-26T21:28:57.140286+0000","last_undegraded":"2024-04-26T21:28:57.140286+0000","last_fullsized":"2024-04-26T21:28:57.140286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:27:10.408234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148845+0000","last_change":"2024-04-26T21:27:02.872456+0000","last_active":"2024-04-26T21:28:57.148845+0000","last_peered":"2024-04-26T21:28:57.148845+0000","last_clean":"2024-04-26T21:28:57.148845+0000","last_became_active":"2024-04-26T21:27:02.872064+0000","last_became_peered":"2024-04-26T21:27:02.872064+0000","last_unstale":"2024-04-26T21:28:57.148845+0000","last_undegraded":"2024-04-26T21:28:57.148845+0000","last_fullsized":"2024-04-26T21:28:57.148845+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:09:33.511164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120361+0000","last_change":"2024-04-26T21:27:07.386148+0000","last_active":"2024-04-26T21:28:57.120361+0000","last_peered":"2024-04-26T21:28:57.120361+0000","last_clean":"2024-04-26T21:28:57.120361+0000","last_became_active":"2024-04-26T21:27:07.385787+0000","last_became_peered":"2024-04-26T21:27:07.385787+0000","last_unstale":"2024-04-26T21:28:57.120361+0000","last_undegraded":"2024-04-26T21:28:57.120361+0000","last_fullsized":"2024-04-26T21:28:57.120361+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:24:57.776660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089128+0000","last_change":"2024-04-26T21:27:02.313725+0000","last_active":"2024-04-26T21:28:57.089128+0000","last_peered":"2024-04-26T21:28:57.089128+0000","last_clean":"2024-04-26T21:28:57.089128+0000","last_became_active":"2024-04-26T21:27:02.313350+0000","last_became_peered":"2024-04-26T21:27:02.313350+0000","last_unstale":"2024-04-26T21:28:57.089128+0000","last_undegraded":"2024-04-26T21:28:57.089128+0000","last_fullsized":"2024-04-26T21:28:57.089128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:21:30.608067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141462+0000","last_change":"2024-04-26T21:27:56.856753+0000","last_active":"2024-04-26T21:28:57.141462+0000","last_peered":"2024-04-26T21:28:57.141462+0000","last_clean":"2024-04-26T21:28:57.141462+0000","last_became_active":"2024-04-26T21:27:56.856297+0000","last_became_peered":"2024-04-26T21:27:56.856297+0000","last_unstale":"2024-04-26T21:28:57.141462+0000","last_undegraded":"2024-04-26T21:28:57.141462+0000","last_fullsized":"2024-04-26T21:28:57.141462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:11:54.650077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.095669+0000","last_change":"2024-04-26T21:27:07.378091+0000","last_active":"2024-04-26T21:28:57.095669+0000","last_peered":"2024-04-26T21:28:57.095669+0000","last_clean":"2024-04-26T21:28:57.095669+0000","last_became_active":"2024-04-26T21:27:07.377671+0000","last_became_peered":"2024-04-26T21:27:07.377671+0000","last_unstale":"2024-04-26T21:28:57.095669+0000","last_undegraded":"2024-04-26T21:28:57.095669+0000","last_fullsized":"2024-04-26T21:28:57.095669+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:41:36.551655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'3","reported_seq":145,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:15.894870+0000","last_change":"2024-04-26T21:27:02.294738+0000","last_active":"2024-04-26T21:29:15.894870+0000","last_peered":"2024-04-26T21:29:15.894870+0000","last_clean":"2024-04-26T21:29:15.894870+0000","last_became_active":"2024-04-26T21:27:02.294338+0000","last_became_peered":"2024-04-26T21:27:02.294338+0000","last_unstale":"2024-04-26T21:29:15.894870+0000","last_undegraded":"2024-04-26T21:29:15.894870+0000","last_fullsized":"2024-04-26T21:29:15.894870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:10:25.309107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141783+0000","last_change":"2024-04-26T21:27:02.861332+0000","last_active":"2024-04-26T21:28:57.141783+0000","last_peered":"2024-04-26T21:28:57.141783+0000","last_clean":"2024-04-26T21:28:57.141783+0000","last_became_active":"2024-04-26T21:27:02.857865+0000","last_became_peered":"2024-04-26T21:27:02.857865+0000","last_unstale":"2024-04-26T21:28:57.141783+0000","last_undegraded":"2024-04-26T21:28:57.141783+0000","last_fullsized":"2024-04-26T21:28:57.141783+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:48.493086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150359+0000","last_change":"2024-04-26T21:27:07.383219+0000","last_active":"2024-04-26T21:28:57.150359+0000","last_peered":"2024-04-26T21:28:57.150359+0000","last_clean":"2024-04-26T21:28:57.150359+0000","last_became_active":"2024-04-26T21:27:07.382820+0000","last_became_peered":"2024-04-26T21:27:07.382820+0000","last_unstale":"2024-04-26T21:28:57.150359+0000","last_undegraded":"2024-04-26T21:28:57.150359+0000","last_fullsized":"2024-04-26T21:28:57.150359+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:52:07.752124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":47,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083444+0000","last_change":"2024-04-26T21:27:02.307746+0000","last_active":"2024-04-26T21:28:57.083444+0000","last_peered":"2024-04-26T21:28:57.083444+0000","last_clean":"2024-04-26T21:28:57.083444+0000","last_became_active":"2024-04-26T21:27:02.307406+0000","last_became_peered":"2024-04-26T21:27:02.307406+0000","last_unstale":"2024-04-26T21:28:57.083444+0000","last_undegraded":"2024-04-26T21:28:57.083444+0000","last_fullsized":"2024-04-26T21:28:57.083444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:59:13.799727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088903+0000","last_change":"2024-04-26T21:27:02.861194+0000","last_active":"2024-04-26T21:28:57.088903+0000","last_peered":"2024-04-26T21:28:57.088903+0000","last_clean":"2024-04-26T21:28:57.088903+0000","last_became_active":"2024-04-26T21:27:02.860871+0000","last_became_peered":"2024-04-26T21:27:02.860871+0000","last_unstale":"2024-04-26T21:28:57.088903+0000","last_undegraded":"2024-04-26T21:28:57.088903+0000","last_fullsized":"2024-04-26T21:28:57.088903+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:32:56.829027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.144000+0000","last_change":"2024-04-26T21:28:57.143451+0000","last_active":"2024-04-26T21:28:57.144000+0000","last_peered":"2024-04-26T21:28:57.144000+0000","last_clean":"2024-04-26T21:28:57.144000+0000","last_became_active":"2024-04-26T21:28:57.142896+0000","last_became_peered":"2024-04-26T21:28:57.142896+0000","last_unstale":"2024-04-26T21:28:57.144000+0000","last_undegraded":"2024-04-26T21:28:57.144000+0000","last_fullsized":"2024-04-26T21:28:57.144000+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:10:40.800757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,3,1],"acting":[10,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084171+0000","last_change":"2024-04-26T21:27:02.301711+0000","last_active":"2024-04-26T21:28:57.084171+0000","last_peered":"2024-04-26T21:28:57.084171+0000","last_clean":"2024-04-26T21:28:57.084171+0000","last_became_active":"2024-04-26T21:27:02.301429+0000","last_became_peered":"2024-04-26T21:27:02.301429+0000","last_unstale":"2024-04-26T21:28:57.084171+0000","last_undegraded":"2024-04-26T21:28:57.084171+0000","last_fullsized":"2024-04-26T21:28:57.084171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:07:50.876824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120651+0000","last_change":"2024-04-26T21:27:02.854372+0000","last_active":"2024-04-26T21:28:57.120651+0000","last_peered":"2024-04-26T21:28:57.120651+0000","last_clean":"2024-04-26T21:28:57.120651+0000","last_became_active":"2024-04-26T21:27:02.853826+0000","last_became_peered":"2024-04-26T21:27:02.853826+0000","last_unstale":"2024-04-26T21:28:57.120651+0000","last_undegraded":"2024-04-26T21:28:57.120651+0000","last_fullsized":"2024-04-26T21:28:57.120651+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:04:08.253384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083595+0000","last_change":"2024-04-26T21:27:07.506684+0000","last_active":"2024-04-26T21:28:57.083595+0000","last_peered":"2024-04-26T21:28:57.083595+0000","last_clean":"2024-04-26T21:28:57.083595+0000","last_became_active":"2024-04-26T21:27:07.506203+0000","last_became_peered":"2024-04-26T21:27:07.506203+0000","last_unstale":"2024-04-26T21:28:57.083595+0000","last_undegraded":"2024-04-26T21:28:57.083595+0000","last_fullsized":"2024-04-26T21:28:57.083595+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:42:48.580106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.094179+0000","last_change":"2024-04-26T21:27:02.306265+0000","last_active":"2024-04-26T21:28:57.094179+0000","last_peered":"2024-04-26T21:28:57.094179+0000","last_clean":"2024-04-26T21:28:57.094179+0000","last_became_active":"2024-04-26T21:27:02.305941+0000","last_became_peered":"2024-04-26T21:27:02.305941+0000","last_unstale":"2024-04-26T21:28:57.094179+0000","last_undegraded":"2024-04-26T21:28:57.094179+0000","last_fullsized":"2024-04-26T21:28:57.094179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:44:29.076833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088626+0000","last_change":"2024-04-26T21:27:02.871916+0000","last_active":"2024-04-26T21:28:57.088626+0000","last_peered":"2024-04-26T21:28:57.088626+0000","last_clean":"2024-04-26T21:28:57.088626+0000","last_became_active":"2024-04-26T21:27:02.871517+0000","last_became_peered":"2024-04-26T21:27:02.871517+0000","last_unstale":"2024-04-26T21:28:57.088626+0000","last_undegraded":"2024-04-26T21:28:57.088626+0000","last_fullsized":"2024-04-26T21:28:57.088626+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:51:41.361757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148245+0000","last_change":"2024-04-26T21:27:07.383422+0000","last_active":"2024-04-26T21:28:57.148245+0000","last_peered":"2024-04-26T21:28:57.148245+0000","last_clean":"2024-04-26T21:28:57.148245+0000","last_became_active":"2024-04-26T21:27:07.382727+0000","last_became_peered":"2024-04-26T21:27:07.382727+0000","last_unstale":"2024-04-26T21:28:57.148245+0000","last_undegraded":"2024-04-26T21:28:57.148245+0000","last_fullsized":"2024-04-26T21:28:57.148245+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:20:22.002658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141798+0000","last_change":"2024-04-26T21:27:02.308081+0000","last_active":"2024-04-26T21:28:57.141798+0000","last_peered":"2024-04-26T21:28:57.141798+0000","last_clean":"2024-04-26T21:28:57.141798+0000","last_became_active":"2024-04-26T21:27:02.307715+0000","last_became_peered":"2024-04-26T21:27:02.307715+0000","last_unstale":"2024-04-26T21:28:57.141798+0000","last_undegraded":"2024-04-26T21:28:57.141798+0000","last_fullsized":"2024-04-26T21:28:57.141798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:28.967817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088500+0000","last_change":"2024-04-26T21:27:02.859527+0000","last_active":"2024-04-26T21:28:57.088500+0000","last_peered":"2024-04-26T21:28:57.088500+0000","last_clean":"2024-04-26T21:28:57.088500+0000","last_became_active":"2024-04-26T21:27:02.859173+0000","last_became_peered":"2024-04-26T21:27:02.859173+0000","last_unstale":"2024-04-26T21:28:57.088500+0000","last_undegraded":"2024-04-26T21:28:57.088500+0000","last_fullsized":"2024-04-26T21:28:57.088500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:24.811630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147125+0000","last_change":"2024-04-26T21:27:07.509778+0000","last_active":"2024-04-26T21:28:57.147125+0000","last_peered":"2024-04-26T21:28:57.147125+0000","last_clean":"2024-04-26T21:28:57.147125+0000","last_became_active":"2024-04-26T21:27:07.508344+0000","last_became_peered":"2024-04-26T21:27:07.508344+0000","last_unstale":"2024-04-26T21:28:57.147125+0000","last_undegraded":"2024-04-26T21:28:57.147125+0000","last_fullsized":"2024-04-26T21:28:57.147125+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:38:05.926717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.143833+0000","last_change":"2024-04-26T21:28:57.143833+0000","last_active":"2024-04-26T21:28:57.143833+0000","last_peered":"2024-04-26T21:28:57.143833+0000","last_clean":"2024-04-26T21:28:57.143833+0000","last_became_active":"2024-04-26T21:28:57.142433+0000","last_became_peered":"2024-04-26T21:28:57.142433+0000","last_unstale":"2024-04-26T21:28:57.143833+0000","last_undegraded":"2024-04-26T21:28:57.143833+0000","last_fullsized":"2024-04-26T21:28:57.143833+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:41:45.191169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":146,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:29:16.896126+0000","last_change":"2024-04-26T21:27:02.317413+0000","last_active":"2024-04-26T21:29:16.896126+0000","last_peered":"2024-04-26T21:29:16.896126+0000","last_clean":"2024-04-26T21:29:16.896126+0000","last_became_active":"2024-04-26T21:27:02.316510+0000","last_became_peered":"2024-04-26T21:27:02.316510+0000","last_unstale":"2024-04-26T21:29:16.896126+0000","last_undegraded":"2024-04-26T21:29:16.896126+0000","last_fullsized":"2024-04-26T21:29:16.896126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:52:49.156366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":24,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141147+0000","last_change":"2024-04-26T21:27:07.373434+0000","last_active":"2024-04-26T21:28:57.141147+0000","last_peered":"2024-04-26T21:28:57.141147+0000","last_clean":"2024-04-26T21:28:57.141147+0000","last_became_active":"2024-04-26T21:27:07.372973+0000","last_became_peered":"2024-04-26T21:27:07.372973+0000","last_unstale":"2024-04-26T21:28:57.141147+0000","last_undegraded":"2024-04-26T21:28:57.141147+0000","last_fullsized":"2024-04-26T21:28:57.141147+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:08:04.022651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.133271+0000","last_change":"2024-04-26T21:27:02.862734+0000","last_active":"2024-04-26T21:28:57.133271+0000","last_peered":"2024-04-26T21:28:57.133271+0000","last_clean":"2024-04-26T21:28:57.133271+0000","last_became_active":"2024-04-26T21:27:02.862334+0000","last_became_peered":"2024-04-26T21:27:02.862334+0000","last_unstale":"2024-04-26T21:28:57.133271+0000","last_undegraded":"2024-04-26T21:28:57.133271+0000","last_fullsized":"2024-04-26T21:28:57.133271+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:37:09.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":37,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089528+0000","last_change":"2024-04-26T21:27:02.302471+0000","last_active":"2024-04-26T21:28:57.089528+0000","last_peered":"2024-04-26T21:28:57.089528+0000","last_clean":"2024-04-26T21:28:57.089528+0000","last_became_active":"2024-04-26T21:27:02.302055+0000","last_became_peered":"2024-04-26T21:27:02.302055+0000","last_unstale":"2024-04-26T21:28:57.089528+0000","last_undegraded":"2024-04-26T21:28:57.089528+0000","last_fullsized":"2024-04-26T21:28:57.089528+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:29:14.649670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":367,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141165+0000","last_change":"2024-04-26T21:25:41.647633+0000","last_active":"2024-04-26T21:28:57.141165+0000","last_peered":"2024-04-26T21:28:57.141165+0000","last_clean":"2024-04-26T21:28:57.141165+0000","last_became_active":"2024-04-26T21:25:41.647091+0000","last_became_peered":"2024-04-26T21:25:41.647091+0000","last_unstale":"2024-04-26T21:28:57.141165+0000","last_undegraded":"2024-04-26T21:28:57.141165+0000","last_fullsized":"2024-04-26T21:28:57.141165+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:21:43.547929+0000","last_clean_scrub_stamp":"2024-04-26T21:21:43.547929+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:34:37.666065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.153287+0000","last_change":"2024-04-26T21:28:57.152820+0000","last_active":"2024-04-26T21:28:57.153287+0000","last_peered":"2024-04-26T21:28:57.153287+0000","last_clean":"2024-04-26T21:28:57.153287+0000","last_became_active":"2024-04-26T21:28:57.152453+0000","last_became_peered":"2024-04-26T21:28:57.152453+0000","last_unstale":"2024-04-26T21:28:57.153287+0000","last_undegraded":"2024-04-26T21:28:57.153287+0000","last_fullsized":"2024-04-26T21:28:57.153287+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:16:47.634651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.147632+0000","last_change":"2024-04-26T21:27:02.874116+0000","last_active":"2024-04-26T21:28:57.147632+0000","last_peered":"2024-04-26T21:28:57.147632+0000","last_clean":"2024-04-26T21:28:57.147632+0000","last_became_active":"2024-04-26T21:27:02.871334+0000","last_became_peered":"2024-04-26T21:27:02.871334+0000","last_unstale":"2024-04-26T21:28:57.147632+0000","last_undegraded":"2024-04-26T21:28:57.147632+0000","last_fullsized":"2024-04-26T21:28:57.147632+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:34:28.835061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120084+0000","last_change":"2024-04-26T21:27:56.847702+0000","last_active":"2024-04-26T21:28:57.120084+0000","last_peered":"2024-04-26T21:28:57.120084+0000","last_clean":"2024-04-26T21:28:57.120084+0000","last_became_active":"2024-04-26T21:27:56.844034+0000","last_became_peered":"2024-04-26T21:27:56.844034+0000","last_unstale":"2024-04-26T21:28:57.120084+0000","last_undegraded":"2024-04-26T21:28:57.120084+0000","last_fullsized":"2024-04-26T21:28:57.120084+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:25:54.631297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120284+0000","last_change":"2024-04-26T21:27:07.373894+0000","last_active":"2024-04-26T21:28:57.120284+0000","last_peered":"2024-04-26T21:28:57.120284+0000","last_clean":"2024-04-26T21:28:57.120284+0000","last_became_active":"2024-04-26T21:27:07.373338+0000","last_became_peered":"2024-04-26T21:27:07.373338+0000","last_unstale":"2024-04-26T21:28:57.120284+0000","last_undegraded":"2024-04-26T21:28:57.120284+0000","last_fullsized":"2024-04-26T21:28:57.120284+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:57:49.636493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":19,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120527+0000","last_change":"2024-04-26T21:27:56.858975+0000","last_active":"2024-04-26T21:28:57.120527+0000","last_peered":"2024-04-26T21:28:57.120527+0000","last_clean":"2024-04-26T21:28:57.120527+0000","last_became_active":"2024-04-26T21:27:56.850022+0000","last_became_peered":"2024-04-26T21:27:56.850022+0000","last_unstale":"2024-04-26T21:28:57.120527+0000","last_undegraded":"2024-04-26T21:28:57.120527+0000","last_fullsized":"2024-04-26T21:28:57.120527+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:43:31.860906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.148649+0000","last_change":"2024-04-26T21:27:02.869831+0000","last_active":"2024-04-26T21:28:57.148649+0000","last_peered":"2024-04-26T21:28:57.148649+0000","last_clean":"2024-04-26T21:28:57.148649+0000","last_became_active":"2024-04-26T21:27:02.869375+0000","last_became_peered":"2024-04-26T21:27:02.869375+0000","last_unstale":"2024-04-26T21:28:57.148649+0000","last_undegraded":"2024-04-26T21:28:57.148649+0000","last_fullsized":"2024-04-26T21:28:57.148649+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:01:20.571408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.132837+0000","last_change":"2024-04-26T21:27:07.507562+0000","last_active":"2024-04-26T21:28:57.132837+0000","last_peered":"2024-04-26T21:28:57.132837+0000","last_clean":"2024-04-26T21:28:57.132837+0000","last_became_active":"2024-04-26T21:27:07.507063+0000","last_became_peered":"2024-04-26T21:27:07.507063+0000","last_unstale":"2024-04-26T21:28:57.132837+0000","last_undegraded":"2024-04-26T21:28:57.132837+0000","last_fullsized":"2024-04-26T21:28:57.132837+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T08:42:58.192617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'9","reported_seq":41,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.094689+0000","last_change":"2024-04-26T21:27:02.315360+0000","last_active":"2024-04-26T21:28:57.094689+0000","last_peered":"2024-04-26T21:28:57.094689+0000","last_clean":"2024-04-26T21:28:57.094689+0000","last_became_active":"2024-04-26T21:27:02.314817+0000","last_became_peered":"2024-04-26T21:27:02.314817+0000","last_unstale":"2024-04-26T21:28:57.094689+0000","last_undegraded":"2024-04-26T21:28:57.094689+0000","last_fullsized":"2024-04-26T21:28:57.094689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:23:12.045363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.145131+0000","last_change":"2024-04-26T21:28:57.145131+0000","last_active":"2024-04-26T21:28:57.145131+0000","last_peered":"2024-04-26T21:28:57.145131+0000","last_clean":"2024-04-26T21:28:57.145131+0000","last_became_active":"2024-04-26T21:28:57.144059+0000","last_became_peered":"2024-04-26T21:28:57.144059+0000","last_unstale":"2024-04-26T21:28:57.145131+0000","last_undegraded":"2024-04-26T21:28:57.145131+0000","last_fullsized":"2024-04-26T21:28:57.145131+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:11:12.442030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140055+0000","last_change":"2024-04-26T21:27:07.362016+0000","last_active":"2024-04-26T21:28:57.140055+0000","last_peered":"2024-04-26T21:28:57.140055+0000","last_clean":"2024-04-26T21:28:57.140055+0000","last_became_active":"2024-04-26T21:27:07.361360+0000","last_became_peered":"2024-04-26T21:27:07.361360+0000","last_unstale":"2024-04-26T21:28:57.140055+0000","last_undegraded":"2024-04-26T21:28:57.140055+0000","last_fullsized":"2024-04-26T21:28:57.140055+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:02:03.196293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089626+0000","last_change":"2024-04-26T21:27:02.308088+0000","last_active":"2024-04-26T21:28:57.089626+0000","last_peered":"2024-04-26T21:28:57.089626+0000","last_clean":"2024-04-26T21:28:57.089626+0000","last_became_active":"2024-04-26T21:27:02.307727+0000","last_became_peered":"2024-04-26T21:27:02.307727+0000","last_unstale":"2024-04-26T21:28:57.089626+0000","last_undegraded":"2024-04-26T21:28:57.089626+0000","last_fullsized":"2024-04-26T21:28:57.089626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:23:02.132125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":42,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.155086+0000","last_change":"2024-04-26T21:28:57.153650+0000","last_active":"2024-04-26T21:28:57.155086+0000","last_peered":"2024-04-26T21:28:57.155086+0000","last_clean":"2024-04-26T21:28:57.155086+0000","last_became_active":"2024-04-26T21:28:57.152541+0000","last_became_peered":"2024-04-26T21:28:57.152541+0000","last_unstale":"2024-04-26T21:28:57.155086+0000","last_undegraded":"2024-04-26T21:28:57.155086+0000","last_fullsized":"2024-04-26T21:28:57.155086+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:44:53.236806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089207+0000","last_change":"2024-04-26T21:27:07.506609+0000","last_active":"2024-04-26T21:28:57.089207+0000","last_peered":"2024-04-26T21:28:57.089207+0000","last_clean":"2024-04-26T21:28:57.089207+0000","last_became_active":"2024-04-26T21:27:07.506100+0000","last_became_peered":"2024-04-26T21:27:07.506100+0000","last_unstale":"2024-04-26T21:28:57.089207+0000","last_undegraded":"2024-04-26T21:28:57.089207+0000","last_fullsized":"2024-04-26T21:28:57.089207+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:44:26.535809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142174+0000","last_change":"2024-04-26T21:27:02.297432+0000","last_active":"2024-04-26T21:28:57.142174+0000","last_peered":"2024-04-26T21:28:57.142174+0000","last_clean":"2024-04-26T21:28:57.142174+0000","last_became_active":"2024-04-26T21:27:02.297081+0000","last_became_peered":"2024-04-26T21:27:02.297081+0000","last_unstale":"2024-04-26T21:28:57.142174+0000","last_undegraded":"2024-04-26T21:28:57.142174+0000","last_fullsized":"2024-04-26T21:28:57.142174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:08.139469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083414+0000","last_change":"2024-04-26T21:27:02.866653+0000","last_active":"2024-04-26T21:28:57.083414+0000","last_peered":"2024-04-26T21:28:57.083414+0000","last_clean":"2024-04-26T21:28:57.083414+0000","last_became_active":"2024-04-26T21:27:02.866275+0000","last_became_peered":"2024-04-26T21:27:02.866275+0000","last_unstale":"2024-04-26T21:28:57.083414+0000","last_undegraded":"2024-04-26T21:28:57.083414+0000","last_fullsized":"2024-04-26T21:28:57.083414+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:04:39.089972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.143381+0000","last_change":"2024-04-26T21:28:57.142863+0000","last_active":"2024-04-26T21:28:57.143381+0000","last_peered":"2024-04-26T21:28:57.143381+0000","last_clean":"2024-04-26T21:28:57.143381+0000","last_became_active":"2024-04-26T21:28:57.141745+0000","last_became_peered":"2024-04-26T21:28:57.141745+0000","last_unstale":"2024-04-26T21:28:57.143381+0000","last_undegraded":"2024-04-26T21:28:57.143381+0000","last_fullsized":"2024-04-26T21:28:57.143381+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:42:07.261644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":35,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134498+0000","last_change":"2024-04-26T21:27:02.307250+0000","last_active":"2024-04-26T21:28:57.134498+0000","last_peered":"2024-04-26T21:28:57.134498+0000","last_clean":"2024-04-26T21:28:57.134498+0000","last_became_active":"2024-04-26T21:27:02.303532+0000","last_became_peered":"2024-04-26T21:27:02.303532+0000","last_unstale":"2024-04-26T21:28:57.134498+0000","last_undegraded":"2024-04-26T21:28:57.134498+0000","last_fullsized":"2024-04-26T21:28:57.134498+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:38:36.265188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120235+0000","last_change":"2024-04-26T21:27:02.858818+0000","last_active":"2024-04-26T21:28:57.120235+0000","last_peered":"2024-04-26T21:28:57.120235+0000","last_clean":"2024-04-26T21:28:57.120235+0000","last_became_active":"2024-04-26T21:27:02.858059+0000","last_became_peered":"2024-04-26T21:27:02.858059+0000","last_unstale":"2024-04-26T21:28:57.120235+0000","last_undegraded":"2024-04-26T21:28:57.120235+0000","last_fullsized":"2024-04-26T21:28:57.120235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:00:02.159334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088650+0000","last_change":"2024-04-26T21:27:07.381273+0000","last_active":"2024-04-26T21:28:57.088650+0000","last_peered":"2024-04-26T21:28:57.088650+0000","last_clean":"2024-04-26T21:28:57.088650+0000","last_became_active":"2024-04-26T21:27:07.380891+0000","last_became_peered":"2024-04-26T21:27:07.380891+0000","last_unstale":"2024-04-26T21:28:57.088650+0000","last_undegraded":"2024-04-26T21:28:57.088650+0000","last_fullsized":"2024-04-26T21:28:57.088650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:20:39.641577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":33,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140420+0000","last_change":"2024-04-26T21:27:02.301363+0000","last_active":"2024-04-26T21:28:57.140420+0000","last_peered":"2024-04-26T21:28:57.140420+0000","last_clean":"2024-04-26T21:28:57.140420+0000","last_became_active":"2024-04-26T21:27:02.301059+0000","last_became_peered":"2024-04-26T21:27:02.301059+0000","last_unstale":"2024-04-26T21:28:57.140420+0000","last_undegraded":"2024-04-26T21:28:57.140420+0000","last_fullsized":"2024-04-26T21:28:57.140420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:17.586190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134340+0000","last_change":"2024-04-26T21:27:02.860612+0000","last_active":"2024-04-26T21:28:57.134340+0000","last_peered":"2024-04-26T21:28:57.134340+0000","last_clean":"2024-04-26T21:28:57.134340+0000","last_became_active":"2024-04-26T21:27:02.857285+0000","last_became_peered":"2024-04-26T21:27:02.857285+0000","last_unstale":"2024-04-26T21:28:57.134340+0000","last_undegraded":"2024-04-26T21:28:57.134340+0000","last_fullsized":"2024-04-26T21:28:57.134340+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:27:11.968818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089342+0000","last_change":"2024-04-26T21:27:07.506425+0000","last_active":"2024-04-26T21:28:57.089342+0000","last_peered":"2024-04-26T21:28:57.089342+0000","last_clean":"2024-04-26T21:28:57.089342+0000","last_became_active":"2024-04-26T21:27:07.505968+0000","last_became_peered":"2024-04-26T21:27:07.505968+0000","last_unstale":"2024-04-26T21:28:57.089342+0000","last_undegraded":"2024-04-26T21:28:57.089342+0000","last_fullsized":"2024-04-26T21:28:57.089342+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:40:59.914560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.083503+0000","last_change":"2024-04-26T21:27:02.294635+0000","last_active":"2024-04-26T21:28:57.083503+0000","last_peered":"2024-04-26T21:28:57.083503+0000","last_clean":"2024-04-26T21:28:57.083503+0000","last_became_active":"2024-04-26T21:27:02.294311+0000","last_became_peered":"2024-04-26T21:27:02.294311+0000","last_unstale":"2024-04-26T21:28:57.083503+0000","last_undegraded":"2024-04-26T21:28:57.083503+0000","last_fullsized":"2024-04-26T21:28:57.083503+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:00:47.906546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141796+0000","last_change":"2024-04-26T21:27:02.875475+0000","last_active":"2024-04-26T21:28:57.141796+0000","last_peered":"2024-04-26T21:28:57.141796+0000","last_clean":"2024-04-26T21:28:57.141796+0000","last_became_active":"2024-04-26T21:27:02.875137+0000","last_became_peered":"2024-04-26T21:27:02.875137+0000","last_unstale":"2024-04-26T21:28:57.141796+0000","last_undegraded":"2024-04-26T21:28:57.141796+0000","last_fullsized":"2024-04-26T21:28:57.141796+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:08:04.988973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141646+0000","last_change":"2024-04-26T21:27:07.372927+0000","last_active":"2024-04-26T21:28:57.141646+0000","last_peered":"2024-04-26T21:28:57.141646+0000","last_clean":"2024-04-26T21:28:57.141646+0000","last_became_active":"2024-04-26T21:27:07.372371+0000","last_became_peered":"2024-04-26T21:27:07.372371+0000","last_unstale":"2024-04-26T21:28:57.141646+0000","last_undegraded":"2024-04-26T21:28:57.141646+0000","last_fullsized":"2024-04-26T21:28:57.141646+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T02:12:41.607167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140712+0000","last_change":"2024-04-26T21:27:02.295379+0000","last_active":"2024-04-26T21:28:57.140712+0000","last_peered":"2024-04-26T21:28:57.140712+0000","last_clean":"2024-04-26T21:28:57.140712+0000","last_became_active":"2024-04-26T21:27:02.295077+0000","last_became_peered":"2024-04-26T21:27:02.295077+0000","last_unstale":"2024-04-26T21:28:57.140712+0000","last_undegraded":"2024-04-26T21:28:57.140712+0000","last_fullsized":"2024-04-26T21:28:57.140712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:06:58.177960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141479+0000","last_change":"2024-04-26T21:27:02.855998+0000","last_active":"2024-04-26T21:28:57.141479+0000","last_peered":"2024-04-26T21:28:57.141479+0000","last_clean":"2024-04-26T21:28:57.141479+0000","last_became_active":"2024-04-26T21:27:02.855636+0000","last_became_peered":"2024-04-26T21:27:02.855636+0000","last_unstale":"2024-04-26T21:28:57.141479+0000","last_undegraded":"2024-04-26T21:28:57.141479+0000","last_fullsized":"2024-04-26T21:28:57.141479+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T09:19:02.988946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.084647+0000","last_change":"2024-04-26T21:27:07.378980+0000","last_active":"2024-04-26T21:28:57.084647+0000","last_peered":"2024-04-26T21:28:57.084647+0000","last_clean":"2024-04-26T21:28:57.084647+0000","last_became_active":"2024-04-26T21:27:07.378374+0000","last_became_peered":"2024-04-26T21:27:07.378374+0000","last_unstale":"2024-04-26T21:28:57.084647+0000","last_undegraded":"2024-04-26T21:28:57.084647+0000","last_fullsized":"2024-04-26T21:28:57.084647+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:28:31.845297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.089873+0000","last_change":"2024-04-26T21:27:02.313801+0000","last_active":"2024-04-26T21:28:57.089873+0000","last_peered":"2024-04-26T21:28:57.089873+0000","last_clean":"2024-04-26T21:28:57.089873+0000","last_became_active":"2024-04-26T21:27:02.313540+0000","last_became_peered":"2024-04-26T21:27:02.313540+0000","last_unstale":"2024-04-26T21:28:57.089873+0000","last_undegraded":"2024-04-26T21:28:57.089873+0000","last_fullsized":"2024-04-26T21:28:57.089873+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:52:38.612261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140890+0000","last_change":"2024-04-26T21:27:02.864151+0000","last_active":"2024-04-26T21:28:57.140890+0000","last_peered":"2024-04-26T21:28:57.140890+0000","last_clean":"2024-04-26T21:28:57.140890+0000","last_became_active":"2024-04-26T21:27:02.863818+0000","last_became_peered":"2024-04-26T21:27:02.863818+0000","last_unstale":"2024-04-26T21:28:57.140890+0000","last_undegraded":"2024-04-26T21:28:57.140890+0000","last_fullsized":"2024-04-26T21:28:57.140890+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:04:26.215370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088627+0000","last_change":"2024-04-26T21:27:07.365821+0000","last_active":"2024-04-26T21:28:57.088627+0000","last_peered":"2024-04-26T21:28:57.088627+0000","last_clean":"2024-04-26T21:28:57.088627+0000","last_became_active":"2024-04-26T21:27:07.365347+0000","last_became_peered":"2024-04-26T21:27:07.365347+0000","last_unstale":"2024-04-26T21:28:57.088627+0000","last_undegraded":"2024-04-26T21:28:57.088627+0000","last_fullsized":"2024-04-26T21:28:57.088627+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:06.315068+0000","last_clean_scrub_stamp":"2024-04-26T21:27:06.315068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T21:40:11.596908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":34,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150203+0000","last_change":"2024-04-26T21:27:02.310641+0000","last_active":"2024-04-26T21:28:57.150203+0000","last_peered":"2024-04-26T21:28:57.150203+0000","last_clean":"2024-04-26T21:28:57.150203+0000","last_became_active":"2024-04-26T21:27:02.308569+0000","last_became_peered":"2024-04-26T21:27:02.308569+0000","last_unstale":"2024-04-26T21:28:57.150203+0000","last_undegraded":"2024-04-26T21:28:57.150203+0000","last_fullsized":"2024-04-26T21:28:57.150203+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:01.267526+0000","last_clean_scrub_stamp":"2024-04-26T21:27:01.267526+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T00:40:20.988098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141041+0000","last_change":"2024-04-26T21:27:02.854976+0000","last_active":"2024-04-26T21:28:57.141041+0000","last_peered":"2024-04-26T21:28:57.141041+0000","last_clean":"2024-04-26T21:28:57.141041+0000","last_became_active":"2024-04-26T21:27:02.854611+0000","last_became_peered":"2024-04-26T21:27:02.854611+0000","last_unstale":"2024-04-26T21:28:57.141041+0000","last_undegraded":"2024-04-26T21:28:57.141041+0000","last_fullsized":"2024-04-26T21:28:57.141041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T04:42:57.065657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120299+0000","last_change":"2024-04-26T21:27:02.859017+0000","last_active":"2024-04-26T21:28:57.120299+0000","last_peered":"2024-04-26T21:28:57.120299+0000","last_clean":"2024-04-26T21:28:57.120299+0000","last_became_active":"2024-04-26T21:27:02.858507+0000","last_became_peered":"2024-04-26T21:27:02.858507+0000","last_unstale":"2024-04-26T21:28:57.120299+0000","last_undegraded":"2024-04-26T21:28:57.120299+0000","last_fullsized":"2024-04-26T21:28:57.120299+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:47:14.850891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":36,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140440+0000","last_change":"2024-04-26T21:27:56.841251+0000","last_active":"2024-04-26T21:28:57.140440+0000","last_peered":"2024-04-26T21:28:57.140440+0000","last_clean":"2024-04-26T21:28:57.140440+0000","last_became_active":"2024-04-26T21:27:56.840862+0000","last_became_peered":"2024-04-26T21:27:56.840862+0000","last_unstale":"2024-04-26T21:28:57.140440+0000","last_undegraded":"2024-04-26T21:28:57.140440+0000","last_fullsized":"2024-04-26T21:28:57.140440+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T03:36:41.460343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.140223+0000","last_change":"2024-04-26T21:27:02.868817+0000","last_active":"2024-04-26T21:28:57.140223+0000","last_peered":"2024-04-26T21:28:57.140223+0000","last_clean":"2024-04-26T21:28:57.140223+0000","last_became_active":"2024-04-26T21:27:02.868557+0000","last_became_peered":"2024-04-26T21:27:02.868557+0000","last_unstale":"2024-04-26T21:28:57.140223+0000","last_undegraded":"2024-04-26T21:28:57.140223+0000","last_fullsized":"2024-04-26T21:28:57.140223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:34:55.088889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.085900+0000","last_change":"2024-04-26T21:27:02.876029+0000","last_active":"2024-04-26T21:28:57.085900+0000","last_peered":"2024-04-26T21:28:57.085900+0000","last_clean":"2024-04-26T21:28:57.085900+0000","last_became_active":"2024-04-26T21:27:02.875656+0000","last_became_peered":"2024-04-26T21:27:02.875656+0000","last_unstale":"2024-04-26T21:28:57.085900+0000","last_undegraded":"2024-04-26T21:28:57.085900+0000","last_fullsized":"2024-04-26T21:28:57.085900+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:55:47.317581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.150726+0000","last_change":"2024-04-26T21:27:02.873818+0000","last_active":"2024-04-26T21:28:57.150726+0000","last_peered":"2024-04-26T21:28:57.150726+0000","last_clean":"2024-04-26T21:28:57.150726+0000","last_became_active":"2024-04-26T21:27:02.873459+0000","last_became_peered":"2024-04-26T21:27:02.873459+0000","last_unstale":"2024-04-26T21:28:57.150726+0000","last_undegraded":"2024-04-26T21:28:57.150726+0000","last_fullsized":"2024-04-26T21:28:57.150726+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T06:47:07.259187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120728+0000","last_change":"2024-04-26T21:27:02.854901+0000","last_active":"2024-04-26T21:28:57.120728+0000","last_peered":"2024-04-26T21:28:57.120728+0000","last_clean":"2024-04-26T21:28:57.120728+0000","last_became_active":"2024-04-26T21:27:02.854547+0000","last_became_peered":"2024-04-26T21:27:02.854547+0000","last_unstale":"2024-04-26T21:28:57.120728+0000","last_undegraded":"2024-04-26T21:28:57.120728+0000","last_fullsized":"2024-04-26T21:28:57.120728+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:28:24.320868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.141047+0000","last_change":"2024-04-26T21:27:02.856616+0000","last_active":"2024-04-26T21:28:57.141047+0000","last_peered":"2024-04-26T21:28:57.141047+0000","last_clean":"2024-04-26T21:28:57.141047+0000","last_became_active":"2024-04-26T21:27:02.856264+0000","last_became_peered":"2024-04-26T21:27:02.856264+0000","last_unstale":"2024-04-26T21:28:57.141047+0000","last_undegraded":"2024-04-26T21:28:57.141047+0000","last_fullsized":"2024-04-26T21:28:57.141047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T23:04:43.374534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134194+0000","last_change":"2024-04-26T21:27:02.859587+0000","last_active":"2024-04-26T21:28:57.134194+0000","last_peered":"2024-04-26T21:28:57.134194+0000","last_clean":"2024-04-26T21:28:57.134194+0000","last_became_active":"2024-04-26T21:27:02.859136+0000","last_became_peered":"2024-04-26T21:27:02.859136+0000","last_unstale":"2024-04-26T21:28:57.134194+0000","last_undegraded":"2024-04-26T21:28:57.134194+0000","last_fullsized":"2024-04-26T21:28:57.134194+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:09.370319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.088809+0000","last_change":"2024-04-26T21:27:02.864265+0000","last_active":"2024-04-26T21:28:57.088809+0000","last_peered":"2024-04-26T21:28:57.088809+0000","last_clean":"2024-04-26T21:28:57.088809+0000","last_became_active":"2024-04-26T21:27:02.863920+0000","last_became_peered":"2024-04-26T21:27:02.863920+0000","last_unstale":"2024-04-26T21:28:57.088809+0000","last_undegraded":"2024-04-26T21:28:57.088809+0000","last_fullsized":"2024-04-26T21:28:57.088809+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T22:01:13.941544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":29,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.134222+0000","last_change":"2024-04-26T21:27:02.860189+0000","last_active":"2024-04-26T21:28:57.134222+0000","last_peered":"2024-04-26T21:28:57.134222+0000","last_clean":"2024-04-26T21:28:57.134222+0000","last_became_active":"2024-04-26T21:27:02.859843+0000","last_became_peered":"2024-04-26T21:27:02.859843+0000","last_unstale":"2024-04-26T21:28:57.134222+0000","last_undegraded":"2024-04-26T21:28:57.134222+0000","last_fullsized":"2024-04-26T21:28:57.134222+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:14:53.490628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.142017+0000","last_change":"2024-04-26T21:27:02.865459+0000","last_active":"2024-04-26T21:28:57.142017+0000","last_peered":"2024-04-26T21:28:57.142017+0000","last_clean":"2024-04-26T21:28:57.142017+0000","last_became_active":"2024-04-26T21:27:02.863602+0000","last_became_peered":"2024-04-26T21:27:02.863602+0000","last_unstale":"2024-04-26T21:28:57.142017+0000","last_undegraded":"2024-04-26T21:28:57.142017+0000","last_fullsized":"2024-04-26T21:28:57.142017+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T05:28:03.733858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.149656+0000","last_change":"2024-04-26T21:27:02.873711+0000","last_active":"2024-04-26T21:28:57.149656+0000","last_peered":"2024-04-26T21:28:57.149656+0000","last_clean":"2024-04-26T21:28:57.149656+0000","last_became_active":"2024-04-26T21:27:02.873349+0000","last_became_peered":"2024-04-26T21:27:02.873349+0000","last_unstale":"2024-04-26T21:28:57.149656+0000","last_undegraded":"2024-04-26T21:28:57.149656+0000","last_fullsized":"2024-04-26T21:28:57.149656+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T07:01:10.841536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-04-26T21:28:57.120452+0000","last_change":"2024-04-26T21:27:02.852625+0000","last_active":"2024-04-26T21:28:57.120452+0000","last_peered":"2024-04-26T21:28:57.120452+0000","last_clean":"2024-04-26T21:28:57.120452+0000","last_became_active":"2024-04-26T21:27:02.852177+0000","last_became_peered":"2024-04-26T21:27:02.852177+0000","last_unstale":"2024-04-26T21:28:57.120452+0000","last_undegraded":"2024-04-26T21:28:57.120452+0000","last_fullsized":"2024-04-26T21:28:57.120452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T21:27:02.271863+0000","last_clean_scrub_stamp":"2024-04-26T21:27:02.271863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-28T01:11:52.046678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":59286,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":683,"num_read_kb":405,"num_write":164,"num_write_kb":109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":22,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":177858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318314,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":528427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579951,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28740,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716412,"statfs":{"total":95995035648,"available":95965605888,"internally_reserved":0,"allocated":1761280,"data_stored":1171349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743476,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1097728,"data_stored":532737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005115,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1089536,"data_stored":528539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266752,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32832,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712320,"statfs":{"total":95995035648,"available":95961415680,"internally_reserved":0,"allocated":1687552,"data_stored":1124079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462983,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1073152,"data_stored":523817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1134592,"data_stored":579403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36416,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708736,"statfs":{"total":95995035648,"available":95957745664,"internally_reserved":0,"allocated":1097728,"data_stored":525205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1159168,"data_stored":580753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1085440,"data_stored":524011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888146,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1085440,"data_stored":528467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724620,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37000,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708152,"statfs":{"total":95995035648,"available":95957147648,"internally_reserved":0,"allocated":1695744,"data_stored":1119667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":55676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":57026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":57254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T21:29:21.211 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T21:29:21.212 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T21:29:21.212 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T21:29:21.212 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T21:29:21.330 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:21 smithi001 ceph-mon[28054]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:21.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:21 smithi186 ceph-mon[32991]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:21.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:21 smithi038 ceph-mon[32724]: pgmap v397: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:21.749 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:22 smithi186 ceph-mon[32991]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:22.479 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:22 smithi001 ceph-mon[28054]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:22.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:22 smithi038 ceph-mon[32724]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T21:29:23.380 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:23 smithi001 ceph-mon[28054]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:23.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:23 smithi186 ceph-mon[32991]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:23.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:23 smithi038 ceph-mon[32724]: pgmap v398: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.896+0000 7f269f200700 1 -- 172.21.15.1:0/1968693521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26a0108750 msgr2=0x7f26a0110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.896+0000 7f269f200700 1 --2- 172.21.15.1:0/1968693521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26a0108750 0x7f26a0110290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f269800b0a0 tx=0x7f269800b3b0 comp rx=0 tx=0).stop 2024-04-26T21:29:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.896+0000 7f269f200700 1 -- 172.21.15.1:0/1968693521 shutdown_connections 2024-04-26T21:29:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.896+0000 7f269f200700 1 --2- 172.21.15.1:0/1968693521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26a0108750 0x7f26a0110290 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.896+0000 7f269f200700 1 --2- 172.21.15.1:0/1968693521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a006bbf0 0x7f26a006bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.896+0000 7f269f200700 1 --2- 172.21.15.1:0/1968693521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a006c5a0 0x7f26a0108020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.896+0000 7f269f200700 1 -- 172.21.15.1:0/1968693521 >> 172.21.15.1:0/1968693521 conn(0x7f26a0079160 msgr2=0x7f26a0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:23.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.896+0000 7f269f200700 1 -- 172.21.15.1:0/1968693521 shutdown_connections 2024-04-26T21:29:23.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.896+0000 7f269f200700 1 -- 172.21.15.1:0/1968693521 wait complete. 2024-04-26T21:29:23.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.898+0000 7f269f200700 1 Processor -- start 2024-04-26T21:29:23.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.898+0000 7f269f200700 1 -- start start 2024-04-26T21:29:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.898+0000 7f269f200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a006bbf0 0x7f26a01025b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.898+0000 7f269f200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26a006c5a0 0x7f26a0102af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.898+0000 7f269f200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a0108750 0x7f26a0103030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.898+0000 7f269f200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f26a01135b0 con 0x7f26a006bbf0 2024-04-26T21:29:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.898+0000 7f269f200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f26a0113430 con 0x7f26a0108750 2024-04-26T21:29:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.898+0000 7f269f200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f26a0113730 con 0x7f26a006c5a0 2024-04-26T21:29:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a006bbf0 0x7f26a01025b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269de00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a006bbf0 0x7f26a01025b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44788/0 (socket says 172.21.15.1:44788) 2024-04-26T21:29:23.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269de00700 1 -- 172.21.15.1:0/4137816321 learned_addr learned my addr 172.21.15.1:0/4137816321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269d400700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26a006c5a0 0x7f26a0102af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269e800700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a0108750 0x7f26a0103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269de00700 1 -- 172.21.15.1:0/4137816321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26a006c5a0 msgr2=0x7f26a0102af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269de00700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26a006c5a0 0x7f26a0102af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269de00700 1 -- 172.21.15.1:0/4137816321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a0108750 msgr2=0x7f26a0103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269de00700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a0108750 0x7f26a0103030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269de00700 1 -- 172.21.15.1:0/4137816321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a01a7ef0 con 0x7f26a006bbf0 2024-04-26T21:29:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.899+0000 7f269e800700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a0108750 0x7f26a0103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:23.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.900+0000 7f269de00700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a006bbf0 0x7f26a01025b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f269000e9a0 tx=0x7f269000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.900+0000 7f2696a00700 1 -- 172.21.15.1:0/4137816321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269000ca10 con 0x7f26a006bbf0 2024-04-26T21:29:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.900+0000 7f269f200700 1 -- 172.21.15.1:0/4137816321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a01a81e0 con 0x7f26a006bbf0 2024-04-26T21:29:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.900+0000 7f2696a00700 1 -- 172.21.15.1:0/4137816321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f269000cbb0 con 0x7f26a006bbf0 2024-04-26T21:29:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.900+0000 7f2696a00700 1 -- 172.21.15.1:0/4137816321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690018680 con 0x7f26a006bbf0 2024-04-26T21:29:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.900+0000 7f269f200700 1 -- 172.21.15.1:0/4137816321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a01a8720 con 0x7f26a006bbf0 2024-04-26T21:29:23.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.901+0000 7f269f200700 1 -- 172.21.15.1:0/4137816321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a006a110 con 0x7f26a006bbf0 2024-04-26T21:29:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.903+0000 7f2696a00700 1 -- 172.21.15.1:0/4137816321 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f2690010430 con 0x7f26a006bbf0 2024-04-26T21:29:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.904+0000 7f2696a00700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f26840a3cd0 0x7f26840a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.905+0000 7f2696a00700 1 -- 172.21.15.1:0/4137816321 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f26900d0b50 con 0x7f26a006bbf0 2024-04-26T21:29:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.905+0000 7f269d400700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f26840a3cd0 0x7f26840a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:23.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.906+0000 7f269d400700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f26840a3cd0 0x7f26840a6190 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f26a0108110 tx=0x7f268c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:23.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:23.910+0000 7f2696a00700 1 -- 172.21.15.1:0/4137816321 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f269009cf70 con 0x7f26a006bbf0 2024-04-26T21:29:24.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.198+0000 7f269f200700 1 -- 172.21.15.1:0/4137816321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f26a002e490 con 0x7f26a006bbf0 2024-04-26T21:29:24.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.199+0000 7f2696a00700 1 -- 172.21.15.1:0/4137816321 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f269009c870 con 0x7f26a006bbf0 2024-04-26T21:29:24.200 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:24.200 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 -- 172.21.15.1:0/4137816321 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f26840a3cd0 msgr2=0x7f26840a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f26840a3cd0 0x7f26840a6190 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f26a0108110 tx=0x7f268c009260 comp rx=0 tx=0).stop 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 -- 172.21.15.1:0/4137816321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a006bbf0 msgr2=0x7f26a01025b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a006bbf0 0x7f26a01025b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f269000e9a0 tx=0x7f269000ecb0 comp rx=0 tx=0).stop 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 -- 172.21.15.1:0/4137816321 shutdown_connections 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f26840a3cd0 0x7f26840a6190 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a0108750 0x7f26a0103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26a006bbf0 0x7f26a01025b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 --2- 172.21.15.1:0/4137816321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26a006c5a0 0x7f26a0102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 -- 172.21.15.1:0/4137816321 >> 172.21.15.1:0/4137816321 conn(0x7f26a0079160 msgr2=0x7f26a0101cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:24.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.202+0000 7f269f200700 1 -- 172.21.15.1:0/4137816321 shutdown_connections 2024-04-26T21:29:24.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:24.203+0000 7f269f200700 1 -- 172.21.15.1:0/4137816321 wait complete. 2024-04-26T21:29:25.000 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T21:29:25.001 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:29:25.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.232+0000 7f6321000640 1 -- 172.21.15.1:0/1744140334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f631c109f80 msgr2=0x7f631c10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:25.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.232+0000 7f6321000640 1 --2- 172.21.15.1:0/1744140334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f631c109f80 0x7f631c10a400 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6308009900 tx=0x7f630802a410 comp rx=0 tx=0).stop 2024-04-26T21:29:25.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.232+0000 7f6321000640 1 -- 172.21.15.1:0/1744140334 shutdown_connections 2024-04-26T21:29:25.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.232+0000 7f6321000640 1 --2- 172.21.15.1:0/1744140334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f631c10a940 0x7f631c1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.232+0000 7f6321000640 1 --2- 172.21.15.1:0/1744140334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f631c109f80 0x7f631c10a400 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.232+0000 7f6321000640 1 --2- 172.21.15.1:0/1744140334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f631c108d90 0x7f631c1091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.232+0000 7f6321000640 1 -- 172.21.15.1:0/1744140334 >> 172.21.15.1:0/1744140334 conn(0x7f631c077d30 msgr2=0x7f631c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:25.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.232+0000 7f6321000640 1 -- 172.21.15.1:0/1744140334 shutdown_connections 2024-04-26T21:29:25.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.232+0000 7f6321000640 1 -- 172.21.15.1:0/1744140334 wait complete. 2024-04-26T21:29:25.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.233+0000 7f6321000640 1 Processor -- start 2024-04-26T21:29:25.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.233+0000 7f6321000640 1 -- start start 2024-04-26T21:29:25.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f6321000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f631c108d90 0x7f631c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:25.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f6321000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f631c109f80 0x7f631c19f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:25.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f6321000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f631c10a940 0x7f631c1a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:25.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f6321000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f631c1148c0 con 0x7f631c109f80 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f6321000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f631c114740 con 0x7f631c10a940 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f6321000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f631c114a40 con 0x7f631c108d90 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f631aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f631c109f80 0x7f631c19f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f631b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f631c108d90 0x7f631c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f631aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f631c109f80 0x7f631c19f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44796/0 (socket says 172.21.15.1:44796) 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f631aa00640 1 -- 172.21.15.1:0/3632131512 learned_addr learned my addr 172.21.15.1:0/3632131512 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f631be00640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f631c10a940 0x7f631c1a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f631aa00640 1 -- 172.21.15.1:0/3632131512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f631c108d90 msgr2=0x7f631c19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f631aa00640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f631c108d90 0x7f631c19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.234+0000 7f631aa00640 1 -- 172.21.15.1:0/3632131512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f631c10a940 msgr2=0x7f631c1a64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.235+0000 7f631aa00640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f631c10a940 0x7f631c1a64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.235+0000 7f631aa00640 1 -- 172.21.15.1:0/3632131512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631c1a6b50 con 0x7f631c109f80 2024-04-26T21:29:25.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.235+0000 7f631be00640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f631c10a940 0x7f631c1a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:25.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.235+0000 7f631aa00640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f631c109f80 0x7f631c19f510 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6308006010 tx=0x7f63080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:25.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.235+0000 7f6313e00640 1 -- 172.21.15.1:0/3632131512 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630803f030 con 0x7f631c109f80 2024-04-26T21:29:25.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.235+0000 7f6313e00640 1 -- 172.21.15.1:0/3632131512 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6308031910 con 0x7f631c109f80 2024-04-26T21:29:25.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.235+0000 7f6313e00640 1 -- 172.21.15.1:0/3632131512 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308035070 con 0x7f631c109f80 2024-04-26T21:29:25.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.235+0000 7f6321000640 1 -- 172.21.15.1:0/3632131512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631c1a6de0 con 0x7f631c109f80 2024-04-26T21:29:25.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.235+0000 7f6321000640 1 -- 172.21.15.1:0/3632131512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f631c1a7270 con 0x7f631c109f80 2024-04-26T21:29:25.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.237+0000 7f6321000640 1 -- 172.21.15.1:0/3632131512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f631c110050 con 0x7f631c109f80 2024-04-26T21:29:25.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.239+0000 7f6313e00640 1 -- 172.21.15.1:0/3632131512 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f6308042030 con 0x7f631c109f80 2024-04-26T21:29:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.240+0000 7f6313e00640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f63000a3cf0 0x7f63000a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.240+0000 7f6313e00640 1 -- 172.21.15.1:0/3632131512 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f63080f0e40 con 0x7f631c109f80 2024-04-26T21:29:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.241+0000 7f631b400640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f63000a3cf0 0x7f63000a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:25.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.241+0000 7f631b400640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f63000a3cf0 0x7f63000a61b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f630c0095f0 tx=0x7f630c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.245+0000 7f6313e00640 1 -- 172.21.15.1:0/3632131512 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f63080bcc10 con 0x7f631c109f80 2024-04-26T21:29:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:25 smithi001 ceph-mon[28054]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:25 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4137816321' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T21:29:25.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:25 smithi186 ceph-mon[32991]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:25 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4137816321' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T21:29:25.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.454+0000 7f6321000640 1 -- 172.21.15.1:0/3632131512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f631c02dd40 con 0x7f631c109f80 2024-04-26T21:29:25.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.457+0000 7f6313e00640 1 -- 172.21.15.1:0/3632131512 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24767 (secure 0 0 0) 0x7f63080c0f10 con 0x7f631c109f80 2024-04-26T21:29:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:25.458 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":95,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:28:57.077487+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7},{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:29:25.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.459+0000 7f6321000640 1 -- 172.21.15.1:0/3632131512 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f63000a3cf0 msgr2=0x7f63000a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:25.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f63000a3cf0 0x7f63000a61b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f630c0095f0 tx=0x7f630c005f90 comp rx=0 tx=0).stop 2024-04-26T21:29:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 -- 172.21.15.1:0/3632131512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f631c109f80 msgr2=0x7f631c19f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f631c109f80 0x7f631c19f510 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6308006010 tx=0x7f63080042f0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 -- 172.21.15.1:0/3632131512 shutdown_connections 2024-04-26T21:29:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f63000a3cf0 0x7f63000a61b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f631c10a940 0x7f631c1a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f631c109f80 0x7f631c19f510 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 --2- 172.21.15.1:0/3632131512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f631c108d90 0x7f631c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 -- 172.21.15.1:0/3632131512 >> 172.21.15.1:0/3632131512 conn(0x7f631c077d30 msgr2=0x7f631c10d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 -- 172.21.15.1:0/3632131512 shutdown_connections 2024-04-26T21:29:25.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.460+0000 7f6321000640 1 -- 172.21.15.1:0/3632131512 wait complete. 2024-04-26T21:29:25.483 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:27:00.381106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:27:01.743898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T21:27:05.341216+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T21:29:25.483 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:29:25.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:25 smithi038 ceph-mon[32724]: pgmap v399: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:25.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:25 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4137816321' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T21:29:25.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.700+0000 7f0900a00640 1 -- 172.21.15.1:0/3331620446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08fc075390 msgr2=0x7f08fc075810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:25.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.700+0000 7f0900a00640 1 --2- 172.21.15.1:0/3331620446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08fc075390 0x7f08fc075810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f08e4009900 tx=0x7f08e402a3f0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.700+0000 7f0900a00640 1 -- 172.21.15.1:0/3331620446 shutdown_connections 2024-04-26T21:29:25.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.700+0000 7f0900a00640 1 --2- 172.21.15.1:0/3331620446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08fc075390 0x7f08fc075810 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.700+0000 7f0900a00640 1 --2- 172.21.15.1:0/3331620446 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc10a660 0x7f08fc10caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.700+0000 7f0900a00640 1 --2- 172.21.15.1:0/3331620446 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08fc076970 0x7f08fc074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.700+0000 7f0900a00640 1 -- 172.21.15.1:0/3331620446 >> 172.21.15.1:0/3331620446 conn(0x7f08fc0fdd10 msgr2=0x7f08fc100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:25.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.700+0000 7f0900a00640 1 -- 172.21.15.1:0/3331620446 shutdown_connections 2024-04-26T21:29:25.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.701+0000 7f0900a00640 1 -- 172.21.15.1:0/3331620446 wait complete. 2024-04-26T21:29:25.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.702+0000 7f0900a00640 1 Processor -- start 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.702+0000 7f0900a00640 1 -- start start 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.702+0000 7f0900a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08fc075390 0x7f08fc19ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.702+0000 7f0900a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08fc076970 0x7f08fc19b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.702+0000 7f0900a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc10a660 0x7f08fc1a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.702+0000 7f0900a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc110500 con 0x7f08fc10a660 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.702+0000 7f0900a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc110380 con 0x7f08fc076970 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.702+0000 7f0900a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc110680 con 0x7f08fc075390 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08f1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08fc076970 0x7f08fc19b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08fb200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc10a660 0x7f08fc1a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08fa800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08fc075390 0x7f08fc19ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08f1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08fc076970 0x7f08fc19b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37460/0 (socket says 172.21.15.1:37460) 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08fa800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08fc075390 0x7f08fc19ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50730/0 (socket says 172.21.15.1:50730) 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08f1e00640 1 -- 172.21.15.1:0/3570358603 learned_addr learned my addr 172.21.15.1:0/3570358603 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:25.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08f1e00640 1 -- 172.21.15.1:0/3570358603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08fc075390 msgr2=0x7f08fc19ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08f1e00640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08fc075390 0x7f08fc19ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08f1e00640 1 -- 172.21.15.1:0/3570358603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc10a660 msgr2=0x7f08fc1a2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08f1e00640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc10a660 0x7f08fc1a2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.703+0000 7f08f1e00640 1 -- 172.21.15.1:0/3570358603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08fc1a2830 con 0x7f08fc076970 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.704+0000 7f08f1e00640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08fc076970 0x7f08fc19b160 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f08e4005ab0 tx=0x7f08e4003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.704+0000 7f08f3e00640 1 -- 172.21.15.1:0/3570358603 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e403f070 con 0x7f08fc076970 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.704+0000 7f08f3e00640 1 -- 172.21.15.1:0/3570358603 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08e4031c50 con 0x7f08fc076970 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.704+0000 7f08f3e00640 1 -- 172.21.15.1:0/3570358603 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4035030 con 0x7f08fc076970 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.704+0000 7f0900a00640 1 -- 172.21.15.1:0/3570358603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08fc1a2ac0 con 0x7f08fc076970 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.704+0000 7f0900a00640 1 -- 172.21.15.1:0/3570358603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08fc1a3000 con 0x7f08fc076970 2024-04-26T21:29:25.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.705+0000 7f0900a00640 1 -- 172.21.15.1:0/3570358603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08fc109390 con 0x7f08fc076970 2024-04-26T21:29:25.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.708+0000 7f08f3e00640 1 -- 172.21.15.1:0/3570358603 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f08e4042070 con 0x7f08fc076970 2024-04-26T21:29:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.709+0000 7f08f3e00640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f08e00a3cc0 0x7f08e00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.709+0000 7f08f3e00640 1 -- 172.21.15.1:0/3570358603 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f08e40f1200 con 0x7f08fc076970 2024-04-26T21:29:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.709+0000 7f08fa800640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f08e00a3cc0 0x7f08e00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.710+0000 7f08fa800640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f08e00a3cc0 0x7f08e00a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f08fc19c0f0 tx=0x7f08ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:25.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.714+0000 7f08f3e00640 1 -- 172.21.15.1:0/3570358603 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f08e40bcfd0 con 0x7f08fc076970 2024-04-26T21:29:25.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.921+0000 7f0900a00640 1 -- 172.21.15.1:0/3570358603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f08fc02dd20 con 0x7f08fc076970 2024-04-26T21:29:25.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.922+0000 7f08f3e00640 1 -- 172.21.15.1:0/3570358603 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f08e40c12d0 con 0x7f08fc076970 2024-04-26T21:29:25.923 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:29:25.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 -- 172.21.15.1:0/3570358603 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f08e00a3cc0 msgr2=0x7f08e00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:25.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f08e00a3cc0 0x7f08e00a6180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f08fc19c0f0 tx=0x7f08ec005f90 comp rx=0 tx=0).stop 2024-04-26T21:29:25.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 -- 172.21.15.1:0/3570358603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08fc076970 msgr2=0x7f08fc19b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:25.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08fc076970 0x7f08fc19b160 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f08e4005ab0 tx=0x7f08e4003b90 comp rx=0 tx=0).stop 2024-04-26T21:29:25.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 -- 172.21.15.1:0/3570358603 shutdown_connections 2024-04-26T21:29:25.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f08e00a3cc0 0x7f08e00a6180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08fc10a660 0x7f08fc1a2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08fc076970 0x7f08fc19b160 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 --2- 172.21.15.1:0/3570358603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08fc075390 0x7f08fc19ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 -- 172.21.15.1:0/3570358603 >> 172.21.15.1:0/3570358603 conn(0x7f08fc0fdd10 msgr2=0x7f08fc106560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 -- 172.21.15.1:0/3570358603 shutdown_connections 2024-04-26T21:29:25.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:25.925+0000 7f0900a00640 1 -- 172.21.15.1:0/3570358603 wait complete. 2024-04-26T21:29:25.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:29:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.153+0000 7f2761e00640 1 -- 172.21.15.1:0/2542520791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f275c1049b0 msgr2=0x7f275c104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.153+0000 7f2761e00640 1 --2- 172.21.15.1:0/2542520791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f275c1049b0 0x7f275c104dd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f274c009900 tx=0x7f274c02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.154+0000 7f2761e00640 1 -- 172.21.15.1:0/2542520791 shutdown_connections 2024-04-26T21:29:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.154+0000 7f2761e00640 1 --2- 172.21.15.1:0/2542520791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f275c106560 0x7f275c10cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.154+0000 7f2761e00640 1 --2- 172.21.15.1:0/2542520791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f275c105ba0 0x7f275c106020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.154+0000 7f2761e00640 1 --2- 172.21.15.1:0/2542520791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f275c1049b0 0x7f275c104dd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.154+0000 7f2761e00640 1 -- 172.21.15.1:0/2542520791 >> 172.21.15.1:0/2542520791 conn(0x7f275c0fff40 msgr2=0x7f275c102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.154+0000 7f2761e00640 1 -- 172.21.15.1:0/2542520791 shutdown_connections 2024-04-26T21:29:26.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.154+0000 7f2761e00640 1 -- 172.21.15.1:0/2542520791 wait complete. 2024-04-26T21:29:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.155+0000 7f2761e00640 1 Processor -- start 2024-04-26T21:29:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.155+0000 7f2761e00640 1 -- start start 2024-04-26T21:29:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2761e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f275c1049b0 0x7f275c19ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2761e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f275c105ba0 0x7f275c19b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2761e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f275c106560 0x7f275c1a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2761e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f275c110530 con 0x7f275c1049b0 2024-04-26T21:29:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2761e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f275c1103b0 con 0x7f275c105ba0 2024-04-26T21:29:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2761e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f275c1106b0 con 0x7f275c106560 2024-04-26T21:29:26.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2760a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f275c1049b0 0x7f275c19ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2760a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f275c1049b0 0x7f275c19ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44832/0 (socket says 172.21.15.1:44832) 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2761400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f275c106560 0x7f275c1a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2760a00640 1 -- 172.21.15.1:0/2489936842 learned_addr learned my addr 172.21.15.1:0/2489936842 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.156+0000 7f2753e00640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f275c105ba0 0x7f275c19b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f2761400640 1 -- 172.21.15.1:0/2489936842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f275c105ba0 msgr2=0x7f275c19b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f2761400640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f275c105ba0 0x7f275c19b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f2761400640 1 -- 172.21.15.1:0/2489936842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f275c1049b0 msgr2=0x7f275c19ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f2761400640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f275c1049b0 0x7f275c19ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f2761400640 1 -- 172.21.15.1:0/2489936842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275c1a2860 con 0x7f275c106560 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f2761400640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f275c106560 0x7f275c1a21c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2754009fd0 tx=0x7f275400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f275a000640 1 -- 172.21.15.1:0/2489936842 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275400ce20 con 0x7f275c106560 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f275a000640 1 -- 172.21.15.1:0/2489936842 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2754005020 con 0x7f275c106560 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f2761e00640 1 -- 172.21.15.1:0/2489936842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275c1a2b50 con 0x7f275c106560 2024-04-26T21:29:26.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f275a000640 1 -- 172.21.15.1:0/2489936842 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2754010690 con 0x7f275c106560 2024-04-26T21:29:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.157+0000 7f2761e00640 1 -- 172.21.15.1:0/2489936842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f275c1a30a0 con 0x7f275c106560 2024-04-26T21:29:26.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.159+0000 7f2761e00640 1 -- 172.21.15.1:0/2489936842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f275c10bda0 con 0x7f275c106560 2024-04-26T21:29:26.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.161+0000 7f275a000640 1 -- 172.21.15.1:0/2489936842 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f27540054d0 con 0x7f275c106560 2024-04-26T21:29:26.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.163+0000 7f275a000640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f27400a3c40 0x7f27400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:26.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.163+0000 7f275a000640 1 -- 172.21.15.1:0/2489936842 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f27540d1150 con 0x7f275c106560 2024-04-26T21:29:26.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.163+0000 7f2760a00640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f27400a3c40 0x7f27400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:26.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.164+0000 7f2760a00640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f27400a3c40 0x7f27400a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f274c006010 tx=0x7f274c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.167+0000 7f275a000640 1 -- 172.21.15.1:0/2489936842 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f275409d570 con 0x7f275c106560 2024-04-26T21:29:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:26 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3632131512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:29:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:26 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3570358603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:29:26.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.379+0000 7f2761e00640 1 -- 172.21.15.1:0/2489936842 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f275c02dd20 con 0x7f275c106560 2024-04-26T21:29:26.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.380+0000 7f275a000640 1 -- 172.21.15.1:0/2489936842 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f275409ce70 con 0x7f275c106560 2024-04-26T21:29:26.380 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:29:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.382+0000 7f2761e00640 1 -- 172.21.15.1:0/2489936842 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f27400a3c40 msgr2=0x7f27400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.382+0000 7f2761e00640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f27400a3c40 0x7f27400a6100 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f274c006010 tx=0x7f274c005a20 comp rx=0 tx=0).stop 2024-04-26T21:29:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.382+0000 7f2761e00640 1 -- 172.21.15.1:0/2489936842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f275c106560 msgr2=0x7f275c1a21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:26.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.382+0000 7f2761e00640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f275c106560 0x7f275c1a21c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2754009fd0 tx=0x7f275400ef10 comp rx=0 tx=0).stop 2024-04-26T21:29:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.382+0000 7f2761e00640 1 -- 172.21.15.1:0/2489936842 shutdown_connections 2024-04-26T21:29:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.383+0000 7f2761e00640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f27400a3c40 0x7f27400a6100 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.383+0000 7f2761e00640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f275c106560 0x7f275c1a21c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.383+0000 7f2761e00640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f275c105ba0 0x7f275c19b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.383+0000 7f2761e00640 1 --2- 172.21.15.1:0/2489936842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f275c1049b0 0x7f275c19ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.383+0000 7f2761e00640 1 -- 172.21.15.1:0/2489936842 >> 172.21.15.1:0/2489936842 conn(0x7f275c0fff40 msgr2=0x7f275c108f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.383+0000 7f2761e00640 1 -- 172.21.15.1:0/2489936842 shutdown_connections 2024-04-26T21:29:26.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.383+0000 7f2761e00640 1 -- 172.21.15.1:0/2489936842 wait complete. 2024-04-26T21:29:26.403 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:29:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:26 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3632131512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:29:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:26 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3570358603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:29:26.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:26 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3632131512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:29:26.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:26 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3570358603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:29:26.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.623+0000 7f2dede00640 1 -- 172.21.15.1:0/3926087996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de8106dd0 msgr2=0x7f2de81091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:26.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.623+0000 7f2dede00640 1 --2- 172.21.15.1:0/3926087996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de8106dd0 0x7f2de81091c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009ca0 tx=0x7f2de002a8c0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.623+0000 7f2dede00640 1 -- 172.21.15.1:0/3926087996 shutdown_connections 2024-04-26T21:29:26.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.623+0000 7f2dede00640 1 --2- 172.21.15.1:0/3926087996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2de81020d0 0x7f2de8102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.623+0000 7f2dede00640 1 --2- 172.21.15.1:0/3926087996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de8106dd0 0x7f2de81091c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.623+0000 7f2dede00640 1 --2- 172.21.15.1:0/3926087996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2de8068f50 0x7f2de8101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.623+0000 7f2dede00640 1 -- 172.21.15.1:0/3926087996 >> 172.21.15.1:0/3926087996 conn(0x7f2de80fbf30 msgr2=0x7f2de80fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:26.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.623+0000 7f2dede00640 1 -- 172.21.15.1:0/3926087996 shutdown_connections 2024-04-26T21:29:26.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.623+0000 7f2dede00640 1 -- 172.21.15.1:0/3926087996 wait complete. 2024-04-26T21:29:26.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.624+0000 7f2dede00640 1 Processor -- start 2024-04-26T21:29:26.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.624+0000 7f2dede00640 1 -- start start 2024-04-26T21:29:26.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.624+0000 7f2dede00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de8068f50 0x7f2de819ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:26.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.624+0000 7f2dede00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2de81020d0 0x7f2de819b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:26.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.624+0000 7f2dede00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2de8106dd0 0x7f2de81a2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:26.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2dede00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2de810e330 con 0x7f2de81020d0 2024-04-26T21:29:26.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2dede00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2de810e1b0 con 0x7f2de8068f50 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2dede00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2de810e4b0 con 0x7f2de8106dd0 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2deca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de8068f50 0x7f2de819ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2de7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2de81020d0 0x7f2de819b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2deca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de8068f50 0x7f2de819ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37484/0 (socket says 172.21.15.1:37484) 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2deca00640 1 -- 172.21.15.1:0/3971883744 learned_addr learned my addr 172.21.15.1:0/3971883744 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2ded400640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2de8106dd0 0x7f2de81a2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2de7e00640 1 -- 172.21.15.1:0/3971883744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2de8106dd0 msgr2=0x7f2de81a2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2de7e00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2de8106dd0 0x7f2de81a2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2de7e00640 1 -- 172.21.15.1:0/3971883744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de8068f50 msgr2=0x7f2de819ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2de7e00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de8068f50 0x7f2de819ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2de7e00640 1 -- 172.21.15.1:0/3971883744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de81a27a0 con 0x7f2de81020d0 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.625+0000 7f2ded400640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2de8106dd0 0x7f2de81a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.626+0000 7f2de7e00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2de81020d0 0x7f2de819b0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400e9c0 tx=0x7f2dd400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.626+0000 7f2de5600640 1 -- 172.21.15.1:0/3971883744 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd400ccc0 con 0x7f2de81020d0 2024-04-26T21:29:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.626+0000 7f2de5600640 1 -- 172.21.15.1:0/3971883744 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dd40035f0 con 0x7f2de81020d0 2024-04-26T21:29:26.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.626+0000 7f2dede00640 1 -- 172.21.15.1:0/3971883744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de81a2a90 con 0x7f2de81020d0 2024-04-26T21:29:26.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.626+0000 7f2de5600640 1 -- 172.21.15.1:0/3971883744 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd400ce60 con 0x7f2de81020d0 2024-04-26T21:29:26.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.626+0000 7f2dede00640 1 -- 172.21.15.1:0/3971883744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de81a2f90 con 0x7f2de81020d0 2024-04-26T21:29:26.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.627+0000 7f2dede00640 1 -- 172.21.15.1:0/3971883744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de81a3220 con 0x7f2de81020d0 2024-04-26T21:29:26.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.629+0000 7f2de5600640 1 -- 172.21.15.1:0/3971883744 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f2dd4004000 con 0x7f2de81020d0 2024-04-26T21:29:26.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.631+0000 7f2de5600640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2dd00a3c30 0x7f2dd00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:26.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.631+0000 7f2de5600640 1 -- 172.21.15.1:0/3971883744 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f2dd40d0d60 con 0x7f2de81020d0 2024-04-26T21:29:26.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.631+0000 7f2deca00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2dd00a3c30 0x7f2dd00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.632+0000 7f2deca00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2dd00a3c30 0x7f2dd00a60f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0058c0 tx=0x7f2ddc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:26.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.636+0000 7f2de5600640 1 -- 172.21.15.1:0/3971883744 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2dd409d180 con 0x7f2de81020d0 2024-04-26T21:29:26.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.844+0000 7f2dede00640 1 -- 172.21.15.1:0/3971883744 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2de81a33f0 con 0x7f2de81020d0 2024-04-26T21:29:26.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.845+0000 7f2de5600640 1 -- 172.21.15.1:0/3971883744 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f2dd409ca80 con 0x7f2de81020d0 2024-04-26T21:29:26.846 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.848+0000 7f2dede00640 1 -- 172.21.15.1:0/3971883744 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2dd00a3c30 msgr2=0x7f2dd00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2dd00a3c30 0x7f2dd00a60f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0058c0 tx=0x7f2ddc005f90 comp rx=0 tx=0).stop 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 -- 172.21.15.1:0/3971883744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2de81020d0 msgr2=0x7f2de819b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2de81020d0 0x7f2de819b0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400e9c0 tx=0x7f2dd400ee80 comp rx=0 tx=0).stop 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 -- 172.21.15.1:0/3971883744 shutdown_connections 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2dd00a3c30 0x7f2dd00a60f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2de8106dd0 0x7f2de81a2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2de81020d0 0x7f2de819b0c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 --2- 172.21.15.1:0/3971883744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2de8068f50 0x7f2de819ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 -- 172.21.15.1:0/3971883744 >> 172.21.15.1:0/3971883744 conn(0x7f2de80fbf30 msgr2=0x7f2de80fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 -- 172.21.15.1:0/3971883744 shutdown_connections 2024-04-26T21:29:26.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:26.849+0000 7f2dede00640 1 -- 172.21.15.1:0/3971883744 wait complete. 2024-04-26T21:29:26.872 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T21:29:27.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.088+0000 7f9efca00640 1 -- 172.21.15.1:0/2281610139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ef810ab40 msgr2=0x7f9ef81113d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:27.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.088+0000 7f9efca00640 1 --2- 172.21.15.1:0/2281610139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ef810ab40 0x7f9ef81113d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009be0 tx=0x7f9ef002a6d0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.088+0000 7f9efca00640 1 -- 172.21.15.1:0/2281610139 shutdown_connections 2024-04-26T21:29:27.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.088+0000 7f9efca00640 1 --2- 172.21.15.1:0/2281610139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ef810ab40 0x7f9ef81113d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.088+0000 7f9efca00640 1 --2- 172.21.15.1:0/2281610139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef810a180 0x7f9ef810a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.088+0000 7f9efca00640 1 --2- 172.21.15.1:0/2281610139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ef8108f90 0x7f9ef81093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.088+0000 7f9efca00640 1 -- 172.21.15.1:0/2281610139 >> 172.21.15.1:0/2281610139 conn(0x7f9ef8077f40 msgr2=0x7f9ef807a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:27.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.088+0000 7f9efca00640 1 -- 172.21.15.1:0/2281610139 shutdown_connections 2024-04-26T21:29:27.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.088+0000 7f9efca00640 1 -- 172.21.15.1:0/2281610139 wait complete. 2024-04-26T21:29:27.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.089+0000 7f9efca00640 1 Processor -- start 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.089+0000 7f9efca00640 1 -- start start 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.089+0000 7f9efca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ef8108f90 0x7f9ef819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.089+0000 7f9efca00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef810a180 0x7f9ef819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.089+0000 7f9efca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ef810ab40 0x7f9ef81a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.089+0000 7f9efca00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef8114ad0 con 0x7f9ef810a180 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.089+0000 7f9efca00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef8114950 con 0x7f9ef810ab40 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.089+0000 7f9efca00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef8114c50 con 0x7f9ef8108f90 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.090+0000 7f9ef5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ef8108f90 0x7f9ef819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.090+0000 7f9ef5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ef8108f90 0x7f9ef819f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50786/0 (socket says 172.21.15.1:50786) 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.090+0000 7f9ef5e00640 1 -- 172.21.15.1:0/4156956215 learned_addr learned my addr 172.21.15.1:0/4156956215 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.090+0000 7f9ef6800640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ef810ab40 0x7f9ef81a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.090+0000 7f9ef5400640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef810a180 0x7f9ef819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:27.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.090+0000 7f9ef5e00640 1 -- 172.21.15.1:0/4156956215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ef810ab40 msgr2=0x7f9ef81a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.090+0000 7f9ef5e00640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ef810ab40 0x7f9ef81a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.090+0000 7f9ef5e00640 1 -- 172.21.15.1:0/4156956215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef810a180 msgr2=0x7f9ef819f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.090+0000 7f9ef5e00640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef810a180 0x7f9ef819f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.090+0000 7f9ef5e00640 1 -- 172.21.15.1:0/4156956215 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef81a6db0 con 0x7f9ef8108f90 2024-04-26T21:29:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.091+0000 7f9ef5e00640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ef8108f90 0x7f9ef819f190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9ee000e9a0 tx=0x7f9ee000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.091+0000 7f9eeea00640 1 -- 172.21.15.1:0/4156956215 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee000cd80 con 0x7f9ef8108f90 2024-04-26T21:29:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.091+0000 7f9efca00640 1 -- 172.21.15.1:0/4156956215 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef81a70a0 con 0x7f9ef8108f90 2024-04-26T21:29:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.091+0000 7f9eeea00640 1 -- 172.21.15.1:0/4156956215 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ee00108e0 con 0x7f9ef8108f90 2024-04-26T21:29:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.091+0000 7f9eeea00640 1 -- 172.21.15.1:0/4156956215 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0018960 con 0x7f9ef8108f90 2024-04-26T21:29:27.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.091+0000 7f9efca00640 1 -- 172.21.15.1:0/4156956215 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef81a7640 con 0x7f9ef8108f90 2024-04-26T21:29:27.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.092+0000 7f9efca00640 1 -- 172.21.15.1:0/4156956215 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ef8110300 con 0x7f9ef8108f90 2024-04-26T21:29:27.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.094+0000 7f9eeea00640 1 -- 172.21.15.1:0/4156956215 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9ee0010a90 con 0x7f9ef8108f90 2024-04-26T21:29:27.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.096+0000 7f9eeea00640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9edc0a3c40 0x7f9edc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:27.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.096+0000 7f9eeea00640 1 -- 172.21.15.1:0/4156956215 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f9ee00d1410 con 0x7f9ef8108f90 2024-04-26T21:29:27.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.096+0000 7f9ef5400640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9edc0a3c40 0x7f9edc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:27.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.097+0000 7f9ef5400640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9edc0a3c40 0x7f9edc0a6100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80095f0 tx=0x7f9ee8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:27.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.098+0000 7f9eeea00640 1 -- 172.21.15.1:0/4156956215 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9ee0026070 con 0x7f9ef8108f90 2024-04-26T21:29:27.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.308+0000 7f9efca00640 1 -- 172.21.15.1:0/4156956215 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9ef8063530 con 0x7f9ef8108f90 2024-04-26T21:29:27.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.309+0000 7f9eeea00640 1 -- 172.21.15.1:0/4156956215 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f9ee009d830 con 0x7f9ef8108f90 2024-04-26T21:29:27.310 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:29:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.312+0000 7f9efca00640 1 -- 172.21.15.1:0/4156956215 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9edc0a3c40 msgr2=0x7f9edc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.312+0000 7f9efca00640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9edc0a3c40 0x7f9edc0a6100 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80095f0 tx=0x7f9ee8005f90 comp rx=0 tx=0).stop 2024-04-26T21:29:27.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.312+0000 7f9efca00640 1 -- 172.21.15.1:0/4156956215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ef8108f90 msgr2=0x7f9ef819f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.312+0000 7f9efca00640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ef8108f90 0x7f9ef819f190 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9ee000e9a0 tx=0x7f9ee000ee60 comp rx=0 tx=0).stop 2024-04-26T21:29:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.313+0000 7f9efca00640 1 -- 172.21.15.1:0/4156956215 shutdown_connections 2024-04-26T21:29:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.313+0000 7f9efca00640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f9edc0a3c40 0x7f9edc0a6100 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.313+0000 7f9efca00640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ef810ab40 0x7f9ef81a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.313+0000 7f9efca00640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9ef810a180 0x7f9ef819f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.313+0000 7f9efca00640 1 --2- 172.21.15.1:0/4156956215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ef8108f90 0x7f9ef819f190 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.313+0000 7f9efca00640 1 -- 172.21.15.1:0/4156956215 >> 172.21.15.1:0/4156956215 conn(0x7f9ef8077f40 msgr2=0x7f9ef810d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.313+0000 7f9efca00640 1 -- 172.21.15.1:0/4156956215 shutdown_connections 2024-04-26T21:29:27.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.313+0000 7f9efca00640 1 -- 172.21.15.1:0/4156956215 wait complete. 2024-04-26T21:29:27.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:29:27.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:27 smithi186 ceph-mon[32991]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:27.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:27 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2489936842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:29:27.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:27 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3971883744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:29:27.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.550+0000 7fd75ca00640 1 -- 172.21.15.1:0/474398269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd758103990 msgr2=0x7fd758103e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:27.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.550+0000 7fd75ca00640 1 --2- 172.21.15.1:0/474398269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd758103990 0x7fd758103e10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd740009900 tx=0x7fd74002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.551+0000 7fd75ca00640 1 -- 172.21.15.1:0/474398269 shutdown_connections 2024-04-26T21:29:27.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.551+0000 7fd75ca00640 1 --2- 172.21.15.1:0/474398269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd758104350 0x7fd75810abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.551+0000 7fd75ca00640 1 --2- 172.21.15.1:0/474398269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd758103990 0x7fd758103e10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.551+0000 7fd75ca00640 1 --2- 172.21.15.1:0/474398269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7581027a0 0x7fd758102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.551+0000 7fd75ca00640 1 -- 172.21.15.1:0/474398269 >> 172.21.15.1:0/474398269 conn(0x7fd7580fdd10 msgr2=0x7fd758100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:27.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.551+0000 7fd75ca00640 1 -- 172.21.15.1:0/474398269 shutdown_connections 2024-04-26T21:29:27.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.551+0000 7fd75ca00640 1 -- 172.21.15.1:0/474398269 wait complete. 2024-04-26T21:29:27.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.552+0000 7fd75ca00640 1 Processor -- start 2024-04-26T21:29:27.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.552+0000 7fd75ca00640 1 -- start start 2024-04-26T21:29:27.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.552+0000 7fd75ca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7581027a0 0x7fd75819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:27.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.552+0000 7fd75ca00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd758103990 0x7fd75819aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:27.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd75ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd758104350 0x7fd7581a2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:27.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd75ca00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd75810e2d0 con 0x7fd758103990 2024-04-26T21:29:27.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd75ca00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd75810e150 con 0x7fd7581027a0 2024-04-26T21:29:27.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd75ca00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd75810e450 con 0x7fd758104350 2024-04-26T21:29:27.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd755e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd758103990 0x7fd75819aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:27.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd756800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7581027a0 0x7fd75819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:27.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd756800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7581027a0 0x7fd75819aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37522/0 (socket says 172.21.15.1:37522) 2024-04-26T21:29:27.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd756800640 1 -- 172.21.15.1:0/2974909968 learned_addr learned my addr 172.21.15.1:0/2974909968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:27.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd755e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd758103990 0x7fd75819aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44880/0 (socket says 172.21.15.1:44880) 2024-04-26T21:29:27.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd757200640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd758104350 0x7fd7581a2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:27.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd756800640 1 -- 172.21.15.1:0/2974909968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd758104350 msgr2=0x7fd7581a2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:27.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd756800640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd758104350 0x7fd7581a2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd756800640 1 -- 172.21.15.1:0/2974909968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd758103990 msgr2=0x7fd75819aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:27.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd756800640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd758103990 0x7fd75819aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.553+0000 7fd756800640 1 -- 172.21.15.1:0/2974909968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7581a26d0 con 0x7fd7581027a0 2024-04-26T21:29:27.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.554+0000 7fd756800640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7581027a0 0x7fd75819aab0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd74800ebc0 tx=0x7fd74800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:27.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.554+0000 7fd74f400640 1 -- 172.21.15.1:0/2974909968 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748003b70 con 0x7fd7581027a0 2024-04-26T21:29:27.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.554+0000 7fd74f400640 1 -- 172.21.15.1:0/2974909968 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd748003d10 con 0x7fd7581027a0 2024-04-26T21:29:27.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.554+0000 7fd74f400640 1 -- 172.21.15.1:0/2974909968 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd748019800 con 0x7fd7581027a0 2024-04-26T21:29:27.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.554+0000 7fd75ca00640 1 -- 172.21.15.1:0/2974909968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7581a29c0 con 0x7fd7581027a0 2024-04-26T21:29:27.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.554+0000 7fd75ca00640 1 -- 172.21.15.1:0/2974909968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7581a2f30 con 0x7fd7581027a0 2024-04-26T21:29:27.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.555+0000 7fd75ca00640 1 -- 172.21.15.1:0/2974909968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7581a31c0 con 0x7fd7581027a0 2024-04-26T21:29:27.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.557+0000 7fd74f400640 1 -- 172.21.15.1:0/2974909968 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fd7480199a0 con 0x7fd7581027a0 2024-04-26T21:29:27.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.558+0000 7fd74f400640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd73c0a3d70 0x7fd73c0a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:27.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.559+0000 7fd74f400640 1 -- 172.21.15.1:0/2974909968 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fd7480d12a0 con 0x7fd7581027a0 2024-04-26T21:29:27.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.559+0000 7fd755e00640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd73c0a3d70 0x7fd73c0a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:27.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.560+0000 7fd755e00640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd73c0a3d70 0x7fd73c0a6230 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd740000c00 tx=0x7fd7400314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:27.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.563+0000 7fd74f400640 1 -- 172.21.15.1:0/2974909968 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd74809d6c0 con 0x7fd7581027a0 2024-04-26T21:29:27.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:27 smithi001 ceph-mon[28054]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:27.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:27 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2489936842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:29:27.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:27 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3971883744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:29:27.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:27 smithi038 ceph-mon[32724]: pgmap v400: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:27.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:27 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2489936842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:29:27.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:27 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3971883744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:29:27.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.815+0000 7fd75ca00640 1 -- 172.21.15.1:0/2974909968 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7581a3390 con 0x7fd7581027a0 2024-04-26T21:29:27.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.816+0000 7fd74f400640 1 -- 172.21.15.1:0/2974909968 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11857 (secure 0 0 0) 0x7fd74809cfc0 con 0x7fd7581027a0 2024-04-26T21:29:27.817 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:27.818 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:27:02.795246+0000","modified":"2024-04-26T21:27:28.957703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24572,"mds_2":24548},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24548":{"gid":24548,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24548,24572,24602]},"id":1}]} 2024-04-26T21:29:27.818 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 23 2024-04-26T21:29:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 -- 172.21.15.1:0/2974909968 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd73c0a3d70 msgr2=0x7fd73c0a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd73c0a3d70 0x7fd73c0a6230 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd740000c00 tx=0x7fd7400314a0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 -- 172.21.15.1:0/2974909968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7581027a0 msgr2=0x7fd75819aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7581027a0 0x7fd75819aab0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd74800ebc0 tx=0x7fd74800c6a0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 -- 172.21.15.1:0/2974909968 shutdown_connections 2024-04-26T21:29:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd73c0a3d70 0x7fd73c0a6230 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd758104350 0x7fd7581a2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd758103990 0x7fd75819aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 --2- 172.21.15.1:0/2974909968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7581027a0 0x7fd75819aab0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:27.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 -- 172.21.15.1:0/2974909968 >> 172.21.15.1:0/2974909968 conn(0x7fd7580fdd10 msgr2=0x7fd758074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:27.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.819+0000 7fd75ca00640 1 -- 172.21.15.1:0/2974909968 shutdown_connections 2024-04-26T21:29:27.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:27.820+0000 7fd75ca00640 1 -- 172.21.15.1:0/2974909968 wait complete. 2024-04-26T21:29:27.840 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:29:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.053+0000 7f7b8be00640 1 -- 172.21.15.1:0/798057345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b8c109f80 msgr2=0x7f7b8c10a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.053+0000 7f7b8be00640 1 --2- 172.21.15.1:0/798057345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b8c109f80 0x7f7b8c10a400 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009900 tx=0x7f7b7802a3f0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.053+0000 7f7b8be00640 1 -- 172.21.15.1:0/798057345 shutdown_connections 2024-04-26T21:29:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.053+0000 7f7b8be00640 1 --2- 172.21.15.1:0/798057345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b8c109f80 0x7f7b8c10a400 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.053+0000 7f7b8be00640 1 --2- 172.21.15.1:0/798057345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b8c10a940 0x7f7b8c1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.053+0000 7f7b8be00640 1 --2- 172.21.15.1:0/798057345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b8c108d90 0x7f7b8c1091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.053+0000 7f7b8be00640 1 -- 172.21.15.1:0/798057345 >> 172.21.15.1:0/798057345 conn(0x7f7b8c077d30 msgr2=0x7f7b8c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:28.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.053+0000 7f7b8be00640 1 -- 172.21.15.1:0/798057345 shutdown_connections 2024-04-26T21:29:28.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.053+0000 7f7b8be00640 1 -- 172.21.15.1:0/798057345 wait complete. 2024-04-26T21:29:28.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.054+0000 7f7b8be00640 1 Processor -- start 2024-04-26T21:29:28.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.054+0000 7f7b8be00640 1 -- start start 2024-04-26T21:29:28.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b8c108d90 0x7f7b8c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b8c109f80 0x7f7b8c19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b8c10a940 0x7f7b8c1a6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8be00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c1148a0 con 0x7f7b8c108d90 2024-04-26T21:29:28.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8be00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c114720 con 0x7f7b8c10a940 2024-04-26T21:29:28.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c114a20 con 0x7f7b8c109f80 2024-04-26T21:29:28.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b8c108d90 0x7f7b8c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b8c109f80 0x7f7b8c19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b8c109f80 0x7f7b8c19f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42536/0 (socket says 172.21.15.1:42536) 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b8c108d90 0x7f7b8c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40116/0 (socket says 172.21.15.1:40116) 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8a000640 1 -- 172.21.15.1:0/3838900068 learned_addr learned my addr 172.21.15.1:0/3838900068 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.055+0000 7f7b8b400640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b8c10a940 0x7f7b8c1a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b8a000640 1 -- 172.21.15.1:0/3838900068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b8c10a940 msgr2=0x7f7b8c1a6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b8a000640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b8c10a940 0x7f7b8c1a6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b8a000640 1 -- 172.21.15.1:0/3838900068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b8c108d90 msgr2=0x7f7b8c19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b8a000640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b8c108d90 0x7f7b8c19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b8a000640 1 -- 172.21.15.1:0/3838900068 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b8c1a6ae0 con 0x7f7b8c109f80 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b8a000640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b8c109f80 0x7f7b8c19f400 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7b78005940 tx=0x7f7b780042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b83400640 1 -- 172.21.15.1:0/3838900068 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7803f030 con 0x7f7b8c109f80 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b83400640 1 -- 172.21.15.1:0/3838900068 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b78003cb0 con 0x7f7b8c109f80 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b8be00640 1 -- 172.21.15.1:0/3838900068 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b8c1a6d70 con 0x7f7b8c109f80 2024-04-26T21:29:28.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b83400640 1 -- 172.21.15.1:0/3838900068 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78035070 con 0x7f7b8c109f80 2024-04-26T21:29:28.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.056+0000 7f7b8be00640 1 -- 172.21.15.1:0/3838900068 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b8c1a7250 con 0x7f7b8c109f80 2024-04-26T21:29:28.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.057+0000 7f7b8be00640 1 -- 172.21.15.1:0/3838900068 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b8c1100b0 con 0x7f7b8c109f80 2024-04-26T21:29:28.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.060+0000 7f7b83400640 1 -- 172.21.15.1:0/3838900068 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f7b78042030 con 0x7f7b8c109f80 2024-04-26T21:29:28.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.061+0000 7f7b83400640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7b700a3d70 0x7f7b700a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.061+0000 7f7b83400640 1 -- 172.21.15.1:0/3838900068 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f7b780f11d0 con 0x7f7b8c109f80 2024-04-26T21:29:28.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.061+0000 7f7b8aa00640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7b700a3d70 0x7f7b700a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.062+0000 7f7b8aa00640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7b700a3d70 0x7f7b700a6230 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c109de0 tx=0x7f7b7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:28.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.063+0000 7f7b83400640 1 -- 172.21.15.1:0/3838900068 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7b78003e60 con 0x7f7b8c109f80 2024-04-26T21:29:28.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.269+0000 7f7b8be00640 1 -- 172.21.15.1:0/3838900068 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b8c02dd40 con 0x7f7b8c109f80 2024-04-26T21:29:28.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.272+0000 7f7b83400640 1 -- 172.21.15.1:0/3838900068 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24767 (secure 0 0 0) 0x7f7b780bcfa0 con 0x7f7b8c109f80 2024-04-26T21:29:28.273 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:28.273 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":95,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:28:57.077487+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7},{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:29:28.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.274+0000 7f7b8be00640 1 -- 172.21.15.1:0/3838900068 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7b700a3d70 msgr2=0x7f7b700a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.274+0000 7f7b8be00640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7b700a3d70 0x7f7b700a6230 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c109de0 tx=0x7f7b7c005f90 comp rx=0 tx=0).stop 2024-04-26T21:29:28.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.274+0000 7f7b8be00640 1 -- 172.21.15.1:0/3838900068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b8c109f80 msgr2=0x7f7b8c19f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.274+0000 7f7b8be00640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b8c109f80 0x7f7b8c19f400 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7b78005940 tx=0x7f7b780042f0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.274+0000 7f7b8be00640 1 -- 172.21.15.1:0/3838900068 shutdown_connections 2024-04-26T21:29:28.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.274+0000 7f7b8be00640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7b700a3d70 0x7f7b700a6230 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.274+0000 7f7b8be00640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b8c109f80 0x7f7b8c19f400 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.274+0000 7f7b8be00640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b8c10a940 0x7f7b8c1a6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.274+0000 7f7b8be00640 1 --2- 172.21.15.1:0/3838900068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b8c108d90 0x7f7b8c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.274+0000 7f7b8be00640 1 -- 172.21.15.1:0/3838900068 >> 172.21.15.1:0/3838900068 conn(0x7f7b8c077d30 msgr2=0x7f7b8c10d280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:28.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.275+0000 7f7b8be00640 1 -- 172.21.15.1:0/3838900068 shutdown_connections 2024-04-26T21:29:28.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.275+0000 7f7b8be00640 1 -- 172.21.15.1:0/3838900068 wait complete. 2024-04-26T21:29:28.294 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:27:00.381106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:27:01.743898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T21:27:05.341216+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T21:29:28.295 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:29:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:28 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4156956215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:29:28.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:28 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2974909968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:29:28.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.505+0000 7fd72de00640 1 -- 172.21.15.1:0/4185032998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7281049d0 msgr2=0x7fd728104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.505+0000 7fd72de00640 1 --2- 172.21.15.1:0/4185032998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7281049d0 0x7fd728104df0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd718009be0 tx=0x7fd71802a6d0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.505+0000 7fd72de00640 1 -- 172.21.15.1:0/4185032998 shutdown_connections 2024-04-26T21:29:28.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.505+0000 7fd72de00640 1 --2- 172.21.15.1:0/4185032998 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd728106580 0x7fd72810ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.505+0000 7fd72de00640 1 --2- 172.21.15.1:0/4185032998 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd728105bc0 0x7fd728106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.505+0000 7fd72de00640 1 --2- 172.21.15.1:0/4185032998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7281049d0 0x7fd728104df0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.505+0000 7fd72de00640 1 -- 172.21.15.1:0/4185032998 >> 172.21.15.1:0/4185032998 conn(0x7fd7280fff40 msgr2=0x7fd728102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:28.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.505+0000 7fd72de00640 1 -- 172.21.15.1:0/4185032998 shutdown_connections 2024-04-26T21:29:28.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.505+0000 7fd72de00640 1 -- 172.21.15.1:0/4185032998 wait complete. 2024-04-26T21:29:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.506+0000 7fd72de00640 1 Processor -- start 2024-04-26T21:29:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72de00640 1 -- start start 2024-04-26T21:29:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7281049d0 0x7fd72819ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd728105bc0 0x7fd72819b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd728106580 0x7fd7281a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72de00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd7281104f0 con 0x7fd7281049d0 2024-04-26T21:29:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72de00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd728110370 con 0x7fd728106580 2024-04-26T21:29:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72de00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd728110670 con 0x7fd728105bc0 2024-04-26T21:29:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72ca00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7281049d0 0x7fd72819ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72ca00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7281049d0 0x7fd72819ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40128/0 (socket says 172.21.15.1:40128) 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72ca00640 1 -- 172.21.15.1:0/3829762177 learned_addr learned my addr 172.21.15.1:0/3829762177 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd71fe00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd728105bc0 0x7fd72819b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.507+0000 7fd72d400640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd728106580 0x7fd7281a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd71fe00640 1 -- 172.21.15.1:0/3829762177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd728106580 msgr2=0x7fd7281a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd71fe00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd728106580 0x7fd7281a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd71fe00640 1 -- 172.21.15.1:0/3829762177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7281049d0 msgr2=0x7fd72819ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd71fe00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7281049d0 0x7fd72819ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd71fe00640 1 -- 172.21.15.1:0/3829762177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7281a2870 con 0x7fd728105bc0 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd72ca00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7281049d0 0x7fd72819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd71fe00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd728105bc0 0x7fd72819b1a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd710009fd0 tx=0x7fd71000ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd726000640 1 -- 172.21.15.1:0/3829762177 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71000ccc0 con 0x7fd728105bc0 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd726000640 1 -- 172.21.15.1:0/3829762177 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd71000ee70 con 0x7fd728105bc0 2024-04-26T21:29:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd72de00640 1 -- 172.21.15.1:0/3829762177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7281a2b60 con 0x7fd728105bc0 2024-04-26T21:29:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd726000640 1 -- 172.21.15.1:0/3829762177 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd710004a00 con 0x7fd728105bc0 2024-04-26T21:29:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.508+0000 7fd72de00640 1 -- 172.21.15.1:0/3829762177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7281a30b0 con 0x7fd728105bc0 2024-04-26T21:29:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.510+0000 7fd72de00640 1 -- 172.21.15.1:0/3829762177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd72810bdc0 con 0x7fd728105bc0 2024-04-26T21:29:28.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.512+0000 7fd726000640 1 -- 172.21.15.1:0/3829762177 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fd710004ba0 con 0x7fd728105bc0 2024-04-26T21:29:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.513+0000 7fd726000640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd70c0a3cc0 0x7fd70c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.513+0000 7fd726000640 1 -- 172.21.15.1:0/3829762177 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fd7100d09d0 con 0x7fd728105bc0 2024-04-26T21:29:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.513+0000 7fd72ca00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd70c0a3cc0 0x7fd70c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.514+0000 7fd72ca00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd70c0a3cc0 0x7fd70c0a6180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd718009370 tx=0x7fd718030970 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.515+0000 7fd726000640 1 -- 172.21.15.1:0/3829762177 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd71009cdf0 con 0x7fd728105bc0 2024-04-26T21:29:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:28 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4156956215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:29:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:28 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2974909968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:29:28.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:28 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4156956215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:29:28.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:28 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2974909968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:29:28.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.711+0000 7fd72de00640 1 -- 172.21.15.1:0/3829762177 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd72802dd20 con 0x7fd728105bc0 2024-04-26T21:29:28.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.712+0000 7fd726000640 1 -- 172.21.15.1:0/3829762177 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7fd71009c6f0 con 0x7fd728105bc0 2024-04-26T21:29:28.713 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:29:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.715+0000 7fd72de00640 1 -- 172.21.15.1:0/3829762177 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd70c0a3cc0 msgr2=0x7fd70c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.715+0000 7fd72de00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd70c0a3cc0 0x7fd70c0a6180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd718009370 tx=0x7fd718030970 comp rx=0 tx=0).stop 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.715+0000 7fd72de00640 1 -- 172.21.15.1:0/3829762177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd728105bc0 msgr2=0x7fd72819b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.716+0000 7fd72de00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd728105bc0 0x7fd72819b1a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd710009fd0 tx=0x7fd71000ec20 comp rx=0 tx=0).stop 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.716+0000 7fd72de00640 1 -- 172.21.15.1:0/3829762177 shutdown_connections 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.716+0000 7fd72de00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fd70c0a3cc0 0x7fd70c0a6180 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.716+0000 7fd72de00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd728105bc0 0x7fd72819b1a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.716+0000 7fd72de00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd728106580 0x7fd7281a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.716+0000 7fd72de00640 1 --2- 172.21.15.1:0/3829762177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7281049d0 0x7fd72819ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.716+0000 7fd72de00640 1 -- 172.21.15.1:0/3829762177 >> 172.21.15.1:0/3829762177 conn(0x7fd7280fff40 msgr2=0x7fd728108f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.716+0000 7fd72de00640 1 -- 172.21.15.1:0/3829762177 shutdown_connections 2024-04-26T21:29:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.716+0000 7fd72de00640 1 -- 172.21.15.1:0/3829762177 wait complete. 2024-04-26T21:29:28.735 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:29:28.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.947+0000 7f1d0b200640 1 -- 172.21.15.1:0/560926752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d0c102100 msgr2=0x7f1d0c102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.947+0000 7f1d0b200640 1 --2- 172.21.15.1:0/560926752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d0c102100 0x7f1d0c102580 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009900 tx=0x7f1cf802a3f0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.947+0000 7f1d0b200640 1 -- 172.21.15.1:0/560926752 shutdown_connections 2024-04-26T21:29:28.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.947+0000 7f1d0b200640 1 --2- 172.21.15.1:0/560926752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c106e00 0x7f1d0c1091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.947+0000 7f1d0b200640 1 --2- 172.21.15.1:0/560926752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d0c102100 0x7f1d0c102580 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.947+0000 7f1d0b200640 1 --2- 172.21.15.1:0/560926752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c068f50 0x7f1d0c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.947+0000 7f1d0b200640 1 -- 172.21.15.1:0/560926752 >> 172.21.15.1:0/560926752 conn(0x7f1d0c0fbf40 msgr2=0x7f1d0c0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:28.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.947+0000 7f1d0b200640 1 -- 172.21.15.1:0/560926752 shutdown_connections 2024-04-26T21:29:28.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.947+0000 7f1d0b200640 1 -- 172.21.15.1:0/560926752 wait complete. 2024-04-26T21:29:28.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.948+0000 7f1d0b200640 1 Processor -- start 2024-04-26T21:29:28.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.948+0000 7f1d0b200640 1 -- start start 2024-04-26T21:29:28.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.948+0000 7f1d0b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c068f50 0x7f1d0c19ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.948+0000 7f1d0b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d0c102100 0x7f1d0c19b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.948+0000 7f1d0b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c106e00 0x7f1d0c1a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.948+0000 7f1d0b200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c10e360 con 0x7f1d0c102100 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.948+0000 7f1d0b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c10e1e0 con 0x7f1d0c106e00 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.948+0000 7f1d0b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0c10e4e0 con 0x7f1d0c068f50 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d0a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c106e00 0x7f1d0c1a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d0a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c106e00 0x7f1d0c1a21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37322/0 (socket says 172.21.15.1:37322) 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d09e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c068f50 0x7f1d0c19ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d0a800640 1 -- 172.21.15.1:0/847712501 learned_addr learned my addr 172.21.15.1:0/847712501 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d09400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d0c102100 0x7f1d0c19b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d0a800640 1 -- 172.21.15.1:0/847712501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c068f50 msgr2=0x7f1d0c19ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d0a800640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c068f50 0x7f1d0c19ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d0a800640 1 -- 172.21.15.1:0/847712501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d0c102100 msgr2=0x7f1d0c19b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:28.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d0a800640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d0c102100 0x7f1d0c19b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:28.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d0a800640 1 -- 172.21.15.1:0/847712501 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0c1a2860 con 0x7f1d0c106e00 2024-04-26T21:29:28.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.949+0000 7f1d0a800640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c106e00 0x7f1d0c1a21c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400eb10 tx=0x7f1d0400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:28.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.950+0000 7f1d02a00640 1 -- 172.21.15.1:0/847712501 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d04004b20 con 0x7f1d0c106e00 2024-04-26T21:29:28.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.950+0000 7f1d02a00640 1 -- 172.21.15.1:0/847712501 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d04004cc0 con 0x7f1d0c106e00 2024-04-26T21:29:28.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.950+0000 7f1d0b200640 1 -- 172.21.15.1:0/847712501 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d0c1a2b50 con 0x7f1d0c106e00 2024-04-26T21:29:28.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.950+0000 7f1d02a00640 1 -- 172.21.15.1:0/847712501 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d04010670 con 0x7f1d0c106e00 2024-04-26T21:29:28.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.950+0000 7f1d0b200640 1 -- 172.21.15.1:0/847712501 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d0c1a3050 con 0x7f1d0c106e00 2024-04-26T21:29:28.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.951+0000 7f1d0b200640 1 -- 172.21.15.1:0/847712501 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d0c1a3220 con 0x7f1d0c106e00 2024-04-26T21:29:28.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.953+0000 7f1d02a00640 1 -- 172.21.15.1:0/847712501 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f1d04005020 con 0x7f1d0c106e00 2024-04-26T21:29:28.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.954+0000 7f1d02a00640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1cf00a3c40 0x7f1cf00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:28.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.955+0000 7f1d02a00640 1 -- 172.21.15.1:0/847712501 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f1d040d0fe0 con 0x7f1d0c106e00 2024-04-26T21:29:28.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.955+0000 7f1d09e00640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1cf00a3c40 0x7f1cf00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:28.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.955+0000 7f1d09e00640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1cf00a3c40 0x7f1cf00a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c19c120 tx=0x7f1cfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:28.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:28.959+0000 7f1d02a00640 1 -- 172.21.15.1:0/847712501 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f1d0409d400 con 0x7f1d0c106e00 2024-04-26T21:29:29.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.155+0000 7f1d0b200640 1 -- 172.21.15.1:0/847712501 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1d0c1a3490 con 0x7f1d0c106e00 2024-04-26T21:29:29.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.156+0000 7f1d02a00640 1 -- 172.21.15.1:0/847712501 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f1d0409cd00 con 0x7f1d0c106e00 2024-04-26T21:29:29.157 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:29:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 -- 172.21.15.1:0/847712501 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1cf00a3c40 msgr2=0x7f1cf00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1cf00a3c40 0x7f1cf00a6100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c19c120 tx=0x7f1cfc005f90 comp rx=0 tx=0).stop 2024-04-26T21:29:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 -- 172.21.15.1:0/847712501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c106e00 msgr2=0x7f1d0c1a21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c106e00 0x7f1d0c1a21c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400eb10 tx=0x7f1d0400c2d0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 -- 172.21.15.1:0/847712501 shutdown_connections 2024-04-26T21:29:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f1cf00a3c40 0x7f1cf00a6100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d0c106e00 0x7f1d0c1a21c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d0c102100 0x7f1d0c19b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 --2- 172.21.15.1:0/847712501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c068f50 0x7f1d0c19ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 -- 172.21.15.1:0/847712501 >> 172.21.15.1:0/847712501 conn(0x7f1d0c0fbf40 msgr2=0x7f1d0c0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.159+0000 7f1d0b200640 1 -- 172.21.15.1:0/847712501 shutdown_connections 2024-04-26T21:29:29.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.160+0000 7f1d0b200640 1 -- 172.21.15.1:0/847712501 wait complete. 2024-04-26T21:29:29.177 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:29:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.389+0000 7fcb4d400640 1 -- 172.21.15.1:0/2939961232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb48076970 msgr2=0x7fcb48074dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.389+0000 7fcb4d400640 1 --2- 172.21.15.1:0/2939961232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb48076970 0x7fcb48074dc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcb38009930 tx=0x7fcb3802a420 comp rx=0 tx=0).stop 2024-04-26T21:29:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.389+0000 7fcb4d400640 1 -- 172.21.15.1:0/2939961232 shutdown_connections 2024-04-26T21:29:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.389+0000 7fcb4d400640 1 --2- 172.21.15.1:0/2939961232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb48075390 0x7fcb48075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.389+0000 7fcb4d400640 1 --2- 172.21.15.1:0/2939961232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb4810ed40 0x7fcb48111130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.389+0000 7fcb4d400640 1 --2- 172.21.15.1:0/2939961232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb48076970 0x7fcb48074dc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.389+0000 7fcb4d400640 1 -- 172.21.15.1:0/2939961232 >> 172.21.15.1:0/2939961232 conn(0x7fcb480fdce0 msgr2=0x7fcb48100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:29.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.389+0000 7fcb4d400640 1 -- 172.21.15.1:0/2939961232 shutdown_connections 2024-04-26T21:29:29.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.389+0000 7fcb4d400640 1 -- 172.21.15.1:0/2939961232 wait complete. 2024-04-26T21:29:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.390+0000 7fcb4d400640 1 Processor -- start 2024-04-26T21:29:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb4d400640 1 -- start start 2024-04-26T21:29:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb4d400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb48075390 0x7fcb4819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb4d400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb48076970 0x7fcb4819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb4d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb4810ed40 0x7fcb481a65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb4d400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcb48114890 con 0x7fcb48076970 2024-04-26T21:29:29.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb4d400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb48114710 con 0x7fcb48075390 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb4d400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb48114a10 con 0x7fcb4810ed40 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb47400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb48076970 0x7fcb4819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb47400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb48076970 0x7fcb4819f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40180/0 (socket says 172.21.15.1:40180) 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb47400640 1 -- 172.21.15.1:0/1459109629 learned_addr learned my addr 172.21.15.1:0/1459109629 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb4ca00640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb4810ed40 0x7fcb481a65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.391+0000 7fcb47e00640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb48075390 0x7fcb4819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.392+0000 7fcb47400640 1 -- 172.21.15.1:0/1459109629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb4810ed40 msgr2=0x7fcb481a65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.392+0000 7fcb47400640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb4810ed40 0x7fcb481a65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.392+0000 7fcb47400640 1 -- 172.21.15.1:0/1459109629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb48075390 msgr2=0x7fcb4819eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.392+0000 7fcb47400640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb48075390 0x7fcb4819eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.392+0000 7fcb47400640 1 -- 172.21.15.1:0/1459109629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb481a6c10 con 0x7fcb48076970 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.392+0000 7fcb47400640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb48076970 0x7fcb4819f530 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400e9f0 tx=0x7fcb3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.392+0000 7fcb44c00640 1 -- 172.21.15.1:0/1459109629 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3400cd80 con 0x7fcb48076970 2024-04-26T21:29:29.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.392+0000 7fcb44c00640 1 -- 172.21.15.1:0/1459109629 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb340054d0 con 0x7fcb48076970 2024-04-26T21:29:29.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.392+0000 7fcb4d400640 1 -- 172.21.15.1:0/1459109629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb481a6f00 con 0x7fcb48076970 2024-04-26T21:29:29.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.392+0000 7fcb44c00640 1 -- 172.21.15.1:0/1459109629 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb34010690 con 0x7fcb48076970 2024-04-26T21:29:29.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.393+0000 7fcb4d400640 1 -- 172.21.15.1:0/1459109629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb481a7370 con 0x7fcb48076970 2024-04-26T21:29:29.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.394+0000 7fcb4d400640 1 -- 172.21.15.1:0/1459109629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb48104f10 con 0x7fcb48076970 2024-04-26T21:29:29.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.396+0000 7fcb44c00640 1 -- 172.21.15.1:0/1459109629 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fcb34004e90 con 0x7fcb48076970 2024-04-26T21:29:29.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.398+0000 7fcb44c00640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcb2c0a3c30 0x7fcb2c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:29.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.398+0000 7fcb44c00640 1 -- 172.21.15.1:0/1459109629 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fcb340d0dd0 con 0x7fcb48076970 2024-04-26T21:29:29.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.398+0000 7fcb47e00640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcb2c0a3c30 0x7fcb2c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:29.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.399+0000 7fcb47e00640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcb2c0a3c30 0x7fcb2c0a60f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcb38006010 tx=0x7fcb380314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:29.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.403+0000 7fcb44c00640 1 -- 172.21.15.1:0/1459109629 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcb3409d1f0 con 0x7fcb48076970 2024-04-26T21:29:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:29 smithi186 ceph-mon[32991]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:29 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3838900068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:29:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:29 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3829762177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:29:29.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:29 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/847712501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:29:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:29 smithi001 ceph-mon[28054]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:29 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3838900068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:29:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:29 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3829762177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:29:29.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:29 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/847712501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:29:29.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:29 smithi038 ceph-mon[32724]: pgmap v401: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:29.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:29 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3838900068' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:29:29.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:29 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3829762177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:29:29.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:29 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/847712501' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:29:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.600+0000 7fcb4d400640 1 -- 172.21.15.1:0/1459109629 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcb4802dcf0 con 0x7fcb48076970 2024-04-26T21:29:29.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.601+0000 7fcb44c00640 1 -- 172.21.15.1:0/1459109629 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fcb3409caf0 con 0x7fcb48076970 2024-04-26T21:29:29.602 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T21:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.603+0000 7fcb4d400640 1 -- 172.21.15.1:0/1459109629 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcb2c0a3c30 msgr2=0x7fcb2c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.603+0000 7fcb4d400640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcb2c0a3c30 0x7fcb2c0a60f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcb38006010 tx=0x7fcb380314a0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.603+0000 7fcb4d400640 1 -- 172.21.15.1:0/1459109629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb48076970 msgr2=0x7fcb4819f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.603+0000 7fcb4d400640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb48076970 0x7fcb4819f530 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400e9f0 tx=0x7fcb3400edf0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.603+0000 7fcb4d400640 1 -- 172.21.15.1:0/1459109629 shutdown_connections 2024-04-26T21:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.603+0000 7fcb4d400640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fcb2c0a3c30 0x7fcb2c0a60f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.603+0000 7fcb4d400640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb4810ed40 0x7fcb481a65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.603+0000 7fcb4d400640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb48076970 0x7fcb4819f530 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.603+0000 7fcb4d400640 1 --2- 172.21.15.1:0/1459109629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb48075390 0x7fcb4819eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.603+0000 7fcb4d400640 1 -- 172.21.15.1:0/1459109629 >> 172.21.15.1:0/1459109629 conn(0x7fcb480fdce0 msgr2=0x7fcb481020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.604+0000 7fcb4d400640 1 -- 172.21.15.1:0/1459109629 shutdown_connections 2024-04-26T21:29:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.604+0000 7fcb4d400640 1 -- 172.21.15.1:0/1459109629 wait complete. 2024-04-26T21:29:29.625 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T21:29:29.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.833+0000 7f860fe00640 1 -- 172.21.15.1:0/4142725930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f861010a960 msgr2=0x7f86101111f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:29.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.833+0000 7f860fe00640 1 --2- 172.21.15.1:0/4142725930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f861010a960 0x7f86101111f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8608009ca0 tx=0x7f860802a8b0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.833+0000 7f860fe00640 1 -- 172.21.15.1:0/4142725930 shutdown_connections 2024-04-26T21:29:29.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.833+0000 7f860fe00640 1 --2- 172.21.15.1:0/4142725930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f861010a960 0x7f86101111f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.833+0000 7f860fe00640 1 --2- 172.21.15.1:0/4142725930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8610109fa0 0x7f861010a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.833+0000 7f860fe00640 1 --2- 172.21.15.1:0/4142725930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8610108db0 0x7f86101091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.833+0000 7f860fe00640 1 -- 172.21.15.1:0/4142725930 >> 172.21.15.1:0/4142725930 conn(0x7f8610077d70 msgr2=0x7f861007a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.833+0000 7f860fe00640 1 -- 172.21.15.1:0/4142725930 shutdown_connections 2024-04-26T21:29:29.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.833+0000 7f860fe00640 1 -- 172.21.15.1:0/4142725930 wait complete. 2024-04-26T21:29:29.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.834+0000 7f860fe00640 1 Processor -- start 2024-04-26T21:29:29.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.834+0000 7f860fe00640 1 -- start start 2024-04-26T21:29:29.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8610108db0 0x7f861019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:29.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8610109fa0 0x7f861019f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:29.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f861010a960 0x7f86101a6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:29.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860fe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f86101148c0 con 0x7f8610109fa0 2024-04-26T21:29:29.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860fe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8610114740 con 0x7f861010a960 2024-04-26T21:29:29.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8610114a40 con 0x7f8610108db0 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860e000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8610109fa0 0x7f861019f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860e000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8610109fa0 0x7f861019f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40188/0 (socket says 172.21.15.1:40188) 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860e000640 1 -- 172.21.15.1:0/2176868769 learned_addr learned my addr 172.21.15.1:0/2176868769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860ea00640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8610108db0 0x7f861019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860f400640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f861010a960 0x7f86101a6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860e000640 1 -- 172.21.15.1:0/2176868769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8610108db0 msgr2=0x7f861019eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860e000640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8610108db0 0x7f861019eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860e000640 1 -- 172.21.15.1:0/2176868769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f861010a960 msgr2=0x7f86101a6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860e000640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f861010a960 0x7f86101a6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.835+0000 7f860e000640 1 -- 172.21.15.1:0/2176868769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86101a6b70 con 0x7f8610109fa0 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.836+0000 7f860e000640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8610109fa0 0x7f861019f530 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00e9f0 tx=0x7f85fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.836+0000 7f8607400640 1 -- 172.21.15.1:0/2176868769 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc00cd80 con 0x7f8610109fa0 2024-04-26T21:29:29.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.836+0000 7f8607400640 1 -- 172.21.15.1:0/2176868769 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85fc0108e0 con 0x7f8610109fa0 2024-04-26T21:29:29.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.836+0000 7f8607400640 1 -- 172.21.15.1:0/2176868769 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc018980 con 0x7f8610109fa0 2024-04-26T21:29:29.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.836+0000 7f860fe00640 1 -- 172.21.15.1:0/2176868769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86101a6e60 con 0x7f8610109fa0 2024-04-26T21:29:29.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.836+0000 7f860fe00640 1 -- 172.21.15.1:0/2176868769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86101a72d0 con 0x7f8610109fa0 2024-04-26T21:29:29.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.837+0000 7f860fe00640 1 -- 172.21.15.1:0/2176868769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86101100d0 con 0x7f8610109fa0 2024-04-26T21:29:29.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.840+0000 7f8607400640 1 -- 172.21.15.1:0/2176868769 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f85fc010430 con 0x7f8610109fa0 2024-04-26T21:29:29.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.841+0000 7f8607400640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85f40a3c40 0x7f85f40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:29.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.842+0000 7f8607400640 1 -- 172.21.15.1:0/2176868769 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f85fc0d0c70 con 0x7f8610109fa0 2024-04-26T21:29:29.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.842+0000 7f860ea00640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85f40a3c40 0x7f85f40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:29.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.843+0000 7f860ea00640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85f40a3c40 0x7f85f40a6100 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8600005fd0 tx=0x7f860000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:29.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:29.846+0000 7f8607400640 1 -- 172.21.15.1:0/2176868769 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f85fc09ca40 con 0x7f8610109fa0 2024-04-26T21:29:30.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.054+0000 7f860fe00640 1 -- 172.21.15.1:0/2176868769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f861002dd10 con 0x7f8610109fa0 2024-04-26T21:29:30.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.055+0000 7f8607400640 1 -- 172.21.15.1:0/2176868769 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f85fc0a0d40 con 0x7f8610109fa0 2024-04-26T21:29:30.056 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:29:30.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.058+0000 7f860fe00640 1 -- 172.21.15.1:0/2176868769 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85f40a3c40 msgr2=0x7f85f40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:30.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.058+0000 7f860fe00640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85f40a3c40 0x7f85f40a6100 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8600005fd0 tx=0x7f860000b410 comp rx=0 tx=0).stop 2024-04-26T21:29:30.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.058+0000 7f860fe00640 1 -- 172.21.15.1:0/2176868769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8610109fa0 msgr2=0x7f861019f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:30.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.058+0000 7f860fe00640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8610109fa0 0x7f861019f530 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00e9f0 tx=0x7f85fc00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.058+0000 7f860fe00640 1 -- 172.21.15.1:0/2176868769 shutdown_connections 2024-04-26T21:29:30.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.058+0000 7f860fe00640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f85f40a3c40 0x7f85f40a6100 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.058+0000 7f860fe00640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f861010a960 0x7f86101a6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.058+0000 7f860fe00640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8610109fa0 0x7f861019f530 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.058+0000 7f860fe00640 1 --2- 172.21.15.1:0/2176868769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8610108db0 0x7f861019eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.058+0000 7f860fe00640 1 -- 172.21.15.1:0/2176868769 >> 172.21.15.1:0/2176868769 conn(0x7f8610077d70 msgr2=0x7f861010d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:30.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.059+0000 7f860fe00640 1 -- 172.21.15.1:0/2176868769 shutdown_connections 2024-04-26T21:29:30.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.059+0000 7f860fe00640 1 -- 172.21.15.1:0/2176868769 wait complete. 2024-04-26T21:29:30.079 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:29:30.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.294+0000 7fa998c00640 1 -- 172.21.15.1:0/354778800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa994104990 msgr2=0x7fa994104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:30.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.294+0000 7fa998c00640 1 --2- 172.21.15.1:0/354778800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa994104990 0x7fa994104db0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa984009ca0 tx=0x7fa98402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.294+0000 7fa998c00640 1 -- 172.21.15.1:0/354778800 shutdown_connections 2024-04-26T21:29:30.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.294+0000 7fa998c00640 1 --2- 172.21.15.1:0/354778800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa994105b80 0x7fa994106000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.294+0000 7fa998c00640 1 --2- 172.21.15.1:0/354778800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa994106540 0x7fa99410cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.294+0000 7fa998c00640 1 --2- 172.21.15.1:0/354778800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa994104990 0x7fa994104db0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.294+0000 7fa998c00640 1 -- 172.21.15.1:0/354778800 >> 172.21.15.1:0/354778800 conn(0x7fa9940fff00 msgr2=0x7fa994102340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:30.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.294+0000 7fa998c00640 1 -- 172.21.15.1:0/354778800 shutdown_connections 2024-04-26T21:29:30.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.294+0000 7fa998c00640 1 -- 172.21.15.1:0/354778800 wait complete. 2024-04-26T21:29:30.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.295+0000 7fa998c00640 1 Processor -- start 2024-04-26T21:29:30.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.295+0000 7fa998c00640 1 -- start start 2024-04-26T21:29:30.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa998c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa994104990 0x7fa99419ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:30.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa998c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa994105b80 0x7fa99419b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:30.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa998c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa994106540 0x7fa9941a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:30.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa998c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa994110500 con 0x7fa994105b80 2024-04-26T21:29:30.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa998c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa994110380 con 0x7fa994106540 2024-04-26T21:29:30.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa998c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa994110680 con 0x7fa994104990 2024-04-26T21:29:30.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa993400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa994104990 0x7fa99419ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa98aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa994105b80 0x7fa99419b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa993400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa994104990 0x7fa99419ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42646/0 (socket says 172.21.15.1:42646) 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa993400640 1 -- 172.21.15.1:0/3611035320 learned_addr learned my addr 172.21.15.1:0/3611035320 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.296+0000 7fa993e00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa994106540 0x7fa9941a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa98aa00640 1 -- 172.21.15.1:0/3611035320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa994104990 msgr2=0x7fa99419ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa98aa00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa994104990 0x7fa99419ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa98aa00640 1 -- 172.21.15.1:0/3611035320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa994106540 msgr2=0x7fa9941a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa98aa00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa994106540 0x7fa9941a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa98aa00640 1 -- 172.21.15.1:0/3611035320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9941a2760 con 0x7fa994105b80 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa98aa00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa994105b80 0x7fa99419b080 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00e9f0 tx=0x7fa97c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa990c00640 1 -- 172.21.15.1:0/3611035320 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c00ccc0 con 0x7fa994105b80 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa990c00640 1 -- 172.21.15.1:0/3611035320 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa97c005020 con 0x7fa994105b80 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa998c00640 1 -- 172.21.15.1:0/3611035320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9941a2a50 con 0x7fa994105b80 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa990c00640 1 -- 172.21.15.1:0/3611035320 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c004a00 con 0x7fa994105b80 2024-04-26T21:29:30.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.297+0000 7fa998c00640 1 -- 172.21.15.1:0/3611035320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9941a2f50 con 0x7fa994105b80 2024-04-26T21:29:30.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.298+0000 7fa998c00640 1 -- 172.21.15.1:0/3611035320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa99410bcb0 con 0x7fa994105b80 2024-04-26T21:29:30.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.301+0000 7fa990c00640 1 -- 172.21.15.1:0/3611035320 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fa97c004ba0 con 0x7fa994105b80 2024-04-26T21:29:30.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.302+0000 7fa990c00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9780a3cf0 0x7fa9780a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:30.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.302+0000 7fa990c00640 1 -- 172.21.15.1:0/3611035320 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fa97c0d09a0 con 0x7fa994105b80 2024-04-26T21:29:30.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.302+0000 7fa993400640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9780a3cf0 0x7fa9780a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:30.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.303+0000 7fa993400640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9780a3cf0 0x7fa9780a61b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa984005fe0 tx=0x7fa984005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:30.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.306+0000 7fa990c00640 1 -- 172.21.15.1:0/3611035320 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa97c09cdc0 con 0x7fa994105b80 2024-04-26T21:29:30.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.553+0000 7fa998c00640 1 -- 172.21.15.1:0/3611035320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa99402dcb0 con 0x7fa994105b80 2024-04-26T21:29:30.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.555+0000 7fa990c00640 1 -- 172.21.15.1:0/3611035320 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11857 (secure 0 0 0) 0x7fa97c09c6c0 con 0x7fa994105b80 2024-04-26T21:29:30.556 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:30.556 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:27:02.795246+0000","modified":"2024-04-26T21:27:28.957703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24572,"mds_2":24548},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24548":{"gid":24548,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24548,24572,24602]},"id":1}]} 2024-04-26T21:29:30.557 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 23 2024-04-26T21:29:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 -- 172.21.15.1:0/3611035320 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9780a3cf0 msgr2=0x7fa9780a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9780a3cf0 0x7fa9780a61b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa984005fe0 tx=0x7fa984005f70 comp rx=0 tx=0).stop 2024-04-26T21:29:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 -- 172.21.15.1:0/3611035320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa994105b80 msgr2=0x7fa99419b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa994105b80 0x7fa99419b080 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00e9f0 tx=0x7fa97c00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 -- 172.21.15.1:0/3611035320 shutdown_connections 2024-04-26T21:29:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9780a3cf0 0x7fa9780a61b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa994106540 0x7fa9941a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa994105b80 0x7fa99419b080 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 --2- 172.21.15.1:0/3611035320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa994104990 0x7fa99419ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 -- 172.21.15.1:0/3611035320 >> 172.21.15.1:0/3611035320 conn(0x7fa9940fff00 msgr2=0x7fa994108e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:30.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.558+0000 7fa998c00640 1 -- 172.21.15.1:0/3611035320 shutdown_connections 2024-04-26T21:29:30.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.559+0000 7fa998c00640 1 -- 172.21.15.1:0/3611035320 wait complete. 2024-04-26T21:29:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:30 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1459109629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:29:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:30 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2176868769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:29:30.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:30 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1459109629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:29:30.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:30 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2176868769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:29:30.581 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:29:30.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:30 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1459109629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:29:30.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:30 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2176868769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:29:30.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.790+0000 7f2a9cc00640 1 -- 172.21.15.1:0/610118640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a98109f80 msgr2=0x7f2a9810a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:30.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.790+0000 7f2a9cc00640 1 --2- 172.21.15.1:0/610118640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a98109f80 0x7f2a9810a400 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2a84009ca0 tx=0x7f2a8402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.790+0000 7f2a9cc00640 1 -- 172.21.15.1:0/610118640 shutdown_connections 2024-04-26T21:29:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.790+0000 7f2a9cc00640 1 --2- 172.21.15.1:0/610118640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a9810a940 0x7f2a981111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.790+0000 7f2a9cc00640 1 --2- 172.21.15.1:0/610118640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a98109f80 0x7f2a9810a400 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.790+0000 7f2a9cc00640 1 --2- 172.21.15.1:0/610118640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a98108d90 0x7f2a981091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.790+0000 7f2a9cc00640 1 -- 172.21.15.1:0/610118640 >> 172.21.15.1:0/610118640 conn(0x7f2a98077d30 msgr2=0x7f2a9807a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.790+0000 7f2a9cc00640 1 -- 172.21.15.1:0/610118640 shutdown_connections 2024-04-26T21:29:30.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.790+0000 7f2a9cc00640 1 -- 172.21.15.1:0/610118640 wait complete. 2024-04-26T21:29:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.791+0000 7f2a9cc00640 1 Processor -- start 2024-04-26T21:29:30.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a9cc00640 1 -- start start 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a9cc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a98108d90 0x7f2a9819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a9cc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a98109f80 0x7f2a9819f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a9cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a9810a940 0x7f2a981a6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a9cc00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2a981148f0 con 0x7f2a98109f80 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a9cc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2a98114770 con 0x7f2a98108d90 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a9cc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2a98114a70 con 0x7f2a9810a940 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a96a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a98109f80 0x7f2a9819f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a97400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a98108d90 0x7f2a9819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a97400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a98108d90 0x7f2a9819eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37400/0 (socket says 172.21.15.1:37400) 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a97400640 1 -- 172.21.15.1:0/1667239477 learned_addr learned my addr 172.21.15.1:0/1667239477 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.792+0000 7f2a97e00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a9810a940 0x7f2a981a6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a97400640 1 -- 172.21.15.1:0/1667239477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a9810a940 msgr2=0x7f2a981a6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:30.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a97400640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a9810a940 0x7f2a981a6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a97400640 1 -- 172.21.15.1:0/1667239477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a98109f80 msgr2=0x7f2a9819f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a97400640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a98109f80 0x7f2a9819f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a97400640 1 -- 172.21.15.1:0/1667239477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a981a6ad0 con 0x7f2a98108d90 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a97e00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a9810a940 0x7f2a981a6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a96a00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a98109f80 0x7f2a9819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a97400640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a98108d90 0x7f2a9819eeb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800e9c0 tx=0x7f2a8800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a8fe00640 1 -- 172.21.15.1:0/1667239477 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8800cde0 con 0x7f2a98108d90 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a8fe00640 1 -- 172.21.15.1:0/1667239477 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a880108e0 con 0x7f2a98108d90 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a9cc00640 1 -- 172.21.15.1:0/1667239477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a981a6dc0 con 0x7f2a98108d90 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.793+0000 7f2a8fe00640 1 -- 172.21.15.1:0/1667239477 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88018960 con 0x7f2a98108d90 2024-04-26T21:29:30.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.794+0000 7f2a9cc00640 1 -- 172.21.15.1:0/1667239477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a981a7280 con 0x7f2a98108d90 2024-04-26T21:29:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.795+0000 7f2a9cc00640 1 -- 172.21.15.1:0/1667239477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a98110050 con 0x7f2a98108d90 2024-04-26T21:29:30.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.797+0000 7f2a8fe00640 1 -- 172.21.15.1:0/1667239477 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f2a8801f020 con 0x7f2a98108d90 2024-04-26T21:29:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.798+0000 7f2a8fe00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2a7c0a3d70 0x7f2a7c0a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.798+0000 7f2a8fe00640 1 -- 172.21.15.1:0/1667239477 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f2a880d1370 con 0x7f2a98108d90 2024-04-26T21:29:30.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.799+0000 7f2a96a00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2a7c0a3d70 0x7f2a7c0a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:30.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.799+0000 7f2a96a00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2a7c0a3d70 0x7f2a7c0a6230 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2a84005e30 tx=0x7f2a84004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:30.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:30.801+0000 7f2a8fe00640 1 -- 172.21.15.1:0/1667239477 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2a88026070 con 0x7f2a98108d90 2024-04-26T21:29:31.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.008+0000 7f2a9cc00640 1 -- 172.21.15.1:0/1667239477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a9802dd40 con 0x7f2a98108d90 2024-04-26T21:29:31.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.011+0000 7f2a8fe00640 1 -- 172.21.15.1:0/1667239477 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24767 (secure 0 0 0) 0x7f2a8809d790 con 0x7f2a98108d90 2024-04-26T21:29:31.012 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:31.012 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":95,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:28:57.077487+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7},{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:29:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 -- 172.21.15.1:0/1667239477 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2a7c0a3d70 msgr2=0x7f2a7c0a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2a7c0a3d70 0x7f2a7c0a6230 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2a84005e30 tx=0x7f2a84004a40 comp rx=0 tx=0).stop 2024-04-26T21:29:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 -- 172.21.15.1:0/1667239477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a98108d90 msgr2=0x7f2a9819eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a98108d90 0x7f2a9819eeb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800e9c0 tx=0x7f2a8800ee80 comp rx=0 tx=0).stop 2024-04-26T21:29:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 -- 172.21.15.1:0/1667239477 shutdown_connections 2024-04-26T21:29:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f2a7c0a3d70 0x7f2a7c0a6230 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a9810a940 0x7f2a981a6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a98109f80 0x7f2a9819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 --2- 172.21.15.1:0/1667239477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a98108d90 0x7f2a9819eeb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 -- 172.21.15.1:0/1667239477 >> 172.21.15.1:0/1667239477 conn(0x7f2a98077d30 msgr2=0x7f2a9810d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:31.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 -- 172.21.15.1:0/1667239477 shutdown_connections 2024-04-26T21:29:31.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.014+0000 7f2a9cc00640 1 -- 172.21.15.1:0/1667239477 wait complete. 2024-04-26T21:29:31.036 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:29:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.254+0000 7fb43b200640 1 -- 172.21.15.1:0/1828333151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb43c108d90 msgr2=0x7fb43c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.254+0000 7fb43b200640 1 --2- 172.21.15.1:0/1828333151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb43c108d90 0x7fb43c1091b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009900 tx=0x7fb42c02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.254+0000 7fb43b200640 1 -- 172.21.15.1:0/1828333151 shutdown_connections 2024-04-26T21:29:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.254+0000 7fb43b200640 1 --2- 172.21.15.1:0/1828333151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c10a940 0x7fb43c1111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.254+0000 7fb43b200640 1 --2- 172.21.15.1:0/1828333151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb43c109f80 0x7fb43c10a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.254+0000 7fb43b200640 1 --2- 172.21.15.1:0/1828333151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb43c108d90 0x7fb43c1091b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.254+0000 7fb43b200640 1 -- 172.21.15.1:0/1828333151 >> 172.21.15.1:0/1828333151 conn(0x7fb43c077d30 msgr2=0x7fb43c07a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:31.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.254+0000 7fb43b200640 1 -- 172.21.15.1:0/1828333151 shutdown_connections 2024-04-26T21:29:31.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.254+0000 7fb43b200640 1 -- 172.21.15.1:0/1828333151 wait complete. 2024-04-26T21:29:31.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.256+0000 7fb43b200640 1 Processor -- start 2024-04-26T21:29:31.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.256+0000 7fb43b200640 1 -- start start 2024-04-26T21:29:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.256+0000 7fb43b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c108d90 0x7fb43c19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.256+0000 7fb43b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb43c109f80 0x7fb43c19f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb43b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb43c10a940 0x7fb43c1a6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb43b200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c114910 con 0x7fb43c10a940 2024-04-26T21:29:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb43b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c114790 con 0x7fb43c108d90 2024-04-26T21:29:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb43b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb43c114a90 con 0x7fb43c109f80 2024-04-26T21:29:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb439e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c108d90 0x7fb43c19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb43a800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb43c10a940 0x7fb43c1a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb439e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c108d90 0x7fb43c19eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37418/0 (socket says 172.21.15.1:37418) 2024-04-26T21:29:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb439e00640 1 -- 172.21.15.1:0/755867180 learned_addr learned my addr 172.21.15.1:0/755867180 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:31.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb431400640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb43c109f80 0x7fb43c19f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb439e00640 1 -- 172.21.15.1:0/755867180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb43c109f80 msgr2=0x7fb43c19f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb439e00640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb43c109f80 0x7fb43c19f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb439e00640 1 -- 172.21.15.1:0/755867180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb43c10a940 msgr2=0x7fb43c1a6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb439e00640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb43c10a940 0x7fb43c1a6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.257+0000 7fb439e00640 1 -- 172.21.15.1:0/755867180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43c1a6af0 con 0x7fb43c108d90 2024-04-26T21:29:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.258+0000 7fb439e00640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c108d90 0x7fb43c19eed0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb42c031500 tx=0x7fb42c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.258+0000 7fb433400640 1 -- 172.21.15.1:0/755867180 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c03f070 con 0x7fb43c108d90 2024-04-26T21:29:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.258+0000 7fb433400640 1 -- 172.21.15.1:0/755867180 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb42c031c50 con 0x7fb43c108d90 2024-04-26T21:29:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.258+0000 7fb43b200640 1 -- 172.21.15.1:0/755867180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43c1a6d80 con 0x7fb43c108d90 2024-04-26T21:29:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.258+0000 7fb433400640 1 -- 172.21.15.1:0/755867180 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c035030 con 0x7fb43c108d90 2024-04-26T21:29:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.258+0000 7fb43b200640 1 -- 172.21.15.1:0/755867180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb43c1a71c0 con 0x7fb43c108d90 2024-04-26T21:29:31.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.259+0000 7fb43b200640 1 -- 172.21.15.1:0/755867180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb43c110050 con 0x7fb43c108d90 2024-04-26T21:29:31.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.261+0000 7fb433400640 1 -- 172.21.15.1:0/755867180 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fb42c042070 con 0x7fb43c108d90 2024-04-26T21:29:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.263+0000 7fb433400640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb4200a3cf0 0x7fb4200a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.263+0000 7fb433400640 1 -- 172.21.15.1:0/755867180 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fb42c0f10f0 con 0x7fb43c108d90 2024-04-26T21:29:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.263+0000 7fb431400640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb4200a3cf0 0x7fb4200a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.264+0000 7fb431400640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb4200a3cf0 0x7fb4200a61b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb43c1a03e0 tx=0x7fb424005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:31.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.268+0000 7fb433400640 1 -- 172.21.15.1:0/755867180 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb42c0bcec0 con 0x7fb43c108d90 2024-04-26T21:29:31.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:31 smithi001 ceph-mon[28054]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:31.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:31 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3611035320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:29:31.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:31 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1667239477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:29:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.522+0000 7fb43b200640 1 -- 172.21.15.1:0/755867180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb43c02dcf0 con 0x7fb43c108d90 2024-04-26T21:29:31.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.523+0000 7fb433400640 1 -- 172.21.15.1:0/755867180 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11857 (secure 0 0 0) 0x7fb42c0c11c0 con 0x7fb43c108d90 2024-04-26T21:29:31.524 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:31.524 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:27:02.795246+0000","modified":"2024-04-26T21:27:28.957703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24572,"mds_2":24548},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24548":{"gid":24548,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24548,24572,24602]},"id":1}]} 2024-04-26T21:29:31.524 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 23 2024-04-26T21:29:31.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.525+0000 7fb43b200640 1 -- 172.21.15.1:0/755867180 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb4200a3cf0 msgr2=0x7fb4200a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.525+0000 7fb43b200640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb4200a3cf0 0x7fb4200a61b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb43c1a03e0 tx=0x7fb424005f90 comp rx=0 tx=0).stop 2024-04-26T21:29:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.525+0000 7fb43b200640 1 -- 172.21.15.1:0/755867180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c108d90 msgr2=0x7fb43c19eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.525+0000 7fb43b200640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c108d90 0x7fb43c19eed0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb42c031500 tx=0x7fb42c003c10 comp rx=0 tx=0).stop 2024-04-26T21:29:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.526+0000 7fb43b200640 1 -- 172.21.15.1:0/755867180 shutdown_connections 2024-04-26T21:29:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.526+0000 7fb43b200640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb4200a3cf0 0x7fb4200a61b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.526+0000 7fb43b200640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb43c109f80 0x7fb43c19f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.526+0000 7fb43b200640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb43c10a940 0x7fb43c1a6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.526+0000 7fb43b200640 1 --2- 172.21.15.1:0/755867180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb43c108d90 0x7fb43c19eed0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.526+0000 7fb43b200640 1 -- 172.21.15.1:0/755867180 >> 172.21.15.1:0/755867180 conn(0x7fb43c077d30 msgr2=0x7fb43c10d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.526+0000 7fb43b200640 1 -- 172.21.15.1:0/755867180 shutdown_connections 2024-04-26T21:29:31.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.526+0000 7fb43b200640 1 -- 172.21.15.1:0/755867180 wait complete. 2024-04-26T21:29:31.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T21:27:02.795246+0000', 'modified': '2024-04-26T21:27:28.957703+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24602, 'mds_1': 24572, 'mds_2': 24548}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24515': {'gid': 24515, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6835/3801556553', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6834', 'nonce': 3801556553}, {'type': 'v1', 'addr': '172.21.15.38:6835', 'nonce': 3801556553}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6837/3582845013', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 3582845013}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 3582845013}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6839/2477017203', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 2477017203}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 2477017203}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24548': {'gid': 24548, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.38:6837/1033703980', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6836', 'nonce': 1033703980}, {'type': 'v1', 'addr': '172.21.15.38:6837', 'nonce': 1033703980}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24572': {'gid': 24572, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.1:6839/338659174', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6838', 'nonce': 338659174}, {'type': 'v1', 'addr': '172.21.15.1:6839', 'nonce': 338659174}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24602': {'gid': 24602, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.38:6839/1675711617', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 1675711617}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 1675711617}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24602, 'qdb_cluster': [24548, 24572, 24602]} 2024-04-26T21:29:31.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T21:29:31.545 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-26T21:29:31.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:31 smithi038 ceph-mon[32724]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:31.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:31 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3611035320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:29:31.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:31 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1667239477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:29:31.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:31 smithi186 ceph-mon[32991]: pgmap v402: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:31.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:31 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3611035320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:29:31.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:31 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1667239477' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:29:31.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.762+0000 7f985b200640 1 -- 172.21.15.1:0/3346835788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985c10a980 msgr2=0x7f985c111210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.762+0000 7f985b200640 1 --2- 172.21.15.1:0/3346835788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985c10a980 0x7f985c111210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9854009ca0 tx=0x7f985402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.762+0000 7f985b200640 1 -- 172.21.15.1:0/3346835788 shutdown_connections 2024-04-26T21:29:31.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.762+0000 7f985b200640 1 --2- 172.21.15.1:0/3346835788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985c10a980 0x7f985c111210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.762+0000 7f985b200640 1 --2- 172.21.15.1:0/3346835788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f985c109fc0 0x7f985c10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.762+0000 7f985b200640 1 --2- 172.21.15.1:0/3346835788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c108dd0 0x7f985c1091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.762+0000 7f985b200640 1 -- 172.21.15.1:0/3346835788 >> 172.21.15.1:0/3346835788 conn(0x7f985c077d70 msgr2=0x7f985c07a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.762+0000 7f985b200640 1 -- 172.21.15.1:0/3346835788 shutdown_connections 2024-04-26T21:29:31.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.762+0000 7f985b200640 1 -- 172.21.15.1:0/3346835788 wait complete. 2024-04-26T21:29:31.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.763+0000 7f985b200640 1 Processor -- start 2024-04-26T21:29:31.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f985b200640 1 -- start start 2024-04-26T21:29:31.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f985b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f985c108dd0 0x7f985c19ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f985b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985c109fc0 0x7f985c19f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f985b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c10a980 0x7f985c1a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f985b200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f985c1148e0 con 0x7f985c109fc0 2024-04-26T21:29:31.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f985b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f985c114760 con 0x7f985c108dd0 2024-04-26T21:29:31.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f985b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f985c114a60 con 0x7f985c10a980 2024-04-26T21:29:31.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f9859400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985c109fc0 0x7f985c19f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f9859e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f985c108dd0 0x7f985c19ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f9859400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985c109fc0 0x7f985c19f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40258/0 (socket says 172.21.15.1:40258) 2024-04-26T21:29:31.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f9859400640 1 -- 172.21.15.1:0/1939578537 learned_addr learned my addr 172.21.15.1:0/1939578537 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:31.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.764+0000 7f985a800640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c10a980 0x7f985c1a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.765+0000 7f9859400640 1 -- 172.21.15.1:0/1939578537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c10a980 msgr2=0x7f985c1a6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.765+0000 7f9859400640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c10a980 0x7f985c1a6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.765+0000 7f9859400640 1 -- 172.21.15.1:0/1939578537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f985c108dd0 msgr2=0x7f985c19ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.765+0000 7f9859400640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f985c108dd0 0x7f985c19ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.765+0000 7f9859400640 1 -- 172.21.15.1:0/1939578537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c1a6ba0 con 0x7f985c109fc0 2024-04-26T21:29:31.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.765+0000 7f9859400640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985c109fc0 0x7f985c19f4c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f984400e9f0 tx=0x7f984400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:31.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.765+0000 7f9852a00640 1 -- 172.21.15.1:0/1939578537 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984400cd80 con 0x7f985c109fc0 2024-04-26T21:29:31.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.765+0000 7f9852a00640 1 -- 172.21.15.1:0/1939578537 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98440108e0 con 0x7f985c109fc0 2024-04-26T21:29:31.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.765+0000 7f985b200640 1 -- 172.21.15.1:0/1939578537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f985c1a6e90 con 0x7f985c109fc0 2024-04-26T21:29:31.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.766+0000 7f9852a00640 1 -- 172.21.15.1:0/1939578537 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9844018980 con 0x7f985c109fc0 2024-04-26T21:29:31.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.766+0000 7f985b200640 1 -- 172.21.15.1:0/1939578537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f985c1a73a0 con 0x7f985c109fc0 2024-04-26T21:29:31.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.769+0000 7f9863e0a740 1 Processor -- start 2024-04-26T21:29:31.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.769+0000 7f9852a00640 1 -- 172.21.15.1:0/1939578537 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9844010430 con 0x7f985c109fc0 2024-04-26T21:29:31.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.769+0000 7f9863e0a740 1 -- start start 2024-04-26T21:29:31.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.769+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261f966e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.769+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261f9b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.769+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f9be20 0x564261f9e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.769+0000 7f9863e0a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564261efa600 con 0x564261f9be20 2024-04-26T21:29:31.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.769+0000 7f9863e0a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564261e97100 con 0x564261f96cb0 2024-04-26T21:29:31.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.769+0000 7f9863e0a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564261ef65c0 con 0x564261f962c0 2024-04-26T21:29:31.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f985a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261f966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f9859400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f9be20 0x564261f9e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f985a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261f966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42708/0 (socket says 172.21.15.1:42708) 2024-04-26T21:29:31.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f985a800640 1 -- 172.21.15.1:0/924906788 learned_addr learned my addr 172.21.15.1:0/924906788 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:31.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f9859e00640 1 --2- 172.21.15.1:0/924906788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261f9b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f985a800640 1 -- 172.21.15.1:0/924906788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 msgr2=0x564261f9b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f985a800640 1 --2- 172.21.15.1:0/924906788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261f9b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f985a800640 1 -- 172.21.15.1:0/924906788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f9be20 msgr2=0x564261f9e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f985a800640 1 --2- 172.21.15.1:0/924906788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f9be20 0x564261f9e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f985a800640 1 -- 172.21.15.1:0/924906788 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564261c0c2d0 con 0x564261f962c0 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f9859e00640 1 --2- 172.21.15.1:0/924906788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261f9b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.770+0000 7f985a800640 1 --2- 172.21.15.1:0/924906788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261f966e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9854009ca0 tx=0x7f9854005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec4401f318abadba server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.771+0000 7f9852a00640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98400a3cb0 0x7f98400a6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.771+0000 7f984b400640 1 -- 172.21.15.1:0/924906788 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854031650 con 0x564261f962c0 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.771+0000 7f984b400640 1 -- 172.21.15.1:0/924906788 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9854036030 con 0x564261f962c0 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.771+0000 7f984b400640 1 -- 172.21.15.1:0/924906788 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854004730 con 0x564261f962c0 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.771+0000 7f9852a00640 1 -- 172.21.15.1:0/1939578537 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f98440d0c50 con 0x7f985c109fc0 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.771+0000 7f9859e00640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98400a3cb0 0x7f98400a6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.771+0000 7f9863e0a740 1 -- 172.21.15.1:0/924906788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 msgr2=0x564261f966e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.771+0000 7f9863e0a740 1 --2- 172.21.15.1:0/924906788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261f966e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9854009ca0 tx=0x7f9854005ef0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.772+0000 7f9859e00640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98400a3cb0 0x7f98400a6170 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x564261f970b0 tx=0x7f984c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.772+0000 7f9863e0a740 1 -- 172.21.15.1:0/924906788 shutdown_connections 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.772+0000 7f9863e0a740 1 --2- 172.21.15.1:0/924906788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f9be20 0x564261f9e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.772+0000 7f9863e0a740 1 --2- 172.21.15.1:0/924906788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261f9b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.772+0000 7f9863e0a740 1 --2- 172.21.15.1:0/924906788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261f966e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.772+0000 7f9863e0a740 1 -- 172.21.15.1:0/924906788 >> 172.21.15.1:0/924906788 conn(0x564261f2ad10 msgr2=0x564261f91d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.772+0000 7f9863e0a740 1 -- 172.21.15.1:0/924906788 shutdown_connections 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.772+0000 7f9863e0a740 1 -- 172.21.15.1:0/924906788 wait complete. 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.772+0000 7f9863e0a740 1 -- start start 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.772+0000 7f9863e0a740 1 Processor -- start 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261fa05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f96cb0 0x564261fa0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f9be20 0x564261fa7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9863e0a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564261f9ebf0 con 0x564261f96cb0 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9863e0a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564261e81f70 con 0x564261f9be20 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9863e0a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564261f9ed70 con 0x564261f962c0 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9863e0a740 10 client.0 _ll_register_callbacks cb 0x564261ef0480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9859400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f96cb0 0x564261fa0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9859e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261fa05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f985a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f9be20 0x564261fa7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9859e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261fa05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42720/0 (socket says 172.21.15.1:42720) 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.773+0000 7f9859e00640 1 -- 172.21.15.1:0/2080135952 learned_addr learned my addr 172.21.15.1:0/2080135952 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f985a800640 1 -- 172.21.15.1:0/2080135952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 msgr2=0x564261fa05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f985a800640 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261fa05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f985a800640 1 -- 172.21.15.1:0/2080135952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f96cb0 msgr2=0x564261fa0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f985a800640 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f96cb0 0x564261fa0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f985a800640 1 -- 172.21.15.1:0/2080135952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564261cd66b0 con 0x564261f9be20 2024-04-26T21:29:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f9859400640 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f96cb0 0x564261fa0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f985a800640 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f9be20 0x564261fa7b20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9854031a20 tx=0x7f9854031780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40472a76ff520b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f983fe00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f983fe00640 1 -- 172.21.15.1:0/2080135952 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854052020 con 0x564261f9be20 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f9863e0a740 1 -- 172.21.15.1:0/2080135952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9854036030 con 0x564261f9be20 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.774+0000 7f983fe00640 1 -- 172.21.15.1:0/2080135952 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f9854003b70 con 0x564261f9be20 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f983fe00640 1 -- 172.21.15.1:0/2080135952 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f985404a040 con 0x564261f9be20 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f9863e0a740 10 client.25199 fetch_fsmap learned FSMap version 23 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f9863e0a740 1 -- 172.21.15.1:0/2080135952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x564261f9ebf0 con 0x564261f9be20 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f983fe00640 1 -- 172.21.15.1:0/2080135952 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9854036030 con 0x564261f9be20 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f983fe00640 1 -- 172.21.15.1:0/2080135952 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f985404d030 con 0x564261f9be20 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f9863e0a740 10 client.25199 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:29:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f9863e0a740 10 client.25199 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f9863e0a740 20 client.25199 populate_metadata read hostname 'smithi001' 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f9863e0a740 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x564261facf00 0x564261fae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f9863e0a740 4 client.25199 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.775+0000 7f9863e0a740 1 -- 172.21.15.1:0/2080135952 --> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564261f08210 con 0x564261facf00 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.776+0000 7f9859e00640 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x564261facf00 0x564261fae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.776+0000 7f9859e00640 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x564261facf00 0x564261fae370 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b66a969845e259e3 server_cookie=70273ce47f82c63 in_seq=0 out_seq=0 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.776+0000 7f983fe00640 10 client.25199 ms_handle_connect on v2:172.21.15.38:6836/1033703980 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.777+0000 7f983fe00640 1 -- 172.21.15.1:0/2080135952 <== mds.2 v2:172.21.15.38:6836/1033703980 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564261f08210 con 0x564261facf00 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.777+0000 7f983fe00640 10 client.25199 handle_command_reply: tid=0 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.777+0000 7f9863e0a740 1 client.25199 shutdown 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.777+0000 7f9863e0a740 1 -- 172.21.15.1:0/2080135952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f9be20 msgr2=0x564261fa7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.777+0000 7f9863e0a740 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f9be20 0x564261fa7b20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9854031a20 tx=0x7f9854031780 comp rx=0 tx=0).stop 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.778+0000 7f9863e0a740 1 -- 172.21.15.1:0/2080135952 shutdown_connections 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.778+0000 7f9863e0a740 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x564261facf00 0x564261fae370 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.778+0000 7f9863e0a740 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f9be20 0x564261fa7b20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.778+0000 7f9863e0a740 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f96cb0 0x564261fa0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.778+0000 7f9863e0a740 1 --2- 172.21.15.1:0/2080135952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f962c0 0x564261fa05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.778+0000 7f983fe00640 0 client.25199 ms_handle_reset on v2:172.21.15.38:6836/1033703980 2024-04-26T21:29:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.778+0000 7f9863e0a740 1 -- 172.21.15.1:0/2080135952 >> 172.21.15.1:0/2080135952 conn(0x564261f2ad10 msgr2=0x564261f90f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.778+0000 7f9863e0a740 1 -- 172.21.15.1:0/2080135952 shutdown_connections 2024-04-26T21:29:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.778+0000 7f9863e0a740 1 -- 172.21.15.1:0/2080135952 wait complete. 2024-04-26T21:29:31.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.778+0000 7f9863e0a740 20 client.25199 trim_cache size 0 max 16384 2024-04-26T21:29:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.798+0000 7f9863e0a740 1 Processor -- start 2024-04-26T21:29:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.798+0000 7f9863e0a740 1 -- start start 2024-04-26T21:29:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.798+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f962c0 0x564261fa05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.798+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f96cb0 0x564261fa0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.798+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f9be20 0x564261fa7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.798+0000 7f9863e0a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564261fac6b0 con 0x564261f9be20 2024-04-26T21:29:31.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.798+0000 7f9863e0a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564261facd80 con 0x564261f962c0 2024-04-26T21:29:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.798+0000 7f9863e0a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564261fad450 con 0x564261f96cb0 2024-04-26T21:29:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f9859e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f962c0 0x564261fa05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f985a800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f9be20 0x564261fa7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f9859e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f962c0 0x564261fa05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37476/0 (socket says 172.21.15.1:37476) 2024-04-26T21:29:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f9859e00640 1 -- 172.21.15.1:0/3574226584 learned_addr learned my addr 172.21.15.1:0/3574226584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:31.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f9859400640 1 --2- 172.21.15.1:0/3574226584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f96cb0 0x564261fa0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f9859e00640 1 -- 172.21.15.1:0/3574226584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f96cb0 msgr2=0x564261fa0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f9859e00640 1 --2- 172.21.15.1:0/3574226584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f96cb0 0x564261fa0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f9859e00640 1 -- 172.21.15.1:0/3574226584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f9be20 msgr2=0x564261fa7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f9859e00640 1 --2- 172.21.15.1:0/3574226584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f9be20 0x564261fa7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f9859e00640 1 -- 172.21.15.1:0/3574226584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564261fae6a0 con 0x564261f962c0 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.799+0000 7f9859e00640 1 --2- 172.21.15.1:0/3574226584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f962c0 0x564261fa05a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f984c009fd0 tx=0x7f984c01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=952304fb87ae87b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f984aa00640 1 -- 172.21.15.1:0/3574226584 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c01dd80 con 0x564261f962c0 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f984aa00640 1 -- 172.21.15.1:0/3574226584 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f984c02c5a0 con 0x564261f962c0 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f984aa00640 1 -- 172.21.15.1:0/3574226584 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c02a9e0 con 0x564261f962c0 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f9863e0a740 1 -- 172.21.15.1:0/3574226584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f962c0 msgr2=0x564261fa05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f9863e0a740 1 --2- 172.21.15.1:0/3574226584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f962c0 0x564261fa05a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f984c009fd0 tx=0x7f984c01b000 comp rx=0 tx=0).stop 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f9863e0a740 1 -- 172.21.15.1:0/3574226584 shutdown_connections 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f9863e0a740 1 --2- 172.21.15.1:0/3574226584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f96cb0 0x564261fa0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f9863e0a740 1 --2- 172.21.15.1:0/3574226584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f9be20 0x564261fa7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f9863e0a740 1 --2- 172.21.15.1:0/3574226584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f962c0 0x564261fa05a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f9863e0a740 1 -- 172.21.15.1:0/3574226584 >> 172.21.15.1:0/3574226584 conn(0x564261f2ad10 msgr2=0x564261f980b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f9863e0a740 1 -- 172.21.15.1:0/3574226584 shutdown_connections 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.800+0000 7f9863e0a740 1 -- 172.21.15.1:0/3574226584 wait complete. 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.801+0000 7f9863e0a740 1 -- start start 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.801+0000 7f9863e0a740 1 Processor -- start 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.801+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f962c0 0x564261fa3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.801+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261fa3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.801+0000 7f9863e0a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f9be20 0x564261fa6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.801+0000 7f9863e0a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564261fae1f0 con 0x564261f962c0 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.801+0000 7f9863e0a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564261fadb20 con 0x564261f96cb0 2024-04-26T21:29:31.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.801+0000 7f9863e0a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564261fae8c0 con 0x564261f9be20 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9863e0a740 10 client.0 _ll_register_callbacks cb 0x7f98440d45d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f962c0 0x564261fa3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261fa3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f962c0 0x564261fa3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40310/0 (socket says 172.21.15.1:40310) 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859e00640 1 -- 172.21.15.1:0/670306407 learned_addr learned my addr 172.21.15.1:0/670306407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261fa3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37492/0 (socket says 172.21.15.1:37492) 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f985a800640 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f9be20 0x564261fa6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859e00640 1 -- 172.21.15.1:0/670306407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f9be20 msgr2=0x564261fa6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859e00640 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f9be20 0x564261fa6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859e00640 1 -- 172.21.15.1:0/670306407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 msgr2=0x564261fa3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859e00640 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261fa3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859e00640 1 -- 172.21.15.1:0/670306407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564261fa70f0 con 0x564261f962c0 2024-04-26T21:29:31.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f985a800640 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f9be20 0x564261fa6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.802+0000 7f9859400640 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261fa3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.803+0000 7f9859e00640 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f962c0 0x564261fa3610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f984c0098c0 tx=0x7f984c0098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383eb2db57a6c2f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.803+0000 7f983fe00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:29:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.803+0000 7f983fe00640 1 -- 172.21.15.1:0/670306407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c02c0f0 con 0x564261f962c0 2024-04-26T21:29:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.803+0000 7f983fe00640 1 -- 172.21.15.1:0/670306407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f984c02c290 con 0x564261f962c0 2024-04-26T21:29:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.803+0000 7f9863e0a740 1 -- 172.21.15.1:0/670306407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f984c02c5a0 con 0x564261f962c0 2024-04-26T21:29:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.803+0000 7f983fe00640 1 -- 172.21.15.1:0/670306407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f984c02a760 con 0x564261f962c0 2024-04-26T21:29:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.803+0000 7f9863e0a740 10 client.15063 fetch_fsmap learned FSMap version 23 2024-04-26T21:29:31.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.803+0000 7f9863e0a740 1 -- 172.21.15.1:0/670306407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5642620131c0 con 0x564261f962c0 2024-04-26T21:29:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.804+0000 7f983fe00640 1 -- 172.21.15.1:0/670306407 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f984c02c5a0 con 0x564261f962c0 2024-04-26T21:29:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.804+0000 7f983fe00640 1 -- 172.21.15.1:0/670306407 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f984c0348b0 con 0x564261f962c0 2024-04-26T21:29:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.804+0000 7f9863e0a740 10 client.15063 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:29:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.804+0000 7f9863e0a740 10 client.15063 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T21:29:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.804+0000 7f9863e0a740 20 client.15063 populate_metadata read hostname 'smithi001' 2024-04-26T21:29:31.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.804+0000 7f9863e0a740 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x564262017a60 0x564262016770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.804+0000 7f9863e0a740 4 client.15063 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.804+0000 7f9863e0a740 1 -- 172.21.15.1:0/670306407 --> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] -- command(tid 0: {"prefix": "status"}) -- 0x564261f02080 con 0x564262017a60 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.804+0000 7f9859400640 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x564262017a60 0x564262016770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.805+0000 7f9859400640 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x564262017a60 0x564262016770 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d96c4554c0c9f915 server_cookie=94cc4dd98ee7318b in_seq=0 out_seq=0 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.805+0000 7f983fe00640 10 client.15063 ms_handle_connect on v2:172.21.15.38:6836/1033703980 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.805+0000 7f983fe00640 1 -- 172.21.15.1:0/670306407 <== mds.2 v2:172.21.15.38:6836/1033703980 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564261f02080 con 0x564262017a60 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.805+0000 7f983fe00640 10 client.15063 handle_command_reply: tid=0 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.805+0000 7f9863e0a740 1 client.15063 shutdown 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.805+0000 7f9863e0a740 1 -- 172.21.15.1:0/670306407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f962c0 msgr2=0x564261fa3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.805+0000 7f9863e0a740 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f962c0 0x564261fa3610 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f984c0098c0 tx=0x7f984c0098f0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f9863e0a740 1 -- 172.21.15.1:0/670306407 shutdown_connections 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f9863e0a740 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x564262017a60 0x564262016770 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f9863e0a740 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564261f9be20 0x564261fa6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f9863e0a740 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564261f96cb0 0x564261fa3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f9863e0a740 1 --2- 172.21.15.1:0/670306407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564261f962c0 0x564261fa3610 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f983fe00640 0 client.15063 ms_handle_reset on v2:172.21.15.38:6836/1033703980 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f9863e0a740 1 -- 172.21.15.1:0/670306407 >> 172.21.15.1:0/670306407 conn(0x564261f2ad10 msgr2=0x564261f980b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f9863e0a740 1 -- 172.21.15.1:0/670306407 shutdown_connections 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f9863e0a740 1 -- 172.21.15.1:0/670306407 wait complete. 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f9863e0a740 20 client.15063 trim_cache size 0 max 16384 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 2, 2024-04-26T21:29:31.811 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24548, 2024-04-26T21:29:31.812 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T21:29:31.812 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T21:29:31.812 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T21:29:31.812 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 123.34187902399999, 2024-04-26T21:29:31.812 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 23, 2024-04-26T21:29:31.812 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 95, 2024-04-26T21:29:31.812 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T21:29:31.812 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 169.75196840999999 2024-04-26T21:29:31.812 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f985b200640 1 -- 172.21.15.1:0/1939578537 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98400a3cb0 msgr2=0x7f98400a6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f985b200640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98400a3cb0 0x7f98400a6170 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x564261f970b0 tx=0x7f984c005f90 comp rx=0 tx=0).stop 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f985b200640 1 -- 172.21.15.1:0/1939578537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985c109fc0 msgr2=0x7f985c19f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.806+0000 7f985b200640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985c109fc0 0x7f985c19f4c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f984400e9f0 tx=0x7f984400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.807+0000 7f985b200640 1 -- 172.21.15.1:0/1939578537 shutdown_connections 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.807+0000 7f985b200640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98400a3cb0 0x7f98400a6170 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.807+0000 7f985b200640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c10a980 0x7f985c1a6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.807+0000 7f985b200640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f985c109fc0 0x7f985c19f4c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.807+0000 7f985b200640 1 --2- 172.21.15.1:0/1939578537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f985c108dd0 0x7f985c19ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.807+0000 7f985b200640 1 -- 172.21.15.1:0/1939578537 >> 172.21.15.1:0/1939578537 conn(0x7f985c077d70 msgr2=0x7f985c10d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:31.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.807+0000 7f985b200640 1 -- 172.21.15.1:0/1939578537 shutdown_connections 2024-04-26T21:29:31.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:31.807+0000 7f985b200640 1 -- 172.21.15.1:0/1939578537 wait complete. 2024-04-26T21:29:31.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-26T21:29:32.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.031+0000 7ffab8000640 1 -- 172.21.15.1:0/844098435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab01067f0 msgr2=0x7ffab010d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.031+0000 7ffab8000640 1 --2- 172.21.15.1:0/844098435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab01067f0 0x7ffab010d080 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8009be0 tx=0x7ffaa802a6c0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.031+0000 7ffab8000640 1 -- 172.21.15.1:0/844098435 shutdown_connections 2024-04-26T21:29:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.031+0000 7ffab8000640 1 --2- 172.21.15.1:0/844098435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab01067f0 0x7ffab010d080 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.031+0000 7ffab8000640 1 --2- 172.21.15.1:0/844098435 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffab0105e30 0x7ffab01062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.031+0000 7ffab8000640 1 --2- 172.21.15.1:0/844098435 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffab0104c40 0x7ffab0105060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.031+0000 7ffab8000640 1 -- 172.21.15.1:0/844098435 >> 172.21.15.1:0/844098435 conn(0x7ffab01001b0 msgr2=0x7ffab01025f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.032+0000 7ffab8000640 1 -- 172.21.15.1:0/844098435 shutdown_connections 2024-04-26T21:29:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.032+0000 7ffab8000640 1 -- 172.21.15.1:0/844098435 wait complete. 2024-04-26T21:29:32.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab8000640 1 Processor -- start 2024-04-26T21:29:32.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab8000640 1 -- start start 2024-04-26T21:29:32.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab8000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffab0104c40 0x7ffab019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab8000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffab0105e30 0x7ffab019b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab8000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab01067f0 0x7ffab01a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab8000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffab0110750 con 0x7ffab0105e30 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab8000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffab01105d0 con 0x7ffab0104c40 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab8000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffab01108d0 con 0x7ffab01067f0 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab4c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffab0105e30 0x7ffab019b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab5600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffab0104c40 0x7ffab019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab5600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffab0104c40 0x7ffab019adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37500/0 (socket says 172.21.15.1:37500) 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.033+0000 7ffab5600640 1 -- 172.21.15.1:0/3598500441 learned_addr learned my addr 172.21.15.1:0/3598500441 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.034+0000 7ffab6000640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab01067f0 0x7ffab01a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.034+0000 7ffab5600640 1 -- 172.21.15.1:0/3598500441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab01067f0 msgr2=0x7ffab01a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.034+0000 7ffab5600640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab01067f0 0x7ffab01a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.034+0000 7ffab5600640 1 -- 172.21.15.1:0/3598500441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffab0105e30 msgr2=0x7ffab019b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.034+0000 7ffab5600640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffab0105e30 0x7ffab019b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.034+0000 7ffab5600640 1 -- 172.21.15.1:0/3598500441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab01a29e0 con 0x7ffab0104c40 2024-04-26T21:29:32.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.034+0000 7ffab4c00640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffab0105e30 0x7ffab019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:32.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.034+0000 7ffab5600640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffab0104c40 0x7ffab019adc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ffaa400e9f0 tx=0x7ffaa400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.034+0000 7ffaae000640 1 -- 172.21.15.1:0/3598500441 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa400ccc0 con 0x7ffab0104c40 2024-04-26T21:29:32.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.035+0000 7ffaae000640 1 -- 172.21.15.1:0/3598500441 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaa40035f0 con 0x7ffab0104c40 2024-04-26T21:29:32.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.035+0000 7ffab8000640 1 -- 172.21.15.1:0/3598500441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab01a2cd0 con 0x7ffab0104c40 2024-04-26T21:29:32.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.035+0000 7ffaae000640 1 -- 172.21.15.1:0/3598500441 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa400ce60 con 0x7ffab0104c40 2024-04-26T21:29:32.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.035+0000 7ffab8000640 1 -- 172.21.15.1:0/3598500441 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab01a31d0 con 0x7ffab0104c40 2024-04-26T21:29:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.038+0000 7ffabaf94740 1 Processor -- start 2024-04-26T21:29:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.038+0000 7ffabaf94740 1 -- start start 2024-04-26T21:29:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.038+0000 7ffaae000640 1 -- 172.21.15.1:0/3598500441 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7ffaa4004000 con 0x7ffab0104c40 2024-04-26T21:29:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.038+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b54fb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.038+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b55005e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.038+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b55030e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.038+0000 7ffabaf94740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5642b545f600 con 0x5642b54fbae0 2024-04-26T21:29:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.038+0000 7ffabaf94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5642b53fc100 con 0x5642b5500c50 2024-04-26T21:29:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.038+0000 7ffabaf94740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5642b545b5c0 con 0x5642b54fb0f0 2024-04-26T21:29:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab6000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b54fb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab5600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b55005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab6000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b54fb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42760/0 (socket says 172.21.15.1:42760) 2024-04-26T21:29:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab6000640 1 -- 172.21.15.1:0/71653451 learned_addr learned my addr 172.21.15.1:0/71653451 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab4c00640 1 --2- 172.21.15.1:0/71653451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b55030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab6000640 1 -- 172.21.15.1:0/71653451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 msgr2=0x5642b55030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab6000640 1 --2- 172.21.15.1:0/71653451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b55030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab6000640 1 -- 172.21.15.1:0/71653451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 msgr2=0x5642b55005e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab6000640 1 --2- 172.21.15.1:0/71653451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b55005e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab6000640 1 -- 172.21.15.1:0/71653451 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642b51712d0 con 0x5642b54fb0f0 2024-04-26T21:29:32.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.039+0000 7ffab6000640 1 --2- 172.21.15.1:0/71653451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b54fb510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffaa802acb0 tx=0x7ffaa8005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4acc73723e5816a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.040+0000 7ffa9e600640 1 -- 172.21.15.1:0/71653451 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8041030 con 0x5642b54fb0f0 2024-04-26T21:29:32.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.040+0000 7ffa9e600640 1 -- 172.21.15.1:0/71653451 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaa8030660 con 0x5642b54fb0f0 2024-04-26T21:29:32.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.040+0000 7ffa9e600640 1 -- 172.21.15.1:0/71653451 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa8036040 con 0x5642b54fb0f0 2024-04-26T21:29:32.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.040+0000 7ffaae000640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffa980a3c40 0x7ffa980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.040+0000 7ffaae000640 1 -- 172.21.15.1:0/3598500441 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7ffaa40d0bd0 con 0x7ffab0104c40 2024-04-26T21:29:32.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.040+0000 7ffab5600640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffa980a3c40 0x7ffa980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffabaf94740 1 -- 172.21.15.1:0/71653451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 msgr2=0x5642b54fb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffabaf94740 1 --2- 172.21.15.1:0/71653451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b54fb510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffaa802acb0 tx=0x7ffaa8005ec0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffabaf94740 1 -- 172.21.15.1:0/71653451 shutdown_connections 2024-04-26T21:29:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffabaf94740 1 --2- 172.21.15.1:0/71653451 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b55030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffabaf94740 1 --2- 172.21.15.1:0/71653451 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b55005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffabaf94740 1 --2- 172.21.15.1:0/71653451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b54fb510 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffabaf94740 1 -- 172.21.15.1:0/71653451 >> 172.21.15.1:0/71653451 conn(0x5642b548fd10 msgr2=0x5642b54f6b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffabaf94740 1 -- 172.21.15.1:0/71653451 shutdown_connections 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffabaf94740 1 -- 172.21.15.1:0/71653451 wait complete. 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffab5600640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffa980a3c40 0x7ffa980a6100 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x5642b54fbee0 tx=0x7ffaa40d4680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.041+0000 7ffabaf94740 1 -- start start 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffabaf94740 1 Processor -- start 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b550dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b550e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b5515270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffabaf94740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5642b5503ba0 con 0x5642b54fbae0 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffabaf94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5642b53e6f70 con 0x5642b5500c50 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffabaf94740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5642b5503d20 con 0x5642b54fb0f0 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffabaf94740 10 client.0 _ll_register_callbacks cb 0x5642b5455480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffab4c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b550e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffab4c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b550e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40346/0 (socket says 172.21.15.1:40346) 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.042+0000 7ffab4c00640 1 -- 172.21.15.1:0/228431786 learned_addr learned my addr 172.21.15.1:0/228431786 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.043+0000 7ffab6000640 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b550dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.043+0000 7ffab6000640 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b5515270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.043+0000 7ffab4c00640 1 -- 172.21.15.1:0/228431786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 msgr2=0x5642b550dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.043+0000 7ffab4c00640 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b550dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.043+0000 7ffab4c00640 1 -- 172.21.15.1:0/228431786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 msgr2=0x5642b5515270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.043+0000 7ffab4c00640 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b5515270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.043+0000 7ffab4c00640 1 -- 172.21.15.1:0/228431786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642b523b6b0 con 0x5642b54fbae0 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.043+0000 7ffab6000640 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b5515270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.043+0000 7ffab4c00640 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b550e230 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0012b10 tx=0x7ffaa000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce079461ef226e27 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.043+0000 7ffa92a00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.044+0000 7ffa92a00640 1 -- 172.21.15.1:0/228431786 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa000fae0 con 0x5642b54fbae0 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.044+0000 7ffabaf94740 1 -- 172.21.15.1:0/228431786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaa8030660 con 0x5642b54fbae0 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.044+0000 7ffa92a00640 1 -- 172.21.15.1:0/228431786 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7ffaa000fc80 con 0x5642b54fbae0 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.044+0000 7ffa92a00640 1 -- 172.21.15.1:0/228431786 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ffaa0025050 con 0x5642b54fbae0 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.044+0000 7ffabaf94740 10 client.15075 fetch_fsmap learned FSMap version 23 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.044+0000 7ffabaf94740 1 -- 172.21.15.1:0/228431786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5642b5518370 con 0x5642b54fbae0 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.044+0000 7ffa92a00640 1 -- 172.21.15.1:0/228431786 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaa8030660 con 0x5642b54fbae0 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.044+0000 7ffa92a00640 1 -- 172.21.15.1:0/228431786 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7ffaa0003c00 con 0x5642b54fbae0 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.045+0000 7ffabaf94740 10 client.15075 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:29:32.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.045+0000 7ffabaf94740 10 client.15075 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T21:29:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.045+0000 7ffabaf94740 20 client.15075 populate_metadata read hostname 'smithi001' 2024-04-26T21:29:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.045+0000 7ffabaf94740 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x5642b551a9d0 0x5642b551be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.045+0000 7ffabaf94740 4 client.15075 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:29:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.045+0000 7ffabaf94740 1 -- 172.21.15.1:0/228431786 --> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642b546d210 con 0x5642b551a9d0 2024-04-26T21:29:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.045+0000 7ffab6000640 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x5642b551a9d0 0x5642b551be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.045+0000 7ffab6000640 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x5642b551a9d0 0x5642b551be40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf955c906f8e18b0 server_cookie=db6af7652987e566 in_seq=0 out_seq=0 2024-04-26T21:29:32.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.045+0000 7ffa92a00640 10 client.15075 ms_handle_connect on v2:172.21.15.1:6838/338659174 2024-04-26T21:29:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.046+0000 7ffa92a00640 1 -- 172.21.15.1:0/228431786 <== mds.1 v2:172.21.15.1:6838/338659174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5642b546d210 con 0x5642b551a9d0 2024-04-26T21:29:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.046+0000 7ffa92a00640 10 client.15075 handle_command_reply: tid=0 2024-04-26T21:29:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.046+0000 7ffabaf94740 1 client.15075 shutdown 2024-04-26T21:29:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffabaf94740 1 -- 172.21.15.1:0/228431786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 msgr2=0x5642b550e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffabaf94740 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b550e230 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0012b10 tx=0x7ffaa000f3c0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffabaf94740 1 -- 172.21.15.1:0/228431786 shutdown_connections 2024-04-26T21:29:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffabaf94740 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x5642b551a9d0 0x5642b551be40 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffabaf94740 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b5515270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffabaf94740 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b550e230 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffabaf94740 1 --2- 172.21.15.1:0/228431786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b550dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffabaf94740 1 -- 172.21.15.1:0/228431786 >> 172.21.15.1:0/228431786 conn(0x5642b548fd10 msgr2=0x5642b54f5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffa92a00640 0 client.15075 ms_handle_reset on v2:172.21.15.1:6838/338659174 2024-04-26T21:29:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffabaf94740 1 -- 172.21.15.1:0/228431786 shutdown_connections 2024-04-26T21:29:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.047+0000 7ffabaf94740 1 -- 172.21.15.1:0/228431786 wait complete. 2024-04-26T21:29:32.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.048+0000 7ffabaf94740 20 client.15075 trim_cache size 0 max 16384 2024-04-26T21:29:32.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.067+0000 7ffabaf94740 1 Processor -- start 2024-04-26T21:29:32.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffabaf94740 1 -- start start 2024-04-26T21:29:32.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fb0f0 0x5642b54fc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b54fbae0 0x5642b5515230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b5500c50 0x5642b54f5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffabaf94740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5642b551a180 con 0x5642b54fb0f0 2024-04-26T21:29:32.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffabaf94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5642b551a850 con 0x5642b54fbae0 2024-04-26T21:29:32.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffabaf94740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5642b551af20 con 0x5642b5500c50 2024-04-26T21:29:32.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffab4c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b54fbae0 0x5642b5515230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffab6000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fb0f0 0x5642b54fc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffab4c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b54fbae0 0x5642b5515230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37532/0 (socket says 172.21.15.1:37532) 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.068+0000 7ffab4c00640 1 -- 172.21.15.1:0/3182764625 learned_addr learned my addr 172.21.15.1:0/3182764625 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffab6000640 1 --2- 172.21.15.1:0/3182764625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b5500c50 0x5642b54f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffab4c00640 1 -- 172.21.15.1:0/3182764625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b5500c50 msgr2=0x5642b54f5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffab4c00640 1 --2- 172.21.15.1:0/3182764625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b5500c50 0x5642b54f5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffab4c00640 1 -- 172.21.15.1:0/3182764625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fb0f0 msgr2=0x5642b54fc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffab4c00640 1 --2- 172.21.15.1:0/3182764625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fb0f0 0x5642b54fc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffab4c00640 1 -- 172.21.15.1:0/3182764625 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642b5518190 con 0x5642b54fbae0 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffab6000640 1 --2- 172.21.15.1:0/3182764625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b5500c50 0x5642b54f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffab4c00640 1 --2- 172.21.15.1:0/3182764625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b54fbae0 0x5642b5515230 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0010480 tx=0x7ffaa0010900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95bd9e8850f21219 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffa9dc00640 1 -- 172.21.15.1:0/3182764625 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0022040 con 0x5642b54fbae0 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffa9dc00640 1 -- 172.21.15.1:0/3182764625 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaa00109f0 con 0x5642b54fbae0 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.069+0000 7ffa9dc00640 1 -- 172.21.15.1:0/3182764625 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0005530 con 0x5642b54fbae0 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.070+0000 7ffabaf94740 1 -- 172.21.15.1:0/3182764625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b54fbae0 msgr2=0x5642b5515230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.070+0000 7ffabaf94740 1 --2- 172.21.15.1:0/3182764625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b54fbae0 0x5642b5515230 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0010480 tx=0x7ffaa0010900 comp rx=0 tx=0).stop 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.070+0000 7ffabaf94740 1 -- 172.21.15.1:0/3182764625 shutdown_connections 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.070+0000 7ffabaf94740 1 --2- 172.21.15.1:0/3182764625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b5500c50 0x5642b54f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.070+0000 7ffabaf94740 1 --2- 172.21.15.1:0/3182764625 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b54fbae0 0x5642b5515230 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.070+0000 7ffabaf94740 1 --2- 172.21.15.1:0/3182764625 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fb0f0 0x5642b54fc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.070+0000 7ffabaf94740 1 -- 172.21.15.1:0/3182764625 >> 172.21.15.1:0/3182764625 conn(0x5642b548fd10 msgr2=0x5642b54fcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.070+0000 7ffabaf94740 1 -- 172.21.15.1:0/3182764625 shutdown_connections 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.070+0000 7ffabaf94740 1 -- 172.21.15.1:0/3182764625 wait complete. 2024-04-26T21:29:32.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.070+0000 7ffabaf94740 1 -- start start 2024-04-26T21:29:32.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffabaf94740 1 Processor -- start 2024-04-26T21:29:32.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b5510ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b5511420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffabaf94740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b5514440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffabaf94740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5642b551bcc0 con 0x5642b54fbae0 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffabaf94740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5642b551b5f0 con 0x5642b5500c50 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffabaf94740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5642b551c390 con 0x5642b54fb0f0 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffabaf94740 10 client.0 _ll_register_callbacks cb 0x7ffaa0009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffab4c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b5511420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffab4c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b5511420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40364/0 (socket says 172.21.15.1:40364) 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffab4c00640 1 -- 172.21.15.1:0/1931095754 learned_addr learned my addr 172.21.15.1:0/1931095754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.071+0000 7ffab6000640 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b5510ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffab4c00640 1 -- 172.21.15.1:0/1931095754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 msgr2=0x5642b5510ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffab4c00640 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b5510ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffab6000640 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b5514440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffab4c00640 1 -- 172.21.15.1:0/1931095754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 msgr2=0x5642b5514440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffab4c00640 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b5514440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffab4c00640 1 -- 172.21.15.1:0/1931095754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642b5514a30 con 0x5642b54fbae0 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffab6000640 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b5514440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffab4c00640 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b5511420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0010520 tx=0x7ffaa0004fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3894dc668b50a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffa92a00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffa92a00640 1 -- 172.21.15.1:0/1931095754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa00052b0 con 0x5642b54fbae0 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffa92a00640 1 -- 172.21.15.1:0/1931095754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7ffaa0005450 con 0x5642b54fbae0 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.072+0000 7ffabaf94740 1 -- 172.21.15.1:0/1931095754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaa00109f0 con 0x5642b54fbae0 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffa92a00640 1 -- 172.21.15.1:0/1931095754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ffaa0005650 con 0x5642b54fbae0 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffabaf94740 10 client.15081 fetch_fsmap learned FSMap version 23 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffabaf94740 1 -- 172.21.15.1:0/1931095754 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5642b5580e40 con 0x5642b54fbae0 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffa92a00640 1 -- 172.21.15.1:0/1931095754 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaa00109f0 con 0x5642b54fbae0 2024-04-26T21:29:32.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffa92a00640 1 -- 172.21.15.1:0/1931095754 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7ffaa0033450 con 0x5642b54fbae0 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffabaf94740 10 client.15081 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffabaf94740 10 client.15081 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffabaf94740 20 client.15081 populate_metadata read hostname 'smithi001' 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffabaf94740 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x5642b55856f0 0x5642b5584400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffabaf94740 4 client.15081 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffabaf94740 1 -- 172.21.15.1:0/1931095754 --> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] -- command(tid 0: {"prefix": "status"}) -- 0x5642b5467080 con 0x5642b55856f0 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.073+0000 7ffab6000640 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x5642b55856f0 0x5642b5584400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.074+0000 7ffab6000640 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x5642b55856f0 0x5642b5584400 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6801ea060b506aaa server_cookie=8ed3dc0cb2da5f29 in_seq=0 out_seq=0 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.074+0000 7ffa92a00640 10 client.15081 ms_handle_connect on v2:172.21.15.1:6838/338659174 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.074+0000 7ffa92a00640 1 -- 172.21.15.1:0/1931095754 <== mds.1 v2:172.21.15.1:6838/338659174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5642b5467080 con 0x5642b55856f0 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.074+0000 7ffa92a00640 10 client.15081 handle_command_reply: tid=0 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.074+0000 7ffabaf94740 1 client.15081 shutdown 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.074+0000 7ffabaf94740 1 -- 172.21.15.1:0/1931095754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 msgr2=0x5642b5511420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.074+0000 7ffabaf94740 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b5511420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0010520 tx=0x7ffaa0004fb0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffabaf94740 1 -- 172.21.15.1:0/1931095754 shutdown_connections 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffabaf94740 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x5642b55856f0 0x5642b5584400 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffabaf94740 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642b5500c50 0x5642b5514440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffabaf94740 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5642b54fbae0 0x5642b5511420 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffabaf94740 1 --2- 172.21.15.1:0/1931095754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5642b54fb0f0 0x5642b5510ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffa92a00640 0 client.15081 ms_handle_reset on v2:172.21.15.1:6838/338659174 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffabaf94740 1 -- 172.21.15.1:0/1931095754 >> 172.21.15.1:0/1931095754 conn(0x5642b548fd10 msgr2=0x5642b54f5620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffabaf94740 1 -- 172.21.15.1:0/1931095754 shutdown_connections 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffabaf94740 1 -- 172.21.15.1:0/1931095754 wait complete. 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffabaf94740 20 client.15081 trim_cache size 0 max 16384 2024-04-26T21:29:32.080 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 1, 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24572, 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 124.62270709400001, 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 23, 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 95, 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T21:29:32.081 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 162.87103985900001 2024-04-26T21:29:32.082 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:29:32.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffab8000640 1 -- 172.21.15.1:0/3598500441 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffa980a3c40 msgr2=0x7ffa980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffab8000640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffa980a3c40 0x7ffa980a6100 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x5642b54fbee0 tx=0x7ffaa40d4680 comp rx=0 tx=0).stop 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffab8000640 1 -- 172.21.15.1:0/3598500441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffab0104c40 msgr2=0x7ffab019adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.075+0000 7ffab8000640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffab0104c40 0x7ffab019adc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ffaa400e9f0 tx=0x7ffaa400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.076+0000 7ffab8000640 1 -- 172.21.15.1:0/3598500441 shutdown_connections 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.076+0000 7ffab8000640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffa980a3c40 0x7ffa980a6100 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.076+0000 7ffab8000640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffab01067f0 0x7ffab01a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.076+0000 7ffab8000640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffab0105e30 0x7ffab019b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.076+0000 7ffab8000640 1 --2- 172.21.15.1:0/3598500441 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffab0104c40 0x7ffab019adc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.076+0000 7ffab8000640 1 -- 172.21.15.1:0/3598500441 >> 172.21.15.1:0/3598500441 conn(0x7ffab01001b0 msgr2=0x7ffab0109180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.076+0000 7ffab8000640 1 -- 172.21.15.1:0/3598500441 shutdown_connections 2024-04-26T21:29:32.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.076+0000 7ffab8000640 1 -- 172.21.15.1:0/3598500441 wait complete. 2024-04-26T21:29:32.092 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T21:29:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.291+0000 7facf3200640 1 -- 172.21.15.1:0/2633760884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facf4108dd0 msgr2=0x7facf41091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.291+0000 7facf3200640 1 --2- 172.21.15.1:0/2633760884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facf4108dd0 0x7facf41091f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7face4009ca0 tx=0x7face402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.291+0000 7facf3200640 1 -- 172.21.15.1:0/2633760884 shutdown_connections 2024-04-26T21:29:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.291+0000 7facf3200640 1 --2- 172.21.15.1:0/2633760884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facf410a980 0x7facf4111210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.291+0000 7facf3200640 1 --2- 172.21.15.1:0/2633760884 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4109fc0 0x7facf410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.291+0000 7facf3200640 1 --2- 172.21.15.1:0/2633760884 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facf4108dd0 0x7facf41091f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.291+0000 7facf3200640 1 -- 172.21.15.1:0/2633760884 >> 172.21.15.1:0/2633760884 conn(0x7facf4077d70 msgr2=0x7facf407a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.291+0000 7facf3200640 1 -- 172.21.15.1:0/2633760884 shutdown_connections 2024-04-26T21:29:32.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.291+0000 7facf3200640 1 -- 172.21.15.1:0/2633760884 wait complete. 2024-04-26T21:29:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.292+0000 7facf3200640 1 Processor -- start 2024-04-26T21:29:32.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.292+0000 7facf3200640 1 -- start start 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.292+0000 7facf3200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4108dd0 0x7facf419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.292+0000 7facf3200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facf4109fc0 0x7facf419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.292+0000 7facf3200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facf410a980 0x7facf41a6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.292+0000 7facf3200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7facf4114900 con 0x7facf410a980 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.292+0000 7facf3200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7facf4114780 con 0x7facf4108dd0 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.292+0000 7facf3200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7facf4114a80 con 0x7facf4109fc0 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf2800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facf410a980 0x7facf41a6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4108dd0 0x7facf419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf2800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facf410a980 0x7facf41a6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40388/0 (socket says 172.21.15.1:40388) 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf2800640 1 -- 172.21.15.1:0/1778728260 learned_addr learned my addr 172.21.15.1:0/1778728260 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf1400640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facf4109fc0 0x7facf419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf2800640 1 -- 172.21.15.1:0/1778728260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facf4109fc0 msgr2=0x7facf419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf2800640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facf4109fc0 0x7facf419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf2800640 1 -- 172.21.15.1:0/1778728260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4108dd0 msgr2=0x7facf419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf2800640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4108dd0 0x7facf419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf2800640 1 -- 172.21.15.1:0/1778728260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf41a6bd0 con 0x7facf410a980 2024-04-26T21:29:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.293+0000 7facf2800640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facf410a980 0x7facf41a6560 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7facec00e970 tx=0x7facec00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.294+0000 7faceaa00640 1 -- 172.21.15.1:0/1778728260 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec00ccc0 con 0x7facf410a980 2024-04-26T21:29:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.294+0000 7faceaa00640 1 -- 172.21.15.1:0/1778728260 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facec00ce60 con 0x7facf410a980 2024-04-26T21:29:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.294+0000 7faceaa00640 1 -- 172.21.15.1:0/1778728260 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec010430 con 0x7facf410a980 2024-04-26T21:29:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.294+0000 7facf3200640 1 -- 172.21.15.1:0/1778728260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf41a6ec0 con 0x7facf410a980 2024-04-26T21:29:32.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.294+0000 7facf3200640 1 -- 172.21.15.1:0/1778728260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf41a7380 con 0x7facf410a980 2024-04-26T21:29:32.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.297+0000 7facfb972740 1 Processor -- start 2024-04-26T21:29:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.297+0000 7faceaa00640 1 -- 172.21.15.1:0/1778728260 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7facec004f90 con 0x7facf410a980 2024-04-26T21:29:32.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.298+0000 7facfb972740 1 -- start start 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.298+0000 7facfb972740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56358449e2c0 0x56358449e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.298+0000 7facfb972740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.298+0000 7facfb972740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635844a3e20 0x5635844a62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.298+0000 7facfb972740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x563584402600 con 0x5635844a3e20 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.298+0000 7facfb972740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56358439f100 con 0x56358449e2c0 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.298+0000 7facfb972740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5635843fe5c0 con 0x56358449ecb0 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635844a3e20 0x5635844a62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844a37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42822/0 (socket says 172.21.15.1:42822) 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1e00640 1 -- 172.21.15.1:0/1346781236 learned_addr learned my addr 172.21.15.1:0/1346781236 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:32.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1400640 1 --2- 172.21.15.1:0/1346781236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56358449e2c0 0x56358449e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1e00640 1 -- 172.21.15.1:0/1346781236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56358449e2c0 msgr2=0x56358449e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1e00640 1 --2- 172.21.15.1:0/1346781236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56358449e2c0 0x56358449e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1e00640 1 -- 172.21.15.1:0/1346781236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635844a3e20 msgr2=0x5635844a62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1e00640 1 --2- 172.21.15.1:0/1346781236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635844a3e20 0x5635844a62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1e00640 1 -- 172.21.15.1:0/1346781236 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635841142d0 con 0x56358449ecb0 2024-04-26T21:29:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7faceaa00640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7facd80a3c30 0x7facd80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.299+0000 7facf1e00640 1 --2- 172.21.15.1:0/1346781236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844a37b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7face4009ca0 tx=0x7face4009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac48f3e321e66687 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.300+0000 7faceaa00640 1 -- 172.21.15.1:0/1778728260 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7facec0d0d30 con 0x7facf410a980 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.300+0000 7facf1400640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7facd80a3c30 0x7facd80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.300+0000 7facdf000640 1 -- 172.21.15.1:0/1346781236 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4031450 con 0x56358449ecb0 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.300+0000 7facdf000640 1 -- 172.21.15.1:0/1346781236 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7face4036030 con 0x56358449ecb0 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.300+0000 7facdf000640 1 -- 172.21.15.1:0/1346781236 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face40043d0 con 0x56358449ecb0 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facfb972740 1 -- 172.21.15.1:0/1346781236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 msgr2=0x5635844a37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facfb972740 1 --2- 172.21.15.1:0/1346781236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844a37b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7face4009ca0 tx=0x7face4009360 comp rx=0 tx=0).stop 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facf1400640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7facd80a3c30 0x7facd80a60f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7face0009fd0 tx=0x7face0009770 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facfb972740 1 -- 172.21.15.1:0/1346781236 shutdown_connections 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facfb972740 1 --2- 172.21.15.1:0/1346781236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844a37b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facfb972740 1 --2- 172.21.15.1:0/1346781236 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635844a3e20 0x5635844a62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facfb972740 1 --2- 172.21.15.1:0/1346781236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56358449e2c0 0x56358449e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facfb972740 1 -- 172.21.15.1:0/1346781236 >> 172.21.15.1:0/1346781236 conn(0x563584432d10 msgr2=0x563584499d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facfb972740 1 -- 172.21.15.1:0/1346781236 shutdown_connections 2024-04-26T21:29:32.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facfb972740 1 -- 172.21.15.1:0/1346781236 wait complete. 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.301+0000 7facfb972740 1 -- start start 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.302+0000 7facfb972740 1 Processor -- start 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.302+0000 7facfb972740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449e2c0 0x5635844a8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.302+0000 7facfb972740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449ecb0 0x5635844a8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.302+0000 7facfb972740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 0x5635844afab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.302+0000 7facfb972740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5635844a6bf0 con 0x56358449ecb0 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.302+0000 7facfb972740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563584389f70 con 0x5635844a3e20 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.302+0000 7facfb972740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5635844a6d70 con 0x56358449e2c0 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.302+0000 7facfb972740 10 client.0 _ll_register_callbacks cb 0x5635843f8480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449ecb0 0x5635844a8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449e2c0 0x5635844a8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449ecb0 0x5635844a8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40400/0 (socket says 172.21.15.1:40400) 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1400640 1 -- 172.21.15.1:0/2144047162 learned_addr learned my addr 172.21.15.1:0/2144047162 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:32.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf2800640 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 0x5635844afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1400640 1 -- 172.21.15.1:0/2144047162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449e2c0 msgr2=0x5635844a8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1400640 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449e2c0 0x5635844a8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1400640 1 -- 172.21.15.1:0/2144047162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 msgr2=0x5635844afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1400640 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 0x5635844afab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1400640 1 -- 172.21.15.1:0/2144047162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635841de6b0 con 0x56358449ecb0 2024-04-26T21:29:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1e00640 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449e2c0 0x5635844a8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:32.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf2800640 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 0x5635844afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facf1400640 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449ecb0 0x5635844a8a70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7face0011a30 tx=0x7face0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c006e7fb0ad8a81 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.303+0000 7facd3400640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.304+0000 7facd3400640 1 -- 172.21.15.1:0/2144047162 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face002e060 con 0x56358449ecb0 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.304+0000 7facd3400640 1 -- 172.21.15.1:0/2144047162 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7face0009040 con 0x56358449ecb0 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.304+0000 7facfb972740 1 -- 172.21.15.1:0/2144047162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7face4036030 con 0x56358449ecb0 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.304+0000 7facd3400640 1 -- 172.21.15.1:0/2144047162 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7face00219b0 con 0x56358449ecb0 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.304+0000 7facfb972740 10 client.15093 fetch_fsmap learned FSMap version 23 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.304+0000 7facfb972740 1 -- 172.21.15.1:0/2144047162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5635844b2b00 con 0x56358449ecb0 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.304+0000 7facd3400640 1 -- 172.21.15.1:0/2144047162 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7face4036030 con 0x56358449ecb0 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.305+0000 7facd3400640 1 -- 172.21.15.1:0/2144047162 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7face001d070 con 0x56358449ecb0 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.305+0000 7facfb972740 10 client.15093 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.305+0000 7facfb972740 10 client.15093 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.305+0000 7facfb972740 20 client.15093 populate_metadata read hostname 'smithi001' 2024-04-26T21:29:32.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.305+0000 7facfb972740 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5635844b51f0 0x5635844b6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.305+0000 7facfb972740 4 client.15093 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.305+0000 7facfb972740 1 -- 172.21.15.1:0/2144047162 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563584410210 con 0x5635844b51f0 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.305+0000 7facf1e00640 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5635844b51f0 0x5635844b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.306+0000 7facf1e00640 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5635844b51f0 0x5635844b6660 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19d471b6ad99e00c server_cookie=220b4f44a0d59d85 in_seq=0 out_seq=0 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.306+0000 7facd3400640 10 client.15093 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facd3400640 1 -- 172.21.15.1:0/2144047162 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563584410210 con 0x5635844b51f0 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facd3400640 10 client.15093 handle_command_reply: tid=0 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 client.15093 shutdown 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 -- 172.21.15.1:0/2144047162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449ecb0 msgr2=0x5635844a8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449ecb0 0x5635844a8a70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7face0011a30 tx=0x7face0004730 comp rx=0 tx=0).stop 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 -- 172.21.15.1:0/2144047162 shutdown_connections 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5635844b51f0 0x5635844b6660 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 0x5635844afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449ecb0 0x5635844a8a70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 --2- 172.21.15.1:0/2144047162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449e2c0 0x5635844a8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facd3400640 0 client.15093 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:29:32.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 -- 172.21.15.1:0/2144047162 >> 172.21.15.1:0/2144047162 conn(0x563584432d10 msgr2=0x563584498f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 -- 172.21.15.1:0/2144047162 shutdown_connections 2024-04-26T21:29:32.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 1 -- 172.21.15.1:0/2144047162 wait complete. 2024-04-26T21:29:32.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.307+0000 7facfb972740 20 client.15093 trim_cache size 0 max 16384 2024-04-26T21:29:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.327+0000 7facfb972740 1 Processor -- start 2024-04-26T21:29:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.327+0000 7facfb972740 1 -- start start 2024-04-26T21:29:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facfb972740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449e2c0 0x5635844a6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facfb972740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844afa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facfb972740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 0x5635844a8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facfb972740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5635844b49a0 con 0x56358449e2c0 2024-04-26T21:29:32.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facfb972740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5635844b5070 con 0x5635844a3e20 2024-04-26T21:29:32.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facfb972740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5635844b5740 con 0x56358449ecb0 2024-04-26T21:29:32.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facf1400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facf2800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 0x5635844a8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facf1400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844afa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42834/0 (socket says 172.21.15.1:42834) 2024-04-26T21:29:32.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facf1400640 1 -- 172.21.15.1:0/3738585021 learned_addr learned my addr 172.21.15.1:0/3738585021 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:32.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facf1e00640 1 --2- 172.21.15.1:0/3738585021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449e2c0 0x5635844a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facf1400640 1 -- 172.21.15.1:0/3738585021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 msgr2=0x5635844a8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facf1400640 1 --2- 172.21.15.1:0/3738585021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 0x5635844a8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facf1400640 1 -- 172.21.15.1:0/3738585021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449e2c0 msgr2=0x5635844a6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facf1400640 1 --2- 172.21.15.1:0/3738585021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449e2c0 0x5635844a6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.328+0000 7facf1400640 1 -- 172.21.15.1:0/3738585021 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635844a3380 con 0x56358449ecb0 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.329+0000 7facf1400640 1 --2- 172.21.15.1:0/3738585021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844afa70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7face0009c50 tx=0x7face0005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=882b7bd11cf014a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.329+0000 7facde600640 1 -- 172.21.15.1:0/3738585021 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face002e040 con 0x56358449ecb0 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.329+0000 7facde600640 1 -- 172.21.15.1:0/3738585021 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7face0021800 con 0x56358449ecb0 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.329+0000 7facde600640 1 -- 172.21.15.1:0/3738585021 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face001e5f0 con 0x56358449ecb0 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 -- 172.21.15.1:0/3738585021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 msgr2=0x5635844afa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 --2- 172.21.15.1:0/3738585021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844afa70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7face0009c50 tx=0x7face0005db0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 -- 172.21.15.1:0/3738585021 shutdown_connections 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 --2- 172.21.15.1:0/3738585021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844afa70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 --2- 172.21.15.1:0/3738585021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5635844a3e20 0x5635844a8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 --2- 172.21.15.1:0/3738585021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56358449e2c0 0x5635844a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 -- 172.21.15.1:0/3738585021 >> 172.21.15.1:0/3738585021 conn(0x563584432d10 msgr2=0x5635844a00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 -- 172.21.15.1:0/3738585021 shutdown_connections 2024-04-26T21:29:32.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 -- 172.21.15.1:0/3738585021 wait complete. 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 -- start start 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.330+0000 7facfb972740 1 Processor -- start 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facfb972740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56358449e2c0 0x5635844ab130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facfb972740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facfb972740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635844a3e20 0x5635844ae690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facfb972740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5635844b64e0 con 0x5635844a3e20 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facfb972740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5635844b5e10 con 0x56358449e2c0 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facfb972740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5635844b6bb0 con 0x56358449ecb0 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facfb972740 10 client.0 _ll_register_callbacks cb 0x7face0003f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facf1400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facf1400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844ab670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42836/0 (socket says 172.21.15.1:42836) 2024-04-26T21:29:32.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facf1400640 1 -- 172.21.15.1:0/170540481 learned_addr learned my addr 172.21.15.1:0/170540481 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facf1e00640 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56358449e2c0 0x5635844ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.331+0000 7facf2800640 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635844a3e20 0x5635844ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facf1400640 1 -- 172.21.15.1:0/170540481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56358449e2c0 msgr2=0x5635844ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facf1400640 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56358449e2c0 0x5635844ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facf1400640 1 -- 172.21.15.1:0/170540481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635844a3e20 msgr2=0x5635844ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facf1400640 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635844a3e20 0x5635844ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facf1400640 1 -- 172.21.15.1:0/170540481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635844aec80 con 0x56358449ecb0 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facf1400640 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844ab670 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7face0005bc0 tx=0x7face0004180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec3780dc2eff9807 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facd3400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facd3400640 1 -- 172.21.15.1:0/170540481 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face003d020 con 0x56358449ecb0 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facfb972740 1 -- 172.21.15.1:0/170540481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7face0021800 con 0x56358449ecb0 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facd3400640 1 -- 172.21.15.1:0/170540481 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7face0009040 con 0x56358449ecb0 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facd3400640 1 -- 172.21.15.1:0/170540481 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7face0048050 con 0x56358449ecb0 2024-04-26T21:29:32.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facfb972740 10 client.25228 fetch_fsmap learned FSMap version 23 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.332+0000 7facfb972740 1 -- 172.21.15.1:0/170540481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x56358451b1a0 con 0x56358449ecb0 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.333+0000 7facd3400640 1 -- 172.21.15.1:0/170540481 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7face0021800 con 0x56358449ecb0 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.333+0000 7facd3400640 1 -- 172.21.15.1:0/170540481 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7face001d040 con 0x56358449ecb0 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.333+0000 7facfb972740 10 client.25228 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.333+0000 7facfb972740 10 client.25228 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.333+0000 7facfb972740 20 client.25228 populate_metadata read hostname 'smithi001' 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.333+0000 7facfb972740 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56358451fbd0 0x56358451c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.333+0000 7facfb972740 4 client.25228 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.333+0000 7facfb972740 1 -- 172.21.15.1:0/170540481 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "status"}) -- 0x56358440a080 con 0x56358451fbd0 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.333+0000 7facf1e00640 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56358451fbd0 0x56358451c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facf1e00640 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56358451fbd0 0x56358451c6e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4dfacf37e2c03bb server_cookie=2b9c48fc9486fe85 in_seq=0 out_seq=0 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facd3400640 10 client.25228 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facd3400640 1 -- 172.21.15.1:0/170540481 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56358440a080 con 0x56358451fbd0 2024-04-26T21:29:32.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facd3400640 10 client.25228 handle_command_reply: tid=0 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facfb972740 1 client.25228 shutdown 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facfb972740 1 -- 172.21.15.1:0/170540481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 msgr2=0x5635844ab670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facfb972740 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844ab670 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7face0005bc0 tx=0x7face0004180 comp rx=0 tx=0).stop 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facfb972740 1 -- 172.21.15.1:0/170540481 shutdown_connections 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facfb972740 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56358451fbd0 0x56358451c6e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facfb972740 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56358449ecb0 0x5635844ab670 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facfb972740 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5635844a3e20 0x5635844ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facfb972740 1 --2- 172.21.15.1:0/170540481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56358449e2c0 0x5635844ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facfb972740 1 -- 172.21.15.1:0/170540481 >> 172.21.15.1:0/170540481 conn(0x563584432d10 msgr2=0x5635844a8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.334+0000 7facd3400640 0 client.25228 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facfb972740 1 -- 172.21.15.1:0/170540481 shutdown_connections 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facfb972740 1 -- 172.21.15.1:0/170540481 wait complete. 2024-04-26T21:29:32.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facfb972740 20 client.25228 trim_cache size 0 max 16384 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24602, 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 149.03958280099999, 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 23, 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 95, 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T21:29:32.340 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 156.073885789 2024-04-26T21:29:32.341 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:29:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facf3200640 1 -- 172.21.15.1:0/1778728260 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7facd80a3c30 msgr2=0x7facd80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facf3200640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7facd80a3c30 0x7facd80a60f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7face0009fd0 tx=0x7face0009770 comp rx=0 tx=0).stop 2024-04-26T21:29:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facf3200640 1 -- 172.21.15.1:0/1778728260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facf410a980 msgr2=0x7facf41a6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:32.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facf3200640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facf410a980 0x7facf41a6560 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7facec00e970 tx=0x7facec00ed70 comp rx=0 tx=0).stop 2024-04-26T21:29:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facf3200640 1 -- 172.21.15.1:0/1778728260 shutdown_connections 2024-04-26T21:29:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facf3200640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7facd80a3c30 0x7facd80a60f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facf3200640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facf4109fc0 0x7facf419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facf3200640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7facf410a980 0x7facf41a6560 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facf3200640 1 --2- 172.21.15.1:0/1778728260 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7facf4108dd0 0x7facf419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.335+0000 7facf3200640 1 -- 172.21.15.1:0/1778728260 >> 172.21.15.1:0/1778728260 conn(0x7facf4077d70 msgr2=0x7facf410d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.336+0000 7facf3200640 1 -- 172.21.15.1:0/1778728260 shutdown_connections 2024-04-26T21:29:32.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:32.336+0000 7facf3200640 1 -- 172.21.15.1:0/1778728260 wait complete. 2024-04-26T21:29:32.350 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:32 smithi001 ceph-mon[28054]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:32.351 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:32 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/755867180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:29:32.353 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T21:29:32.353 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T21:29:32.363 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:29:32.363 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-26T21:29:32.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:32 smithi038 ceph-mon[32724]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:32.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:32 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/755867180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:29:32.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:32 smithi186 ceph-mon[32991]: pgmap v403: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:32.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:32 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/755867180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:29:32.821 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:34.766 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:34 smithi001 ceph-mon[28054]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:34.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:34 smithi038 ceph-mon[32724]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:34.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:34 smithi186 ceph-mon[32991]: pgmap v404: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.290+0000 7f943dc00700 1 -- 172.21.15.1:0/935453657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9438105020 msgr2=0x7f943810b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.290+0000 7f943dc00700 1 --2- 172.21.15.1:0/935453657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9438105020 0x7f943810b8d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f943000b0d0 tx=0x7f943000b3e0 comp rx=0 tx=0).stop 2024-04-26T21:29:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.291+0000 7f943dc00700 1 -- 172.21.15.1:0/935453657 shutdown_connections 2024-04-26T21:29:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.291+0000 7f943dc00700 1 --2- 172.21.15.1:0/935453657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9438103480 0x7f94381038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.291+0000 7f943dc00700 1 --2- 172.21.15.1:0/935453657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9438105020 0x7f943810b8d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.291+0000 7f943dc00700 1 --2- 172.21.15.1:0/935453657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9438104680 0x7f9438104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.291+0000 7f943dc00700 1 -- 172.21.15.1:0/935453657 >> 172.21.15.1:0/935453657 conn(0x7f94380fea20 msgr2=0x7f9438100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.291+0000 7f943dc00700 1 -- 172.21.15.1:0/935453657 shutdown_connections 2024-04-26T21:29:35.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.291+0000 7f943dc00700 1 -- 172.21.15.1:0/935453657 wait complete. 2024-04-26T21:29:35.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.292+0000 7f943dc00700 1 Processor -- start 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.292+0000 7f943dc00700 1 -- start start 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.292+0000 7f943dc00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9438103480 0x7f943819b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.292+0000 7f943dc00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9438104680 0x7f943819bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.292+0000 7f943dc00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9438105020 0x7f94381a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.292+0000 7f943dc00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f943810ef70 con 0x7f9438105020 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.292+0000 7f943dc00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f943810edf0 con 0x7f9438103480 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.292+0000 7f943dc00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f943810f0f0 con 0x7f9438104680 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.293+0000 7f9436a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9438103480 0x7f943819b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.293+0000 7f9436a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9438103480 0x7f943819b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37626/0 (socket says 172.21.15.1:37626) 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.293+0000 7f9436a00700 1 -- 172.21.15.1:0/2877529044 learned_addr learned my addr 172.21.15.1:0/2877529044 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.293+0000 7f9437400700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9438105020 0x7f94381a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:35.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.293+0000 7f9436000700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9438104680 0x7f943819bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.293+0000 7f9436a00700 1 -- 172.21.15.1:0/2877529044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9438104680 msgr2=0x7f943819bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.293+0000 7f9436a00700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9438104680 0x7f943819bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:35.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.293+0000 7f9436a00700 1 -- 172.21.15.1:0/2877529044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9438105020 msgr2=0x7f94381a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.293+0000 7f9436a00700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9438105020 0x7f94381a2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.293+0000 7f9436a00700 1 -- 172.21.15.1:0/2877529044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94381a3430 con 0x7f9438103480 2024-04-26T21:29:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.294+0000 7f9436a00700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9438103480 0x7f943819b7d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f942800e950 tx=0x7f942800ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.294+0000 7f942f400700 1 -- 172.21.15.1:0/2877529044 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9428004230 con 0x7f9438103480 2024-04-26T21:29:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.294+0000 7f942f400700 1 -- 172.21.15.1:0/2877529044 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94280043d0 con 0x7f9438103480 2024-04-26T21:29:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.294+0000 7f943dc00700 1 -- 172.21.15.1:0/2877529044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94381a3720 con 0x7f9438103480 2024-04-26T21:29:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.294+0000 7f942f400700 1 -- 172.21.15.1:0/2877529044 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9428003d60 con 0x7f9438103480 2024-04-26T21:29:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.294+0000 7f943dc00700 1 -- 172.21.15.1:0/2877529044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94381a3c10 con 0x7f9438103480 2024-04-26T21:29:35.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.295+0000 7f943dc00700 1 -- 172.21.15.1:0/2877529044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f943806a110 con 0x7f9438103480 2024-04-26T21:29:35.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.299+0000 7f942f400700 1 -- 172.21.15.1:0/2877529044 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f942800c7e0 con 0x7f9438103480 2024-04-26T21:29:35.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.300+0000 7f942f400700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f941c0a3cd0 0x7f941c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.301+0000 7f942f400700 1 -- 172.21.15.1:0/2877529044 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f94280d0c30 con 0x7f9438103480 2024-04-26T21:29:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.301+0000 7f9436000700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f941c0a3cd0 0x7f941c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:35.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.302+0000 7f9436000700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f941c0a3cd0 0x7f941c0a6190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9424009680 tx=0x7f9424009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:35.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.305+0000 7f942f400700 1 -- 172.21.15.1:0/2877529044 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f942809d050 con 0x7f9438103480 2024-04-26T21:29:35.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.565+0000 7f943dc00700 1 -- 172.21.15.1:0/2877529044 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f9438076d70 con 0x7f941c0a3cd0 2024-04-26T21:29:35.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.567+0000 7f942f400700 1 -- 172.21.15.1:0/2877529044 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f9438076d70 con 0x7f941c0a3cd0 2024-04-26T21:29:35.569 INFO:teuthology.orchestra.run.smithi001.stdout:Backend: cephadm 2024-04-26T21:29:35.569 INFO:teuthology.orchestra.run.smithi001.stdout:Available: Yes 2024-04-26T21:29:35.569 INFO:teuthology.orchestra.run.smithi001.stdout:Paused: No 2024-04-26T21:29:35.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 -- 172.21.15.1:0/2877529044 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f941c0a3cd0 msgr2=0x7f941c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:35.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f941c0a3cd0 0x7f941c0a6190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9424009680 tx=0x7f9424009260 comp rx=0 tx=0).stop 2024-04-26T21:29:35.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 -- 172.21.15.1:0/2877529044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9438103480 msgr2=0x7f943819b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:35.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9438103480 0x7f943819b7d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f942800e950 tx=0x7f942800ec60 comp rx=0 tx=0).stop 2024-04-26T21:29:35.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 -- 172.21.15.1:0/2877529044 shutdown_connections 2024-04-26T21:29:35.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f941c0a3cd0 0x7f941c0a6190 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:35.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9438105020 0x7f94381a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:35.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9438103480 0x7f943819b7d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:35.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 --2- 172.21.15.1:0/2877529044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9438104680 0x7f943819bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:35.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 -- 172.21.15.1:0/2877529044 >> 172.21.15.1:0/2877529044 conn(0x7f94380fea20 msgr2=0x7f9438075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:35.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.570+0000 7f943dc00700 1 -- 172.21.15.1:0/2877529044 shutdown_connections 2024-04-26T21:29:35.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:35.571+0000 7f943dc00700 1 -- 172.21.15.1:0/2877529044 wait complete. 2024-04-26T21:29:36.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-26T21:29:36.755 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:36 smithi001 ceph-mon[28054]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:36.756 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:36 smithi001 ceph-mon[28054]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:36.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:36 smithi038 ceph-mon[32724]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:36.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:36 smithi038 ceph-mon[32724]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:36.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:36 smithi186 ceph-mon[32991]: pgmap v405: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:36.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:36 smithi186 ceph-mon[32991]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:38.776 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:38 smithi001 ceph-mon[28054]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:38.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:38 smithi038 ceph-mon[32724]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:38.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:38 smithi186 ceph-mon[32991]: pgmap v406: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.309+0000 7f5f42c00700 1 -- 172.21.15.1:0/2767747026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c101d00 msgr2=0x7f5f3c102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.309+0000 7f5f42c00700 1 --2- 172.21.15.1:0/2767747026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c101d00 0x7f5f3c102120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c0098f0 tx=0x7f5f2c009c00 comp rx=0 tx=0).stop 2024-04-26T21:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.309+0000 7f5f42c00700 1 -- 172.21.15.1:0/2767747026 shutdown_connections 2024-04-26T21:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.309+0000 7f5f42c00700 1 --2- 172.21.15.1:0/2767747026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f3c103020 0x7f5f3c1034a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.309+0000 7f5f42c00700 1 --2- 172.21.15.1:0/2767747026 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c101d00 0x7f5f3c102120 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.309+0000 7f5f42c00700 1 --2- 172.21.15.1:0/2767747026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f3c102660 0x7f5f3c102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.309+0000 7f5f42c00700 1 -- 172.21.15.1:0/2767747026 >> 172.21.15.1:0/2767747026 conn(0x7f5f3c0fd920 msgr2=0x7f5f3c0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.309+0000 7f5f42c00700 1 -- 172.21.15.1:0/2767747026 shutdown_connections 2024-04-26T21:29:39.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.309+0000 7f5f42c00700 1 -- 172.21.15.1:0/2767747026 wait complete. 2024-04-26T21:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.311+0000 7f5f42c00700 1 Processor -- start 2024-04-26T21:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.311+0000 7f5f42c00700 1 -- start start 2024-04-26T21:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.311+0000 7f5f42c00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c101d00 0x7f5f3c19b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f42c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f3c102660 0x7f5f3c19be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f42c00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f3c103020 0x7f5f3c1a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f42c00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c111130 con 0x7f5f3c103020 2024-04-26T21:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f42c00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c110fb0 con 0x7f5f3c101d00 2024-04-26T21:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f42c00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c1112b0 con 0x7f5f3c102660 2024-04-26T21:29:39.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f3be00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c101d00 0x7f5f3c19b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f3be00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c101d00 0x7f5f3c19b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39702/0 (socket says 172.21.15.1:39702) 2024-04-26T21:29:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f3be00700 1 -- 172.21.15.1:0/3868569544 learned_addr learned my addr 172.21.15.1:0/3868569544 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f33400700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f3c102660 0x7f5f3c19be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f40c00700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f3c103020 0x7f5f3c1a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f3be00700 1 -- 172.21.15.1:0/3868569544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f3c102660 msgr2=0x7f5f3c19be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f3be00700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f3c102660 0x7f5f3c19be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f3be00700 1 -- 172.21.15.1:0/3868569544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f3c103020 msgr2=0x7f5f3c1a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f3be00700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f3c103020 0x7f5f3c1a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:39.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.312+0000 7f5f3be00700 1 -- 172.21.15.1:0/3868569544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3c1a3540 con 0x7f5f3c101d00 2024-04-26T21:29:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.313+0000 7f5f3be00700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c101d00 0x7f5f3c19b8e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c011040 tx=0x7f5f2c005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.313+0000 7f5f39600700 1 -- 172.21.15.1:0/3868569544 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c027070 con 0x7f5f3c101d00 2024-04-26T21:29:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.313+0000 7f5f39600700 1 -- 172.21.15.1:0/3868569544 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f2c00f930 con 0x7f5f3c101d00 2024-04-26T21:29:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.313+0000 7f5f39600700 1 -- 172.21.15.1:0/3868569544 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0049c0 con 0x7f5f3c101d00 2024-04-26T21:29:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.313+0000 7f5f42c00700 1 -- 172.21.15.1:0/3868569544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3c1a37d0 con 0x7f5f3c101d00 2024-04-26T21:29:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.313+0000 7f5f42c00700 1 -- 172.21.15.1:0/3868569544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f3c1a3bd0 con 0x7f5f3c101d00 2024-04-26T21:29:39.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.314+0000 7f5f42c00700 1 -- 172.21.15.1:0/3868569544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f3c06a110 con 0x7f5f3c101d00 2024-04-26T21:29:39.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.317+0000 7f5f39600700 1 -- 172.21.15.1:0/3868569544 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f5f2c02a070 con 0x7f5f3c101d00 2024-04-26T21:29:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.318+0000 7f5f39600700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5f200a3d50 0x7f5f200a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.318+0000 7f5f39600700 1 -- 172.21.15.1:0/3868569544 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f5f2c0d1850 con 0x7f5f3c101d00 2024-04-26T21:29:39.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.319+0000 7f5f33400700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5f200a3d50 0x7f5f200a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:39.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.319+0000 7f5f33400700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5f200a3d50 0x7f5f200a6210 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009680 tx=0x7f5f24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:39.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.321+0000 7f5f39600700 1 -- 172.21.15.1:0/3868569544 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5f2c031080 con 0x7f5f3c101d00 2024-04-26T21:29:39.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.574+0000 7f5f42c00700 1 -- 172.21.15.1:0/3868569544 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f5f3c063860 con 0x7f5f200a3d50 2024-04-26T21:29:39.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.590+0000 7f5f39600700 1 -- 172.21.15.1:0/3868569544 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f5f3c063860 con 0x7f5f200a3d50 2024-04-26T21:29:39.593 INFO:teuthology.orchestra.run.smithi001.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-26T21:29:39.593 INFO:teuthology.orchestra.run.smithi001.stdout:mds.a smithi001 running (2m) 51s ago 2m 12.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 57763fb7fce3 2024-04-26T21:29:39.593 INFO:teuthology.orchestra.run.smithi001.stdout:mds.b smithi038 running (2m) 94s ago 2m 26.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 1ac8426148a5 2024-04-26T21:29:39.593 INFO:teuthology.orchestra.run.smithi001.stdout:mds.c smithi186 running (3m) 94s ago 3m 18.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 f83a8c0b275a 2024-04-26T21:29:39.593 INFO:teuthology.orchestra.run.smithi001.stdout:mds.d smithi001 running (3m) 51s ago 3m 12.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 bdead4d57d83 2024-04-26T21:29:39.593 INFO:teuthology.orchestra.run.smithi001.stdout:mds.e smithi038 running (3m) 94s ago 3m 18.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 ef0064fcc81e 2024-04-26T21:29:39.593 INFO:teuthology.orchestra.run.smithi001.stdout:mds.f smithi186 running (3m) 94s ago 3m 14.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 1fbd5fa336a0 2024-04-26T21:29:39.593 INFO:teuthology.orchestra.run.smithi001.stdout:mds.g smithi001 running (2m) 51s ago 2m 16.4M - 19.0.0-3306-gbc612ffb 55dc54019ad8 fa524eb6c02a 2024-04-26T21:29:39.593 INFO:teuthology.orchestra.run.smithi001.stdout:mds.h smithi038 running (2m) 94s ago 2m 16.3M - 19.0.0-3306-gbc612ffb 55dc54019ad8 eaecce66b2ab 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:mds.i smithi186 running (2m) 94s ago 2m 18.5M - 19.0.0-3306-gbc612ffb 55dc54019ad8 b230804d290a 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:mds.j smithi001 running (3m) 51s ago 3m 12.6M - 19.0.0-3306-gbc612ffb 55dc54019ad8 31aa4a5f93ef 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:mds.k smithi038 running (3m) 94s ago 3m 14.0M - 19.0.0-3306-gbc612ffb 55dc54019ad8 aa549981267a 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:mds.l smithi186 running (3m) 94s ago 3m 14.2M - 19.0.0-3306-gbc612ffb 55dc54019ad8 15b4c1050def 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.x smithi001 *:9283,8765 running (11m) 51s ago 11m 515M - 19.0.0-3306-gbc612ffb 55dc54019ad8 7e9c0a8e9d3d 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.y smithi038 *:8443,8765 running (9m) 94s ago 9m 454M - 19.0.0-3306-gbc612ffb 55dc54019ad8 3a51d5f39ccf 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:mgr.z smithi186 *:8443,8765 running (9m) 94s ago 9m 453M - 19.0.0-3306-gbc612ffb 55dc54019ad8 f6020480e4a4 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:mon.a smithi001 running (11m) 51s ago 11m 49.2M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 acb0745845a2 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:mon.b smithi038 running (9m) 94s ago 9m 40.7M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 c3b8a2fd1e24 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:mon.c smithi186 running (9m) 94s ago 9m 45.3M 2048M 19.0.0-3306-gbc612ffb 55dc54019ad8 2aa078f643b1 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 smithi001 running (8m) 51s ago 8m 52.0M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 5e5a962be261 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 smithi038 running (8m) 94s ago 8m 47.3M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 cd0f4da29464 2024-04-26T21:29:39.594 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 smithi186 running (8m) 94s ago 8m 46.2M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 0abb392a08c6 2024-04-26T21:29:39.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 smithi001 running (7m) 51s ago 7m 51.4M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 d01e8d00b130 2024-04-26T21:29:39.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 smithi038 running (7m) 94s ago 7m 46.9M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 5019cd953c5c 2024-04-26T21:29:39.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 smithi186 running (6m) 94s ago 6m 53.5M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 ffe90ca75bf4 2024-04-26T21:29:39.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6 smithi001 running (6m) 51s ago 6m 42.1M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 01567e331cc2 2024-04-26T21:29:39.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7 smithi038 running (5m) 94s ago 5m 51.7M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 1c4f4afb1f5b 2024-04-26T21:29:39.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8 smithi186 running (5m) 94s ago 5m 54.9M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 a34c1233047a 2024-04-26T21:29:39.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9 smithi001 running (4m) 51s ago 4m 48.4M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 b247478b9757 2024-04-26T21:29:39.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10 smithi038 running (4m) 94s ago 4m 43.1M 4273M 19.0.0-3306-gbc612ffb 55dc54019ad8 9c3c231c5509 2024-04-26T21:29:39.595 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11 smithi186 running (3m) 94s ago 3m 41.1M 4272M 19.0.0-3306-gbc612ffb 55dc54019ad8 fdc4c1f150d4 2024-04-26T21:29:39.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.593+0000 7f5f42c00700 1 -- 172.21.15.1:0/3868569544 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5f200a3d50 msgr2=0x7f5f200a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:39.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.593+0000 7f5f42c00700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5f200a3d50 0x7f5f200a6210 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009680 tx=0x7f5f24009260 comp rx=0 tx=0).stop 2024-04-26T21:29:39.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.593+0000 7f5f42c00700 1 -- 172.21.15.1:0/3868569544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c101d00 msgr2=0x7f5f3c19b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:39.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.593+0000 7f5f42c00700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c101d00 0x7f5f3c19b8e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c011040 tx=0x7f5f2c005270 comp rx=0 tx=0).stop 2024-04-26T21:29:39.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.594+0000 7f5f42c00700 1 -- 172.21.15.1:0/3868569544 shutdown_connections 2024-04-26T21:29:39.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.594+0000 7f5f42c00700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5f200a3d50 0x7f5f200a6210 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:39.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.594+0000 7f5f42c00700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f3c103020 0x7f5f3c1a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:39.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.594+0000 7f5f42c00700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5f3c101d00 0x7f5f3c19b8e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:39.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.594+0000 7f5f42c00700 1 --2- 172.21.15.1:0/3868569544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f3c102660 0x7f5f3c19be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:39.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.594+0000 7f5f42c00700 1 -- 172.21.15.1:0/3868569544 >> 172.21.15.1:0/3868569544 conn(0x7f5f3c0fd920 msgr2=0x7f5f3c100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:39.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.594+0000 7f5f42c00700 1 -- 172.21.15.1:0/3868569544 shutdown_connections 2024-04-26T21:29:39.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:39.594+0000 7f5f42c00700 1 -- 172.21.15.1:0/3868569544 wait complete. 2024-04-26T21:29:40.391 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-26T21:29:40.757 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:40 smithi186 ceph-mon[32991]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:40.757 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:40 smithi186 ceph-mon[32991]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:40.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:40 smithi001 ceph-mon[28054]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:40.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:40 smithi001 ceph-mon[28054]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:40 smithi038 ceph-mon[32724]: pgmap v407: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:40.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:40 smithi038 ceph-mon[32724]: from='client.25250 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:40.858 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:42.710 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:42 smithi001 ceph-mon[28054]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:42.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:42 smithi038 ceph-mon[32724]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:42.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:42 smithi186 ceph-mon[32991]: pgmap v408: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.279+0000 7ffaef200700 1 -- 172.21.15.1:0/2929627800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf01045c0 msgr2=0x7ffaf0104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.279+0000 7ffaef200700 1 --2- 172.21.15.1:0/2929627800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf01045c0 0x7ffaf0104a20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0099c0 tx=0x7ffadc009cd0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.279+0000 7ffaef200700 1 -- 172.21.15.1:0/2929627800 shutdown_connections 2024-04-26T21:29:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.279+0000 7ffaef200700 1 --2- 172.21.15.1:0/2929627800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0104f60 0x7ffaf010b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.279+0000 7ffaef200700 1 --2- 172.21.15.1:0/2929627800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf01033c0 0x7ffaf01037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.279+0000 7ffaef200700 1 --2- 172.21.15.1:0/2929627800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf01045c0 0x7ffaf0104a20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.279+0000 7ffaef200700 1 -- 172.21.15.1:0/2929627800 >> 172.21.15.1:0/2929627800 conn(0x7ffaf00fe960 msgr2=0x7ffaf0100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:43.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.279+0000 7ffaef200700 1 -- 172.21.15.1:0/2929627800 shutdown_connections 2024-04-26T21:29:43.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.279+0000 7ffaef200700 1 -- 172.21.15.1:0/2929627800 wait complete. 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.280+0000 7ffaef200700 1 Processor -- start 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.280+0000 7ffaef200700 1 -- start start 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaef200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf01033c0 0x7ffaf00771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaef200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf01045c0 0x7ffaf0075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaef200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0104f60 0x7ffaf0075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaef200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf010ee90 con 0x7ffaf01045c0 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaef200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf010ed10 con 0x7ffaf01033c0 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaef200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf010f010 con 0x7ffaf0104f60 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaede00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf01033c0 0x7ffaf00771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaee800700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0104f60 0x7ffaf0075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaede00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf01033c0 0x7ffaf00771c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39720/0 (socket says 172.21.15.1:39720) 2024-04-26T21:29:43.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaede00700 1 -- 172.21.15.1:0/374678230 learned_addr learned my addr 172.21.15.1:0/374678230 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaed400700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf01045c0 0x7ffaf0075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaede00700 1 -- 172.21.15.1:0/374678230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0104f60 msgr2=0x7ffaf0075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaede00700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0104f60 0x7ffaf0075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaede00700 1 -- 172.21.15.1:0/374678230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf01045c0 msgr2=0x7ffaf0075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaede00700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf01045c0 0x7ffaf0075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.281+0000 7ffaede00700 1 -- 172.21.15.1:0/374678230 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf0076600 con 0x7ffaf01033c0 2024-04-26T21:29:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.282+0000 7ffaede00700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf01033c0 0x7ffaf00771c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffae000e970 tx=0x7ffae000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.282+0000 7ffae6a00700 1 -- 172.21.15.1:0/374678230 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae0004220 con 0x7ffaf01033c0 2024-04-26T21:29:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.282+0000 7ffae6a00700 1 -- 172.21.15.1:0/374678230 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffae00043c0 con 0x7ffaf01033c0 2024-04-26T21:29:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.282+0000 7ffae6a00700 1 -- 172.21.15.1:0/374678230 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae0003ce0 con 0x7ffaf01033c0 2024-04-26T21:29:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.282+0000 7ffaef200700 1 -- 172.21.15.1:0/374678230 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaf01a36b0 con 0x7ffaf01033c0 2024-04-26T21:29:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.282+0000 7ffaef200700 1 -- 172.21.15.1:0/374678230 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaf01a3b70 con 0x7ffaf01033c0 2024-04-26T21:29:43.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.283+0000 7ffaef200700 1 -- 172.21.15.1:0/374678230 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaf006a110 con 0x7ffaf01033c0 2024-04-26T21:29:43.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.286+0000 7ffae6a00700 1 -- 172.21.15.1:0/374678230 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7ffae000c7e0 con 0x7ffaf01033c0 2024-04-26T21:29:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.287+0000 7ffae6a00700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffad40a3d50 0x7ffad40a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.287+0000 7ffae6a00700 1 -- 172.21.15.1:0/374678230 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7ffae00d0c50 con 0x7ffaf01033c0 2024-04-26T21:29:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.287+0000 7ffaed400700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffad40a3d50 0x7ffad40a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.288+0000 7ffaed400700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffad40a3d50 0x7ffad40a6210 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000c00 tx=0x7ffadc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:43.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.289+0000 7ffae6a00700 1 -- 172.21.15.1:0/374678230 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ffae009d100 con 0x7ffaf01033c0 2024-04-26T21:29:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.534+0000 7ffaef200700 1 -- 172.21.15.1:0/374678230 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7ffaf0063860 con 0x7ffad40a3d50 2024-04-26T21:29:43.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.546+0000 7ffae6a00700 1 -- 172.21.15.1:0/374678230 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7ffaf0063860 con 0x7ffad40a3d50 2024-04-26T21:29:43.548 INFO:teuthology.orchestra.run.smithi001.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-26T21:29:43.548 INFO:teuthology.orchestra.run.smithi001.stdout:mds.all 12/12 98s ago 3m smithi001=a;smithi001=d;smithi001=g;smithi001=j;smithi038=b;smithi038=e;smithi038=h;smithi038=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-26T21:29:43.548 INFO:teuthology.orchestra.run.smithi001.stdout:mgr 3/3 98s ago 9m smithi001=x;smithi038=y;smithi186=z;count:3 2024-04-26T21:29:43.548 INFO:teuthology.orchestra.run.smithi001.stdout:mon 3/3 98s ago 9m smithi001:172.21.15.1=a;smithi038:172.21.15.38=b;smithi186:172.21.15.186=c;count:3 2024-04-26T21:29:43.548 INFO:teuthology.orchestra.run.smithi001.stdout:osd 12 98s ago - 2024-04-26T21:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 -- 172.21.15.1:0/374678230 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffad40a3d50 msgr2=0x7ffad40a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffad40a3d50 0x7ffad40a6210 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffadc000c00 tx=0x7ffadc005fb0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 -- 172.21.15.1:0/374678230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf01033c0 msgr2=0x7ffaf00771c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf01033c0 0x7ffaf00771c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffae000e970 tx=0x7ffae000ec80 comp rx=0 tx=0).stop 2024-04-26T21:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 -- 172.21.15.1:0/374678230 shutdown_connections 2024-04-26T21:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ffad40a3d50 0x7ffad40a6210 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffaf0104f60 0x7ffaf0075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffaf01033c0 0x7ffaf00771c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 --2- 172.21.15.1:0/374678230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaf01045c0 0x7ffaf0075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:43.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 -- 172.21.15.1:0/374678230 >> 172.21.15.1:0/374678230 conn(0x7ffaf00fe960 msgr2=0x7ffaf0107990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:43.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 -- 172.21.15.1:0/374678230 shutdown_connections 2024-04-26T21:29:43.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:43.549+0000 7ffaef200700 1 -- 172.21.15.1:0/374678230 wait complete. 2024-04-26T21:29:44.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-26T21:29:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:44 smithi001 ceph-mon[28054]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:44 smithi001 ceph-mon[28054]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:44.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:44 smithi038 ceph-mon[32724]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:44.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:44 smithi038 ceph-mon[32724]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:44.870 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:44.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:44 smithi186 ceph-mon[32991]: pgmap v409: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:44.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:44 smithi186 ceph-mon[32991]: from='client.25256 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:46.768 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:46 smithi001 ceph-mon[28054]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:46.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:46 smithi038 ceph-mon[32724]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:46.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:46 smithi186 ceph-mon[32991]: pgmap v410: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.071+0000 7f9621000700 1 -- 172.21.15.1:0/2897389525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f961c0799e0 msgr2=0x7f961c07a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.071+0000 7f9621000700 1 --2- 172.21.15.1:0/2897389525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f961c0799e0 0x7f961c07a260 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f961400b0a0 tx=0x7f961400b3b0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.071+0000 7f9621000700 1 -- 172.21.15.1:0/2897389525 shutdown_connections 2024-04-26T21:29:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.071+0000 7f9621000700 1 --2- 172.21.15.1:0/2897389525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f961c077dd0 0x7f961c0781f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.071+0000 7f9621000700 1 --2- 172.21.15.1:0/2897389525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f961c0799e0 0x7f961c07a260 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.071+0000 7f9621000700 1 --2- 172.21.15.1:0/2897389525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f961c079020 0x7f961c0794a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.071+0000 7f9621000700 1 -- 172.21.15.1:0/2897389525 >> 172.21.15.1:0/2897389525 conn(0x7f961c100b60 msgr2=0x7f961c102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.071+0000 7f9621000700 1 -- 172.21.15.1:0/2897389525 shutdown_connections 2024-04-26T21:29:47.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.071+0000 7f9621000700 1 -- 172.21.15.1:0/2897389525 wait complete. 2024-04-26T21:29:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.072+0000 7f9621000700 1 Processor -- start 2024-04-26T21:29:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.073+0000 7f9621000700 1 -- start start 2024-04-26T21:29:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.073+0000 7f9621000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f961c077dd0 0x7f961c19fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.073+0000 7f9621000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f961c079020 0x7f961c19ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.073+0000 7f9621000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f961c0799e0 0x7f961c1a6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:47.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.073+0000 7f9621000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f961c1154b0 con 0x7f961c0799e0 2024-04-26T21:29:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.073+0000 7f9621000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f961c115330 con 0x7f961c079020 2024-04-26T21:29:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.073+0000 7f9621000700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f961c115630 con 0x7f961c077dd0 2024-04-26T21:29:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.073+0000 7f961be00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f961c0799e0 0x7f961c1a6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.073+0000 7f9612a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f961c079020 0x7f961c19ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.073+0000 7f961b400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f961c077dd0 0x7f961c19fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.074+0000 7f9612a00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f961c079020 0x7f961c19ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39744/0 (socket says 172.21.15.1:39744) 2024-04-26T21:29:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.074+0000 7f9612a00700 1 -- 172.21.15.1:0/2112404016 learned_addr learned my addr 172.21.15.1:0/2112404016 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:47.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.074+0000 7f961b400700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f961c077dd0 0x7f961c19fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:57566/0 (socket says 172.21.15.1:57566) 2024-04-26T21:29:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.074+0000 7f9612a00700 1 -- 172.21.15.1:0/2112404016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f961c077dd0 msgr2=0x7f961c19fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.074+0000 7f9612a00700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f961c077dd0 0x7f961c19fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.074+0000 7f9612a00700 1 -- 172.21.15.1:0/2112404016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f961c0799e0 msgr2=0x7f961c1a6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.074+0000 7f9612a00700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f961c0799e0 0x7f961c1a6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.074+0000 7f9612a00700 1 -- 172.21.15.1:0/2112404016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f961c1a7670 con 0x7f961c079020 2024-04-26T21:29:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.074+0000 7f961b400700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f961c077dd0 0x7f961c19fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.074+0000 7f9612a00700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f961c079020 0x7f961c19ff50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f960400e9a0 tx=0x7f960400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:47.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.075+0000 7f9618c00700 1 -- 172.21.15.1:0/2112404016 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960400ca10 con 0x7f961c079020 2024-04-26T21:29:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.075+0000 7f9621000700 1 -- 172.21.15.1:0/2112404016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f961c1a7960 con 0x7f961c079020 2024-04-26T21:29:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.075+0000 7f9618c00700 1 -- 172.21.15.1:0/2112404016 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f960400cbb0 con 0x7f961c079020 2024-04-26T21:29:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.075+0000 7f9618c00700 1 -- 172.21.15.1:0/2112404016 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9604018680 con 0x7f961c079020 2024-04-26T21:29:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.075+0000 7f9621000700 1 -- 172.21.15.1:0/2112404016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f961c1a7ef0 con 0x7f961c079020 2024-04-26T21:29:47.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.076+0000 7f9621000700 1 -- 172.21.15.1:0/2112404016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f961c06a110 con 0x7f961c079020 2024-04-26T21:29:47.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.079+0000 7f9618c00700 1 -- 172.21.15.1:0/2112404016 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f9604010430 con 0x7f961c079020 2024-04-26T21:29:47.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.080+0000 7f9618c00700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f96000a3cd0 0x7f96000a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.080+0000 7f9618c00700 1 -- 172.21.15.1:0/2112404016 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f96040d0d10 con 0x7f961c079020 2024-04-26T21:29:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.080+0000 7f961b400700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f96000a3cd0 0x7f96000a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:47.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.081+0000 7f961b400700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f96000a3cd0 0x7f96000a6190 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f960c011490 tx=0x7f960c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:47.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.082+0000 7f9618c00700 1 -- 172.21.15.1:0/2112404016 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f960409d130 con 0x7f961c079020 2024-04-26T21:29:47.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.323+0000 7f9621000700 1 -- 172.21.15.1:0/2112404016 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f961c063860 con 0x7f96000a3cd0 2024-04-26T21:29:47.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.326+0000 7f9618c00700 1 -- 172.21.15.1:0/2112404016 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f961c063860 con 0x7f96000a3cd0 2024-04-26T21:29:47.328 INFO:teuthology.orchestra.run.smithi001.stdout:HOST ADDR LABELS STATUS 2024-04-26T21:29:47.328 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 172.21.15.1 2024-04-26T21:29:47.328 INFO:teuthology.orchestra.run.smithi001.stdout:smithi038 172.21.15.38 2024-04-26T21:29:47.328 INFO:teuthology.orchestra.run.smithi001.stdout:smithi186 172.21.15.186 2024-04-26T21:29:47.328 INFO:teuthology.orchestra.run.smithi001.stdout:3 hosts in cluster 2024-04-26T21:29:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 -- 172.21.15.1:0/2112404016 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f96000a3cd0 msgr2=0x7f96000a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f96000a3cd0 0x7f96000a6190 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f960c011490 tx=0x7f960c009260 comp rx=0 tx=0).stop 2024-04-26T21:29:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 -- 172.21.15.1:0/2112404016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f961c079020 msgr2=0x7f961c19ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f961c079020 0x7f961c19ff50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f960400e9a0 tx=0x7f960400ecb0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 -- 172.21.15.1:0/2112404016 shutdown_connections 2024-04-26T21:29:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f96000a3cd0 0x7f96000a6190 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f961c077dd0 0x7f961c19fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f961c0799e0 0x7f961c1a6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 --2- 172.21.15.1:0/2112404016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f961c079020 0x7f961c19ff50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 -- 172.21.15.1:0/2112404016 >> 172.21.15.1:0/2112404016 conn(0x7f961c100b60 msgr2=0x7f961c102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.329+0000 7f9621000700 1 -- 172.21.15.1:0/2112404016 shutdown_connections 2024-04-26T21:29:47.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:47.330+0000 7f9621000700 1 -- 172.21.15.1:0/2112404016 wait complete. 2024-04-26T21:29:48.198 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-26T21:29:48.684 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:48.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:48 smithi001 ceph-mon[28054]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:48.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:48 smithi001 ceph-mon[28054]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:48 smithi038 ceph-mon[32724]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:48.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:48 smithi038 ceph-mon[32724]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:48.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:48 smithi186 ceph-mon[32991]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:48.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:48 smithi186 ceph-mon[32991]: pgmap v411: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:50.801 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:50 smithi001 ceph-mon[28054]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:50.801 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:50 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:50 smithi038 ceph-mon[32724]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:51.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:50 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:51.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.097+0000 7fa9a8000700 1 -- 172.21.15.1:0/3223257959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a0105070 msgr2=0x7fa9a010b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.097+0000 7fa9a8000700 1 --2- 172.21.15.1:0/3223257959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a0105070 0x7fa9a010b920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa99800b0d0 tx=0x7fa99800b3e0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.097+0000 7fa9a8000700 1 -- 172.21.15.1:0/3223257959 shutdown_connections 2024-04-26T21:29:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.097+0000 7fa9a8000700 1 --2- 172.21.15.1:0/3223257959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a0105070 0x7fa9a010b920 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.097+0000 7fa9a8000700 1 --2- 172.21.15.1:0/3223257959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9a01034d0 0x7fa9a01038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.097+0000 7fa9a8000700 1 --2- 172.21.15.1:0/3223257959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9a01046d0 0x7fa9a0104b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.097+0000 7fa9a8000700 1 -- 172.21.15.1:0/3223257959 >> 172.21.15.1:0/3223257959 conn(0x7fa9a00fea70 msgr2=0x7fa9a0100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.097+0000 7fa9a8000700 1 -- 172.21.15.1:0/3223257959 shutdown_connections 2024-04-26T21:29:51.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.098+0000 7fa9a8000700 1 -- 172.21.15.1:0/3223257959 wait complete. 2024-04-26T21:29:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.099+0000 7fa9a8000700 1 Processor -- start 2024-04-26T21:29:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.099+0000 7fa9a8000700 1 -- start start 2024-04-26T21:29:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.099+0000 7fa9a8000700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9a01034d0 0x7fa9a019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:51.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.099+0000 7fa9a8000700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9a01046d0 0x7fa9a019bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.099+0000 7fa9a8000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a0105070 0x7fa9a01a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.099+0000 7fa9a8000700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a010efc0 con 0x7fa9a01034d0 2024-04-26T21:29:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.099+0000 7fa9a8000700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a010ee40 con 0x7fa9a0105070 2024-04-26T21:29:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.099+0000 7fa9a8000700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a010f140 con 0x7fa9a01046d0 2024-04-26T21:29:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a5600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9a01034d0 0x7fa9a019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a4c00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9a01046d0 0x7fa9a019bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a5600700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9a01034d0 0x7fa9a019b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52536/0 (socket says 172.21.15.1:52536) 2024-04-26T21:29:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a5600700 1 -- 172.21.15.1:0/106140869 learned_addr learned my addr 172.21.15.1:0/106140869 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:51.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a6000700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a0105070 0x7fa9a01a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a5600700 1 -- 172.21.15.1:0/106140869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9a01046d0 msgr2=0x7fa9a019bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a5600700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9a01046d0 0x7fa9a019bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a5600700 1 -- 172.21.15.1:0/106140869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a0105070 msgr2=0x7fa9a01a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a5600700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a0105070 0x7fa9a01a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a5600700 1 -- 172.21.15.1:0/106140869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a01a3420 con 0x7fa9a01034d0 2024-04-26T21:29:51.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.100+0000 7fa9a6000700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a0105070 0x7fa9a01a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:51.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.101+0000 7fa9a5600700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9a01034d0 0x7fa9a019b7c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa99400d9b0 tx=0x7fa99400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:51.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.101+0000 7fa99e000700 1 -- 172.21.15.1:0/106140869 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994016070 con 0x7fa9a01034d0 2024-04-26T21:29:51.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.101+0000 7fa9a8000700 1 -- 172.21.15.1:0/106140869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a01a3710 con 0x7fa9a01034d0 2024-04-26T21:29:51.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.101+0000 7fa99e000700 1 -- 172.21.15.1:0/106140869 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa99400be00 con 0x7fa9a01034d0 2024-04-26T21:29:51.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.101+0000 7fa99e000700 1 -- 172.21.15.1:0/106140869 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa994010e30 con 0x7fa9a01034d0 2024-04-26T21:29:51.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.101+0000 7fa9a8000700 1 -- 172.21.15.1:0/106140869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a01a3ca0 con 0x7fa9a01034d0 2024-04-26T21:29:51.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.103+0000 7fa9a8000700 1 -- 172.21.15.1:0/106140869 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a006a110 con 0x7fa9a01034d0 2024-04-26T21:29:51.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.106+0000 7fa99e000700 1 -- 172.21.15.1:0/106140869 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fa994019070 con 0x7fa9a01034d0 2024-04-26T21:29:51.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.108+0000 7fa99e000700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9880a3cd0 0x7fa9880a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:51.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.108+0000 7fa99e000700 1 -- 172.21.15.1:0/106140869 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fa9940d0390 con 0x7fa9a01034d0 2024-04-26T21:29:51.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.108+0000 7fa9a4c00700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9880a3cd0 0x7fa9880a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:51.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.109+0000 7fa9a4c00700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9880a3cd0 0x7fa9880a6190 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009be0 tx=0x7fa98c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:51.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.116+0000 7fa99e000700 1 -- 172.21.15.1:0/106140869 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fa994020070 con 0x7fa9a01034d0 2024-04-26T21:29:51.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:50 smithi186 ceph-mon[32991]: pgmap v412: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:29:51.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:50 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:29:51.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.379+0000 7fa9a8000700 1 -- 172.21.15.1:0/106140869 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fa9a0076d70 con 0x7fa9880a3cd0 2024-04-26T21:29:51.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.382+0000 7fa99e000700 1 -- 172.21.15.1:0/106140869 <== mgr.14152 v2:172.21.15.1:6800/2786715792 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fa9a0076d70 con 0x7fa9880a3cd0 2024-04-26T21:29:51.383 INFO:teuthology.orchestra.run.smithi001.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-26T21:29:51.383 INFO:teuthology.orchestra.run.smithi001.stdout:smithi001 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513000D480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-04-26T21:29:51.383 INFO:teuthology.orchestra.run.smithi001.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 3726G No 4m ago Has a FileSystem, LVM detected 2024-04-26T21:29:51.383 INFO:teuthology.orchestra.run.smithi001.stdout:smithi186 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CB400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-26T21:29:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 -- 172.21.15.1:0/106140869 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9880a3cd0 msgr2=0x7fa9880a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9880a3cd0 0x7fa9880a6190 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009be0 tx=0x7fa98c009260 comp rx=0 tx=0).stop 2024-04-26T21:29:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 -- 172.21.15.1:0/106140869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9a01034d0 msgr2=0x7fa9a019b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9a01034d0 0x7fa9a019b7c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa99400d9b0 tx=0x7fa99400dcc0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 -- 172.21.15.1:0/106140869 shutdown_connections 2024-04-26T21:29:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fa9880a3cd0 0x7fa9880a6190 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9a0105070 0x7fa9a01a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa9a01034d0 0x7fa9a019b7c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 --2- 172.21.15.1:0/106140869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9a01046d0 0x7fa9a019bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:51.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 -- 172.21.15.1:0/106140869 >> 172.21.15.1:0/106140869 conn(0x7fa9a00fea70 msgr2=0x7fa9a0075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:51.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 -- 172.21.15.1:0/106140869 shutdown_connections 2024-04-26T21:29:51.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:51.385+0000 7fa9a8000700 1 -- 172.21.15.1:0/106140869 wait complete. 2024-04-26T21:29:52.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:52 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:52 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:52 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:52 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:52 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:52 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.144 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:52 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.144 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:52 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.144 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:52 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.144 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:52 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.144 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:52 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.144 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:52 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:52 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:52 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:52 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:52 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:52 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:52 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:52.312 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-26T21:29:52.323 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:29:52.323 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-26T21:29:53.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:53 smithi001 ceph-mon[28054]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:53.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:53 smithi001 ceph-mon[28054]: pgmap v413: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:53.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:53 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:53 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:53 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:53 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:53 smithi038 ceph-mon[32724]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:53.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:53 smithi038 ceph-mon[32724]: pgmap v413: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:53.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:53 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:53 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:53 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:53 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.378 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:53.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:53 smithi186 ceph-mon[32991]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:29:53.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:53 smithi186 ceph-mon[32991]: pgmap v413: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:29:53.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:53 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:53 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:53 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:53.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:53 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:55.192 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:55 smithi001 ceph-mon[28054]: pgmap v414: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:55.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:55 smithi038 ceph-mon[32724]: pgmap v414: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:55.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:55 smithi186 ceph-mon[32991]: pgmap v414: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:55.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.713+0000 7fe04ee00700 1 -- 172.21.15.1:0/1364012345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048103e40 msgr2=0x7fe048106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:55.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.713+0000 7fe04ee00700 1 --2- 172.21.15.1:0/1364012345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048103e40 0x7fe048106230 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009990 tx=0x7fe03c009ca0 comp rx=0 tx=0).stop 2024-04-26T21:29:55.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.713+0000 7fe04ee00700 1 -- 172.21.15.1:0/1364012345 shutdown_connections 2024-04-26T21:29:55.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.713+0000 7fe04ee00700 1 --2- 172.21.15.1:0/1364012345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048103e40 0x7fe048106230 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:55.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.713+0000 7fe04ee00700 1 --2- 172.21.15.1:0/1364012345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0481090a0 0x7fe04810b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:55.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.713+0000 7fe04ee00700 1 --2- 172.21.15.1:0/1364012345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe048106770 0x7fe048108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:55.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.713+0000 7fe04ee00700 1 -- 172.21.15.1:0/1364012345 >> 172.21.15.1:0/1364012345 conn(0x7fe0480fd850 msgr2=0x7fe0480ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:55.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.713+0000 7fe04ee00700 1 -- 172.21.15.1:0/1364012345 shutdown_connections 2024-04-26T21:29:55.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.713+0000 7fe04ee00700 1 -- 172.21.15.1:0/1364012345 wait complete. 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.714+0000 7fe04ee00700 1 Processor -- start 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.715+0000 7fe04ee00700 1 -- start start 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.715+0000 7fe04ee00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048103e40 0x7fe04819b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.715+0000 7fe04ee00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe048106770 0x7fe04819bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.715+0000 7fe04ee00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0481090a0 0x7fe0481a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.715+0000 7fe04ee00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe04810ee30 con 0x7fe0481090a0 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.715+0000 7fe04ee00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe04810ecb0 con 0x7fe048106770 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.715+0000 7fe04ee00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe04810efb0 con 0x7fe048103e40 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.715+0000 7fe04da00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048103e40 0x7fe04819b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.715+0000 7fe04d000700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe048106770 0x7fe04819bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.715+0000 7fe04da00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048103e40 0x7fe04819b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52312/0 (socket says 172.21.15.1:52312) 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.716+0000 7fe04da00700 1 -- 172.21.15.1:0/2551762514 learned_addr learned my addr 172.21.15.1:0/2551762514 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:55.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.716+0000 7fe04e400700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0481090a0 0x7fe0481a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:55.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.716+0000 7fe04da00700 1 -- 172.21.15.1:0/2551762514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe048106770 msgr2=0x7fe04819bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:55.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.716+0000 7fe04da00700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe048106770 0x7fe04819bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:55.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.716+0000 7fe04da00700 1 -- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0481090a0 msgr2=0x7fe0481a2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:55.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.716+0000 7fe04da00700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0481090a0 0x7fe0481a2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:55.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.716+0000 7fe04da00700 1 -- 172.21.15.1:0/2551762514 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0481a32b0 con 0x7fe048103e40 2024-04-26T21:29:55.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.716+0000 7fe04e400700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0481090a0 0x7fe0481a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:29:55.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.716+0000 7fe04da00700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048103e40 0x7fe04819b650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe03c0059d0 tx=0x7fe03c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:55.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.717+0000 7fe046000700 1 -- 172.21.15.1:0/2551762514 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c027070 con 0x7fe048103e40 2024-04-26T21:29:55.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.717+0000 7fe046000700 1 -- 172.21.15.1:0/2551762514 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe03c00fb80 con 0x7fe048103e40 2024-04-26T21:29:55.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.717+0000 7fe046000700 1 -- 172.21.15.1:0/2551762514 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c017650 con 0x7fe048103e40 2024-04-26T21:29:55.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.717+0000 7fe04ee00700 1 -- 172.21.15.1:0/2551762514 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0481a3540 con 0x7fe048103e40 2024-04-26T21:29:55.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.717+0000 7fe04ee00700 1 -- 172.21.15.1:0/2551762514 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0481a3940 con 0x7fe048103e40 2024-04-26T21:29:55.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.718+0000 7fe04ee00700 1 -- 172.21.15.1:0/2551762514 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe04806a110 con 0x7fe048103e40 2024-04-26T21:29:55.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.722+0000 7fe046000700 1 -- 172.21.15.1:0/2551762514 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fe03c009d10 con 0x7fe048103e40 2024-04-26T21:29:55.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.724+0000 7fe046000700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe0300a3d50 0x7fe0300a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.724+0000 7fe046000700 1 -- 172.21.15.1:0/2551762514 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fe03c0d1a70 con 0x7fe048103e40 2024-04-26T21:29:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.724+0000 7fe04d000700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe0300a3d50 0x7fe0300a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.724+0000 7fe046000700 1 -- 172.21.15.1:0/2551762514 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fe03c013070 con 0x7fe048103e40 2024-04-26T21:29:55.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:55.725+0000 7fe04d000700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe0300a3d50 0x7fe0300a6210 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe034009c60 tx=0x7fe034009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:56.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.002+0000 7fe04ee00700 1 -- 172.21.15.1:0/2551762514 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fe04802e490 con 0x7fe048103e40 2024-04-26T21:29:56.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.003+0000 7fe046000700 1 -- 172.21.15.1:0/2551762514 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3042 (secure 0 0 0) 0x7fe03c09df20 con 0x7fe048103e40 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:e23 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:legacy client fscid: 1 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:Filesystem 'cephfs' (1) 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:fs_name cephfs 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 23 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-04-26T21:27:02.795246+0000 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-04-26T21:27:28.957703+0000 2024-04-26T21:29:56.004 INFO:teuthology.orchestra.run.smithi001.stdout:tableserver 0 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:root 0 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:session_timeout 300 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:session_autoclose 300 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:max_file_size 1099511627776 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:max_xattr_size 65536 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:required_client_features {} 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure 0 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:last_failure_osd_epoch 0 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:max_mds 3 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:in 0,1,2 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:up {0=24602,1=24572,2=24548} 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:failed 2024-04-26T21:29:56.005 INFO:teuthology.orchestra.run.smithi001.stdout:damaged 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:stopped 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:data_pools [3,4] 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:metadata_pool 2 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:inline_data disabled 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:balancer 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:bal_rank_mask -1 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:standby_count_wanted 0 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:qdb_cluster leader: 24602 members: 24548,24572,24602 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b{0:24602} state up:active seq 3 addr [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.i{0:24547} state up:standby-replay seq 1 addr [v2:172.21.15.186:6838/2477017203,v1:172.21.15.186:6839/2477017203] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.g{1:24572} state up:active seq 11 addr [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c{1:24523} state up:standby-replay seq 1 addr [v2:172.21.15.186:6836/3582845013,v1:172.21.15.186:6837/3582845013] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.h{2:24548} state up:active seq 13 addr [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.006 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.e{2:24515} state up:standby-replay seq 1 addr [v2:172.21.15.38:6834/3801556553,v1:172.21.15.38:6835/3801556553] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:56.007 INFO:teuthology.orchestra.run.smithi001.stdout:Standby daemons: 2024-04-26T21:29:56.007 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:29:56.007 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.j{-1:14385} state up:standby seq 1 addr [v2:172.21.15.1:6834/2332941873,v1:172.21.15.1:6835/2332941873] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.007 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a{-1:14469} state up:standby seq 1 addr [v2:172.21.15.1:6840/2045981022,v1:172.21.15.1:6841/2045981022] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.007 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.l{-1:24451} state up:standby seq 1 addr [v2:172.21.15.186:6832/1838831066,v1:172.21.15.186:6833/1838831066] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.007 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.k{-1:24467} state up:standby seq 1 addr [v2:172.21.15.38:6832/1237951568,v1:172.21.15.38:6833/1237951568] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.007 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d{-1:24503} state up:standby seq 1 addr [v2:172.21.15.1:6836/532288494,v1:172.21.15.1:6837/532288494] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.007 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.f{-1:24511} state up:standby seq 1 addr [v2:172.21.15.186:6834/1778623983,v1:172.21.15.186:6835/1778623983] compat {c=[1],r=[1],i=[1fff]}] 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 -- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe0300a3d50 msgr2=0x7fe0300a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe0300a3d50 0x7fe0300a6210 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe034009c60 tx=0x7fe034009260 comp rx=0 tx=0).stop 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 -- 172.21.15.1:0/2551762514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048103e40 msgr2=0x7fe04819b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048103e40 0x7fe04819b650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe03c0059d0 tx=0x7fe03c0047b0 comp rx=0 tx=0).stop 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 -- 172.21.15.1:0/2551762514 shutdown_connections 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fe0300a3d50 0x7fe0300a6210 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe048103e40 0x7fe04819b650 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0481090a0 0x7fe0481a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 --2- 172.21.15.1:0/2551762514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe048106770 0x7fe04819bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 -- 172.21.15.1:0/2551762514 >> 172.21.15.1:0/2551762514 conn(0x7fe0480fd850 msgr2=0x7fe048103580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 -- 172.21.15.1:0/2551762514 shutdown_connections 2024-04-26T21:29:56.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:56.007+0000 7fe04ee00700 1 -- 172.21.15.1:0/2551762514 wait complete. 2024-04-26T21:29:56.009 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 23 2024-04-26T21:29:56.979 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-26T21:29:57.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:56 smithi186 ceph-mon[32991]: pgmap v415: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:56 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:57.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:56 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2551762514' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T21:29:57.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:56 smithi001 ceph-mon[28054]: pgmap v415: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:57.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:57.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:56 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:57.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:56 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2551762514' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T21:29:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:56 smithi038 ceph-mon[32724]: pgmap v415: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:29:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:56 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:57.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:56 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2551762514' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-26T21:29:57.465 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:29:58.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:58.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:58.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:57 smithi186 ceph-mon[32991]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:58.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:58.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:58.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:57 smithi001 ceph-mon[28054]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:58.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:29:58.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:29:58.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:57 smithi038 ceph-mon[32724]: from='mgr.14152 172.21.15.1:0/1133517404' entity='mgr.x' 2024-04-26T21:29:59.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:29:58 smithi186 ceph-mon[32991]: pgmap v416: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:59.301 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:29:58 smithi001 ceph-mon[28054]: pgmap v416: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:59.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:29:58 smithi038 ceph-mon[32724]: pgmap v416: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:29:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.869+0000 7ff346200700 1 -- 172.21.15.1:0/3407772976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff340101d80 msgr2=0x7ff3401021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.869+0000 7ff346200700 1 --2- 172.21.15.1:0/3407772976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff340101d80 0x7ff3401021a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff330009990 tx=0x7ff330009ca0 comp rx=0 tx=0).stop 2024-04-26T21:29:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.870+0000 7ff346200700 1 -- 172.21.15.1:0/3407772976 shutdown_connections 2024-04-26T21:29:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.870+0000 7ff346200700 1 --2- 172.21.15.1:0/3407772976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3401030a0 0x7ff340103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.870+0000 7ff346200700 1 --2- 172.21.15.1:0/3407772976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff340101d80 0x7ff3401021a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.870+0000 7ff346200700 1 --2- 172.21.15.1:0/3407772976 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3401026e0 0x7ff340102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.870+0000 7ff346200700 1 -- 172.21.15.1:0/3407772976 >> 172.21.15.1:0/3407772976 conn(0x7ff3400fd920 msgr2=0x7ff3400ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:29:59.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.870+0000 7ff346200700 1 -- 172.21.15.1:0/3407772976 shutdown_connections 2024-04-26T21:29:59.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.870+0000 7ff346200700 1 -- 172.21.15.1:0/3407772976 wait complete. 2024-04-26T21:29:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.871+0000 7ff346200700 1 Processor -- start 2024-04-26T21:29:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.871+0000 7ff346200700 1 -- start start 2024-04-26T21:29:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.871+0000 7ff346200700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff340101d80 0x7ff34019b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.871+0000 7ff346200700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3401026e0 0x7ff34019bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.871+0000 7ff346200700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3401030a0 0x7ff3401a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff346200700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff340111110 con 0x7ff3401030a0 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff346200700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff340110f90 con 0x7ff3401026e0 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff346200700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff340111290 con 0x7ff340101d80 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33ea00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3401026e0 0x7ff34019bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33f400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff340101d80 0x7ff34019b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33ea00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3401026e0 0x7ff34019bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37898/0 (socket says 172.21.15.1:37898) 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33ea00700 1 -- 172.21.15.1:0/400180772 learned_addr learned my addr 172.21.15.1:0/400180772 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33fe00700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3401030a0 0x7ff3401a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33f400700 1 -- 172.21.15.1:0/400180772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3401026e0 msgr2=0x7ff34019bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33f400700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3401026e0 0x7ff34019bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33f400700 1 -- 172.21.15.1:0/400180772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3401030a0 msgr2=0x7ff3401a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33f400700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3401030a0 0x7ff3401a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33f400700 1 -- 172.21.15.1:0/400180772 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3401a3420 con 0x7ff340101d80 2024-04-26T21:29:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.872+0000 7ff33ea00700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3401026e0 0x7ff34019bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:29:59.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.873+0000 7ff33f400700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff340101d80 0x7ff34019b7c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff3300098e0 tx=0x7ff330009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:59.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.873+0000 7ff337e00700 1 -- 172.21.15.1:0/400180772 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330026070 con 0x7ff340101d80 2024-04-26T21:29:59.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.873+0000 7ff337e00700 1 -- 172.21.15.1:0/400180772 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff330021410 con 0x7ff340101d80 2024-04-26T21:29:59.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.873+0000 7ff337e00700 1 -- 172.21.15.1:0/400180772 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330017650 con 0x7ff340101d80 2024-04-26T21:29:59.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.873+0000 7ff346200700 1 -- 172.21.15.1:0/400180772 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3401a36b0 con 0x7ff340101d80 2024-04-26T21:29:59.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.873+0000 7ff346200700 1 -- 172.21.15.1:0/400180772 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3401a3ab0 con 0x7ff340101d80 2024-04-26T21:29:59.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.874+0000 7ff346200700 1 -- 172.21.15.1:0/400180772 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff34006a110 con 0x7ff340101d80 2024-04-26T21:29:59.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.877+0000 7ff337e00700 1 -- 172.21.15.1:0/400180772 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7ff330029070 con 0x7ff340101d80 2024-04-26T21:29:59.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.878+0000 7ff337e00700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3240a3cd0 0x7ff3240a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:29:59.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.878+0000 7ff337e00700 1 -- 172.21.15.1:0/400180772 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7ff3300d1c00 con 0x7ff340101d80 2024-04-26T21:29:59.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.878+0000 7ff33ea00700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3240a3cd0 0x7ff3240a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:29:59.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.879+0000 7ff33ea00700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3240a3cd0 0x7ff3240a6190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff32c011550 tx=0x7ff32c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:29:59.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:29:59.880+0000 7ff337e00700 1 -- 172.21.15.1:0/400180772 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7ff330030070 con 0x7ff340101d80 2024-04-26T21:30:00.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.109+0000 7ff346200700 1 -- 172.21.15.1:0/400180772 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7ff34002e6a0 con 0x7ff340101d80 2024-04-26T21:30:00.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.110+0000 7ff337e00700 1 -- 172.21.15.1:0/400180772 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v95) ==== 56+0+5677 (secure 0 0 0) 0x7ff34002e6a0 con 0x7ff340101d80 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 95 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-04-26T21:17:46.700596+0000 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:modified 2024-04-26T21:28:57.077487+0000 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:crush_version 27 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:full_ratio 0.95 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:backfillfull_ratio 0.9 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:nearfull_ratio 0.85 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:require_min_compat_client luminous 2024-04-26T21:30:00.112 INFO:teuthology.orchestra.run.smithi001.stdout:min_compat_client luminous 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:require_osd_release squid 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:stretch_mode_enabled false 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 91 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:max_osd 12 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:osd.0 up in weight 1 up_from 8 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6802/3716675555,v1:172.21.15.1:6803/3716675555] [v2:172.21.15.1:6804/3716675555,v1:172.21.15.1:6805/3716675555] exists,up 17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6800/2314678612,v1:172.21.15.38:6801/2314678612] [v2:172.21.15.38:6802/2314678612,v1:172.21.15.38:6803/2314678612] exists,up bf91a985-daf1-49bb-9ad9-d5eaabbe6896 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:osd.2 up in weight 1 up_from 18 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6800/2555417177,v1:172.21.15.186:6801/2555417177] [v2:172.21.15.186:6802/2555417177,v1:172.21.15.186:6803/2555417177] exists,up 02453d9d-0297-4356-a632-9b786e930f63 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:osd.3 up in weight 1 up_from 26 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6810/851547730,v1:172.21.15.1:6811/851547730] [v2:172.21.15.1:6812/851547730,v1:172.21.15.1:6813/851547730] exists,up 418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:osd.4 up in weight 1 up_from 34 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6808/2096224,v1:172.21.15.38:6809/2096224] [v2:172.21.15.38:6810/2096224,v1:172.21.15.38:6811/2096224] exists,up e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:osd.5 up in weight 1 up_from 39 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6808/577720361,v1:172.21.15.186:6809/577720361] [v2:172.21.15.186:6810/577720361,v1:172.21.15.186:6811/577720361] exists,up fb2ab549-8bab-42e1-8a6a-3a41b28301c8 2024-04-26T21:30:00.113 INFO:teuthology.orchestra.run.smithi001.stdout:osd.6 up in weight 1 up_from 47 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6818/1579861250,v1:172.21.15.1:6819/1579861250] [v2:172.21.15.1:6820/1579861250,v1:172.21.15.1:6821/1579861250] exists,up 55da783d-0f5f-479b-b13f-80ba6bd8eb72 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:osd.7 up in weight 1 up_from 53 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6816/826797783,v1:172.21.15.38:6817/826797783] [v2:172.21.15.38:6818/826797783,v1:172.21.15.38:6819/826797783] exists,up dfdf3d21-894f-4b0d-9055-152108098b66 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:osd.8 up in weight 1 up_from 61 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6816/758273463,v1:172.21.15.186:6817/758273463] [v2:172.21.15.186:6818/758273463,v1:172.21.15.186:6819/758273463] exists,up 74279a09-821c-4a4f-94b9-1ac7a56d6cf7 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:osd.9 up in weight 1 up_from 69 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.1:6826/4290308342,v1:172.21.15.1:6827/4290308342] [v2:172.21.15.1:6828/4290308342,v1:172.21.15.1:6829/4290308342] exists,up f7bd72f3-4a1d-4aca-b57f-9939e36d9786 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:osd.10 up in weight 1 up_from 74 up_thru 94 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6824/4252488524,v1:172.21.15.38:6825/4252488524] [v2:172.21.15.38:6826/4252488524,v1:172.21.15.38:6827/4252488524] exists,up 03733579-57ef-431d-a55f-d01a168480a2 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:osd.11 up in weight 1 up_from 82 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6824/2966559392,v1:172.21.15.186:6825/2966559392] [v2:172.21.15.186:6826/2966559392,v1:172.21.15.186:6827/2966559392] exists,up bb81b189-ee25-44fd-bd06-ba73245f065c 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 2.0 [7,11] 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 2.5 [8,11] 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 2.16 [2,4] 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 2.1a [8,1] 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 2.1d [2,4] 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.0 [4,8,5,9] 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.6 [3,7,4,6] 2024-04-26T21:30:00.114 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.7 [3,9] 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.13 [4,9] 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.21 [4,8] 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 3.37 [5,6] 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 4.6 [9,11] 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 4.9 [5,10] 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 4.f [9,8] 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:pg_upmap_items 4.10 [9,8] 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/3822624839 expires 2024-04-27T21:18:55.151699+0000 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:6801/3159681858 expires 2024-04-27T21:18:55.151699+0000 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:6800/3159681858 expires 2024-04-27T21:18:55.151699+0000 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:6800/304032214 expires 2024-04-27T21:18:20.022040+0000 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/678258621 expires 2024-04-27T21:18:20.022040+0000 2024-04-26T21:30:00.115 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/2647613577 expires 2024-04-27T21:18:55.151699+0000 2024-04-26T21:30:00.116 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/278183393 expires 2024-04-27T21:18:20.022040+0000 2024-04-26T21:30:00.116 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:6801/304032214 expires 2024-04-27T21:18:20.022040+0000 2024-04-26T21:30:00.116 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/1051804891 expires 2024-04-27T21:18:20.022040+0000 2024-04-26T21:30:00.116 INFO:teuthology.orchestra.run.smithi001.stdout:blocklist 172.21.15.1:0/3004159799 expires 2024-04-27T21:18:55.151699+0000 2024-04-26T21:30:00.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 -- 172.21.15.1:0/400180772 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3240a3cd0 msgr2=0x7ff3240a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:00.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3240a3cd0 0x7ff3240a6190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff32c011550 tx=0x7ff32c009260 comp rx=0 tx=0).stop 2024-04-26T21:30:00.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 -- 172.21.15.1:0/400180772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff340101d80 msgr2=0x7ff34019b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:00.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff340101d80 0x7ff34019b7c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff3300098e0 tx=0x7ff330009580 comp rx=0 tx=0).stop 2024-04-26T21:30:00.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 -- 172.21.15.1:0/400180772 shutdown_connections 2024-04-26T21:30:00.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff3240a3cd0 0x7ff3240a6190 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:00.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff340101d80 0x7ff34019b7c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:00.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3401030a0 0x7ff3401a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:00.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 --2- 172.21.15.1:0/400180772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3401026e0 0x7ff34019bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:00.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 -- 172.21.15.1:0/400180772 >> 172.21.15.1:0/400180772 conn(0x7ff3400fd920 msgr2=0x7ff3400ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:00.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 -- 172.21.15.1:0/400180772 shutdown_connections 2024-04-26T21:30:00.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:00.113+0000 7ff346200700 1 -- 172.21.15.1:0/400180772 wait complete. 2024-04-26T21:30:01.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:00 smithi001 ceph-mon[28054]: pgmap v417: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:30:01.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:00 smithi001 ceph-mon[28054]: overall HEALTH_OK 2024-04-26T21:30:01.004 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:00 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/400180772' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T21:30:01.055 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-26T21:30:01.067 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:30:01.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:00 smithi186 ceph-mon[32991]: pgmap v417: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:30:01.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:00 smithi186 ceph-mon[32991]: overall HEALTH_OK 2024-04-26T21:30:01.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:00 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/400180772' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T21:30:01.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.293+0000 7f98c3e00640 1 -- 172.21.15.1:0/2221218261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c4106550 msgr2=0x7f98c410cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:01.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.293+0000 7f98c3e00640 1 --2- 172.21.15.1:0/2221218261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c4106550 0x7f98c410cde0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009ca0 tx=0x7f98bc02a8b0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.293+0000 7f98c3e00640 1 -- 172.21.15.1:0/2221218261 shutdown_connections 2024-04-26T21:30:01.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.293+0000 7f98c3e00640 1 --2- 172.21.15.1:0/2221218261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c4106550 0x7f98c410cde0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.293+0000 7f98c3e00640 1 --2- 172.21.15.1:0/2221218261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c4105b90 0x7f98c4106010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.293+0000 7f98c3e00640 1 --2- 172.21.15.1:0/2221218261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98c41049a0 0x7f98c4104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.293+0000 7f98c3e00640 1 -- 172.21.15.1:0/2221218261 >> 172.21.15.1:0/2221218261 conn(0x7f98c40fff10 msgr2=0x7f98c4102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:01.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.293+0000 7f98c3e00640 1 -- 172.21.15.1:0/2221218261 shutdown_connections 2024-04-26T21:30:01.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.293+0000 7f98c3e00640 1 -- 172.21.15.1:0/2221218261 wait complete. 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.294+0000 7f98c3e00640 1 Processor -- start 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.294+0000 7f98c3e00640 1 -- start start 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.294+0000 7f98c3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c41049a0 0x7f98c419aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.294+0000 7f98c3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98c4105b90 0x7f98c419af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.294+0000 7f98c3e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c4106550 0x7f98c41a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.294+0000 7f98c3e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4110500 con 0x7f98c4106550 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.294+0000 7f98c3e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4110380 con 0x7f98c41049a0 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.294+0000 7f98c3e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98c4110680 con 0x7f98c4105b90 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c41049a0 0x7f98c419aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c41049a0 0x7f98c419aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37916/0 (socket says 172.21.15.1:37916) 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c2a00640 1 -- 172.21.15.1:0/980837096 learned_addr learned my addr 172.21.15.1:0/980837096 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:01.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c3400640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c4106550 0x7f98c41a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c2000640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98c4105b90 0x7f98c419af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c2a00640 1 -- 172.21.15.1:0/980837096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98c4105b90 msgr2=0x7f98c419af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c2a00640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98c4105b90 0x7f98c419af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c2a00640 1 -- 172.21.15.1:0/980837096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c4106550 msgr2=0x7f98c41a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c2a00640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c4106550 0x7f98c41a1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c2a00640 1 -- 172.21.15.1:0/980837096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c41a2650 con 0x7f98c41049a0 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.295+0000 7f98c2000640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98c4105b90 0x7f98c419af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.296+0000 7f98c2a00640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c41049a0 0x7f98c419aa30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f98b400d770 tx=0x7f98b400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.296+0000 7f98bb400640 1 -- 172.21.15.1:0/980837096 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4017070 con 0x7f98c41049a0 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.296+0000 7f98bb400640 1 -- 172.21.15.1:0/980837096 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98b4009c30 con 0x7f98c41049a0 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.296+0000 7f98bb400640 1 -- 172.21.15.1:0/980837096 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b4003cc0 con 0x7f98c41049a0 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.296+0000 7f98c3e00640 1 -- 172.21.15.1:0/980837096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c41a2940 con 0x7f98c41049a0 2024-04-26T21:30:01.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.296+0000 7f98c3e00640 1 -- 172.21.15.1:0/980837096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c41a2e40 con 0x7f98c41049a0 2024-04-26T21:30:01.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.297+0000 7f98c3e00640 1 -- 172.21.15.1:0/980837096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c40506c0 con 0x7f98c41049a0 2024-04-26T21:30:01.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.299+0000 7f98bb400640 1 -- 172.21.15.1:0/980837096 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f98b4003e60 con 0x7f98c41049a0 2024-04-26T21:30:01.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.301+0000 7f98bb400640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98a80a3cf0 0x7f98a80a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:01.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.301+0000 7f98bb400640 1 -- 172.21.15.1:0/980837096 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f98b40d02d0 con 0x7f98c41049a0 2024-04-26T21:30:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.301+0000 7f98c2000640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98a80a3cf0 0x7f98a80a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:01.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.302+0000 7f98c2000640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98a80a3cf0 0x7f98a80a61b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f98b00095f0 tx=0x7f98b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:01.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.306+0000 7f98bb400640 1 -- 172.21.15.1:0/980837096 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f98b4009de0 con 0x7f98c41049a0 2024-04-26T21:30:01.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:00 smithi038 ceph-mon[32724]: pgmap v417: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-26T21:30:01.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:00 smithi038 ceph-mon[32724]: overall HEALTH_OK 2024-04-26T21:30:01.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:00 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/400180772' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T21:30:01.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.513+0000 7f98c3e00640 1 -- 172.21.15.1:0/980837096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98c402dba0 con 0x7f98c41049a0 2024-04-26T21:30:01.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.515+0000 7f98bb400640 1 -- 172.21.15.1:0/980837096 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24767 (secure 0 0 0) 0x7f98b40bc070 con 0x7f98c41049a0 2024-04-26T21:30:01.516 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:30:01.517 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":95,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:28:57.077487+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7},{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 -- 172.21.15.1:0/980837096 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98a80a3cf0 msgr2=0x7f98a80a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98a80a3cf0 0x7f98a80a61b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f98b00095f0 tx=0x7f98b0005f90 comp rx=0 tx=0).stop 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 -- 172.21.15.1:0/980837096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c41049a0 msgr2=0x7f98c419aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c41049a0 0x7f98c419aa30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f98b400d770 tx=0x7f98b400dc30 comp rx=0 tx=0).stop 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 -- 172.21.15.1:0/980837096 shutdown_connections 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f98a80a3cf0 0x7f98a80a61b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98c4105b90 0x7f98c419af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98c4106550 0x7f98c41a1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 --2- 172.21.15.1:0/980837096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98c41049a0 0x7f98c419aa30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 -- 172.21.15.1:0/980837096 >> 172.21.15.1:0/980837096 conn(0x7f98c40fff10 msgr2=0x7f98c4108dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 -- 172.21.15.1:0/980837096 shutdown_connections 2024-04-26T21:30:01.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.518+0000 7f98c3e00640 1 -- 172.21.15.1:0/980837096 wait complete. 2024-04-26T21:30:01.538 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:27:00.381106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:27:01.743898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T21:27:05.341216+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T21:30:01.539 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:30:01.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.754+0000 7fb2f3e00640 1 -- 172.21.15.1:0/1746595080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2f4103230 msgr2=0x7fb2f4105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.754+0000 7fb2f3e00640 1 --2- 172.21.15.1:0/1746595080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2f4103230 0x7fb2f4105620 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009900 tx=0x7fb2e402a3f0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.755+0000 7fb2f3e00640 1 -- 172.21.15.1:0/1746595080 shutdown_connections 2024-04-26T21:30:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.755+0000 7fb2f3e00640 1 --2- 172.21.15.1:0/1746595080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4105b60 0x7fb2f4107f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.755+0000 7fb2f3e00640 1 --2- 172.21.15.1:0/1746595080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f4108490 0x7fb2f410a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.755+0000 7fb2f3e00640 1 --2- 172.21.15.1:0/1746595080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2f4103230 0x7fb2f4105620 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.755+0000 7fb2f3e00640 1 -- 172.21.15.1:0/1746595080 >> 172.21.15.1:0/1746595080 conn(0x7fb2f40fcc20 msgr2=0x7fb2f40ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.755+0000 7fb2f3e00640 1 -- 172.21.15.1:0/1746595080 shutdown_connections 2024-04-26T21:30:01.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.755+0000 7fb2f3e00640 1 -- 172.21.15.1:0/1746595080 wait complete. 2024-04-26T21:30:01.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.756+0000 7fb2f3e00640 1 Processor -- start 2024-04-26T21:30:01.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.756+0000 7fb2f3e00640 1 -- start start 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.756+0000 7fb2f3e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f4103230 0x7fb2f4198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.756+0000 7fb2f3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4105b60 0x7fb2f4198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.756+0000 7fb2f3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2f4108490 0x7fb2f419fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.756+0000 7fb2f3e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f410e330 con 0x7fb2f4103230 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.756+0000 7fb2f3e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f410e1b0 con 0x7fb2f4108490 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.756+0000 7fb2f3e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f410e4b0 con 0x7fb2f4105b60 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.757+0000 7fb2f2a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f4103230 0x7fb2f4198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.757+0000 7fb2f3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2f4108490 0x7fb2f419fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.757+0000 7fb2f2000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4105b60 0x7fb2f4198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.757+0000 7fb2f3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2f4108490 0x7fb2f419fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37944/0 (socket says 172.21.15.1:37944) 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.757+0000 7fb2f3400640 1 -- 172.21.15.1:0/740432286 learned_addr learned my addr 172.21.15.1:0/740432286 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:01.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.757+0000 7fb2f3400640 1 -- 172.21.15.1:0/740432286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4105b60 msgr2=0x7fb2f4198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.757+0000 7fb2f3400640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4105b60 0x7fb2f4198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.757+0000 7fb2f3400640 1 -- 172.21.15.1:0/740432286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f4103230 msgr2=0x7fb2f4198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.757+0000 7fb2f3400640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f4103230 0x7fb2f4198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.757+0000 7fb2f3400640 1 -- 172.21.15.1:0/740432286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f41a0660 con 0x7fb2f4108490 2024-04-26T21:30:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.758+0000 7fb2f3400640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2f4108490 0x7fb2f419fff0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00eaa0 tx=0x7fb2ec00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.758+0000 7fb2f8a00640 1 -- 172.21.15.1:0/740432286 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec00cdb0 con 0x7fb2f4108490 2024-04-26T21:30:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.758+0000 7fb2f8a00640 1 -- 172.21.15.1:0/740432286 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2ec005020 con 0x7fb2f4108490 2024-04-26T21:30:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.758+0000 7fb2f8a00640 1 -- 172.21.15.1:0/740432286 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec0116d0 con 0x7fb2f4108490 2024-04-26T21:30:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.758+0000 7fb2f3e00640 1 -- 172.21.15.1:0/740432286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f41a0950 con 0x7fb2f4108490 2024-04-26T21:30:01.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.758+0000 7fb2f3e00640 1 -- 172.21.15.1:0/740432286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f41a0e50 con 0x7fb2f4108490 2024-04-26T21:30:01.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.759+0000 7fb2f3e00640 1 -- 172.21.15.1:0/740432286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f40fe710 con 0x7fb2f4108490 2024-04-26T21:30:01.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.761+0000 7fb2f8a00640 1 -- 172.21.15.1:0/740432286 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fb2ec0051d0 con 0x7fb2f4108490 2024-04-26T21:30:01.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.763+0000 7fb2f8a00640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb2d80a3cc0 0x7fb2d80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:01.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.763+0000 7fb2f8a00640 1 -- 172.21.15.1:0/740432286 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fb2ec0d0ad0 con 0x7fb2f4108490 2024-04-26T21:30:01.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.763+0000 7fb2f2a00640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb2d80a3cc0 0x7fb2d80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:01.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.764+0000 7fb2f2a00640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb2d80a3cc0 0x7fb2d80a6180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4031500 tx=0x7fb2e4005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:01.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.767+0000 7fb2f8a00640 1 -- 172.21.15.1:0/740432286 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb2ec016030 con 0x7fb2f4108490 2024-04-26T21:30:01.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.981+0000 7fb2f3e00640 1 -- 172.21.15.1:0/740432286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb2f402dd20 con 0x7fb2f4108490 2024-04-26T21:30:01.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.982+0000 7fb2f8a00640 1 -- 172.21.15.1:0/740432286 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7fb2ec09c8a0 con 0x7fb2f4108490 2024-04-26T21:30:01.982 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:30:01.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.984+0000 7fb2f3e00640 1 -- 172.21.15.1:0/740432286 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb2d80a3cc0 msgr2=0x7fb2d80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:01.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.984+0000 7fb2f3e00640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb2d80a3cc0 0x7fb2d80a6180 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4031500 tx=0x7fb2e4005a20 comp rx=0 tx=0).stop 2024-04-26T21:30:01.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.984+0000 7fb2f3e00640 1 -- 172.21.15.1:0/740432286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2f4108490 msgr2=0x7fb2f419fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:01.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.984+0000 7fb2f3e00640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2f4108490 0x7fb2f419fff0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00eaa0 tx=0x7fb2ec00ef60 comp rx=0 tx=0).stop 2024-04-26T21:30:01.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.984+0000 7fb2f3e00640 1 -- 172.21.15.1:0/740432286 shutdown_connections 2024-04-26T21:30:01.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.984+0000 7fb2f3e00640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fb2d80a3cc0 0x7fb2d80a6180 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.984+0000 7fb2f3e00640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4105b60 0x7fb2f4198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.984+0000 7fb2f3e00640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb2f4108490 0x7fb2f419fff0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.984+0000 7fb2f3e00640 1 --2- 172.21.15.1:0/740432286 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2f4103230 0x7fb2f4198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.984+0000 7fb2f3e00640 1 -- 172.21.15.1:0/740432286 >> 172.21.15.1:0/740432286 conn(0x7fb2f40fcc20 msgr2=0x7fb2f4103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.985+0000 7fb2f3e00640 1 -- 172.21.15.1:0/740432286 shutdown_connections 2024-04-26T21:30:01.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:01.985+0000 7fb2f3e00640 1 -- 172.21.15.1:0/740432286 wait complete. 2024-04-26T21:30:02.005 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:30:02.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:01 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/980837096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:30:02.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.227+0000 7f3875e00640 1 -- 172.21.15.1:0/3110362743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387010a630 msgr2=0x7f387010cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.227+0000 7f3875e00640 1 --2- 172.21.15.1:0/3110362743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387010a630 0x7f387010cad0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3868009ca0 tx=0x7f386802a950 comp rx=0 tx=0).stop 2024-04-26T21:30:02.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.227+0000 7f3875e00640 1 -- 172.21.15.1:0/3110362743 shutdown_connections 2024-04-26T21:30:02.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.227+0000 7f3875e00640 1 --2- 172.21.15.1:0/3110362743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387010a630 0x7f387010cad0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.227+0000 7f3875e00640 1 --2- 172.21.15.1:0/3110362743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3870075330 0x7f3870075790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.227+0000 7f3875e00640 1 --2- 172.21.15.1:0/3110362743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3870076910 0x7f3870074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.227+0000 7f3875e00640 1 -- 172.21.15.1:0/3110362743 >> 172.21.15.1:0/3110362743 conn(0x7f38700fdd10 msgr2=0x7f3870100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:02.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.227+0000 7f3875e00640 1 -- 172.21.15.1:0/3110362743 shutdown_connections 2024-04-26T21:30:02.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.227+0000 7f3875e00640 1 -- 172.21.15.1:0/3110362743 wait complete. 2024-04-26T21:30:02.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.228+0000 7f3875e00640 1 Processor -- start 2024-04-26T21:30:02.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.228+0000 7f3875e00640 1 -- start start 2024-04-26T21:30:02.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f3875e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3870075330 0x7f387019abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:02.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f3875e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3870076910 0x7f387019b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:02.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f3875e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387010a630 0x7f38701a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:02.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f3875e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f38701104b0 con 0x7f387010a630 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f3875e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3870110330 con 0x7f3870076910 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f3875e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3870110630 con 0x7f3870075330 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f386fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3870076910 0x7f387019b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f3874a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3870075330 0x7f387019abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f386fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3870076910 0x7f387019b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37964/0 (socket says 172.21.15.1:37964) 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f386fe00640 1 -- 172.21.15.1:0/2649699540 learned_addr learned my addr 172.21.15.1:0/2649699540 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.229+0000 7f3875400640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387010a630 0x7f38701a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f3874a00640 1 -- 172.21.15.1:0/2649699540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3870076910 msgr2=0x7f387019b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f3874a00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3870076910 0x7f387019b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f3874a00640 1 -- 172.21.15.1:0/2649699540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387010a630 msgr2=0x7f38701a2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f3874a00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387010a630 0x7f38701a2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f3874a00640 1 -- 172.21.15.1:0/2649699540 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38701a27d0 con 0x7f3870075330 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f386fe00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3870076910 0x7f387019b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:02.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f3874a00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3870075330 0x7f387019abb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f386400e9f0 tx=0x7f386400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:02.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f386d600640 1 -- 172.21.15.1:0/2649699540 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386400ccc0 con 0x7f3870075330 2024-04-26T21:30:02.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f3875e00640 1 -- 172.21.15.1:0/2649699540 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38701a2ac0 con 0x7f3870075330 2024-04-26T21:30:02.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f386d600640 1 -- 172.21.15.1:0/2649699540 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38640108e0 con 0x7f3870075330 2024-04-26T21:30:02.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.230+0000 7f386d600640 1 -- 172.21.15.1:0/2649699540 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3864018880 con 0x7f3870075330 2024-04-26T21:30:02.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.231+0000 7f3875e00640 1 -- 172.21.15.1:0/2649699540 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38701a3060 con 0x7f3870075330 2024-04-26T21:30:02.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.232+0000 7f3875e00640 1 -- 172.21.15.1:0/2649699540 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38701093f0 con 0x7f3870075330 2024-04-26T21:30:02.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.234+0000 7f386d600640 1 -- 172.21.15.1:0/2649699540 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f3864010430 con 0x7f3870075330 2024-04-26T21:30:02.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.236+0000 7f386d600640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38580a3cc0 0x7f38580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:02.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.236+0000 7f386d600640 1 -- 172.21.15.1:0/2649699540 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f38640d0c60 con 0x7f3870075330 2024-04-26T21:30:02.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.236+0000 7f386fe00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38580a3cc0 0x7f38580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:02.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.237+0000 7f386fe00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38580a3cc0 0x7f38580a6180 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f387019c0c0 tx=0x7f385c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:02.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.240+0000 7f386d600640 1 -- 172.21.15.1:0/2649699540 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f386409ca30 con 0x7f3870075330 2024-04-26T21:30:02.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:01 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/980837096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:30:02.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:01 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/980837096' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:30:02.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.448+0000 7f3875e00640 1 -- 172.21.15.1:0/2649699540 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f387002dad0 con 0x7f3870075330 2024-04-26T21:30:02.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.449+0000 7f386d600640 1 -- 172.21.15.1:0/2649699540 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f387002dad0 con 0x7f3870075330 2024-04-26T21:30:02.450 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:30:02.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.451+0000 7f3875e00640 1 -- 172.21.15.1:0/2649699540 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38580a3cc0 msgr2=0x7f38580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.451+0000 7f3875e00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38580a3cc0 0x7f38580a6180 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f387019c0c0 tx=0x7f385c005f90 comp rx=0 tx=0).stop 2024-04-26T21:30:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.451+0000 7f3875e00640 1 -- 172.21.15.1:0/2649699540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3870075330 msgr2=0x7f387019abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.451+0000 7f3875e00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3870075330 0x7f387019abb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f386400e9f0 tx=0x7f386400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.452+0000 7f3875e00640 1 -- 172.21.15.1:0/2649699540 shutdown_connections 2024-04-26T21:30:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.452+0000 7f3875e00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38580a3cc0 0x7f38580a6180 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.452+0000 7f3875e00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f387010a630 0x7f38701a2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.452+0000 7f3875e00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3870076910 0x7f387019b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.452+0000 7f3875e00640 1 --2- 172.21.15.1:0/2649699540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3870075330 0x7f387019abb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.452+0000 7f3875e00640 1 -- 172.21.15.1:0/2649699540 >> 172.21.15.1:0/2649699540 conn(0x7f38700fdd10 msgr2=0x7f38701065c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.452+0000 7f3875e00640 1 -- 172.21.15.1:0/2649699540 shutdown_connections 2024-04-26T21:30:02.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.452+0000 7f3875e00640 1 -- 172.21.15.1:0/2649699540 wait complete. 2024-04-26T21:30:02.474 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:30:02.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.687+0000 7fce28a00640 1 -- 172.21.15.1:0/1269305131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce24103970 msgr2=0x7fce24103df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.687+0000 7fce28a00640 1 --2- 172.21.15.1:0/1269305131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce24103970 0x7fce24103df0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009860 tx=0x7fce0c02a340 comp rx=0 tx=0).stop 2024-04-26T21:30:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.688+0000 7fce28a00640 1 -- 172.21.15.1:0/1269305131 shutdown_connections 2024-04-26T21:30:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.688+0000 7fce28a00640 1 --2- 172.21.15.1:0/1269305131 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce24104330 0x7fce2410abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.688+0000 7fce28a00640 1 --2- 172.21.15.1:0/1269305131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce24103970 0x7fce24103df0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.688+0000 7fce28a00640 1 --2- 172.21.15.1:0/1269305131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce24102780 0x7fce24102ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.688+0000 7fce28a00640 1 -- 172.21.15.1:0/1269305131 >> 172.21.15.1:0/1269305131 conn(0x7fce240fdcf0 msgr2=0x7fce24100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.688+0000 7fce28a00640 1 -- 172.21.15.1:0/1269305131 shutdown_connections 2024-04-26T21:30:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.688+0000 7fce28a00640 1 -- 172.21.15.1:0/1269305131 wait complete. 2024-04-26T21:30:02.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.689+0000 7fce28a00640 1 Processor -- start 2024-04-26T21:30:02.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.689+0000 7fce28a00640 1 -- start start 2024-04-26T21:30:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce28a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce24102780 0x7fce240766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce28a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce24103970 0x7fce24074d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce28a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce24104330 0x7fce24075260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce28a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fce2410e290 con 0x7fce24102780 2024-04-26T21:30:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce28a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fce2410e110 con 0x7fce24104330 2024-04-26T21:30:02.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce28a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce2410e410 con 0x7fce24103970 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce23400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce24102780 0x7fce240766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce23400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce24102780 0x7fce240766e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53190/0 (socket says 172.21.15.1:53190) 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce23400640 1 -- 172.21.15.1:0/4164898358 learned_addr learned my addr 172.21.15.1:0/4164898358 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce22a00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce24103970 0x7fce24074d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.690+0000 7fce23e00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce24104330 0x7fce24075260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.691+0000 7fce22a00640 1 -- 172.21.15.1:0/4164898358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce24104330 msgr2=0x7fce24075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.691+0000 7fce22a00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce24104330 0x7fce24075260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.691+0000 7fce22a00640 1 -- 172.21.15.1:0/4164898358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce24102780 msgr2=0x7fce240766e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.691+0000 7fce22a00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce24102780 0x7fce240766e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.691+0000 7fce22a00640 1 -- 172.21.15.1:0/4164898358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce24075af0 con 0x7fce24103970 2024-04-26T21:30:02.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.691+0000 7fce23400640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce24102780 0x7fce240766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.691+0000 7fce22a00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce24103970 0x7fce24074d20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fce0c02a860 tx=0x7fce0c032e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.691+0000 7fce1be00640 1 -- 172.21.15.1:0/4164898358 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c037040 con 0x7fce24103970 2024-04-26T21:30:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.692+0000 7fce28a00640 1 -- 172.21.15.1:0/4164898358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce241a2a30 con 0x7fce24103970 2024-04-26T21:30:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.692+0000 7fce1be00640 1 -- 172.21.15.1:0/4164898358 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce0c030c50 con 0x7fce24103970 2024-04-26T21:30:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.692+0000 7fce1be00640 1 -- 172.21.15.1:0/4164898358 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c003850 con 0x7fce24103970 2024-04-26T21:30:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.692+0000 7fce28a00640 1 -- 172.21.15.1:0/4164898358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce241a2f70 con 0x7fce24103970 2024-04-26T21:30:02.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.693+0000 7fce28a00640 1 -- 172.21.15.1:0/4164898358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce24109ac0 con 0x7fce24103970 2024-04-26T21:30:02.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.695+0000 7fce1be00640 1 -- 172.21.15.1:0/4164898358 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fce0c043020 con 0x7fce24103970 2024-04-26T21:30:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.696+0000 7fce1be00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fce080a3c30 0x7fce080a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.697+0000 7fce1be00640 1 -- 172.21.15.1:0/4164898358 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fce0c0efad0 con 0x7fce24103970 2024-04-26T21:30:02.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.697+0000 7fce23400640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fce080a3c30 0x7fce080a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:02.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.698+0000 7fce23400640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fce080a3c30 0x7fce080a60f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fce14011750 tx=0x7fce140096d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:02.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.699+0000 7fce1be00640 1 -- 172.21.15.1:0/4164898358 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fce0c030e00 con 0x7fce24103970 2024-04-26T21:30:02.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.905+0000 7fce28a00640 1 -- 172.21.15.1:0/4164898358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fce24063530 con 0x7fce24103970 2024-04-26T21:30:02.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.906+0000 7fce1be00640 1 -- 172.21.15.1:0/4164898358 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7fce0c0bc080 con 0x7fce24103970 2024-04-26T21:30:02.907 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T21:30:02.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.909+0000 7fce28a00640 1 -- 172.21.15.1:0/4164898358 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fce080a3c30 msgr2=0x7fce080a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.909+0000 7fce28a00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fce080a3c30 0x7fce080a60f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fce14011750 tx=0x7fce140096d0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.909+0000 7fce28a00640 1 -- 172.21.15.1:0/4164898358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce24103970 msgr2=0x7fce24074d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:02.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.909+0000 7fce28a00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce24103970 0x7fce24074d20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fce0c02a860 tx=0x7fce0c032e90 comp rx=0 tx=0).stop 2024-04-26T21:30:02.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.910+0000 7fce28a00640 1 -- 172.21.15.1:0/4164898358 shutdown_connections 2024-04-26T21:30:02.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.910+0000 7fce28a00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fce080a3c30 0x7fce080a60f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.910+0000 7fce28a00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce24103970 0x7fce24074d20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.910+0000 7fce28a00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce24104330 0x7fce24075260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.910+0000 7fce28a00640 1 --2- 172.21.15.1:0/4164898358 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce24102780 0x7fce240766e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:02.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.910+0000 7fce28a00640 1 -- 172.21.15.1:0/4164898358 >> 172.21.15.1:0/4164898358 conn(0x7fce240fdcf0 msgr2=0x7fce24106c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:02.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.910+0000 7fce28a00640 1 -- 172.21.15.1:0/4164898358 shutdown_connections 2024-04-26T21:30:02.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:02.910+0000 7fce28a00640 1 -- 172.21.15.1:0/4164898358 wait complete. 2024-04-26T21:30:02.930 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T21:30:03.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.137+0000 7f0fb7e00640 1 -- 172.21.15.1:0/1527212861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb8075330 msgr2=0x7f0fb8075790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:03.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.137+0000 7f0fb7e00640 1 --2- 172.21.15.1:0/1527212861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb8075330 0x7f0fb8075790 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0009900 tx=0x7f0fa002a480 comp rx=0 tx=0).stop 2024-04-26T21:30:03.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.137+0000 7f0fb7e00640 1 -- 172.21.15.1:0/1527212861 shutdown_connections 2024-04-26T21:30:03.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.137+0000 7f0fb7e00640 1 --2- 172.21.15.1:0/1527212861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb810a650 0x7f0fb810caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.137+0000 7f0fb7e00640 1 --2- 172.21.15.1:0/1527212861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb8075330 0x7f0fb8075790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.137+0000 7f0fb7e00640 1 --2- 172.21.15.1:0/1527212861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fb8076910 0x7f0fb8074d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.137+0000 7f0fb7e00640 1 -- 172.21.15.1:0/1527212861 >> 172.21.15.1:0/1527212861 conn(0x7f0fb80fdd30 msgr2=0x7f0fb8100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:03.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.137+0000 7f0fb7e00640 1 -- 172.21.15.1:0/1527212861 shutdown_connections 2024-04-26T21:30:03.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.138+0000 7f0fb7e00640 1 -- 172.21.15.1:0/1527212861 wait complete. 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.138+0000 7f0fb7e00640 1 Processor -- start 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fb7e00640 1 -- start start 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fb7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fb8075330 0x7f0fb819abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fb7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb8076910 0x7f0fb819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fb7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb810a650 0x7f0fb81a2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fb7e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb81104f0 con 0x7f0fb8076910 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fb7e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb8110370 con 0x7f0fb8075330 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fb7e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb8110670 con 0x7f0fb810a650 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fae000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb8076910 0x7f0fb819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fae000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb8076910 0x7f0fb819b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53214/0 (socket says 172.21.15.1:53214) 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fae000640 1 -- 172.21.15.1:0/778831261 learned_addr learned my addr 172.21.15.1:0/778831261 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:03.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fb7400640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb810a650 0x7f0fb81a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.139+0000 7f0fb6a00640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fb8075330 0x7f0fb819abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.140+0000 7f0fae000640 1 -- 172.21.15.1:0/778831261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb810a650 msgr2=0x7f0fb81a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.140+0000 7f0fae000640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb810a650 0x7f0fb81a2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.140+0000 7f0fae000640 1 -- 172.21.15.1:0/778831261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fb8075330 msgr2=0x7f0fb819abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.140+0000 7f0fae000640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fb8075330 0x7f0fb819abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.140+0000 7f0fae000640 1 -- 172.21.15.1:0/778831261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb81a27e0 con 0x7f0fb8076910 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.140+0000 7f0fb7400640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb810a650 0x7f0fb81a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.140+0000 7f0fae000640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb8076910 0x7f0fb819b100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0005940 tx=0x7f0fa00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.140+0000 7f0fafe00640 1 -- 172.21.15.1:0/778831261 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa003f030 con 0x7f0fb8076910 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.141+0000 7f0fafe00640 1 -- 172.21.15.1:0/778831261 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa0003cb0 con 0x7f0fb8076910 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.141+0000 7f0fb7e00640 1 -- 172.21.15.1:0/778831261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb81a2a70 con 0x7f0fb8076910 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.141+0000 7f0fafe00640 1 -- 172.21.15.1:0/778831261 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa0035070 con 0x7f0fb8076910 2024-04-26T21:30:03.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.141+0000 7f0fb7e00640 1 -- 172.21.15.1:0/778831261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb81a2fb0 con 0x7f0fb8076910 2024-04-26T21:30:03.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.142+0000 7f0fb7e00640 1 -- 172.21.15.1:0/778831261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb8109410 con 0x7f0fb8076910 2024-04-26T21:30:03.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.144+0000 7f0fafe00640 1 -- 172.21.15.1:0/778831261 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f0fa0042030 con 0x7f0fb8076910 2024-04-26T21:30:03.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.146+0000 7f0fafe00640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0f9c0a3cc0 0x7f0f9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:03.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.146+0000 7f0fafe00640 1 -- 172.21.15.1:0/778831261 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f0fa00f0d20 con 0x7f0fb8076910 2024-04-26T21:30:03.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.146+0000 7f0fb6a00640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0f9c0a3cc0 0x7f0f9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:03.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.147+0000 7f0fb6a00640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0f9c0a3cc0 0x7f0f9c0a6180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80095f0 tx=0x7f0fa8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:03.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.150+0000 7f0fafe00640 1 -- 172.21.15.1:0/778831261 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f0fa0003e60 con 0x7f0fb8076910 2024-04-26T21:30:03.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:02 smithi186 ceph-mon[32991]: pgmap v418: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:03.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:02 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/740432286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:30:03.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:02 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2649699540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:30:03.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:02 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4164898358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:30:03.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:02 smithi001 ceph-mon[28054]: pgmap v418: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:03.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:02 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/740432286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:30:03.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:02 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2649699540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:30:03.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:02 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4164898358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:30:03.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:02 smithi038 ceph-mon[32724]: pgmap v418: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:30:03.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:02 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/740432286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:30:03.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:02 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2649699540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:30:03.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:02 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4164898358' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:30:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.358+0000 7f0fb7e00640 1 -- 172.21.15.1:0/778831261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0fb802dd10 con 0x7f0fb8076910 2024-04-26T21:30:03.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.359+0000 7f0fafe00640 1 -- 172.21.15.1:0/778831261 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f0fa00bcaf0 con 0x7f0fb8076910 2024-04-26T21:30:03.360 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 -- 172.21.15.1:0/778831261 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0f9c0a3cc0 msgr2=0x7f0f9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0f9c0a3cc0 0x7f0f9c0a6180 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80095f0 tx=0x7f0fa8005f90 comp rx=0 tx=0).stop 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 -- 172.21.15.1:0/778831261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb8076910 msgr2=0x7f0fb819b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb8076910 0x7f0fb819b100 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0fa0005940 tx=0x7f0fa00042f0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 -- 172.21.15.1:0/778831261 shutdown_connections 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f0f9c0a3cc0 0x7f0f9c0a6180 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb810a650 0x7f0fb81a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb8076910 0x7f0fb819b100 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 --2- 172.21.15.1:0/778831261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0fb8075330 0x7f0fb819abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 -- 172.21.15.1:0/778831261 >> 172.21.15.1:0/778831261 conn(0x7f0fb80fdd30 msgr2=0x7f0fb81065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:03.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 -- 172.21.15.1:0/778831261 shutdown_connections 2024-04-26T21:30:03.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.362+0000 7f0fb7e00640 1 -- 172.21.15.1:0/778831261 wait complete. 2024-04-26T21:30:03.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:30:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.593+0000 7f30a9800640 1 -- 172.21.15.1:0/1604078811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a4108650 msgr2=0x7f30a410aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.593+0000 7f30a9800640 1 --2- 172.21.15.1:0/1604078811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a4108650 0x7f30a410aaa0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f309c009900 tx=0x7f309c02a410 comp rx=0 tx=0).stop 2024-04-26T21:30:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.593+0000 7f30a9800640 1 -- 172.21.15.1:0/1604078811 shutdown_connections 2024-04-26T21:30:03.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.593+0000 7f30a9800640 1 --2- 172.21.15.1:0/1604078811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a4108650 0x7f30a410aaa0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.593+0000 7f30a9800640 1 --2- 172.21.15.1:0/1604078811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a4105d20 0x7f30a4108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.593+0000 7f30a9800640 1 --2- 172.21.15.1:0/1604078811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30a41033f0 0x7f30a41057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.593+0000 7f30a9800640 1 -- 172.21.15.1:0/1604078811 >> 172.21.15.1:0/1604078811 conn(0x7f30a40fce00 msgr2=0x7f30a40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.593+0000 7f30a9800640 1 -- 172.21.15.1:0/1604078811 shutdown_connections 2024-04-26T21:30:03.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.594+0000 7f30a9800640 1 -- 172.21.15.1:0/1604078811 wait complete. 2024-04-26T21:30:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.594+0000 7f30a9800640 1 Processor -- start 2024-04-26T21:30:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.595+0000 7f30a9800640 1 -- start start 2024-04-26T21:30:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.595+0000 7f30a9800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30a41033f0 0x7f30a419ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.595+0000 7f30a9800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a4105d20 0x7f30a419b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.595+0000 7f30a9800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a4108650 0x7f30a41a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.595+0000 7f30a9800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f30a410e4a0 con 0x7f30a4105d20 2024-04-26T21:30:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.595+0000 7f30a9800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f30a410e320 con 0x7f30a4108650 2024-04-26T21:30:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.595+0000 7f30a9800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f30a410e620 con 0x7f30a41033f0 2024-04-26T21:30:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.595+0000 7f30a3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a4108650 0x7f30a41a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:03.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a4108650 0x7f30a41a2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:38022/0 (socket says 172.21.15.1:38022) 2024-04-26T21:30:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30a41033f0 0x7f30a419ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a3400640 1 -- 172.21.15.1:0/1581317974 learned_addr learned my addr 172.21.15.1:0/1581317974 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:03.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a2000640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a4105d20 0x7f30a419b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a3400640 1 -- 172.21.15.1:0/1581317974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30a41033f0 msgr2=0x7f30a419ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a3400640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30a41033f0 0x7f30a419ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a3400640 1 -- 172.21.15.1:0/1581317974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a4105d20 msgr2=0x7f30a419b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a3400640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a4105d20 0x7f30a419b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a3400640 1 -- 172.21.15.1:0/1581317974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a41a2920 con 0x7f30a4108650 2024-04-26T21:30:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a2a00640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30a41033f0 0x7f30a419ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.596+0000 7f30a3400640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a4108650 0x7f30a41a2280 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f309c031500 tx=0x7f309c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:03.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.597+0000 7f309b400640 1 -- 172.21.15.1:0/1581317974 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c03f030 con 0x7f30a4108650 2024-04-26T21:30:03.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.597+0000 7f309b400640 1 -- 172.21.15.1:0/1581317974 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f309c003cb0 con 0x7f30a4108650 2024-04-26T21:30:03.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.597+0000 7f309b400640 1 -- 172.21.15.1:0/1581317974 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c035070 con 0x7f30a4108650 2024-04-26T21:30:03.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.597+0000 7f30a9800640 1 -- 172.21.15.1:0/1581317974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a41a2bb0 con 0x7f30a4108650 2024-04-26T21:30:03.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.597+0000 7f30a9800640 1 -- 172.21.15.1:0/1581317974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a41a30f0 con 0x7f30a4108650 2024-04-26T21:30:03.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.598+0000 7f30a9800640 1 -- 172.21.15.1:0/1581317974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30a410a140 con 0x7f30a4108650 2024-04-26T21:30:03.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.600+0000 7f309b400640 1 -- 172.21.15.1:0/1581317974 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f309c042030 con 0x7f30a4108650 2024-04-26T21:30:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.602+0000 7f309b400640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f30880a3cc0 0x7f30880a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.602+0000 7f309b400640 1 -- 172.21.15.1:0/1581317974 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f309c0f0e20 con 0x7f30a4108650 2024-04-26T21:30:03.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.602+0000 7f30a2a00640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f30880a3cc0 0x7f30880a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:03.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.603+0000 7f30a2a00640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f30880a3cc0 0x7f30880a6180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f30a419c1e0 tx=0x7f3094009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:03.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.606+0000 7f309b400640 1 -- 172.21.15.1:0/1581317974 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f309c0bcbf0 con 0x7f30a4108650 2024-04-26T21:30:03.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.853+0000 7f30a9800640 1 -- 172.21.15.1:0/1581317974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30a402dcb0 con 0x7f30a4108650 2024-04-26T21:30:03.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.855+0000 7f309b400640 1 -- 172.21.15.1:0/1581317974 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11857 (secure 0 0 0) 0x7f309c0c0ef0 con 0x7f30a4108650 2024-04-26T21:30:03.856 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:30:03.856 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:27:02.795246+0000","modified":"2024-04-26T21:27:28.957703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24572,"mds_2":24548},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24548":{"gid":24548,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24548,24572,24602]},"id":1}]} 2024-04-26T21:30:03.856 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 23 2024-04-26T21:30:03.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 -- 172.21.15.1:0/1581317974 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f30880a3cc0 msgr2=0x7f30880a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:03.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f30880a3cc0 0x7f30880a6180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f30a419c1e0 tx=0x7f3094009500 comp rx=0 tx=0).stop 2024-04-26T21:30:03.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 -- 172.21.15.1:0/1581317974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a4108650 msgr2=0x7f30a41a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:03.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a4108650 0x7f30a41a2280 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f309c031500 tx=0x7f309c0042f0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 -- 172.21.15.1:0/1581317974 shutdown_connections 2024-04-26T21:30:03.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f30880a3cc0 0x7f30880a6180 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30a4108650 0x7f30a41a2280 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30a4105d20 0x7f30a419b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 --2- 172.21.15.1:0/1581317974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30a41033f0 0x7f30a419ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:03.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 -- 172.21.15.1:0/1581317974 >> 172.21.15.1:0/1581317974 conn(0x7f30a40fce00 msgr2=0x7f30a4102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:03.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 -- 172.21.15.1:0/1581317974 shutdown_connections 2024-04-26T21:30:03.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:03.858+0000 7f30a9800640 1 -- 172.21.15.1:0/1581317974 wait complete. 2024-04-26T21:30:03.881 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:30:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.085+0000 7f0426c00640 1 -- 172.21.15.1:0/107798298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f042010ed40 msgr2=0x7f0420111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.085+0000 7f0426c00640 1 --2- 172.21.15.1:0/107798298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f042010ed40 0x7f0420111130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0418009890 tx=0x7f041802a360 comp rx=0 tx=0).stop 2024-04-26T21:30:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.085+0000 7f0426c00640 1 -- 172.21.15.1:0/107798298 shutdown_connections 2024-04-26T21:30:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.085+0000 7f0426c00640 1 --2- 172.21.15.1:0/107798298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0420075390 0x7f0420075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.085+0000 7f0426c00640 1 --2- 172.21.15.1:0/107798298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f042010ed40 0x7f0420111130 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.085+0000 7f0426c00640 1 --2- 172.21.15.1:0/107798298 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0420076970 0x7f0420074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.085+0000 7f0426c00640 1 -- 172.21.15.1:0/107798298 >> 172.21.15.1:0/107798298 conn(0x7f04200fdce0 msgr2=0x7f0420100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.085+0000 7f0426c00640 1 -- 172.21.15.1:0/107798298 shutdown_connections 2024-04-26T21:30:04.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.085+0000 7f0426c00640 1 -- 172.21.15.1:0/107798298 wait complete. 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.086+0000 7f0426c00640 1 Processor -- start 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.086+0000 7f0426c00640 1 -- start start 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.086+0000 7f0426c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0420075390 0x7f042019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.086+0000 7f0426c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0420076970 0x7f042019f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.086+0000 7f0426c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f042010ed40 0x7f04201a6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.086+0000 7f0426c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0420114890 con 0x7f0420075390 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.086+0000 7f0426c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0420114710 con 0x7f0420076970 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.086+0000 7f0426c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0420114a10 con 0x7f042010ed40 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0425800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0420075390 0x7f042019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0425800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0420075390 0x7f042019ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53252/0 (socket says 172.21.15.1:53252) 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0425800640 1 -- 172.21.15.1:0/340879266 learned_addr learned my addr 172.21.15.1:0/340879266 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:04.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0424e00640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0420076970 0x7f042019f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0426200640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f042010ed40 0x7f04201a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0425800640 1 -- 172.21.15.1:0/340879266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f042010ed40 msgr2=0x7f04201a6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0425800640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f042010ed40 0x7f04201a6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0425800640 1 -- 172.21.15.1:0/340879266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0420076970 msgr2=0x7f042019f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0425800640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0420076970 0x7f042019f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0425800640 1 -- 172.21.15.1:0/340879266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04201a6a90 con 0x7f0420075390 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0426200640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f042010ed40 0x7f04201a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.087+0000 7f0425800640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0420075390 0x7f042019ee70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f041000e9c0 tx=0x7f041000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.088+0000 7f041e000640 1 -- 172.21.15.1:0/340879266 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041000ccc0 con 0x7f0420075390 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.088+0000 7f041e000640 1 -- 172.21.15.1:0/340879266 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04100054d0 con 0x7f0420075390 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.088+0000 7f041e000640 1 -- 172.21.15.1:0/340879266 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0410010430 con 0x7f0420075390 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.088+0000 7f0426c00640 1 -- 172.21.15.1:0/340879266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04201a6d80 con 0x7f0420075390 2024-04-26T21:30:04.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.088+0000 7f0426c00640 1 -- 172.21.15.1:0/340879266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04201a71c0 con 0x7f0420075390 2024-04-26T21:30:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.089+0000 7f0426c00640 1 -- 172.21.15.1:0/340879266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0420104f10 con 0x7f0420075390 2024-04-26T21:30:04.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.091+0000 7f041e000640 1 -- 172.21.15.1:0/340879266 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f0410004f80 con 0x7f0420075390 2024-04-26T21:30:04.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.092+0000 7f041e000640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04040a3d70 0x7f04040a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.093+0000 7f041e000640 1 -- 172.21.15.1:0/340879266 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f04100d0dd0 con 0x7f0420075390 2024-04-26T21:30:04.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.093+0000 7f0424e00640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04040a3d70 0x7f04040a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.094+0000 7f0424e00640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04040a3d70 0x7f04040a6230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f04201a0380 tx=0x7f040c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:04.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.097+0000 7f041e000640 1 -- 172.21.15.1:0/340879266 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f041009d1f0 con 0x7f0420075390 2024-04-26T21:30:04.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:03 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/778831261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:30:04.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:03 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1581317974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:30:04.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.305+0000 7f0426c00640 1 -- 172.21.15.1:0/340879266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0420063670 con 0x7f0420075390 2024-04-26T21:30:04.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.310+0000 7f041e000640 1 -- 172.21.15.1:0/340879266 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24767 (secure 0 0 0) 0x7f041009caf0 con 0x7f0420075390 2024-04-26T21:30:04.311 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:30:04.311 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":95,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:28:57.077487+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7},{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:30:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 -- 172.21.15.1:0/340879266 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04040a3d70 msgr2=0x7f04040a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04040a3d70 0x7f04040a6230 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f04201a0380 tx=0x7f040c009500 comp rx=0 tx=0).stop 2024-04-26T21:30:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 -- 172.21.15.1:0/340879266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0420075390 msgr2=0x7f042019ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0420075390 0x7f042019ee70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f041000e9c0 tx=0x7f041000ee80 comp rx=0 tx=0).stop 2024-04-26T21:30:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 -- 172.21.15.1:0/340879266 shutdown_connections 2024-04-26T21:30:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f04040a3d70 0x7f04040a6230 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f042010ed40 0x7f04201a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0420076970 0x7f042019f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 --2- 172.21.15.1:0/340879266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0420075390 0x7f042019ee70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 -- 172.21.15.1:0/340879266 >> 172.21.15.1:0/340879266 conn(0x7f04200fdce0 msgr2=0x7f04201020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 -- 172.21.15.1:0/340879266 shutdown_connections 2024-04-26T21:30:04.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.313+0000 7f0426c00640 1 -- 172.21.15.1:0/340879266 wait complete. 2024-04-26T21:30:04.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:03 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/778831261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:30:04.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:03 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1581317974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:30:04.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:03 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/778831261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:30:04.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:03 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1581317974' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:30:04.333 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:27:00.381106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:27:01.743898+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T21:27:05.341216+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T21:30:04.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:30:04.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.522+0000 7f829f200640 1 -- 172.21.15.1:0/2060972405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82a0100a90 msgr2=0x7f82a0100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.522+0000 7f829f200640 1 --2- 172.21.15.1:0/2060972405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82a0100a90 0x7f82a0100eb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8290009900 tx=0x7f829002a410 comp rx=0 tx=0).stop 2024-04-26T21:30:04.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.522+0000 7f829f200640 1 -- 172.21.15.1:0/2060972405 shutdown_connections 2024-04-26T21:30:04.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.522+0000 7f829f200640 1 --2- 172.21.15.1:0/2060972405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a0103a60 0x7f82a0103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.522+0000 7f829f200640 1 --2- 172.21.15.1:0/2060972405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82a0104420 0x7f82a010acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.522+0000 7f829f200640 1 --2- 172.21.15.1:0/2060972405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82a0100a90 0x7f82a0100eb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.522+0000 7f829f200640 1 -- 172.21.15.1:0/2060972405 >> 172.21.15.1:0/2060972405 conn(0x7f82a00fc3c0 msgr2=0x7f82a00fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:04.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.522+0000 7f829f200640 1 -- 172.21.15.1:0/2060972405 shutdown_connections 2024-04-26T21:30:04.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.522+0000 7f829f200640 1 -- 172.21.15.1:0/2060972405 wait complete. 2024-04-26T21:30:04.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.523+0000 7f829f200640 1 Processor -- start 2024-04-26T21:30:04.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.523+0000 7f829f200640 1 -- start start 2024-04-26T21:30:04.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829f200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82a0100a90 0x7f82a01033d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a0103a60 0x7f82a0101a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829f200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82a0104420 0x7f82a0101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829f200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f82a010e360 con 0x7f82a0100a90 2024-04-26T21:30:04.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829f200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f82a010e1e0 con 0x7f82a0104420 2024-04-26T21:30:04.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f82a010e4e0 con 0x7f82a0103a60 2024-04-26T21:30:04.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82a0100a90 0x7f82a01033d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82a0100a90 0x7f82a01033d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53270/0 (socket says 172.21.15.1:53270) 2024-04-26T21:30:04.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829de00640 1 -- 172.21.15.1:0/3921005041 learned_addr learned my addr 172.21.15.1:0/3921005041 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:04.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829e800640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82a0104420 0x7f82a0101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829d400640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a0103a60 0x7f82a0101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829de00640 1 -- 172.21.15.1:0/3921005041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a0103a60 msgr2=0x7f82a0101a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.524+0000 7f829de00640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a0103a60 0x7f82a0101a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.525+0000 7f829de00640 1 -- 172.21.15.1:0/3921005041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82a0104420 msgr2=0x7f82a0101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.525+0000 7f829de00640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82a0104420 0x7f82a0101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.525+0000 7f829de00640 1 -- 172.21.15.1:0/3921005041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a01027e0 con 0x7f82a0100a90 2024-04-26T21:30:04.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.525+0000 7f829de00640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82a0100a90 0x7f82a01033d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8290031490 tx=0x7f8290003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:04.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.525+0000 7f8296a00640 1 -- 172.21.15.1:0/3921005041 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829003f070 con 0x7f82a0100a90 2024-04-26T21:30:04.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.525+0000 7f8296a00640 1 -- 172.21.15.1:0/3921005041 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8290031df0 con 0x7f82a0100a90 2024-04-26T21:30:04.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.525+0000 7f829f200640 1 -- 172.21.15.1:0/3921005041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a01a2b10 con 0x7f82a0100a90 2024-04-26T21:30:04.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.525+0000 7f8296a00640 1 -- 172.21.15.1:0/3921005041 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290035030 con 0x7f82a0100a90 2024-04-26T21:30:04.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.525+0000 7f829f200640 1 -- 172.21.15.1:0/3921005041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a01a3060 con 0x7f82a0100a90 2024-04-26T21:30:04.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.527+0000 7f829f200640 1 -- 172.21.15.1:0/3921005041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82a0109c30 con 0x7f82a0100a90 2024-04-26T21:30:04.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.529+0000 7f8296a00640 1 -- 172.21.15.1:0/3921005041 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f8290042070 con 0x7f82a0100a90 2024-04-26T21:30:04.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.530+0000 7f8296a00640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f82840a3c40 0x7f82840a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.530+0000 7f8296a00640 1 -- 172.21.15.1:0/3921005041 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f82900f1140 con 0x7f82a0100a90 2024-04-26T21:30:04.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.530+0000 7f829d400640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f82840a3c40 0x7f82840a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.531+0000 7f829d400640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f82840a3c40 0x7f82840a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82a0103140 tx=0x7f828c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.534+0000 7f8296a00640 1 -- 172.21.15.1:0/3921005041 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f82900bcf10 con 0x7f82a0100a90 2024-04-26T21:30:04.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.737+0000 7f829f200640 1 -- 172.21.15.1:0/3921005041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f82a002dcf0 con 0x7f82a0100a90 2024-04-26T21:30:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.738+0000 7f8296a00640 1 -- 172.21.15.1:0/3921005041 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) ==== 93+0+10 (secure 0 0 0) 0x7f82900c1210 con 0x7f82a0100a90 2024-04-26T21:30:04.742 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.741+0000 7f829f200640 1 -- 172.21.15.1:0/3921005041 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f82840a3c40 msgr2=0x7f82840a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.741+0000 7f829f200640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f82840a3c40 0x7f82840a6100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82a0103140 tx=0x7f828c005f90 comp rx=0 tx=0).stop 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.741+0000 7f829f200640 1 -- 172.21.15.1:0/3921005041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82a0100a90 msgr2=0x7f82a01033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.741+0000 7f829f200640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82a0100a90 0x7f82a01033d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8290031490 tx=0x7f8290003c80 comp rx=0 tx=0).stop 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.741+0000 7f829f200640 1 -- 172.21.15.1:0/3921005041 shutdown_connections 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.741+0000 7f829f200640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f82840a3c40 0x7f82840a6100 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.741+0000 7f829f200640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82a0103a60 0x7f82a0101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.741+0000 7f829f200640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82a0104420 0x7f82a0101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.741+0000 7f829f200640 1 --2- 172.21.15.1:0/3921005041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82a0100a90 0x7f82a01033d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.741+0000 7f829f200640 1 -- 172.21.15.1:0/3921005041 >> 172.21.15.1:0/3921005041 conn(0x7f82a00fc3c0 msgr2=0x7f82a0106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.742+0000 7f829f200640 1 -- 172.21.15.1:0/3921005041 shutdown_connections 2024-04-26T21:30:04.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.742+0000 7f829f200640 1 -- 172.21.15.1:0/3921005041 wait complete. 2024-04-26T21:30:04.763 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:30:04.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.978+0000 7f22b9a00640 1 -- 172.21.15.1:0/272665385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b410a180 msgr2=0x7f22b410a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.978+0000 7f22b9a00640 1 --2- 172.21.15.1:0/272665385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b410a180 0x7f22b410a600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f229c009900 tx=0x7f229c02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.978+0000 7f22b9a00640 1 -- 172.21.15.1:0/272665385 shutdown_connections 2024-04-26T21:30:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.978+0000 7f22b9a00640 1 --2- 172.21.15.1:0/272665385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b410a180 0x7f22b410a600 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.978+0000 7f22b9a00640 1 --2- 172.21.15.1:0/272665385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b410ab40 0x7f22b41113d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.978+0000 7f22b9a00640 1 --2- 172.21.15.1:0/272665385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b4108f90 0x7f22b41093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.978+0000 7f22b9a00640 1 -- 172.21.15.1:0/272665385 >> 172.21.15.1:0/272665385 conn(0x7f22b4077f40 msgr2=0x7f22b407a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:04.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.979+0000 7f22b9a00640 1 -- 172.21.15.1:0/272665385 shutdown_connections 2024-04-26T21:30:04.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.979+0000 7f22b9a00640 1 -- 172.21.15.1:0/272665385 wait complete. 2024-04-26T21:30:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b9a00640 1 Processor -- start 2024-04-26T21:30:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b9a00640 1 -- start start 2024-04-26T21:30:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b9a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b4108f90 0x7f22b419f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b9a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b410a180 0x7f22b419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b9a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b410ab40 0x7f22b41a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b9a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22b4114b20 con 0x7f22b4108f90 2024-04-26T21:30:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b9a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22b41149a0 con 0x7f22b410ab40 2024-04-26T21:30:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b9a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22b4114ca0 con 0x7f22b410a180 2024-04-26T21:30:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b2a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b4108f90 0x7f22b419f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22abe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b410a180 0x7f22b419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b2a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b4108f90 0x7f22b419f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53280/0 (socket says 172.21.15.1:53280) 2024-04-26T21:30:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.980+0000 7f22b2a00640 1 -- 172.21.15.1:0/3233859709 learned_addr learned my addr 172.21.15.1:0/3233859709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.981+0000 7f22b3400640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b410ab40 0x7f22b41a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.981+0000 7f22b2a00640 1 -- 172.21.15.1:0/3233859709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b410a180 msgr2=0x7f22b419f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.981+0000 7f22b2a00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b410a180 0x7f22b419f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.981+0000 7f22b2a00640 1 -- 172.21.15.1:0/3233859709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b410ab40 msgr2=0x7f22b41a6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.981+0000 7f22b2a00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b410ab40 0x7f22b41a6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.981+0000 7f22b2a00640 1 -- 172.21.15.1:0/3233859709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b41a6de0 con 0x7f22b4108f90 2024-04-26T21:30:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.981+0000 7f22b2a00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b4108f90 0x7f22b419f1c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f22a400e9c0 tx=0x7f22a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:04.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.981+0000 7f22ab400640 1 -- 172.21.15.1:0/3233859709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a400cd80 con 0x7f22b4108f90 2024-04-26T21:30:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.982+0000 7f22b9a00640 1 -- 172.21.15.1:0/3233859709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b41a70d0 con 0x7f22b4108f90 2024-04-26T21:30:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.982+0000 7f22ab400640 1 -- 172.21.15.1:0/3233859709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22a40035f0 con 0x7f22b4108f90 2024-04-26T21:30:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.982+0000 7f22b9a00640 1 -- 172.21.15.1:0/3233859709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b41a75a0 con 0x7f22b4108f90 2024-04-26T21:30:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.982+0000 7f22ab400640 1 -- 172.21.15.1:0/3233859709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a4010630 con 0x7f22b4108f90 2024-04-26T21:30:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.983+0000 7f22b9a00640 1 -- 172.21.15.1:0/3233859709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b4110380 con 0x7f22b4108f90 2024-04-26T21:30:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.985+0000 7f22ab400640 1 -- 172.21.15.1:0/3233859709 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f22a4004000 con 0x7f22b4108f90 2024-04-26T21:30:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.986+0000 7f22ab400640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22980a3c40 0x7f22980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:04.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.986+0000 7f22ab400640 1 -- 172.21.15.1:0/3233859709 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f22a40d0e70 con 0x7f22b4108f90 2024-04-26T21:30:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.987+0000 7f22abe00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22980a3c40 0x7f22980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.987+0000 7f22abe00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22980a3c40 0x7f22980a6100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f229c031500 tx=0x7f229c005a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:04.991+0000 7f22ab400640 1 -- 172.21.15.1:0/3233859709 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f22a409d290 con 0x7f22b4108f90 2024-04-26T21:30:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:04 smithi001 ceph-mon[28054]: pgmap v419: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:30:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:04 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/340879266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:30:05.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:04 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3921005041' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:30:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.200+0000 7f22b9a00640 1 -- 172.21.15.1:0/3233859709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f22b4063530 con 0x7f22b4108f90 2024-04-26T21:30:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.201+0000 7f22ab400640 1 -- 172.21.15.1:0/3233859709 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) ==== 104+0+11 (secure 0 0 0) 0x7f22a409cb90 con 0x7f22b4108f90 2024-04-26T21:30:05.202 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:30:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.203+0000 7f22b9a00640 1 -- 172.21.15.1:0/3233859709 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22980a3c40 msgr2=0x7f22980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.203+0000 7f22b9a00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22980a3c40 0x7f22980a6100 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f229c031500 tx=0x7f229c005a20 comp rx=0 tx=0).stop 2024-04-26T21:30:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.203+0000 7f22b9a00640 1 -- 172.21.15.1:0/3233859709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b4108f90 msgr2=0x7f22b419f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.203+0000 7f22b9a00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b4108f90 0x7f22b419f1c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f22a400e9c0 tx=0x7f22a400ee80 comp rx=0 tx=0).stop 2024-04-26T21:30:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.204+0000 7f22b9a00640 1 -- 172.21.15.1:0/3233859709 shutdown_connections 2024-04-26T21:30:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.204+0000 7f22b9a00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f22980a3c40 0x7f22980a6100 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.204+0000 7f22b9a00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b410a180 0x7f22b419f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.204+0000 7f22b9a00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22b410ab40 0x7f22b41a6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.204+0000 7f22b9a00640 1 --2- 172.21.15.1:0/3233859709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b4108f90 0x7f22b419f1c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.204+0000 7f22b9a00640 1 -- 172.21.15.1:0/3233859709 >> 172.21.15.1:0/3233859709 conn(0x7f22b4077f40 msgr2=0x7f22b410d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.204+0000 7f22b9a00640 1 -- 172.21.15.1:0/3233859709 shutdown_connections 2024-04-26T21:30:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.204+0000 7f22b9a00640 1 -- 172.21.15.1:0/3233859709 wait complete. 2024-04-26T21:30:05.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:04 smithi186 ceph-mon[32991]: pgmap v419: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:30:05.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:04 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/340879266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:30:05.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:04 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3921005041' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:30:05.224 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:30:05.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:04 smithi038 ceph-mon[32724]: pgmap v419: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:30:05.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:04 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/340879266' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:30:05.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:04 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3921005041' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:30:05.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.431+0000 7f5db7000640 1 -- 172.21.15.1:0/4073641934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db0102970 msgr2=0x7f5db0102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:05.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.431+0000 7f5db7000640 1 --2- 172.21.15.1:0/4073641934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db0102970 0x7f5db0102d90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009900 tx=0x7f5da002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.432+0000 7f5db7000640 1 -- 172.21.15.1:0/4073641934 shutdown_connections 2024-04-26T21:30:05.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.432+0000 7f5db7000640 1 --2- 172.21.15.1:0/4073641934 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db0104520 0x7f5db010adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.432+0000 7f5db7000640 1 --2- 172.21.15.1:0/4073641934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5db0103b60 0x7f5db0103fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.432+0000 7f5db7000640 1 --2- 172.21.15.1:0/4073641934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db0102970 0x7f5db0102d90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.432+0000 7f5db7000640 1 -- 172.21.15.1:0/4073641934 >> 172.21.15.1:0/4073641934 conn(0x7f5db00fdec0 msgr2=0x7f5db0100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:05.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.432+0000 7f5db7000640 1 -- 172.21.15.1:0/4073641934 shutdown_connections 2024-04-26T21:30:05.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.432+0000 7f5db7000640 1 -- 172.21.15.1:0/4073641934 wait complete. 2024-04-26T21:30:05.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.433+0000 7f5db7000640 1 Processor -- start 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.433+0000 7f5db7000640 1 -- start start 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.433+0000 7f5db7000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db0102970 0x7f5db019ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5db7000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db0103b60 0x7f5db019b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5db7000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5db0104520 0x7f5db01a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5db7000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5db010e470 con 0x7f5db0103b60 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5db7000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5db010e2f0 con 0x7f5db0104520 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5db7000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5db010e5f0 con 0x7f5db0102970 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5da7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db0103b60 0x7f5db019b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5dafe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db0102970 0x7f5db019ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5db5000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5db0104520 0x7f5db01a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5da7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db0103b60 0x7f5db019b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53300/0 (socket says 172.21.15.1:53300) 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5da7e00640 1 -- 172.21.15.1:0/53358523 learned_addr learned my addr 172.21.15.1:0/53358523 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:05.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5dafe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db0102970 0x7f5db019ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:56266/0 (socket says 172.21.15.1:56266) 2024-04-26T21:30:05.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5db5000640 1 -- 172.21.15.1:0/53358523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db0102970 msgr2=0x7f5db019ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:05.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5db5000640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db0102970 0x7f5db019ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5db5000640 1 -- 172.21.15.1:0/53358523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db0103b60 msgr2=0x7f5db019b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:05.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.434+0000 7f5db5000640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db0103b60 0x7f5db019b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.435+0000 7f5db5000640 1 -- 172.21.15.1:0/53358523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db01a29b0 con 0x7f5db0104520 2024-04-26T21:30:05.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.435+0000 7f5db5000640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5db0104520 0x7f5db01a2310 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009fd0 tx=0x7f5da800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:05.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.435+0000 7f5dad600640 1 -- 172.21.15.1:0/53358523 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da800ce20 con 0x7f5db0104520 2024-04-26T21:30:05.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.435+0000 7f5dad600640 1 -- 172.21.15.1:0/53358523 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5da8005020 con 0x7f5db0104520 2024-04-26T21:30:05.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.435+0000 7f5db7000640 1 -- 172.21.15.1:0/53358523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db01a2ca0 con 0x7f5db0104520 2024-04-26T21:30:05.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.435+0000 7f5dad600640 1 -- 172.21.15.1:0/53358523 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8010690 con 0x7f5db0104520 2024-04-26T21:30:05.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.435+0000 7f5db7000640 1 -- 172.21.15.1:0/53358523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db01a31a0 con 0x7f5db0104520 2024-04-26T21:30:05.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.437+0000 7f5db7000640 1 -- 172.21.15.1:0/53358523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db01a33d0 con 0x7f5db0104520 2024-04-26T21:30:05.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.439+0000 7f5dad600640 1 -- 172.21.15.1:0/53358523 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f5da80054d0 con 0x7f5db0104520 2024-04-26T21:30:05.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.441+0000 7f5dad600640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d940a3c40 0x7f5d940a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.441+0000 7f5dad600640 1 -- 172.21.15.1:0/53358523 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f5da80d1140 con 0x7f5db0104520 2024-04-26T21:30:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.441+0000 7f5dafe00640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d940a3c40 0x7f5d940a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:05.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.442+0000 7f5dafe00640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d940a3c40 0x7f5d940a6100 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5db019c270 tx=0x7f5da00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:05.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.445+0000 7f5dad600640 1 -- 172.21.15.1:0/53358523 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f5da809d560 con 0x7f5db0104520 2024-04-26T21:30:05.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.654+0000 7f5db7000640 1 -- 172.21.15.1:0/53358523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5db01a35a0 con 0x7f5db0104520 2024-04-26T21:30:05.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.654+0000 7f5dad600640 1 -- 172.21.15.1:0/53358523 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) ==== 100+0+11 (secure 0 0 0) 0x7f5da809ce60 con 0x7f5db0104520 2024-04-26T21:30:05.655 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T21:30:05.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.657+0000 7f5db7000640 1 -- 172.21.15.1:0/53358523 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d940a3c40 msgr2=0x7f5d940a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:05.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.657+0000 7f5db7000640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d940a3c40 0x7f5d940a6100 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5db019c270 tx=0x7f5da00314a0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.657+0000 7f5db7000640 1 -- 172.21.15.1:0/53358523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5db0104520 msgr2=0x7f5db01a2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:05.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.657+0000 7f5db7000640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5db0104520 0x7f5db01a2310 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009fd0 tx=0x7f5da800ef10 comp rx=0 tx=0).stop 2024-04-26T21:30:05.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.657+0000 7f5db7000640 1 -- 172.21.15.1:0/53358523 shutdown_connections 2024-04-26T21:30:05.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.657+0000 7f5db7000640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f5d940a3c40 0x7f5d940a6100 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.657+0000 7f5db7000640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5db0104520 0x7f5db01a2310 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.657+0000 7f5db7000640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db0103b60 0x7f5db019b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.657+0000 7f5db7000640 1 --2- 172.21.15.1:0/53358523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db0102970 0x7f5db019ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.657+0000 7f5db7000640 1 -- 172.21.15.1:0/53358523 >> 172.21.15.1:0/53358523 conn(0x7f5db00fdec0 msgr2=0x7f5db0109300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:05.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.658+0000 7f5db7000640 1 -- 172.21.15.1:0/53358523 shutdown_connections 2024-04-26T21:30:05.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.658+0000 7f5db7000640 1 -- 172.21.15.1:0/53358523 wait complete. 2024-04-26T21:30:05.677 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T21:30:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.886+0000 7f92fdc00640 1 -- 172.21.15.1:0/1781533881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92f8103970 msgr2=0x7f92f8105de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.886+0000 7f92fdc00640 1 --2- 172.21.15.1:0/1781533881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92f8103970 0x7f92f8105de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f92e0009900 tx=0x7f92e002a410 comp rx=0 tx=0).stop 2024-04-26T21:30:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.886+0000 7f92fdc00640 1 -- 172.21.15.1:0/1781533881 shutdown_connections 2024-04-26T21:30:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.886+0000 7f92fdc00640 1 --2- 172.21.15.1:0/1781533881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92f8106320 0x7f92f81067a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.886+0000 7f92fdc00640 1 --2- 172.21.15.1:0/1781533881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92f8103970 0x7f92f8105de0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.886+0000 7f92fdc00640 1 --2- 172.21.15.1:0/1781533881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92f8101000 0x7f92f8103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.886+0000 7f92fdc00640 1 -- 172.21.15.1:0/1781533881 >> 172.21.15.1:0/1781533881 conn(0x7f92f80fcc20 msgr2=0x7f92f80ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.886+0000 7f92fdc00640 1 -- 172.21.15.1:0/1781533881 shutdown_connections 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.886+0000 7f92fdc00640 1 -- 172.21.15.1:0/1781533881 wait complete. 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.887+0000 7f92fdc00640 1 Processor -- start 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.887+0000 7f92fdc00640 1 -- start start 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.887+0000 7f92fdc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92f8101000 0x7f92f819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.887+0000 7f92fdc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92f8103970 0x7f92f819f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.887+0000 7f92fdc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92f8106320 0x7f92f81a6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.887+0000 7f92fdc00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f92f81148b0 con 0x7f92f8101000 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.887+0000 7f92fdc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f92f8114730 con 0x7f92f8103970 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.887+0000 7f92fdc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f92f8114a30 con 0x7f92f8106320 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.888+0000 7f92f7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92f8101000 0x7f92f819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.888+0000 7f92f7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92f8101000 0x7f92f819efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53318/0 (socket says 172.21.15.1:53318) 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.888+0000 7f92f7e00640 1 -- 172.21.15.1:0/1809715490 learned_addr learned my addr 172.21.15.1:0/1809715490 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.888+0000 7f92efe00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92f8103970 0x7f92f819f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.888+0000 7f92fd200640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92f8106320 0x7f92f81a6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.888+0000 7f92f7e00640 1 -- 172.21.15.1:0/1809715490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92f8106320 msgr2=0x7f92f81a6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.888+0000 7f92f7e00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92f8106320 0x7f92f81a6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.888+0000 7f92f7e00640 1 -- 172.21.15.1:0/1809715490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92f8103970 msgr2=0x7f92f819f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.888+0000 7f92f7e00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92f8103970 0x7f92f819f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.888+0000 7f92f7e00640 1 -- 172.21.15.1:0/1809715490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f81a6c00 con 0x7f92f8101000 2024-04-26T21:30:05.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.889+0000 7f92f7e00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92f8101000 0x7f92f819efe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f92e800e9f0 tx=0x7f92e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:05.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.889+0000 7f92f5600640 1 -- 172.21.15.1:0/1809715490 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e800cd80 con 0x7f92f8101000 2024-04-26T21:30:05.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.889+0000 7f92f5600640 1 -- 172.21.15.1:0/1809715490 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92e80054d0 con 0x7f92f8101000 2024-04-26T21:30:05.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.889+0000 7f92fdc00640 1 -- 172.21.15.1:0/1809715490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f81a6ef0 con 0x7f92f8101000 2024-04-26T21:30:05.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.889+0000 7f92f5600640 1 -- 172.21.15.1:0/1809715490 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e8010690 con 0x7f92f8101000 2024-04-26T21:30:05.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.889+0000 7f92fdc00640 1 -- 172.21.15.1:0/1809715490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f81a7350 con 0x7f92f8101000 2024-04-26T21:30:05.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.890+0000 7f92fdc00640 1 -- 172.21.15.1:0/1809715490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f80fe710 con 0x7f92f8101000 2024-04-26T21:30:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.894+0000 7f92f5600640 1 -- 172.21.15.1:0/1809715490 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f92e8004e30 con 0x7f92f8101000 2024-04-26T21:30:05.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.895+0000 7f92f5600640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f92dc0a3cc0 0x7f92dc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.895+0000 7f92f5600640 1 -- 172.21.15.1:0/1809715490 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f92e80d0ed0 con 0x7f92f8101000 2024-04-26T21:30:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.895+0000 7f92efe00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f92dc0a3cc0 0x7f92dc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:05.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.896+0000 7f92efe00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f92dc0a3cc0 0x7f92dc0a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f92e0000c00 tx=0x7f92e00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:05.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:05.897+0000 7f92f5600640 1 -- 172.21.15.1:0/1809715490 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f92e8054060 con 0x7f92f8101000 2024-04-26T21:30:06.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.102+0000 7f92fdc00640 1 -- 172.21.15.1:0/1809715490 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f92f802dd40 con 0x7f92f8101000 2024-04-26T21:30:06.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.104+0000 7f92f5600640 1 -- 172.21.15.1:0/1809715490 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v95) ==== 103+0+11 (secure 0 0 0) 0x7f92e809d350 con 0x7f92f8101000 2024-04-26T21:30:06.105 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:30:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 -- 172.21.15.1:0/1809715490 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f92dc0a3cc0 msgr2=0x7f92dc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f92dc0a3cc0 0x7f92dc0a6180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f92e0000c00 tx=0x7f92e00314a0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 -- 172.21.15.1:0/1809715490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92f8101000 msgr2=0x7f92f819efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92f8101000 0x7f92f819efe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f92e800e9f0 tx=0x7f92e800edf0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 -- 172.21.15.1:0/1809715490 shutdown_connections 2024-04-26T21:30:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f92dc0a3cc0 0x7f92dc0a6180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92f8106320 0x7f92f81a6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f92f8103970 0x7f92f819f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 --2- 172.21.15.1:0/1809715490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f92f8101000 0x7f92f819efe0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 -- 172.21.15.1:0/1809715490 >> 172.21.15.1:0/1809715490 conn(0x7f92f80fcc20 msgr2=0x7f92f8101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:06.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 -- 172.21.15.1:0/1809715490 shutdown_connections 2024-04-26T21:30:06.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.106+0000 7f92fdc00640 1 -- 172.21.15.1:0/1809715490 wait complete. 2024-04-26T21:30:06.124 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:30:06.194 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:05 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3233859709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:30:06.194 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:05 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/53358523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:30:06.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:05 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3233859709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:30:06.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:05 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/53358523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:30:06.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:05 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3233859709' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:30:06.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:05 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/53358523' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:30:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.331+0000 7fc6d4c00640 1 -- 172.21.15.1:0/2551528931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc6d0102360 msgr2=0x7fc6d0102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.331+0000 7fc6d4c00640 1 --2- 172.21.15.1:0/2551528931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc6d0102360 0x7fc6d0102be0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009be0 tx=0x7fc6c802a6d0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.331+0000 7fc6d4c00640 1 -- 172.21.15.1:0/2551528931 shutdown_connections 2024-04-26T21:30:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.331+0000 7fc6d4c00640 1 --2- 172.21.15.1:0/2551528931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6d01019a0 0x7fc6d0101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.331+0000 7fc6d4c00640 1 --2- 172.21.15.1:0/2551528931 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc6d0102360 0x7fc6d0102be0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.331+0000 7fc6d4c00640 1 --2- 172.21.15.1:0/2551528931 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d0100c40 0x7fc6d0101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.331+0000 7fc6d4c00640 1 -- 172.21.15.1:0/2551528931 >> 172.21.15.1:0/2551528931 conn(0x7fc6d00fc590 msgr2=0x7fc6d00fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.331+0000 7fc6d4c00640 1 -- 172.21.15.1:0/2551528931 shutdown_connections 2024-04-26T21:30:06.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.331+0000 7fc6d4c00640 1 -- 172.21.15.1:0/2551528931 wait complete. 2024-04-26T21:30:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.332+0000 7fc6d4c00640 1 Processor -- start 2024-04-26T21:30:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.332+0000 7fc6d4c00640 1 -- start start 2024-04-26T21:30:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6d4c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc6d0100c40 0x7fc6d019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6d4c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6d01019a0 0x7fc6d019b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6d4c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d0102360 0x7fc6d01a2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6d4c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d01106c0 con 0x7fc6d0102360 2024-04-26T21:30:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6d4c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d0110540 con 0x7fc6d0100c40 2024-04-26T21:30:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6d4c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d0110840 con 0x7fc6d01019a0 2024-04-26T21:30:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6ce800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d0102360 0x7fc6d01a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:06.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6ce800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d0102360 0x7fc6d01a2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53336/0 (socket says 172.21.15.1:53336) 2024-04-26T21:30:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6ce800640 1 -- 172.21.15.1:0/3573678814 learned_addr learned my addr 172.21.15.1:0/3573678814 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6cde00640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc6d0100c40 0x7fc6d019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.333+0000 7fc6cd400640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6d01019a0 0x7fc6d019b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6ce800640 1 -- 172.21.15.1:0/3573678814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6d01019a0 msgr2=0x7fc6d019b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6ce800640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6d01019a0 0x7fc6d019b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6ce800640 1 -- 172.21.15.1:0/3573678814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc6d0100c40 msgr2=0x7fc6d019ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6ce800640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc6d0100c40 0x7fc6d019ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6ce800640 1 -- 172.21.15.1:0/3573678814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d01a2910 con 0x7fc6d0102360 2024-04-26T21:30:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6ce800640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d0102360 0x7fc6d01a2270 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8004af0 tx=0x7fc6c8030d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6c6a00640 1 -- 172.21.15.1:0/3573678814 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8041020 con 0x7fc6d0102360 2024-04-26T21:30:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6c6a00640 1 -- 172.21.15.1:0/3573678814 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6c80318f0 con 0x7fc6d0102360 2024-04-26T21:30:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6c6a00640 1 -- 172.21.15.1:0/3573678814 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8036040 con 0x7fc6d0102360 2024-04-26T21:30:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6d4c00640 1 -- 172.21.15.1:0/3573678814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d01a2ba0 con 0x7fc6d0102360 2024-04-26T21:30:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.334+0000 7fc6d4c00640 1 -- 172.21.15.1:0/3573678814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d01a3090 con 0x7fc6d0102360 2024-04-26T21:30:06.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.336+0000 7fc6d4c00640 1 -- 172.21.15.1:0/3573678814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d010c750 con 0x7fc6d0102360 2024-04-26T21:30:06.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.337+0000 7fc6c6a00640 1 -- 172.21.15.1:0/3573678814 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc6c8044020 con 0x7fc6d0102360 2024-04-26T21:30:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.339+0000 7fc6c6a00640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc6b40a3c40 0x7fc6b40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.339+0000 7fc6c6a00640 1 -- 172.21.15.1:0/3573678814 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fc6c80f16f0 con 0x7fc6d0102360 2024-04-26T21:30:06.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.339+0000 7fc6cde00640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc6b40a3c40 0x7fc6b40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:06.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.340+0000 7fc6cde00640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc6b40a3c40 0x7fc6b40a6100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc6d019c1d0 tx=0x7fc6c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:06.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.343+0000 7fc6c6a00640 1 -- 172.21.15.1:0/3573678814 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fc6c804b020 con 0x7fc6d0102360 2024-04-26T21:30:06.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.589+0000 7fc6d4c00640 1 -- 172.21.15.1:0/3573678814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6d002dd40 con 0x7fc6d0102360 2024-04-26T21:30:06.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.590+0000 7fc6c6a00640 1 -- 172.21.15.1:0/3573678814 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11857 (secure 0 0 0) 0x7fc6c80bdc20 con 0x7fc6d0102360 2024-04-26T21:30:06.591 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:30:06.591 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:27:02.795246+0000","modified":"2024-04-26T21:27:28.957703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24572,"mds_2":24548},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/2477017203","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2477017203},{"type":"v1","addr":"172.21.15.186:6839","nonce":2477017203}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24548":{"gid":24548,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24548,24572,24602]},"id":1}]} 2024-04-26T21:30:06.592 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 23 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 -- 172.21.15.1:0/3573678814 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc6b40a3c40 msgr2=0x7fc6b40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc6b40a3c40 0x7fc6b40a6100 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc6d019c1d0 tx=0x7fc6c0005f90 comp rx=0 tx=0).stop 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 -- 172.21.15.1:0/3573678814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d0102360 msgr2=0x7fc6d01a2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d0102360 0x7fc6d01a2270 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8004af0 tx=0x7fc6c8030d50 comp rx=0 tx=0).stop 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 -- 172.21.15.1:0/3573678814 shutdown_connections 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc6b40a3c40 0x7fc6b40a6100 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6d01019a0 0x7fc6d019b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d0102360 0x7fc6d01a2270 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 --2- 172.21.15.1:0/3573678814 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc6d0100c40 0x7fc6d019ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 -- 172.21.15.1:0/3573678814 >> 172.21.15.1:0/3573678814 conn(0x7fc6d00fc590 msgr2=0x7fc6d00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:06.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 -- 172.21.15.1:0/3573678814 shutdown_connections 2024-04-26T21:30:06.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.593+0000 7fc6d4c00640 1 -- 172.21.15.1:0/3573678814 wait complete. 2024-04-26T21:30:06.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:30:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.819+0000 7f0206c00640 1 -- 172.21.15.1:0/889375966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0200103960 msgr2=0x7f0200103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.819+0000 7f0206c00640 1 --2- 172.21.15.1:0/889375966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0200103960 0x7f0200103de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009900 tx=0x7f01ec02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.819+0000 7f0206c00640 1 -- 172.21.15.1:0/889375966 shutdown_connections 2024-04-26T21:30:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.819+0000 7f0206c00640 1 --2- 172.21.15.1:0/889375966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0200104320 0x7f020010abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.819+0000 7f0206c00640 1 --2- 172.21.15.1:0/889375966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0200103960 0x7f0200103de0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.819+0000 7f0206c00640 1 --2- 172.21.15.1:0/889375966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0200102770 0x7f0200102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.819+0000 7f0206c00640 1 -- 172.21.15.1:0/889375966 >> 172.21.15.1:0/889375966 conn(0x7f02000fdce0 msgr2=0x7f0200100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.819+0000 7f0206c00640 1 -- 172.21.15.1:0/889375966 shutdown_connections 2024-04-26T21:30:06.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.819+0000 7f0206c00640 1 -- 172.21.15.1:0/889375966 wait complete. 2024-04-26T21:30:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.820+0000 7f0206c00640 1 Processor -- start 2024-04-26T21:30:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.820+0000 7f0206c00640 1 -- start start 2024-04-26T21:30:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0206c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0200102770 0x7f020019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0206c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0200103960 0x7f020019b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0206c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0200104320 0x7f02001a2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0206c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f020010e2f0 con 0x7f0200103960 2024-04-26T21:30:06.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0206c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f020010e170 con 0x7f0200104320 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0206c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f020010e470 con 0x7f0200102770 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0205800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0200102770 0x7f020019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0205800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0200102770 0x7f020019aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:56312/0 (socket says 172.21.15.1:56312) 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0205800640 1 -- 172.21.15.1:0/1162864589 learned_addr learned my addr 172.21.15.1:0/1162864589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0204e00640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0200103960 0x7f020019b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.821+0000 7f0206200640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0200104320 0x7f02001a2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f0205800640 1 -- 172.21.15.1:0/1162864589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0200104320 msgr2=0x7f02001a2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f0205800640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0200104320 0x7f02001a2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f0205800640 1 -- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0200103960 msgr2=0x7f020019b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f0205800640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0200103960 0x7f020019b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f0205800640 1 -- 172.21.15.1:0/1162864589 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02001a26e0 con 0x7f0200102770 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f0204e00640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0200103960 0x7f020019b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f0206200640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0200104320 0x7f02001a2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f0205800640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0200102770 0x7f020019aac0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f01f400e9c0 tx=0x7f01f400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:06.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f01fe000640 1 -- 172.21.15.1:0/1162864589 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f400cd80 con 0x7f0200102770 2024-04-26T21:30:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f0206c00640 1 -- 172.21.15.1:0/1162864589 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02001a29d0 con 0x7f0200102770 2024-04-26T21:30:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f01fe000640 1 -- 172.21.15.1:0/1162864589 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f01f40035f0 con 0x7f0200102770 2024-04-26T21:30:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f01fe000640 1 -- 172.21.15.1:0/1162864589 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f4010690 con 0x7f0200102770 2024-04-26T21:30:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.822+0000 7f0206c00640 1 -- 172.21.15.1:0/1162864589 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02001a2ef0 con 0x7f0200102770 2024-04-26T21:30:06.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.823+0000 7f0206c00640 1 -- 172.21.15.1:0/1162864589 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02001a3180 con 0x7f0200102770 2024-04-26T21:30:06.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.825+0000 7f01fe000640 1 -- 172.21.15.1:0/1162864589 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f01f4004000 con 0x7f0200102770 2024-04-26T21:30:06.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.827+0000 7f01fe000640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01e80a3cf0 0x7f01e80a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:06.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.827+0000 7f01fe000640 1 -- 172.21.15.1:0/1162864589 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f01f40d1110 con 0x7f0200102770 2024-04-26T21:30:06.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.827+0000 7f0204e00640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01e80a3cf0 0x7f01e80a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:06.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.828+0000 7f0204e00640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01e80a3cf0 0x7f01e80a61b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f01ec000c00 tx=0x7f01ec0314c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:06.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:06.829+0000 7f01fe000640 1 -- 172.21.15.1:0/1162864589 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f01f409d530 con 0x7f0200102770 2024-04-26T21:30:07.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.038+0000 7f0206c00640 1 -- 172.21.15.1:0/1162864589 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02001a3840 con 0x7f0200102770 2024-04-26T21:30:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.040+0000 7f01fe000640 1 -- 172.21.15.1:0/1162864589 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) ==== 74+0+24767 (secure 0 0 0) 0x7f01f4020e40 con 0x7f0200102770 2024-04-26T21:30:07.041 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:30:07.041 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":95,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:28:57.077487+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7},{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:30:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.042+0000 7f0206c00640 1 -- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01e80a3cf0 msgr2=0x7f01e80a61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.042+0000 7f0206c00640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01e80a3cf0 0x7f01e80a61b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f01ec000c00 tx=0x7f01ec0314c0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.042+0000 7f0206c00640 1 -- 172.21.15.1:0/1162864589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0200102770 msgr2=0x7f020019aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.042+0000 7f0206c00640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0200102770 0x7f020019aac0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f01f400e9c0 tx=0x7f01f400ee80 comp rx=0 tx=0).stop 2024-04-26T21:30:07.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.042+0000 7f0206c00640 1 -- 172.21.15.1:0/1162864589 shutdown_connections 2024-04-26T21:30:07.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.043+0000 7f0206c00640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f01e80a3cf0 0x7f01e80a61b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.043+0000 7f0206c00640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0200104320 0x7f02001a2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.043+0000 7f0206c00640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0200103960 0x7f020019b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.043+0000 7f0206c00640 1 --2- 172.21.15.1:0/1162864589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0200102770 0x7f020019aac0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.043+0000 7f0206c00640 1 -- 172.21.15.1:0/1162864589 >> 172.21.15.1:0/1162864589 conn(0x7f02000fdce0 msgr2=0x7f0200074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.043+0000 7f0206c00640 1 -- 172.21.15.1:0/1162864589 shutdown_connections 2024-04-26T21:30:07.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.043+0000 7f0206c00640 1 -- 172.21.15.1:0/1162864589 wait complete. 2024-04-26T21:30:07.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-26T21:27:02.795246+0000', 'modified': '2024-04-26T21:27:28.957703+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24602, 'mds_1': 24572, 'mds_2': 24548}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24515': {'gid': 24515, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.38:6835/3801556553', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6834', 'nonce': 3801556553}, {'type': 'v1', 'addr': '172.21.15.38:6835', 'nonce': 3801556553}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6837/3582845013', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 3582845013}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 3582845013}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6839/2477017203', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 2477017203}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 2477017203}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24548': {'gid': 24548, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.38:6837/1033703980', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6836', 'nonce': 1033703980}, {'type': 'v1', 'addr': '172.21.15.38:6837', 'nonce': 1033703980}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24572': {'gid': 24572, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.1:6839/338659174', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6838', 'nonce': 338659174}, {'type': 'v1', 'addr': '172.21.15.1:6839', 'nonce': 338659174}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24602': {'gid': 24602, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.38:6839/1675711617', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.38:6838', 'nonce': 1675711617}, {'type': 'v1', 'addr': '172.21.15.38:6839', 'nonce': 1675711617}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24602, 'qdb_cluster': [24548, 24572, 24602]} 2024-04-26T21:30:07.062 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-26T21:30:07.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-26T21:30:07.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:06 smithi186 ceph-mon[32991]: pgmap v420: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:30:07.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:06 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1809715490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:30:07.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:06 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3573678814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:30:07.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.273+0000 7f38fd000640 1 -- 172.21.15.1:0/694230412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38f8101950 msgr2=0x7f38f8101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.273+0000 7f38fd000640 1 --2- 172.21.15.1:0/694230412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38f8101950 0x7f38f8101db0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009900 tx=0x7f38e002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.274+0000 7f38fd000640 1 -- 172.21.15.1:0/694230412 shutdown_connections 2024-04-26T21:30:07.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.274+0000 7f38fd000640 1 --2- 172.21.15.1:0/694230412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38f81022f0 0x7f38f8102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.274+0000 7f38fd000640 1 --2- 172.21.15.1:0/694230412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38f8101950 0x7f38f8101db0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.274+0000 7f38fd000640 1 --2- 172.21.15.1:0/694230412 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f8100ff0 0x7f38f8101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.274+0000 7f38fd000640 1 -- 172.21.15.1:0/694230412 >> 172.21.15.1:0/694230412 conn(0x7f38f80fcc10 msgr2=0x7f38f80ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.274+0000 7f38fd000640 1 -- 172.21.15.1:0/694230412 shutdown_connections 2024-04-26T21:30:07.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.274+0000 7f38fd000640 1 -- 172.21.15.1:0/694230412 wait complete. 2024-04-26T21:30:07.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.275+0000 7f38fd000640 1 Processor -- start 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.275+0000 7f38fd000640 1 -- start start 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38fd000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38f8100ff0 0x7f38f819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38fd000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f8101950 0x7f38f819b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38fd000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38f81022f0 0x7f38f81a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38fd000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f38f81104a0 con 0x7f38f8101950 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38fd000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8110320 con 0x7f38f81022f0 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38fd000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8110620 con 0x7f38f8100ff0 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38f7400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38f8100ff0 0x7f38f819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38eea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f8101950 0x7f38f819b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38f7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38f81022f0 0x7f38f81a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38f7400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38f8100ff0 0x7f38f819ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:56324/0 (socket says 172.21.15.1:56324) 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38f7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38f81022f0 0x7f38f81a2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:38146/0 (socket says 172.21.15.1:38146) 2024-04-26T21:30:07.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38f7400640 1 -- 172.21.15.1:0/1295402364 learned_addr learned my addr 172.21.15.1:0/1295402364 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38f7400640 1 -- 172.21.15.1:0/1295402364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38f81022f0 msgr2=0x7f38f81a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38f7400640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38f81022f0 0x7f38f81a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38f7400640 1 -- 172.21.15.1:0/1295402364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f8101950 msgr2=0x7f38f819b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38f7400640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f8101950 0x7f38f819b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.276+0000 7f38f7400640 1 -- 172.21.15.1:0/1295402364 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f81a2780 con 0x7f38f8100ff0 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.277+0000 7f38f7400640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38f8100ff0 0x7f38f819ab60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f38e800e9f0 tx=0x7f38e800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.277+0000 7f38f4c00640 1 -- 172.21.15.1:0/1295402364 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e800cd80 con 0x7f38f8100ff0 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.277+0000 7f38fd000640 1 -- 172.21.15.1:0/1295402364 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f81a2a70 con 0x7f38f8100ff0 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.277+0000 7f38f4c00640 1 -- 172.21.15.1:0/1295402364 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e80108e0 con 0x7f38f8100ff0 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.277+0000 7f38f4c00640 1 -- 172.21.15.1:0/1295402364 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e8018980 con 0x7f38f8100ff0 2024-04-26T21:30:07.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.277+0000 7f38fd000640 1 -- 172.21.15.1:0/1295402364 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f81a2f90 con 0x7f38f8100ff0 2024-04-26T21:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.280+0000 7f390147c740 1 Processor -- start 2024-04-26T21:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.280+0000 7f390147c740 1 -- start start 2024-04-26T21:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f390147c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea29782c0 0x557ea29786e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f390147c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea2978cb0 0x557ea297d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f38f4c00640 1 -- 172.21.15.1:0/1295402364 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f38e8010430 con 0x7f38f8100ff0 2024-04-26T21:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f390147c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea297de20 0x557ea29802b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f390147c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x557ea28dc600 con 0x557ea297de20 2024-04-26T21:30:07.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f390147c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557ea2879100 con 0x557ea2978cb0 2024-04-26T21:30:07.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f390147c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557ea28d85c0 con 0x557ea29782c0 2024-04-26T21:30:07.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f38eea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea297de20 0x557ea29802b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f38f7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea29782c0 0x557ea29786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f38eea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea297de20 0x557ea29802b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53382/0 (socket says 172.21.15.1:53382) 2024-04-26T21:30:07.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f38f7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea29782c0 0x557ea29786e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:56338/0 (socket says 172.21.15.1:56338) 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f38eea00640 1 -- 172.21.15.1:0/846504356 learned_addr learned my addr 172.21.15.1:0/846504356 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f38f7400640 1 --2- 172.21.15.1:0/846504356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea2978cb0 0x557ea297d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f38eea00640 1 -- 172.21.15.1:0/846504356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea29782c0 msgr2=0x557ea29786e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.281+0000 7f38eea00640 1 --2- 172.21.15.1:0/846504356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea29782c0 0x557ea29786e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.282+0000 7f38eea00640 1 -- 172.21.15.1:0/846504356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea2978cb0 msgr2=0x557ea297d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.282+0000 7f38eea00640 1 --2- 172.21.15.1:0/846504356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea2978cb0 0x557ea297d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.282+0000 7f38eea00640 1 -- 172.21.15.1:0/846504356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ea25ee2d0 con 0x557ea297de20 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.282+0000 7f38f7e00640 1 --2- 172.21.15.1:0/846504356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea29782c0 0x557ea29786e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.282+0000 7f38eea00640 1 --2- 172.21.15.1:0/846504356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea297de20 0x557ea29802b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009900 tx=0x7f38e0005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e7f6b64672f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.282+0000 7f38ecc00640 1 -- 172.21.15.1:0/846504356 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e0035030 con 0x557ea297de20 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.282+0000 7f38ecc00640 1 -- 172.21.15.1:0/846504356 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e0040070 con 0x557ea297de20 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.282+0000 7f38ecc00640 1 -- 172.21.15.1:0/846504356 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e0003960 con 0x557ea297de20 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.282+0000 7f38f4c00640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38dc0a3c30 0x7f38dc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f38f4c00640 1 -- 172.21.15.1:0/1295402364 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f38e80d0c40 con 0x7f38f8100ff0 2024-04-26T21:30:07.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f38f7e00640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38dc0a3c30 0x7f38dc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f390147c740 1 -- 172.21.15.1:0/846504356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea297de20 msgr2=0x557ea29802b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f390147c740 1 --2- 172.21.15.1:0/846504356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea297de20 0x557ea29802b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009900 tx=0x7f38e0005df0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f390147c740 1 -- 172.21.15.1:0/846504356 shutdown_connections 2024-04-26T21:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f390147c740 1 --2- 172.21.15.1:0/846504356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea297de20 0x557ea29802b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f390147c740 1 --2- 172.21.15.1:0/846504356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea2978cb0 0x557ea297d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f390147c740 1 --2- 172.21.15.1:0/846504356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea29782c0 0x557ea29786e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f390147c740 1 -- 172.21.15.1:0/846504356 >> 172.21.15.1:0/846504356 conn(0x557ea290cd10 msgr2=0x557ea2973d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f390147c740 1 -- 172.21.15.1:0/846504356 shutdown_connections 2024-04-26T21:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f38f7e00640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38dc0a3c30 0x7f38dc0a60f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x557ea2979510 tx=0x7f38f0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.283+0000 7f390147c740 1 -- 172.21.15.1:0/846504356 wait complete. 2024-04-26T21:30:07.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.284+0000 7f390147c740 1 -- start start 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.284+0000 7f390147c740 1 Processor -- start 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.284+0000 7f390147c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2982680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.284+0000 7f390147c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea2978cb0 0x557ea2982bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.284+0000 7f390147c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea297de20 0x557ea2989c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.284+0000 7f390147c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x557ea2980d70 con 0x557ea29782c0 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.284+0000 7f390147c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557ea2863f70 con 0x557ea2978cb0 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.284+0000 7f390147c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557ea2980ef0 con 0x557ea297de20 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f390147c740 10 client.0 _ll_register_callbacks cb 0x557ea28d2480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38f7400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2982680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38f7400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2982680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53390/0 (socket says 172.21.15.1:53390) 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38f7400640 1 -- 172.21.15.1:0/2311561285 learned_addr learned my addr 172.21.15.1:0/2311561285 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38eea00640 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea2978cb0 0x557ea2982bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38f7e00640 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea297de20 0x557ea2989c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38f7400640 1 -- 172.21.15.1:0/2311561285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea297de20 msgr2=0x557ea2989c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38f7400640 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea297de20 0x557ea2989c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38f7400640 1 -- 172.21.15.1:0/2311561285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea2978cb0 msgr2=0x557ea2982bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38f7400640 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea2978cb0 0x557ea2982bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38f7400640 1 -- 172.21.15.1:0/2311561285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ea26b86b0 con 0x557ea29782c0 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38f7400640 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2982680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f38e800e9c0 tx=0x7f38e80d42c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf0c0a7062de60e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.285+0000 7f38e5600640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.286+0000 7f38e5600640 1 -- 172.21.15.1:0/2311561285 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e809c5b0 con 0x557ea29782c0 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.286+0000 7f38e5600640 1 -- 172.21.15.1:0/2311561285 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f38e80bbb80 con 0x557ea29782c0 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.286+0000 7f390147c740 1 -- 172.21.15.1:0/2311561285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38e0040070 con 0x557ea29782c0 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.286+0000 7f38e5600640 1 -- 172.21.15.1:0/2311561285 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f38e809c3d0 con 0x557ea29782c0 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.286+0000 7f390147c740 10 client.15231 fetch_fsmap learned FSMap version 23 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.286+0000 7f390147c740 1 -- 172.21.15.1:0/2311561285 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557ea298cc50 con 0x557ea29782c0 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.286+0000 7f38e5600640 1 -- 172.21.15.1:0/2311561285 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e0040070 con 0x557ea29782c0 2024-04-26T21:30:07.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.286+0000 7f38e5600640 1 -- 172.21.15.1:0/2311561285 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f38e8018b20 con 0x557ea29782c0 2024-04-26T21:30:07.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.286+0000 7f390147c740 10 client.15231 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:30:07.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.287+0000 7f390147c740 10 client.15231 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T21:30:07.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.287+0000 7f390147c740 20 client.15231 populate_metadata read hostname 'smithi001' 2024-04-26T21:30:07.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.287+0000 7f390147c740 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x557ea298f320 0x557ea2990790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.287+0000 7f390147c740 4 client.15231 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:30:07.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.287+0000 7f390147c740 1 -- 172.21.15.1:0/2311561285 --> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ea28ea210 con 0x557ea298f320 2024-04-26T21:30:07.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.287+0000 7f38eea00640 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x557ea298f320 0x557ea2990790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.287+0000 7f38eea00640 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x557ea298f320 0x557ea2990790 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b46589059a7a3252 server_cookie=5ee83bbec5b56a29 in_seq=0 out_seq=0 2024-04-26T21:30:07.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.287+0000 7f38e5600640 10 client.15231 ms_handle_connect on v2:172.21.15.38:6836/1033703980 2024-04-26T21:30:07.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.288+0000 7f38e5600640 1 -- 172.21.15.1:0/2311561285 <== mds.2 v2:172.21.15.38:6836/1033703980 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557ea28ea210 con 0x557ea298f320 2024-04-26T21:30:07.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.288+0000 7f38e5600640 10 client.15231 handle_command_reply: tid=0 2024-04-26T21:30:07.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.288+0000 7f390147c740 1 client.15231 shutdown 2024-04-26T21:30:07.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.288+0000 7f390147c740 1 -- 172.21.15.1:0/2311561285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 msgr2=0x557ea2982680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.288+0000 7f390147c740 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2982680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f38e800e9c0 tx=0x7f38e80d42c0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.289+0000 7f390147c740 1 -- 172.21.15.1:0/2311561285 shutdown_connections 2024-04-26T21:30:07.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.289+0000 7f390147c740 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x557ea298f320 0x557ea2990790 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.289+0000 7f390147c740 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea297de20 0x557ea2989c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.289+0000 7f390147c740 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea2978cb0 0x557ea2982bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.289+0000 7f390147c740 1 --2- 172.21.15.1:0/2311561285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2982680 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.289+0000 7f390147c740 1 -- 172.21.15.1:0/2311561285 >> 172.21.15.1:0/2311561285 conn(0x557ea290cd10 msgr2=0x557ea2972f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.289+0000 7f38e5600640 0 client.15231 ms_handle_reset on v2:172.21.15.38:6836/1033703980 2024-04-26T21:30:07.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.289+0000 7f390147c740 1 -- 172.21.15.1:0/2311561285 shutdown_connections 2024-04-26T21:30:07.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.289+0000 7f390147c740 1 -- 172.21.15.1:0/2311561285 wait complete. 2024-04-26T21:30:07.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.289+0000 7f390147c740 20 client.15231 trim_cache size 0 max 16384 2024-04-26T21:30:07.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.308+0000 7f390147c740 1 Processor -- start 2024-04-26T21:30:07.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.308+0000 7f390147c740 1 -- start start 2024-04-26T21:30:07.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f390147c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea29782c0 0x557ea2982290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f390147c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea2978cb0 0x557ea2989bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f390147c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea297de20 0x557ea29827d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f390147c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x557ea298ead0 con 0x557ea2978cb0 2024-04-26T21:30:07.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f390147c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557ea298f1a0 con 0x557ea29782c0 2024-04-26T21:30:07.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f390147c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557ea298f870 con 0x557ea297de20 2024-04-26T21:30:07.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f38f7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea29782c0 0x557ea2982290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f38f7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea29782c0 0x557ea2982290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:38174/0 (socket says 172.21.15.1:38174) 2024-04-26T21:30:07.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f38f7400640 1 -- 172.21.15.1:0/4182257736 learned_addr learned my addr 172.21.15.1:0/4182257736 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f38eea00640 1 --2- 172.21.15.1:0/4182257736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea2978cb0 0x557ea2989bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f38f7e00640 1 --2- 172.21.15.1:0/4182257736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea297de20 0x557ea29827d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f38eea00640 1 -- 172.21.15.1:0/4182257736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea297de20 msgr2=0x557ea29827d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.309+0000 7f38eea00640 1 --2- 172.21.15.1:0/4182257736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea297de20 0x557ea29827d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.310+0000 7f38eea00640 1 -- 172.21.15.1:0/4182257736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea29782c0 msgr2=0x557ea2982290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.310+0000 7f38eea00640 1 --2- 172.21.15.1:0/4182257736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea29782c0 0x557ea2982290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.310+0000 7f38eea00640 1 -- 172.21.15.1:0/4182257736 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ea296f460 con 0x557ea2978cb0 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.310+0000 7f38f7400640 1 --2- 172.21.15.1:0/4182257736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea29782c0 0x557ea2982290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.310+0000 7f38eea00640 1 --2- 172.21.15.1:0/4182257736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea2978cb0 0x557ea2989bc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f38e0006010 tx=0x7f38e0031b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dbe057d35bde379 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.310+0000 7f38e7e00640 1 -- 172.21.15.1:0/4182257736 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e0035040 con 0x557ea2978cb0 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.310+0000 7f38e7e00640 1 -- 172.21.15.1:0/4182257736 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e0040070 con 0x557ea2978cb0 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.310+0000 7f38e7e00640 1 -- 172.21.15.1:0/4182257736 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e00456d0 con 0x557ea2978cb0 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 -- 172.21.15.1:0/4182257736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea2978cb0 msgr2=0x557ea2989bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 --2- 172.21.15.1:0/4182257736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea2978cb0 0x557ea2989bc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f38e0006010 tx=0x7f38e0031b90 comp rx=0 tx=0).stop 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 -- 172.21.15.1:0/4182257736 shutdown_connections 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 --2- 172.21.15.1:0/4182257736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea297de20 0x557ea29827d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 --2- 172.21.15.1:0/4182257736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea2978cb0 0x557ea2989bc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 --2- 172.21.15.1:0/4182257736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea29782c0 0x557ea2982290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 -- 172.21.15.1:0/4182257736 >> 172.21.15.1:0/4182257736 conn(0x557ea290cd10 msgr2=0x557ea298e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 -- 172.21.15.1:0/4182257736 shutdown_connections 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 -- 172.21.15.1:0/4182257736 wait complete. 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 -- start start 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.311+0000 7f390147c740 1 Processor -- start 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f390147c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2985df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f390147c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea2978cb0 0x557ea2986330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f390147c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea297de20 0x557ea2988e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f390147c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x557ea2990610 con 0x557ea29782c0 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f390147c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x557ea298ff40 con 0x557ea297de20 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f390147c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557ea2990ce0 con 0x557ea2978cb0 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f390147c740 10 client.0 _ll_register_callbacks cb 0x7f38e80d4480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f38f7400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2985df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f38eea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea2978cb0 0x557ea2986330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f38f7400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2985df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53414/0 (socket says 172.21.15.1:53414) 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f38f7400640 1 -- 172.21.15.1:0/98110696 learned_addr learned my addr 172.21.15.1:0/98110696 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.312+0000 7f38f7e00640 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea297de20 0x557ea2988e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38f7400640 1 -- 172.21.15.1:0/98110696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea2978cb0 msgr2=0x557ea2986330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38f7400640 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea2978cb0 0x557ea2986330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38f7400640 1 -- 172.21.15.1:0/98110696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea297de20 msgr2=0x557ea2988e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38f7400640 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea297de20 0x557ea2988e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38f7400640 1 -- 172.21.15.1:0/98110696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ea2989440 con 0x557ea29782c0 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38eea00640 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea2978cb0 0x557ea2986330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38f7400640 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2985df0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f38e80d4530 tx=0x7f38e8098d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b90e6939de2cde6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38e5600640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38e5600640 1 -- 172.21.15.1:0/98110696 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e80bbd80 con 0x557ea29782c0 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38e5600640 1 -- 172.21.15.1:0/98110696 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f38e809bec0 con 0x557ea29782c0 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f390147c740 1 -- 172.21.15.1:0/98110696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38e0040070 con 0x557ea29782c0 2024-04-26T21:30:07.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38e5600640 1 -- 172.21.15.1:0/98110696 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f38e809ca70 con 0x557ea29782c0 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f390147c740 10 client.15237 fetch_fsmap learned FSMap version 23 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f390147c740 1 -- 172.21.15.1:0/98110696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557ea29f5940 con 0x557ea29782c0 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.313+0000 7f38e5600640 1 -- 172.21.15.1:0/98110696 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38e0040070 con 0x557ea29782c0 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.314+0000 7f38e5600640 1 -- 172.21.15.1:0/98110696 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7f38e809b930 con 0x557ea29782c0 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.314+0000 7f390147c740 10 client.15237 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.314+0000 7f390147c740 10 client.15237 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.314+0000 7f390147c740 20 client.15237 populate_metadata read hostname 'smithi001' 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.314+0000 7f390147c740 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x557ea29fa510 0x557ea29f9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.314+0000 7f390147c740 4 client.15237 mds_command: new command op to 24548 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.314+0000 7f390147c740 1 -- 172.21.15.1:0/98110696 --> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] -- command(tid 0: {"prefix": "status"}) -- 0x557ea28e4080 con 0x557ea29fa510 2024-04-26T21:30:07.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.314+0000 7f38eea00640 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x557ea29fa510 0x557ea29f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f38eea00640 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x557ea29fa510 0x557ea29f9220 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=23222c3424800a5e server_cookie=19adaf97cd0fe5ee in_seq=0 out_seq=0 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f38e5600640 10 client.15237 ms_handle_connect on v2:172.21.15.38:6836/1033703980 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f38e5600640 1 -- 172.21.15.1:0/98110696 <== mds.2 v2:172.21.15.38:6836/1033703980 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557ea28e4080 con 0x557ea29fa510 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f38e5600640 10 client.15237 handle_command_reply: tid=0 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 client.15237 shutdown 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 -- 172.21.15.1:0/98110696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 msgr2=0x557ea2985df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2985df0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f38e80d4530 tx=0x7f38e8098d10 comp rx=0 tx=0).stop 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 -- 172.21.15.1:0/98110696 shutdown_connections 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.38:6836/1033703980,v1:172.21.15.38:6837/1033703980] conn(0x557ea29fa510 0x557ea29f9220 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ea2978cb0 0x557ea2986330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557ea297de20 0x557ea2988e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f38e5600640 0 client.15237 ms_handle_reset on v2:172.21.15.38:6836/1033703980 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 --2- 172.21.15.1:0/98110696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557ea29782c0 0x557ea2985df0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 -- 172.21.15.1:0/98110696 >> 172.21.15.1:0/98110696 conn(0x557ea290cd10 msgr2=0x557ea2982290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 -- 172.21.15.1:0/98110696 shutdown_connections 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.315+0000 7f390147c740 1 -- 172.21.15.1:0/98110696 wait complete. 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f390147c740 20 client.15237 trim_cache size 0 max 16384 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 2, 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24548, 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 158.85162076399999, 2024-04-26T21:30:07.321 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 23, 2024-04-26T21:30:07.322 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 95, 2024-04-26T21:30:07.322 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T21:30:07.322 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 205.261706111 2024-04-26T21:30:07.322 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:30:07.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 -- 172.21.15.1:0/1295402364 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38dc0a3c30 msgr2=0x7f38dc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38dc0a3c30 0x7f38dc0a60f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x557ea2979510 tx=0x7f38f0009500 comp rx=0 tx=0).stop 2024-04-26T21:30:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 -- 172.21.15.1:0/1295402364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38f8100ff0 msgr2=0x7f38f819ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38f8100ff0 0x7f38f819ab60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f38e800e9f0 tx=0x7f38e800eeb0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 -- 172.21.15.1:0/1295402364 shutdown_connections 2024-04-26T21:30:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f38dc0a3c30 0x7f38dc0a60f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38f81022f0 0x7f38f81a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38f8101950 0x7f38f819b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 --2- 172.21.15.1:0/1295402364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38f8100ff0 0x7f38f819ab60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 -- 172.21.15.1:0/1295402364 >> 172.21.15.1:0/1295402364 conn(0x7f38f80fcc10 msgr2=0x7f38f80fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.316+0000 7f38fd000640 1 -- 172.21.15.1:0/1295402364 shutdown_connections 2024-04-26T21:30:07.323 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:06 smithi001 ceph-mon[28054]: pgmap v420: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:30:07.324 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:06 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1809715490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:30:07.324 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:06 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3573678814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:30:07.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.317+0000 7f38fd000640 1 -- 172.21.15.1:0/1295402364 wait complete. 2024-04-26T21:30:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:06 smithi038 ceph-mon[32724]: pgmap v420: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-26T21:30:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:06 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1809715490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:30:07.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:06 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3573678814' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:30:07.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-26T21:30:07.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.522+0000 7fc2a7200640 1 -- 172.21.15.1:0/2372566343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2a810a980 msgr2=0x7fc2a8111210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.522+0000 7fc2a7200640 1 --2- 172.21.15.1:0/2372566343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2a810a980 0x7fc2a8111210 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009860 tx=0x7fc2a002a340 comp rx=0 tx=0).stop 2024-04-26T21:30:07.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.522+0000 7fc2a7200640 1 -- 172.21.15.1:0/2372566343 shutdown_connections 2024-04-26T21:30:07.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.522+0000 7fc2a7200640 1 --2- 172.21.15.1:0/2372566343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2a810a980 0x7fc2a8111210 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.522+0000 7fc2a7200640 1 --2- 172.21.15.1:0/2372566343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a8109fc0 0x7fc2a810a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.522+0000 7fc2a7200640 1 --2- 172.21.15.1:0/2372566343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a8108dd0 0x7fc2a81091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.522+0000 7fc2a7200640 1 -- 172.21.15.1:0/2372566343 >> 172.21.15.1:0/2372566343 conn(0x7fc2a8077d70 msgr2=0x7fc2a807a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.522+0000 7fc2a7200640 1 -- 172.21.15.1:0/2372566343 shutdown_connections 2024-04-26T21:30:07.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.522+0000 7fc2a7200640 1 -- 172.21.15.1:0/2372566343 wait complete. 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.523+0000 7fc2a7200640 1 Processor -- start 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.523+0000 7fc2a7200640 1 -- start start 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a7200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2a8108dd0 0x7fc2a819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a7200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a8109fc0 0x7fc2a819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a7200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a810a980 0x7fc2a81a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a7200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a81148e0 con 0x7fc2a8109fc0 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a7200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a8114760 con 0x7fc2a8108dd0 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a7200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a8114a60 con 0x7fc2a810a980 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2a8108dd0 0x7fc2a819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a5400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a8109fc0 0x7fc2a819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2a8108dd0 0x7fc2a819ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:38212/0 (socket says 172.21.15.1:38212) 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a5e00640 1 -- 172.21.15.1:0/3265025704 learned_addr learned my addr 172.21.15.1:0/3265025704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a6800640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a810a980 0x7fc2a81a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a5e00640 1 -- 172.21.15.1:0/3265025704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a810a980 msgr2=0x7fc2a81a6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a5e00640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a810a980 0x7fc2a81a6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a5e00640 1 -- 172.21.15.1:0/3265025704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a8109fc0 msgr2=0x7fc2a819f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a5e00640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a8109fc0 0x7fc2a819f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.524+0000 7fc2a5e00640 1 -- 172.21.15.1:0/3265025704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a81a6ba0 con 0x7fc2a8108dd0 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.525+0000 7fc2a5e00640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2a8108dd0 0x7fc2a819ef80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc29800e9f0 tx=0x7fc29800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.525+0000 7fc29ea00640 1 -- 172.21.15.1:0/3265025704 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29800ccc0 con 0x7fc2a8108dd0 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.525+0000 7fc29ea00640 1 -- 172.21.15.1:0/3265025704 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2980035f0 con 0x7fc2a8108dd0 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.525+0000 7fc29ea00640 1 -- 172.21.15.1:0/3265025704 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29800ce60 con 0x7fc2a8108dd0 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.525+0000 7fc2a7200640 1 -- 172.21.15.1:0/3265025704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a81a6e90 con 0x7fc2a8108dd0 2024-04-26T21:30:07.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.525+0000 7fc2a7200640 1 -- 172.21.15.1:0/3265025704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a81a72a0 con 0x7fc2a8108dd0 2024-04-26T21:30:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.528+0000 7fc2aff79740 1 Processor -- start 2024-04-26T21:30:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2aff79740 1 -- start start 2024-04-26T21:30:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc29ea00640 1 -- 172.21.15.1:0/3265025704 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7fc298004000 con 0x7fc2a8108dd0 2024-04-26T21:30:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a3741162c0 0x55a3741166e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a374116cb0 0x55a37411b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a37411be20 0x55a37411e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2aff79740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a37407a600 con 0x55a37411be20 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2aff79740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a374017100 con 0x55a374116cb0 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2aff79740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a3740765c0 con 0x55a3741162c0 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2a6800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a3741162c0 0x55a3741166e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2a6800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a3741162c0 0x55a3741166e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:56404/0 (socket says 172.21.15.1:56404) 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2a6800640 1 -- 172.21.15.1:0/1261524660 learned_addr learned my addr 172.21.15.1:0/1261524660 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.529+0000 7fc2a5400640 1 --2- 172.21.15.1:0/1261524660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a37411be20 0x55a37411e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.530+0000 7fc2a5e00640 1 --2- 172.21.15.1:0/1261524660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a374116cb0 0x55a37411b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.530+0000 7fc2a6800640 1 -- 172.21.15.1:0/1261524660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a374116cb0 msgr2=0x55a37411b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.530+0000 7fc2a6800640 1 --2- 172.21.15.1:0/1261524660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a374116cb0 0x55a37411b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.530+0000 7fc2a6800640 1 -- 172.21.15.1:0/1261524660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a37411be20 msgr2=0x55a37411e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.530+0000 7fc2a6800640 1 --2- 172.21.15.1:0/1261524660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a37411be20 0x55a37411e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.530+0000 7fc2a6800640 1 -- 172.21.15.1:0/1261524660 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a373d8c2d0 con 0x55a3741162c0 2024-04-26T21:30:07.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.530+0000 7fc2a5400640 1 --2- 172.21.15.1:0/1261524660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a37411be20 0x55a37411e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.530+0000 7fc2a6800640 1 --2- 172.21.15.1:0/1261524660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a3741162c0 0x55a3741166e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0035040 tx=0x7fc2a0005d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1df06c44aee6fb4f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc29ea00640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc28c0a3c30 0x7fc28c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc293400640 1 -- 172.21.15.1:0/1261524660 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0041020 con 0x55a3741162c0 2024-04-26T21:30:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc293400640 1 -- 172.21.15.1:0/1261524660 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a0030e50 con 0x55a3741162c0 2024-04-26T21:30:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc293400640 1 -- 172.21.15.1:0/1261524660 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0037040 con 0x55a3741162c0 2024-04-26T21:30:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc29ea00640 1 -- 172.21.15.1:0/3265025704 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fc2980d0c80 con 0x7fc2a8108dd0 2024-04-26T21:30:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc2a5e00640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc28c0a3c30 0x7fc28c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc2aff79740 1 -- 172.21.15.1:0/1261524660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a3741162c0 msgr2=0x55a3741166e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc2aff79740 1 --2- 172.21.15.1:0/1261524660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a3741162c0 0x55a3741166e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0035040 tx=0x7fc2a0005d20 comp rx=0 tx=0).stop 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc2aff79740 1 -- 172.21.15.1:0/1261524660 shutdown_connections 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc2aff79740 1 --2- 172.21.15.1:0/1261524660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a37411be20 0x55a37411e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc2aff79740 1 --2- 172.21.15.1:0/1261524660 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a374116cb0 0x55a37411b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc2aff79740 1 --2- 172.21.15.1:0/1261524660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a3741162c0 0x55a3741166e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.531+0000 7fc2aff79740 1 -- 172.21.15.1:0/1261524660 >> 172.21.15.1:0/1261524660 conn(0x55a3740aad10 msgr2=0x55a374111d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.532+0000 7fc2a5e00640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc28c0a3c30 0x7fc28c0a60f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55a3741170b0 tx=0x7fc298003e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.532+0000 7fc2aff79740 1 -- 172.21.15.1:0/1261524660 shutdown_connections 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.532+0000 7fc2aff79740 1 -- 172.21.15.1:0/1261524660 wait complete. 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.532+0000 7fc2aff79740 1 -- start start 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.532+0000 7fc2aff79740 1 Processor -- start 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3741162c0 0x55a3741206e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 0x55a374120c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a374127c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2aff79740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a37411ed70 con 0x55a374116cb0 2024-04-26T21:30:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2aff79740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a374001f70 con 0x55a3741162c0 2024-04-26T21:30:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2aff79740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a37411eef0 con 0x55a37411be20 2024-04-26T21:30:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2aff79740 10 client.0 _ll_register_callbacks cb 0x55a374070480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:30:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2a5400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 0x55a374120c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2a5400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 0x55a374120c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53458/0 (socket says 172.21.15.1:53458) 2024-04-26T21:30:07.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2a5400640 1 -- 172.21.15.1:0/1904909245 learned_addr learned my addr 172.21.15.1:0/1904909245 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2a6800640 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a374127c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.533+0000 7fc2a6800640 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3741162c0 0x55a3741206e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc2a5400640 1 -- 172.21.15.1:0/1904909245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 msgr2=0x55a374127c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc2a5400640 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a374127c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc2a5400640 1 -- 172.21.15.1:0/1904909245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3741162c0 msgr2=0x55a3741206e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc2a5400640 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3741162c0 0x55a3741206e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc2a5400640 1 -- 172.21.15.1:0/1904909245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a373e566b0 con 0x55a374116cb0 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc2a6800640 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a374127c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc2a5400640 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 0x55a374120c20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc294011d10 tx=0x7fc29400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a567b0285f201a6a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc287e00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc287e00640 1 -- 172.21.15.1:0/1904909245 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29401f060 con 0x55a374116cb0 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc287e00640 1 -- 172.21.15.1:0/1904909245 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fc294003970 con 0x55a374116cb0 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.534+0000 7fc2aff79740 1 -- 172.21.15.1:0/1904909245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2a0030e50 con 0x55a374116cb0 2024-04-26T21:30:07.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc287e00640 1 -- 172.21.15.1:0/1904909245 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc294026050 con 0x55a374116cb0 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc2aff79740 10 client.15249 fetch_fsmap learned FSMap version 23 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc2aff79740 1 -- 172.21.15.1:0/1904909245 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a37412ad60 con 0x55a374116cb0 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc287e00640 1 -- 172.21.15.1:0/1904909245 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a0030e50 con 0x55a374116cb0 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc287e00640 1 -- 172.21.15.1:0/1904909245 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fc294019030 con 0x55a374116cb0 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc2aff79740 10 client.15249 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc2aff79740 10 client.15249 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc2aff79740 20 client.15249 populate_metadata read hostname 'smithi001' 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc2aff79740 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x55a37412d420 0x55a37412e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc2aff79740 4 client.15249 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.535+0000 7fc2aff79740 1 -- 172.21.15.1:0/1904909245 --> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a374088210 con 0x55a37412d420 2024-04-26T21:30:07.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.536+0000 7fc2a6800640 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x55a37412d420 0x55a37412e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.536+0000 7fc2a6800640 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x55a37412d420 0x55a37412e890 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=35d84044b9669d7a server_cookie=650d36e3f8ea39ce in_seq=0 out_seq=0 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.536+0000 7fc287e00640 10 client.15249 ms_handle_connect on v2:172.21.15.1:6838/338659174 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.537+0000 7fc287e00640 1 -- 172.21.15.1:0/1904909245 <== mds.1 v2:172.21.15.1:6838/338659174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a374088210 con 0x55a37412d420 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.537+0000 7fc287e00640 10 client.15249 handle_command_reply: tid=0 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.537+0000 7fc2aff79740 1 client.15249 shutdown 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.537+0000 7fc2aff79740 1 -- 172.21.15.1:0/1904909245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 msgr2=0x55a374120c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.537+0000 7fc2aff79740 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 0x55a374120c20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc294011d10 tx=0x7fc29400f790 comp rx=0 tx=0).stop 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.537+0000 7fc2aff79740 1 -- 172.21.15.1:0/1904909245 shutdown_connections 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.537+0000 7fc2aff79740 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x55a37412d420 0x55a37412e890 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.537+0000 7fc2aff79740 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a374127c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.538+0000 7fc2aff79740 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 0x55a374120c20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.538+0000 7fc2aff79740 1 --2- 172.21.15.1:0/1904909245 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3741162c0 0x55a3741206e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.537+0000 7fc287e00640 0 client.15249 ms_handle_reset on v2:172.21.15.1:6838/338659174 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.538+0000 7fc2aff79740 1 -- 172.21.15.1:0/1904909245 >> 172.21.15.1:0/1904909245 conn(0x55a3740aad10 msgr2=0x55a374110f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.538+0000 7fc2aff79740 1 -- 172.21.15.1:0/1904909245 shutdown_connections 2024-04-26T21:30:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.538+0000 7fc2aff79740 1 -- 172.21.15.1:0/1904909245 wait complete. 2024-04-26T21:30:07.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.538+0000 7fc2aff79740 20 client.15249 trim_cache size 0 max 16384 2024-04-26T21:30:07.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.558+0000 7fc2aff79740 1 Processor -- start 2024-04-26T21:30:07.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.558+0000 7fc2aff79740 1 -- start start 2024-04-26T21:30:07.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.558+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3741162c0 0x55a37411f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.558+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 0x55a374127c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.558+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a37411e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.558+0000 7fc2aff79740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a37412cbd0 con 0x55a374116cb0 2024-04-26T21:30:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.558+0000 7fc2aff79740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a37412d2a0 con 0x55a3741162c0 2024-04-26T21:30:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.558+0000 7fc2aff79740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a37412d970 con 0x55a37411be20 2024-04-26T21:30:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.559+0000 7fc2a6800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a37411e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.559+0000 7fc2a6800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3741162c0 0x55a37411f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.559+0000 7fc2a6800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a37411e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:56426/0 (socket says 172.21.15.1:56426) 2024-04-26T21:30:07.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.559+0000 7fc2a6800640 1 -- 172.21.15.1:0/3336559354 learned_addr learned my addr 172.21.15.1:0/3336559354 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.559+0000 7fc2a5400640 1 --2- 172.21.15.1:0/3336559354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 0x55a374127c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.559+0000 7fc2a6800640 1 -- 172.21.15.1:0/3336559354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3741162c0 msgr2=0x55a37411f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.559+0000 7fc2a6800640 1 --2- 172.21.15.1:0/3336559354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3741162c0 0x55a37411f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.559+0000 7fc2a6800640 1 -- 172.21.15.1:0/3336559354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 msgr2=0x55a374127c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.559+0000 7fc2a6800640 1 --2- 172.21.15.1:0/3336559354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 0x55a374127c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.559+0000 7fc2a6800640 1 -- 172.21.15.1:0/3336559354 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a37412ab80 con 0x55a37411be20 2024-04-26T21:30:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.560+0000 7fc2a6800640 1 --2- 172.21.15.1:0/3336559354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a37411e210 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00304b0 tx=0x7fc2a002a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f66b43e59ec3dfbd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.560+0000 7fc292a00640 1 -- 172.21.15.1:0/3336559354 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0041040 con 0x55a37411be20 2024-04-26T21:30:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.560+0000 7fc292a00640 1 -- 172.21.15.1:0/3336559354 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a00054e0 con 0x55a37411be20 2024-04-26T21:30:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.560+0000 7fc292a00640 1 -- 172.21.15.1:0/3336559354 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0051040 con 0x55a37411be20 2024-04-26T21:30:07.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 -- 172.21.15.1:0/3336559354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 msgr2=0x55a37411e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 --2- 172.21.15.1:0/3336559354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a37411e210 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00304b0 tx=0x7fc2a002a340 comp rx=0 tx=0).stop 2024-04-26T21:30:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 -- 172.21.15.1:0/3336559354 shutdown_connections 2024-04-26T21:30:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 --2- 172.21.15.1:0/3336559354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a37411be20 0x55a37411e210 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 --2- 172.21.15.1:0/3336559354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a374116cb0 0x55a374127c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 --2- 172.21.15.1:0/3336559354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a3741162c0 0x55a37411f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 -- 172.21.15.1:0/3336559354 >> 172.21.15.1:0/3336559354 conn(0x55a3740aad10 msgr2=0x55a37410cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 -- 172.21.15.1:0/3336559354 shutdown_connections 2024-04-26T21:30:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 -- 172.21.15.1:0/3336559354 wait complete. 2024-04-26T21:30:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 -- start start 2024-04-26T21:30:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.561+0000 7fc2aff79740 1 Processor -- start 2024-04-26T21:30:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a3741162c0 0x55a3741230d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a374116cb0 0x55a374123610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2aff79740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37411be20 0x55a3741265e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2aff79740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a37412e710 con 0x55a3741162c0 2024-04-26T21:30:07.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2aff79740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a37412e040 con 0x55a37411be20 2024-04-26T21:30:07.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2aff79740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a37412ede0 con 0x55a374116cb0 2024-04-26T21:30:07.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2aff79740 10 client.0 _ll_register_callbacks cb 0x7fc294009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:30:07.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2a6800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a3741162c0 0x55a3741230d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2a5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a374116cb0 0x55a374123610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2a6800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a3741162c0 0x55a3741230d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53472/0 (socket says 172.21.15.1:53472) 2024-04-26T21:30:07.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.562+0000 7fc2a6800640 1 -- 172.21.15.1:0/3558765560 learned_addr learned my addr 172.21.15.1:0/3558765560 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc2a6800640 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37411be20 0x55a3741265e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc2a5400640 1 -- 172.21.15.1:0/3558765560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37411be20 msgr2=0x55a3741265e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc2a5400640 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37411be20 0x55a3741265e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc2a5400640 1 -- 172.21.15.1:0/3558765560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a3741162c0 msgr2=0x55a3741230d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc2a5400640 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a3741162c0 0x55a3741230d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc2a5400640 1 -- 172.21.15.1:0/3558765560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a374126bd0 con 0x55a374116cb0 2024-04-26T21:30:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc2a6800640 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37411be20 0x55a3741265e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:30:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc2a5400640 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a374116cb0 0x55a374123610 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc294004900 tx=0x7fc294004980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a46a4783e979e20 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc287e00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:30:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc287e00640 1 -- 172.21.15.1:0/3558765560 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc294004d80 con 0x55a374116cb0 2024-04-26T21:30:07.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc2aff79740 1 -- 172.21.15.1:0/3558765560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2a00054e0 con 0x55a374116cb0 2024-04-26T21:30:07.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.563+0000 7fc287e00640 1 -- 172.21.15.1:0/3558765560 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7fc29400e540 con 0x55a374116cb0 2024-04-26T21:30:07.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc287e00640 1 -- 172.21.15.1:0/3558765560 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc294038050 con 0x55a374116cb0 2024-04-26T21:30:07.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc2aff79740 10 client.25384 fetch_fsmap learned FSMap version 23 2024-04-26T21:30:07.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc2aff79740 1 -- 172.21.15.1:0/3558765560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55a374127f40 con 0x55a374116cb0 2024-04-26T21:30:07.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc287e00640 1 -- 172.21.15.1:0/3558765560 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2a00054e0 con 0x55a374116cb0 2024-04-26T21:30:07.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc287e00640 1 -- 172.21.15.1:0/3558765560 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7fc294019050 con 0x55a374116cb0 2024-04-26T21:30:07.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc2aff79740 10 client.25384 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:30:07.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc2aff79740 10 client.25384 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-26T21:30:07.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc2aff79740 20 client.25384 populate_metadata read hostname 'smithi001' 2024-04-26T21:30:07.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc2aff79740 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x55a374197d30 0x55a374196a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc2aff79740 4 client.25384 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:30:07.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.564+0000 7fc2aff79740 1 -- 172.21.15.1:0/3558765560 --> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] -- command(tid 0: {"prefix": "status"}) -- 0x55a374082080 con 0x55a374197d30 2024-04-26T21:30:07.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.565+0000 7fc2a6800640 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x55a374197d30 0x55a374196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.565+0000 7fc2a6800640 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x55a374197d30 0x55a374196a40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1e50a6f008bdb3c5 server_cookie=f4e96b9ebd8f4651 in_seq=0 out_seq=0 2024-04-26T21:30:07.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.565+0000 7fc287e00640 10 client.25384 ms_handle_connect on v2:172.21.15.1:6838/338659174 2024-04-26T21:30:07.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.565+0000 7fc287e00640 1 -- 172.21.15.1:0/3558765560 <== mds.1 v2:172.21.15.1:6838/338659174 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a374082080 con 0x55a374197d30 2024-04-26T21:30:07.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.565+0000 7fc287e00640 10 client.25384 handle_command_reply: tid=0 2024-04-26T21:30:07.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 client.25384 shutdown 2024-04-26T21:30:07.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 -- 172.21.15.1:0/3558765560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a374116cb0 msgr2=0x55a374123610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a374116cb0 0x55a374123610 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc294004900 tx=0x7fc294004980 comp rx=0 tx=0).stop 2024-04-26T21:30:07.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 -- 172.21.15.1:0/3558765560 shutdown_connections 2024-04-26T21:30:07.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.1:6838/338659174,v1:172.21.15.1:6839/338659174] conn(0x55a374197d30 0x55a374196a40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a374116cb0 0x55a374123610 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a37411be20 0x55a3741265e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 --2- 172.21.15.1:0/3558765560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a3741162c0 0x55a3741230d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 -- 172.21.15.1:0/3558765560 >> 172.21.15.1:0/3558765560 conn(0x55a3740aad10 msgr2=0x55a374118110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc287e00640 0 client.25384 ms_handle_reset on v2:172.21.15.1:6838/338659174 2024-04-26T21:30:07.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 -- 172.21.15.1:0/3558765560 shutdown_connections 2024-04-26T21:30:07.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 1 -- 172.21.15.1:0/3558765560 wait complete. 2024-04-26T21:30:07.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.566+0000 7fc2aff79740 20 client.25384 trim_cache size 0 max 16384 2024-04-26T21:30:07.583 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:30:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:30:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 1, 2024-04-26T21:30:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24572, 2024-04-26T21:30:07.583 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T21:30:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T21:30:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T21:30:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 160.11392327600001, 2024-04-26T21:30:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 23, 2024-04-26T21:30:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 95, 2024-04-26T21:30:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T21:30:07.584 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 198.36225601000001 2024-04-26T21:30:07.584 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:30:07.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 -- 172.21.15.1:0/3265025704 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc28c0a3c30 msgr2=0x7fc28c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc28c0a3c30 0x7fc28c0a60f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55a3741170b0 tx=0x7fc298003e70 comp rx=0 tx=0).stop 2024-04-26T21:30:07.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 -- 172.21.15.1:0/3265025704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2a8108dd0 msgr2=0x7fc2a819ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2a8108dd0 0x7fc2a819ef80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc29800e9f0 tx=0x7fc29800eeb0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 -- 172.21.15.1:0/3265025704 shutdown_connections 2024-04-26T21:30:07.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7fc28c0a3c30 0x7fc28c0a60f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a810a980 0x7fc2a81a6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc2a8109fc0 0x7fc2a819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 --2- 172.21.15.1:0/3265025704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc2a8108dd0 0x7fc2a819ef80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 -- 172.21.15.1:0/3265025704 >> 172.21.15.1:0/3265025704 conn(0x7fc2a8077d70 msgr2=0x7fc2a810d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 -- 172.21.15.1:0/3265025704 shutdown_connections 2024-04-26T21:30:07.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.567+0000 7fc2a7200640 1 -- 172.21.15.1:0/3265025704 wait complete. 2024-04-26T21:30:07.587 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-26T21:30:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.800+0000 7ff75a400640 1 -- 172.21.15.1:0/2513301044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff754106530 msgr2=0x7ff75410cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.800+0000 7ff75a400640 1 --2- 172.21.15.1:0/2513301044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff754106530 0x7ff75410cdc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009ca0 tx=0x7ff74c02a8b0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.800+0000 7ff75a400640 1 -- 172.21.15.1:0/2513301044 shutdown_connections 2024-04-26T21:30:07.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.800+0000 7ff75a400640 1 --2- 172.21.15.1:0/2513301044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff754105b70 0x7ff754105ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.800+0000 7ff75a400640 1 --2- 172.21.15.1:0/2513301044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff754106530 0x7ff75410cdc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.800+0000 7ff75a400640 1 --2- 172.21.15.1:0/2513301044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff754104980 0x7ff754104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.800+0000 7ff75a400640 1 -- 172.21.15.1:0/2513301044 >> 172.21.15.1:0/2513301044 conn(0x7ff7540ffed0 msgr2=0x7ff754102330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.800+0000 7ff75a400640 1 -- 172.21.15.1:0/2513301044 shutdown_connections 2024-04-26T21:30:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.800+0000 7ff75a400640 1 -- 172.21.15.1:0/2513301044 wait complete. 2024-04-26T21:30:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.802+0000 7ff75a400640 1 Processor -- start 2024-04-26T21:30:07.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.802+0000 7ff75a400640 1 -- start start 2024-04-26T21:30:07.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.802+0000 7ff75a400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff754104980 0x7ff75419ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.802+0000 7ff75a400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff754105b70 0x7ff75419b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff75a400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff754106530 0x7ff7541a2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff75a400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff754110500 con 0x7ff754105b70 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff75a400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff754110380 con 0x7ff754104980 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff75a400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff754110680 con 0x7ff754106530 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff759000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff754104980 0x7ff75419ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff759000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff754104980 0x7ff75419ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:38298/0 (socket says 172.21.15.1:38298) 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff759000640 1 -- 172.21.15.1:0/3845399216 learned_addr learned my addr 172.21.15.1:0/3845399216 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff74be00640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff754105b70 0x7ff75419b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff759a00640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff754106530 0x7ff7541a2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff759000640 1 -- 172.21.15.1:0/3845399216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff754106530 msgr2=0x7ff7541a2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff759000640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff754106530 0x7ff7541a2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff759000640 1 -- 172.21.15.1:0/3845399216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff754105b70 msgr2=0x7ff75419b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff759000640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff754105b70 0x7ff75419b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.803+0000 7ff759000640 1 -- 172.21.15.1:0/3845399216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7541a2790 con 0x7ff754104980 2024-04-26T21:30:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.804+0000 7ff759000640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff754104980 0x7ff75419ab70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff74400e9c0 tx=0x7ff74400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.804+0000 7ff752000640 1 -- 172.21.15.1:0/3845399216 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74400cd80 con 0x7ff754104980 2024-04-26T21:30:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.804+0000 7ff75a400640 1 -- 172.21.15.1:0/3845399216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7541a2a80 con 0x7ff754104980 2024-04-26T21:30:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.804+0000 7ff752000640 1 -- 172.21.15.1:0/3845399216 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7440035f0 con 0x7ff754104980 2024-04-26T21:30:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.804+0000 7ff752000640 1 -- 172.21.15.1:0/3845399216 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744010690 con 0x7ff754104980 2024-04-26T21:30:07.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.804+0000 7ff75a400640 1 -- 172.21.15.1:0/3845399216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7541a2fd0 con 0x7ff754104980 2024-04-26T21:30:07.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.807+0000 7ff75e9d8740 1 Processor -- start 2024-04-26T21:30:07.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.807+0000 7ff75e9d8740 1 -- start start 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 0x562868d436e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 0x562868d487b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d4b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff752000640 1 -- 172.21.15.1:0/3845399216 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7ff7440037a0 con 0x7ff754104980 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562868ca7600 con 0x562868d48e20 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562868c44100 con 0x562868d43cb0 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562868ca35c0 con 0x562868d432c0 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff74be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d4b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff759a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 0x562868d436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff74be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d4b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53496/0 (socket says 172.21.15.1:53496) 2024-04-26T21:30:07.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff74be00640 1 -- 172.21.15.1:0/1922081518 learned_addr learned my addr 172.21.15.1:0/1922081518 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff759000640 1 --2- 172.21.15.1:0/1922081518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 0x562868d487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff759a00640 1 -- 172.21.15.1:0/1922081518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 msgr2=0x562868d487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff759a00640 1 --2- 172.21.15.1:0/1922081518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 0x562868d487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff759a00640 1 -- 172.21.15.1:0/1922081518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 msgr2=0x562868d4b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff759a00640 1 --2- 172.21.15.1:0/1922081518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d4b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff759a00640 1 -- 172.21.15.1:0/1922081518 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628689b92d0 con 0x562868d432c0 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff74be00640 1 --2- 172.21.15.1:0/1922081518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d4b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.808+0000 7ff759000640 1 --2- 172.21.15.1:0/1922081518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 0x562868d487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.809+0000 7ff759a00640 1 --2- 172.21.15.1:0/1922081518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 0x562868d436e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff74c006010 tx=0x7ff74c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=193b40109ce99373 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.809+0000 7ff74a000640 1 -- 172.21.15.1:0/1922081518 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c030900 con 0x562868d432c0 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.809+0000 7ff74a000640 1 -- 172.21.15.1:0/1922081518 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff74c036040 con 0x562868d432c0 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.809+0000 7ff74a000640 1 -- 172.21.15.1:0/1922081518 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c004310 con 0x562868d432c0 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.809+0000 7ff752000640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff7380a3c30 0x7ff7380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff752000640 1 -- 172.21.15.1:0/3845399216 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7ff7440d0da0 con 0x7ff754104980 2024-04-26T21:30:07.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff75e9d8740 1 -- 172.21.15.1:0/1922081518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 msgr2=0x562868d436e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/1922081518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 0x562868d436e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff74c006010 tx=0x7ff74c005ef0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff759000640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff7380a3c30 0x7ff7380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff75e9d8740 1 -- 172.21.15.1:0/1922081518 shutdown_connections 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/1922081518 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d4b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/1922081518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 0x562868d487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/1922081518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 0x562868d436e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff75e9d8740 1 -- 172.21.15.1:0/1922081518 >> 172.21.15.1:0/1922081518 conn(0x562868cd7d10 msgr2=0x562868d3ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff75e9d8740 1 -- 172.21.15.1:0/1922081518 shutdown_connections 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.810+0000 7ff75e9d8740 1 -- 172.21.15.1:0/1922081518 wait complete. 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.811+0000 7ff759000640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff7380a3c30 0x7ff7380a60f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x562868d44720 tx=0x7ff7440d43f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.811+0000 7ff75e9d8740 1 -- start start 2024-04-26T21:30:07.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.811+0000 7ff75e9d8740 1 Processor -- start 2024-04-26T21:30:07.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.811+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d432c0 0x562868d4d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.811+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d4dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.811+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d54cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.811+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562868d4bd70 con 0x562868d48e20 2024-04-26T21:30:07.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.811+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562868c2ef70 con 0x562868d432c0 2024-04-26T21:30:07.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.811+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562868d4bef0 con 0x562868d43cb0 2024-04-26T21:30:07.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.811+0000 7ff75e9d8740 10 client.0 _ll_register_callbacks cb 0x562868c9d480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:30:07.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff74be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d4dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff74be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d4dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:56466/0 (socket says 172.21.15.1:56466) 2024-04-26T21:30:07.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff74be00640 1 -- 172.21.15.1:0/247033747 learned_addr learned my addr 172.21.15.1:0/247033747 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff759a00640 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d54cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff759a00640 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d432c0 0x562868d4d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff74be00640 1 -- 172.21.15.1:0/247033747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d432c0 msgr2=0x562868d4d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff74be00640 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d432c0 0x562868d4d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff74be00640 1 -- 172.21.15.1:0/247033747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 msgr2=0x562868d54cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff74be00640 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d54cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff74be00640 1 -- 172.21.15.1:0/247033747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562868a836b0 con 0x562868d43cb0 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff759a00640 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d432c0 0x562868d4d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff74be00640 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d4dc80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff73c012740 tx=0x7ff73c012c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55024a113e91bb5b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.812+0000 7ff742a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.813+0000 7ff742a00640 1 -- 172.21.15.1:0/247033747 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff73c00f8c0 con 0x562868d43cb0 2024-04-26T21:30:07.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.813+0000 7ff75e9d8740 1 -- 172.21.15.1:0/247033747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff74c036040 con 0x562868d43cb0 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.813+0000 7ff742a00640 1 -- 172.21.15.1:0/247033747 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7ff73c00fa60 con 0x562868d43cb0 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.813+0000 7ff742a00640 1 -- 172.21.15.1:0/247033747 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff73c00fc60 con 0x562868d43cb0 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.813+0000 7ff75e9d8740 10 client.25396 fetch_fsmap learned FSMap version 23 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.813+0000 7ff75e9d8740 1 -- 172.21.15.1:0/247033747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562868d57dc0 con 0x562868d43cb0 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.813+0000 7ff742a00640 1 -- 172.21.15.1:0/247033747 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff74c036040 con 0x562868d43cb0 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.813+0000 7ff742a00640 1 -- 172.21.15.1:0/247033747 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7ff73c01c420 con 0x562868d43cb0 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.813+0000 7ff75e9d8740 10 client.25396 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.814+0000 7ff75e9d8740 10 client.25396 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.814+0000 7ff75e9d8740 20 client.25396 populate_metadata read hostname 'smithi001' 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.814+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562868d5a4a0 0x562868d5b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.814+0000 7ff75e9d8740 4 client.25396 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.814+0000 7ff75e9d8740 1 -- 172.21.15.1:0/247033747 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562868cb5210 con 0x562868d5a4a0 2024-04-26T21:30:07.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.814+0000 7ff759a00640 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562868d5a4a0 0x562868d5b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.814+0000 7ff759a00640 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562868d5a4a0 0x562868d5b910 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61d835a7ebf5e586 server_cookie=56ae7bf157ddc7d2 in_seq=0 out_seq=0 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.814+0000 7ff742a00640 10 client.25396 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.815+0000 7ff742a00640 1 -- 172.21.15.1:0/247033747 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562868cb5210 con 0x562868d5a4a0 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.815+0000 7ff742a00640 10 client.25396 handle_command_reply: tid=0 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.815+0000 7ff75e9d8740 1 client.25396 shutdown 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 1 -- 172.21.15.1:0/247033747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 msgr2=0x562868d4dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d4dc80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff73c012740 tx=0x7ff73c012c00 comp rx=0 tx=0).stop 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 1 -- 172.21.15.1:0/247033747 shutdown_connections 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562868d5a4a0 0x562868d5b910 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d4dc80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d54cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/247033747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d432c0 0x562868d4d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff742a00640 0 client.25396 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:07.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 1 -- 172.21.15.1:0/247033747 >> 172.21.15.1:0/247033747 conn(0x562868cd7d10 msgr2=0x562868d3df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 1 -- 172.21.15.1:0/247033747 shutdown_connections 2024-04-26T21:30:07.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 1 -- 172.21.15.1:0/247033747 wait complete. 2024-04-26T21:30:07.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.816+0000 7ff75e9d8740 20 client.25396 trim_cache size 0 max 16384 2024-04-26T21:30:07.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.836+0000 7ff75e9d8740 1 Processor -- start 2024-04-26T21:30:07.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.836+0000 7ff75e9d8740 1 -- start start 2024-04-26T21:30:07.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.836+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d432c0 0x562868d451b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.836+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d456f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.836+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d48e20 0x562868d54c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.836+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562868d59c50 con 0x562868d432c0 2024-04-26T21:30:07.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.836+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562868d5a320 con 0x562868d48e20 2024-04-26T21:30:07.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.836+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562868d5a9f0 con 0x562868d43cb0 2024-04-26T21:30:07.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff759a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d432c0 0x562868d451b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff759a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d432c0 0x562868d451b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53514/0 (socket says 172.21.15.1:53514) 2024-04-26T21:30:07.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff759a00640 1 -- 172.21.15.1:0/2517091655 learned_addr learned my addr 172.21.15.1:0/2517091655 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff74be00640 1 --2- 172.21.15.1:0/2517091655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff759a00640 1 --2- 172.21.15.1:0/2517091655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d48e20 0x562868d54c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff759a00640 1 -- 172.21.15.1:0/2517091655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 msgr2=0x562868d456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff759a00640 1 --2- 172.21.15.1:0/2517091655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff759a00640 1 -- 172.21.15.1:0/2517091655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d48e20 msgr2=0x562868d54c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff759a00640 1 --2- 172.21.15.1:0/2517091655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d48e20 0x562868d54c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff759a00640 1 -- 172.21.15.1:0/2517091655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562868d44090 con 0x562868d432c0 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff74be00640 1 --2- 172.21.15.1:0/2517091655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.837+0000 7ff759a00640 1 --2- 172.21.15.1:0/2517091655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d432c0 0x562868d451b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009ca0 tx=0x7ff74c005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0539e71a270386 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.838+0000 7ff749600640 1 -- 172.21.15.1:0/2517091655 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c05a070 con 0x562868d432c0 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.838+0000 7ff749600640 1 -- 172.21.15.1:0/2517091655 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff74c036070 con 0x562868d432c0 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.838+0000 7ff749600640 1 -- 172.21.15.1:0/2517091655 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c053770 con 0x562868d432c0 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 -- 172.21.15.1:0/2517091655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d432c0 msgr2=0x562868d451b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/2517091655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d432c0 0x562868d451b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009ca0 tx=0x7ff74c005d80 comp rx=0 tx=0).stop 2024-04-26T21:30:07.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 -- 172.21.15.1:0/2517091655 shutdown_connections 2024-04-26T21:30:07.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/2517091655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d43cb0 0x562868d456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/2517091655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d48e20 0x562868d54c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/2517091655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d432c0 0x562868d451b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 -- 172.21.15.1:0/2517091655 >> 172.21.15.1:0/2517091655 conn(0x562868cd7d10 msgr2=0x562868d3e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 -- 172.21.15.1:0/2517091655 shutdown_connections 2024-04-26T21:30:07.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 -- 172.21.15.1:0/2517091655 wait complete. 2024-04-26T21:30:07.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 -- start start 2024-04-26T21:30:07.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.839+0000 7ff75e9d8740 1 Processor -- start 2024-04-26T21:30:07.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 0x562868d506f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 0x562868d50c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff75e9d8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d53c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562868d5b790 con 0x562868d48e20 2024-04-26T21:30:07.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562868d5b0c0 con 0x562868d43cb0 2024-04-26T21:30:07.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff75e9d8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562868d5be60 con 0x562868d432c0 2024-04-26T21:30:07.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff75e9d8740 10 client.0 _ll_register_callbacks cb 0x7ff73c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:30:07.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff759a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d53c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff759a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d53c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53526/0 (socket says 172.21.15.1:53526) 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff759a00640 1 -- 172.21.15.1:0/842307319 learned_addr learned my addr 172.21.15.1:0/842307319 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff759a00640 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 0x562868d506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.840+0000 7ff74be00640 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 0x562868d50c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.841+0000 7ff759a00640 1 -- 172.21.15.1:0/842307319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 msgr2=0x562868d506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.841+0000 7ff759a00640 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 0x562868d506f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.841+0000 7ff759a00640 1 -- 172.21.15.1:0/842307319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 msgr2=0x562868d50c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.841+0000 7ff759a00640 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 0x562868d50c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.841+0000 7ff759a00640 1 -- 172.21.15.1:0/842307319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562868d54240 con 0x562868d48e20 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.841+0000 7ff759a00640 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d53c50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009960 tx=0x7ff74c030680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899f6c1e5c6b3507 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.841+0000 7ff742a00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.841+0000 7ff742a00640 1 -- 172.21.15.1:0/842307319 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c05a070 con 0x562868d48e20 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.841+0000 7ff742a00640 1 -- 172.21.15.1:0/842307319 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7ff74c053610 con 0x562868d48e20 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.841+0000 7ff75e9d8740 1 -- 172.21.15.1:0/842307319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff74c036070 con 0x562868d48e20 2024-04-26T21:30:07.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff742a00640 1 -- 172.21.15.1:0/842307319 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff74c053810 con 0x562868d48e20 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff75e9d8740 10 client.15273 fetch_fsmap learned FSMap version 23 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff75e9d8740 1 -- 172.21.15.1:0/842307319 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562868dc0720 con 0x562868d48e20 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff742a00640 1 -- 172.21.15.1:0/842307319 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff74c036070 con 0x562868d48e20 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff742a00640 1 -- 172.21.15.1:0/842307319 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 23) ==== 8346+0+0 (secure 0 0 0) 0x7ff74c060020 con 0x562868d48e20 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff75e9d8740 10 client.15273 fetch_fsmap finished waiting for FSMap version 23 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff75e9d8740 10 client.15273 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff75e9d8740 20 client.15273 populate_metadata read hostname 'smithi001' 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562868dc4ba0 0x562868dc38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff75e9d8740 4 client.15273 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.842+0000 7ff75e9d8740 1 -- 172.21.15.1:0/842307319 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "status"}) -- 0x562868caf080 con 0x562868dc4ba0 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.843+0000 7ff74be00640 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562868dc4ba0 0x562868dc38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:07.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.843+0000 7ff74be00640 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562868dc4ba0 0x562868dc38b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=692f16fa8060880c server_cookie=21be6ae031bc2aa9 in_seq=0 out_seq=0 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.843+0000 7ff742a00640 10 client.15273 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.843+0000 7ff742a00640 1 -- 172.21.15.1:0/842307319 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x562868caf080 con 0x562868dc4ba0 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.843+0000 7ff742a00640 10 client.15273 handle_command_reply: tid=0 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.843+0000 7ff75e9d8740 1 client.15273 shutdown 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 1 -- 172.21.15.1:0/842307319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 msgr2=0x562868d53c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d53c50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009960 tx=0x7ff74c030680 comp rx=0 tx=0).stop 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 1 -- 172.21.15.1:0/842307319 shutdown_connections 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562868dc4ba0 0x562868dc38b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562868d48e20 0x562868d53c50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562868d43cb0 0x562868d50c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff742a00640 0 client.15273 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 1 --2- 172.21.15.1:0/842307319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562868d432c0 0x562868d506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 1 -- 172.21.15.1:0/842307319 >> 172.21.15.1:0/842307319 conn(0x562868cd7d10 msgr2=0x562868d59230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 1 -- 172.21.15.1:0/842307319 shutdown_connections 2024-04-26T21:30:07.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 1 -- 172.21.15.1:0/842307319 wait complete. 2024-04-26T21:30:07.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.844+0000 7ff75e9d8740 20 client.15273 trim_cache size 0 max 16384 2024-04-26T21:30:07.849 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:30:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "44ef3d6e-0412-11ef-bc93-c7b262605968", 2024-04-26T21:30:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-04-26T21:30:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 24602, 2024-04-26T21:30:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-04-26T21:30:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-04-26T21:30:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-04-26T21:30:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 184.548876549, 2024-04-26T21:30:07.849 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 23, 2024-04-26T21:30:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 95, 2024-04-26T21:30:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 91, 2024-04-26T21:30:07.850 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 191.58317909100001 2024-04-26T21:30:07.850 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:30:07.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 -- 172.21.15.1:0/3845399216 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff7380a3c30 msgr2=0x7ff7380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff7380a3c30 0x7ff7380a60f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x562868d44720 tx=0x7ff7440d43f0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 -- 172.21.15.1:0/3845399216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff754104980 msgr2=0x7ff75419ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff754104980 0x7ff75419ab70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff74400e9c0 tx=0x7ff74400ee80 comp rx=0 tx=0).stop 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 -- 172.21.15.1:0/3845399216 shutdown_connections 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7ff7380a3c30 0x7ff7380a60f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff754106530 0x7ff7541a2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff754105b70 0x7ff75419b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 --2- 172.21.15.1:0/3845399216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff754104980 0x7ff75419ab70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 -- 172.21.15.1:0/3845399216 >> 172.21.15.1:0/3845399216 conn(0x7ff7540ffed0 msgr2=0x7ff754108e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 -- 172.21.15.1:0/3845399216 shutdown_connections 2024-04-26T21:30:07.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:07.845+0000 7ff75a400640 1 -- 172.21.15.1:0/3845399216 wait complete. 2024-04-26T21:30:07.864 INFO:teuthology.run_tasks:Running task sequential... 2024-04-26T21:30:07.879 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-26T21:30:07.879 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-26T21:30:07.879 INFO:teuthology.task.print:Enabling mgr modules 2024-04-26T21:30:07.879 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-26T21:30:07.880 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T21:30:07.881 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:30:07.881 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-26T21:30:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.091+0000 7f61e2400640 1 -- 172.21.15.1:0/2714326176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61dc101050 msgr2=0x7f61dc101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.091+0000 7f61e2400640 1 --2- 172.21.15.1:0/2714326176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61dc101050 0x7f61dc101470 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f61cc009900 tx=0x7f61cc02a400 comp rx=0 tx=0).stop 2024-04-26T21:30:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.091+0000 7f61e2400640 1 -- 172.21.15.1:0/2714326176 shutdown_connections 2024-04-26T21:30:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.091+0000 7f61e2400640 1 --2- 172.21.15.1:0/2714326176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61dc102350 0x7f61dc1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:08.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.091+0000 7f61e2400640 1 --2- 172.21.15.1:0/2714326176 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc1019b0 0x7f61dc101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.091+0000 7f61e2400640 1 --2- 172.21.15.1:0/2714326176 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61dc101050 0x7f61dc101470 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.091+0000 7f61e2400640 1 -- 172.21.15.1:0/2714326176 >> 172.21.15.1:0/2714326176 conn(0x7f61dc0fcc10 msgr2=0x7f61dc0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.091+0000 7f61e2400640 1 -- 172.21.15.1:0/2714326176 shutdown_connections 2024-04-26T21:30:08.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.091+0000 7f61e2400640 1 -- 172.21.15.1:0/2714326176 wait complete. 2024-04-26T21:30:08.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.092+0000 7f61e2400640 1 Processor -- start 2024-04-26T21:30:08.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61e2400640 1 -- start start 2024-04-26T21:30:08.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61e2400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61dc101050 0x7f61dc19ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:08.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61e2400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc1019b0 0x7f61dc19b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:08.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61e2400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61dc102350 0x7f61dc1a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:08.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61e2400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc1104d0 con 0x7f61dc1019b0 2024-04-26T21:30:08.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61e2400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc110350 con 0x7f61dc101050 2024-04-26T21:30:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61e2400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc110650 con 0x7f61dc102350 2024-04-26T21:30:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61d3e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc1019b0 0x7f61dc19b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61e1000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61dc101050 0x7f61dc19ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61d3e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc1019b0 0x7f61dc19b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43494/0 (socket says 172.21.15.1:43494) 2024-04-26T21:30:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61e1000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61dc101050 0x7f61dc19ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37084/0 (socket says 172.21.15.1:37084) 2024-04-26T21:30:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61d3e00640 1 -- 172.21.15.1:0/3107911187 learned_addr learned my addr 172.21.15.1:0/3107911187 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.093+0000 7f61e1a00640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61dc102350 0x7f61dc1a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.094+0000 7f61e1000640 1 -- 172.21.15.1:0/3107911187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61dc102350 msgr2=0x7f61dc1a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.094+0000 7f61e1000640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61dc102350 0x7f61dc1a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:08.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.094+0000 7f61e1000640 1 -- 172.21.15.1:0/3107911187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc1019b0 msgr2=0x7f61dc19b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:08.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.094+0000 7f61e1000640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc1019b0 0x7f61dc19b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:08.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.094+0000 7f61e1000640 1 -- 172.21.15.1:0/3107911187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61dc1a2760 con 0x7f61dc101050 2024-04-26T21:30:08.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.094+0000 7f61e1000640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61dc101050 0x7f61dc19ab40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f61cc006010 tx=0x7f61cc0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:08.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.094+0000 7f61da000640 1 -- 172.21.15.1:0/3107911187 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61cc03f030 con 0x7f61dc101050 2024-04-26T21:30:08.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.094+0000 7f61da000640 1 -- 172.21.15.1:0/3107911187 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61cc0318f0 con 0x7f61dc101050 2024-04-26T21:30:08.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.094+0000 7f61da000640 1 -- 172.21.15.1:0/3107911187 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61cc035070 con 0x7f61dc101050 2024-04-26T21:30:08.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.094+0000 7f61e2400640 1 -- 172.21.15.1:0/3107911187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61dc1a29f0 con 0x7f61dc101050 2024-04-26T21:30:08.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.095+0000 7f61e2400640 1 -- 172.21.15.1:0/3107911187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61dc1a2e90 con 0x7f61dc101050 2024-04-26T21:30:08.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.096+0000 7f61e2400640 1 -- 172.21.15.1:0/3107911187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61dc10bfb0 con 0x7f61dc101050 2024-04-26T21:30:08.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.098+0000 7f61da000640 1 -- 172.21.15.1:0/3107911187 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) ==== 138696+0+0 (secure 0 0 0) 0x7f61cc042030 con 0x7f61dc101050 2024-04-26T21:30:08.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.099+0000 7f61da000640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f61c00a3ca0 0x7f61c00a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:08.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.099+0000 7f61da000640 1 -- 172.21.15.1:0/3107911187 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f61cc0f0da0 con 0x7f61dc101050 2024-04-26T21:30:08.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.100+0000 7f61d3e00640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f61c00a3ca0 0x7f61c00a6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:08.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.100+0000 7f61d3e00640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f61c00a3ca0 0x7f61c00a6160 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f61dc19c050 tx=0x7f61c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:08.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.104+0000 7f61da000640 1 -- 172.21.15.1:0/3107911187 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f61cc0bcb70 con 0x7f61dc101050 2024-04-26T21:30:08.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:07 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1162864589' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:30:08.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:07 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1162864589' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:30:08.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:07 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1162864589' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:30:08.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.368+0000 7f61e2400640 1 -- 172.21.15.1:0/3107911187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f61dc0635d0 con 0x7f61dc101050 2024-04-26T21:30:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.965+0000 7f61da000640 1 -- 172.21.15.1:0/3107911187 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 18) ==== 138713+0+0 (secure 0 0 0) 0x7f61cc0b91e0 con 0x7f61dc101050 2024-04-26T21:30:08.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.966+0000 7f61da000640 1 -- 172.21.15.1:0/3107911187 <== mon.2 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v18) ==== 92+0+0 (secure 0 0 0) 0x7f61cc098ea0 con 0x7f61dc101050 2024-04-26T21:30:08.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.970+0000 7f61e2400640 1 -- 172.21.15.1:0/3107911187 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f61c00a3ca0 msgr2=0x7f61c00a6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:08.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.970+0000 7f61e2400640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f61c00a3ca0 0x7f61c00a6160 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f61dc19c050 tx=0x7f61c4005f90 comp rx=0 tx=0).stop 2024-04-26T21:30:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.971+0000 7f61e2400640 1 -- 172.21.15.1:0/3107911187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61dc101050 msgr2=0x7f61dc19ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.971+0000 7f61e2400640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61dc101050 0x7f61dc19ab40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f61cc006010 tx=0x7f61cc0042f0 comp rx=0 tx=0).stop 2024-04-26T21:30:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.971+0000 7f61e2400640 1 -- 172.21.15.1:0/3107911187 shutdown_connections 2024-04-26T21:30:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.971+0000 7f61e2400640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f61c00a3ca0 0x7f61c00a6160 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.971+0000 7f61e2400640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61dc102350 0x7f61dc1a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.971+0000 7f61e2400640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61dc1019b0 0x7f61dc19b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.971+0000 7f61e2400640 1 --2- 172.21.15.1:0/3107911187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f61dc101050 0x7f61dc19ab40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.971+0000 7f61e2400640 1 -- 172.21.15.1:0/3107911187 >> 172.21.15.1:0/3107911187 conn(0x7f61dc0fcc10 msgr2=0x7f61dc0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.971+0000 7f61e2400640 1 -- 172.21.15.1:0/3107911187 shutdown_connections 2024-04-26T21:30:08.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:08.971+0000 7f61e2400640 1 -- 172.21.15.1:0/3107911187 wait complete. 2024-04-26T21:30:08.992 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-26T21:30:09.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.180+0000 7f3384600640 1 -- 172.21.15.1:0/840751900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f337c107010 msgr2=0x7f337c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:09.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.180+0000 7f3384600640 1 --2- 172.21.15.1:0/840751900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f337c107010 0x7f337c109400 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3374009ca0 tx=0x7f337402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.180+0000 7f3384600640 1 -- 172.21.15.1:0/840751900 shutdown_connections 2024-04-26T21:30:09.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.180+0000 7f3384600640 1 --2- 172.21.15.1:0/840751900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f337c107010 0x7f337c109400 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.180+0000 7f3384600640 1 --2- 172.21.15.1:0/840751900 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f337c102310 0x7f337c102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.180+0000 7f3384600640 1 --2- 172.21.15.1:0/840751900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f337c0693a0 0x7f337c101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.180+0000 7f3384600640 1 -- 172.21.15.1:0/840751900 >> 172.21.15.1:0/840751900 conn(0x7f337c0fc150 msgr2=0x7f337c0fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:09.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.180+0000 7f3384600640 1 -- 172.21.15.1:0/840751900 shutdown_connections 2024-04-26T21:30:09.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.180+0000 7f3384600640 1 -- 172.21.15.1:0/840751900 wait complete. 2024-04-26T21:30:09.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.181+0000 7f3384600640 1 Processor -- start 2024-04-26T21:30:09.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.181+0000 7f3384600640 1 -- start start 2024-04-26T21:30:09.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.181+0000 7f3384600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f337c0693a0 0x7f337c19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:09.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3384600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f337c102310 0x7f337c19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3384600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f337c107010 0x7f337c1a2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3384600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f337c10e570 con 0x7f337c102310 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3384600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f337c10e3f0 con 0x7f337c107010 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3384600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f337c10e6f0 con 0x7f337c0693a0 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3381c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f337c0693a0 0x7f337c19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3381c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f337c0693a0 0x7f337c19ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:34992/0 (socket says 172.21.15.1:34992) 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3381200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f337c102310 0x7f337c19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3381c00640 1 -- 172.21.15.1:0/2139643411 learned_addr learned my addr 172.21.15.1:0/2139643411 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3382600640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f337c107010 0x7f337c1a2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3381c00640 1 -- 172.21.15.1:0/2139643411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f337c107010 msgr2=0x7f337c1a2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3381c00640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f337c107010 0x7f337c1a2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3381c00640 1 -- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f337c102310 msgr2=0x7f337c19b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3381c00640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f337c102310 0x7f337c19b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.182+0000 7f3381c00640 1 -- 172.21.15.1:0/2139643411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c1a29b0 con 0x7f337c0693a0 2024-04-26T21:30:09.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.183+0000 7f3381c00640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f337c0693a0 0x7f337c19ada0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f337000e9c0 tx=0x7f337000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:09.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.183+0000 7f337a000640 1 -- 172.21.15.1:0/2139643411 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337000cde0 con 0x7f337c0693a0 2024-04-26T21:30:09.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.183+0000 7f337a000640 1 -- 172.21.15.1:0/2139643411 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33700108e0 con 0x7f337c0693a0 2024-04-26T21:30:09.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.183+0000 7f3384600640 1 -- 172.21.15.1:0/2139643411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337c1a2ca0 con 0x7f337c0693a0 2024-04-26T21:30:09.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.183+0000 7f337a000640 1 -- 172.21.15.1:0/2139643411 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3370018960 con 0x7f337c0693a0 2024-04-26T21:30:09.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.183+0000 7f3384600640 1 -- 172.21.15.1:0/2139643411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f337c1a31a0 con 0x7f337c0693a0 2024-04-26T21:30:09.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.184+0000 7f3384600640 1 -- 172.21.15.1:0/2139643411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f337c1a33d0 con 0x7f337c0693a0 2024-04-26T21:30:09.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.186+0000 7f337a000640 1 -- 172.21.15.1:0/2139643411 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 138713+0+0 (secure 0 0 0) 0x7f337001f020 con 0x7f337c0693a0 2024-04-26T21:30:09.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.188+0000 7f337a000640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33640a3c80 0x7f33640a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:09.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.188+0000 7f3381200640 1 -- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33640a3c80 msgr2=0x7f33640a6140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:09.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.188+0000 7f337a000640 1 -- 172.21.15.1:0/2139643411 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f33700d0d50 con 0x7f337c0693a0 2024-04-26T21:30:09.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.188+0000 7f3381200640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33640a3c80 0x7f33640a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:30:09.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.191+0000 7f337a000640 1 -- 172.21.15.1:0/2139643411 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3370054060 con 0x7f337c0693a0 2024-04-26T21:30:09.282 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:08 smithi001 ceph-mon[28054]: pgmap v421: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:30:09.282 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:08 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3107911187' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T21:30:09.282 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:08 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T21:30:09.283 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: ignoring --setuser ceph since I am not root 2024-04-26T21:30:09.283 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: ignoring --setgroup ceph since I am not root 2024-04-26T21:30:09.283 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:09.280+0000 7fdb82009200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:30:09.319 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:08 smithi186 ceph-mon[32991]: pgmap v421: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:30:09.319 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:08 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3107911187' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T21:30:09.319 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:08 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T21:30:09.319 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: ignoring --setuser ceph since I am not root 2024-04-26T21:30:09.319 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: ignoring --setgroup ceph since I am not root 2024-04-26T21:30:09.319 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-mgr[34863]: -- 172.21.15.186:0/756594857 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559cb1fbe5a0 con 0x559cb1f92800 2024-04-26T21:30:09.319 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:09.131+0000 7fb50e311200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:30:09.319 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:09.317+0000 7fb50e311200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:30:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:08 smithi038 ceph-mon[32724]: pgmap v421: 129 pgs: 129 active+clean; 635 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-26T21:30:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:08 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3107911187' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T21:30:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:08 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-26T21:30:09.327 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: ignoring --setuser ceph since I am not root 2024-04-26T21:30:09.327 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: ignoring --setgroup ceph since I am not root 2024-04-26T21:30:09.328 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-mgr[34120]: -- 172.21.15.38:0/2192555463 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55de3e5765a0 con 0x55de3e54a800 2024-04-26T21:30:09.328 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:09.144+0000 7f8e651fb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T21:30:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.389+0000 7f3381200640 1 -- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33640a3c80 msgr2=0x7f33640a6140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.389+0000 7f3381200640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33640a3c80 0x7f33640a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:30:09.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.406+0000 7f3384600640 1 -- 172.21.15.1:0/2139643411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f337c1a35a0 con 0x7f337c0693a0 2024-04-26T21:30:09.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.414+0000 7f337a000640 1 -- 172.21.15.1:0/2139643411 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f337009d170 con 0x7f337c0693a0 2024-04-26T21:30:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.418+0000 7f3384600640 1 -- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33640a3c80 msgr2=0x7f33640a6140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:30:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.418+0000 7f3384600640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33640a3c80 0x7f33640a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.418+0000 7f3384600640 1 -- 172.21.15.1:0/2139643411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f337c0693a0 msgr2=0x7f337c19ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.418+0000 7f3384600640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f337c0693a0 0x7f337c19ada0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f337000e9c0 tx=0x7f337000ee80 comp rx=0 tx=0).stop 2024-04-26T21:30:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.419+0000 7f3384600640 1 -- 172.21.15.1:0/2139643411 shutdown_connections 2024-04-26T21:30:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.419+0000 7f3384600640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f33640a3c80 0x7f33640a6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.419+0000 7f3384600640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f337c107010 0x7f337c1a2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.419+0000 7f3384600640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f337c102310 0x7f337c19b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.419+0000 7f3384600640 1 --2- 172.21.15.1:0/2139643411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f337c0693a0 0x7f337c19ada0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.419+0000 7f3384600640 1 -- 172.21.15.1:0/2139643411 >> 172.21.15.1:0/2139643411 conn(0x7f337c0fc150 msgr2=0x7f337c0fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.419+0000 7f3384600640 1 -- 172.21.15.1:0/2139643411 shutdown_connections 2024-04-26T21:30:09.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.419+0000 7f3384600640 1 -- 172.21.15.1:0/2139643411 wait complete. 2024-04-26T21:30:09.447 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-26T21:30:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.650+0000 7f43dfe00640 1 -- 172.21.15.1:0/3030204968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43d8105d50 msgr2=0x7f43d8108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:09.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.650+0000 7f43dfe00640 1 --2- 172.21.15.1:0/3030204968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43d8105d50 0x7f43d8108140 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f43c4009900 tx=0x7f43c402a410 comp rx=0 tx=0).stop 2024-04-26T21:30:09.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.651+0000 7f43dfe00640 1 -- 172.21.15.1:0/3030204968 shutdown_connections 2024-04-26T21:30:09.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.651+0000 7f43dfe00640 1 --2- 172.21.15.1:0/3030204968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d8108680 0x7f43d810aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.651+0000 7f43dfe00640 1 --2- 172.21.15.1:0/3030204968 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43d8105d50 0x7f43d8108140 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.651+0000 7f43dfe00640 1 --2- 172.21.15.1:0/3030204968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43d8103420 0x7f43d8105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.651+0000 7f43dfe00640 1 -- 172.21.15.1:0/3030204968 >> 172.21.15.1:0/3030204968 conn(0x7f43d80fce10 msgr2=0x7f43d80ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:09.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.651+0000 7f43dfe00640 1 -- 172.21.15.1:0/3030204968 shutdown_connections 2024-04-26T21:30:09.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.651+0000 7f43dfe00640 1 -- 172.21.15.1:0/3030204968 wait complete. 2024-04-26T21:30:09.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.652+0000 7f43dfe00640 1 Processor -- start 2024-04-26T21:30:09.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.652+0000 7f43dfe00640 1 -- start start 2024-04-26T21:30:09.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.652+0000 7f43dfe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d8103420 0x7f43d8198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:09.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.652+0000 7f43dfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43d8105d50 0x7f43d8199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:09.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.652+0000 7f43dfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43d8108680 0x7f43d81a0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:09.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.652+0000 7f43dfe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f43d810e530 con 0x7f43d8103420 2024-04-26T21:30:09.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.652+0000 7f43dfe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f43d810e3b0 con 0x7f43d8108680 2024-04-26T21:30:09.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.652+0000 7f43dfe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f43d810e6b0 con 0x7f43d8105d50 2024-04-26T21:30:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.653+0000 7f43dca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43d8105d50 0x7f43d8199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.653+0000 7f43dde00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43d8108680 0x7f43d81a0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.653+0000 7f43dd400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d8103420 0x7f43d8198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.653+0000 7f43dde00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43d8108680 0x7f43d81a0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37128/0 (socket says 172.21.15.1:37128) 2024-04-26T21:30:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.653+0000 7f43dca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43d8105d50 0x7f43d8199160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:35006/0 (socket says 172.21.15.1:35006) 2024-04-26T21:30:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.653+0000 7f43dde00640 1 -- 172.21.15.1:0/1293820416 learned_addr learned my addr 172.21.15.1:0/1293820416 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.653+0000 7f43dde00640 1 -- 172.21.15.1:0/1293820416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43d8105d50 msgr2=0x7f43d8199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.653+0000 7f43dde00640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43d8105d50 0x7f43d8199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.653+0000 7f43dde00640 1 -- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d8103420 msgr2=0x7f43d8198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:09.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.654+0000 7f43dde00640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d8103420 0x7f43d8198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.654+0000 7f43dde00640 1 -- 172.21.15.1:0/1293820416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d81a0830 con 0x7f43d8108680 2024-04-26T21:30:09.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.654+0000 7f43dca00640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43d8105d50 0x7f43d8199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:09.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.654+0000 7f43dd400640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d8103420 0x7f43d8198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:09.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.654+0000 7f43dde00640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43d8108680 0x7f43d81a0190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f43d000eb10 tx=0x7f43d000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:09.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.654+0000 7f43d6000640 1 -- 172.21.15.1:0/1293820416 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d0004b20 con 0x7f43d8108680 2024-04-26T21:30:09.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.654+0000 7f43dfe00640 1 -- 172.21.15.1:0/1293820416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d81a0b20 con 0x7f43d8108680 2024-04-26T21:30:09.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.654+0000 7f43d6000640 1 -- 172.21.15.1:0/1293820416 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43d0004cc0 con 0x7f43d8108680 2024-04-26T21:30:09.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.654+0000 7f43d6000640 1 -- 172.21.15.1:0/1293820416 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d0010670 con 0x7f43d8108680 2024-04-26T21:30:09.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.654+0000 7f43dfe00640 1 -- 172.21.15.1:0/1293820416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d81a1070 con 0x7f43d8108680 2024-04-26T21:30:09.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.655+0000 7f43dfe00640 1 -- 172.21.15.1:0/1293820416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d80fe900 con 0x7f43d8108680 2024-04-26T21:30:09.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.657+0000 7f43d6000640 1 -- 172.21.15.1:0/1293820416 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) ==== 138713+0+0 (secure 0 0 0) 0x7f43d0005020 con 0x7f43d8108680 2024-04-26T21:30:09.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.659+0000 7f43d6000640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43c00a3c80 0x7f43c00a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:09.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.659+0000 7f43dd400640 1 -- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43c00a3c80 msgr2=0x7f43c00a6140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:09.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.659+0000 7f43d6000640 1 -- 172.21.15.1:0/1293820416 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f43d00d1160 con 0x7f43d8108680 2024-04-26T21:30:09.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.659+0000 7f43dd400640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43c00a3c80 0x7f43c00a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:30:09.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.661+0000 7f43d6000640 1 -- 172.21.15.1:0/1293820416 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f43d009d580 con 0x7f43d8108680 2024-04-26T21:30:09.704 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:09.422+0000 7fb50e311200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:30:09.762 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:09.335+0000 7f8e651fb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:30:09.763 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:09.442+0000 7f8e651fb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:30:09.817 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:09.552+0000 7fdb82009200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:30:09.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:09.694+0000 7fdb82009200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T21:30:09.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.860+0000 7f43dd400640 1 -- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43c00a3c80 msgr2=0x7f43c00a6140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:09.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.860+0000 7f43dd400640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43c00a3c80 0x7f43c00a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:30:09.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.871+0000 7f43dfe00640 1 -- 172.21.15.1:0/1293820416 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f43d80635a0 con 0x7f43d8108680 2024-04-26T21:30:09.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.877+0000 7f43d6000640 1 -- 172.21.15.1:0/1293820416 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f43d009ce80 con 0x7f43d8108680 2024-04-26T21:30:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 -- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43c00a3c80 msgr2=0x7f43c00a6140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:30:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43c00a3c80 0x7f43c00a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 -- 172.21.15.1:0/1293820416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43d8108680 msgr2=0x7f43d81a0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43d8108680 0x7f43d81a0190 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f43d000eb10 tx=0x7f43d000c2d0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 -- 172.21.15.1:0/1293820416 shutdown_connections 2024-04-26T21:30:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f43c00a3c80 0x7f43c00a6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43d8105d50 0x7f43d8199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43d8108680 0x7f43d81a0190 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 --2- 172.21.15.1:0/1293820416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f43d8103420 0x7f43d8198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:09.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 -- 172.21.15.1:0/1293820416 >> 172.21.15.1:0/1293820416 conn(0x7f43d80fce10 msgr2=0x7f43d8103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:09.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 -- 172.21.15.1:0/1293820416 shutdown_connections 2024-04-26T21:30:09.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:09.880+0000 7f43dfe00640 1 -- 172.21.15.1:0/1293820416 wait complete. 2024-04-26T21:30:09.901 INFO:teuthology.run_tasks:Running task kclient... 2024-04-26T21:30:09.914 INFO:tasks.kclient:Mounting kernel clients... 2024-04-26T21:30:09.914 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-26T21:30:09.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:30:09.914 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-04-26T21:30:09.969 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-26T21:30:09.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:30:09.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-26T21:30:10.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:30:10.052 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-04-26T21:30:10.077 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:09.762+0000 7f8e651fb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:30:10.077 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:09.850+0000 7f8e651fb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:30:10.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T21:30:10.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-mon[32724]: mgrmap e18: x(active, since 11m), standbys: z, y 2024-04-26T21:30:10.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' 2024-04-26T21:30:10.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:09 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' 2024-04-26T21:30:10.149 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:09 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T21:30:10.149 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:09 smithi001 ceph-mon[28054]: mgrmap e18: x(active, since 11m), standbys: z, y 2024-04-26T21:30:10.149 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:09 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' 2024-04-26T21:30:10.149 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:09 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' 2024-04-26T21:30:10.149 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:10 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:10.132+0000 7fdb82009200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:30:10.152 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-26T21:30:10.152 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-26T21:30:10.152 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T21:30:10.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-26T21:30:10.204 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:09.728+0000 7fb50e311200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T21:30:10.204 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:09.812+0000 7fb50e311200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:30:10.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-26T21:30:10.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-mon[32991]: mgrmap e18: x(active, since 11m), standbys: z, y 2024-04-26T21:30:10.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' 2024-04-26T21:30:10.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:09 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' 2024-04-26T21:30:10.567 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:10 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:10.252+0000 7fdb82009200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T21:30:10.572 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:30:11.334 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:10.989+0000 7fb50e311200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:30:11.334 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:11.157+0000 7fb50e311200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:30:11.452 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:11 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:11.098+0000 7f8e651fb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:30:11.452 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:11 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:11.274+0000 7f8e651fb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:30:11.452 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:11 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:11.451+0000 7f8e651fb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:30:11.635 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:11.333+0000 7fb50e311200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:30:11.941 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:11.634+0000 7fb50e311200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:30:11.941 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:11.730+0000 7fb50e311200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:30:12.056 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:11 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:11.750+0000 7f8e651fb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:30:12.057 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:11 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:11.846+0000 7f8e651fb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:30:12.204 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:11.940+0000 7fb50e311200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:30:12.204 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:12.135+0000 7fb50e311200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:30:12.317 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:11 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:11.907+0000 7fdb82009200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T21:30:12.318 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:12.144+0000 7fdb82009200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T21:30:12.326 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:12.056+0000 7f8e651fb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:30:12.327 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:12.256+0000 7f8e651fb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:30:12.629 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:12.369+0000 7fdb82009200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T21:30:12.891 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:12.771+0000 7fdb82009200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T21:30:12.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.914+0000 7f70b7e00700 1 -- 172.21.15.1:0/2363214946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70b006b130 msgr2=0x7f70b006b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:12.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.914+0000 7f70b7e00700 1 --2- 172.21.15.1:0/2363214946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70b006b130 0x7f70b006b590 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f709c009990 tx=0x7f709c009ca0 comp rx=0 tx=0).stop 2024-04-26T21:30:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.914+0000 7f70b7e00700 1 -- 172.21.15.1:0/2363214946 shutdown_connections 2024-04-26T21:30:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.914+0000 7f70b7e00700 1 --2- 172.21.15.1:0/2363214946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70b006bad0 0x7f70b01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.914+0000 7f70b7e00700 1 --2- 172.21.15.1:0/2363214946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b01094a0 0x7f70b0109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.914+0000 7f70b7e00700 1 --2- 172.21.15.1:0/2363214946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70b006b130 0x7f70b006b590 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.914+0000 7f70b7e00700 1 -- 172.21.15.1:0/2363214946 >> 172.21.15.1:0/2363214946 conn(0x7f70b00792d0 msgr2=0x7f70b00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.915+0000 7f70b7e00700 1 -- 172.21.15.1:0/2363214946 shutdown_connections 2024-04-26T21:30:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.915+0000 7f70b7e00700 1 -- 172.21.15.1:0/2363214946 wait complete. 2024-04-26T21:30:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.916+0000 7f70b7e00700 1 Processor -- start 2024-04-26T21:30:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.916+0000 7f70b7e00700 1 -- start start 2024-04-26T21:30:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.916+0000 7f70b7e00700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70b006b130 0x7f70b019bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.916+0000 7f70b7e00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b006bad0 0x7f70b019c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.916+0000 7f70b7e00700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70b01094a0 0x7f70b01a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.916+0000 7f70b7e00700 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f70b0113600 con 0x7f70b006bad0 2024-04-26T21:30:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.916+0000 7f70b7e00700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f70b0113480 con 0x7f70b01094a0 2024-04-26T21:30:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.916+0000 7f70b7e00700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f70b0113780 con 0x7f70b006b130 2024-04-26T21:30:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70b006b130 0x7f70b019bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5400700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70b006b130 0x7f70b019bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:35020/0 (socket says 172.21.15.1:35020) 2024-04-26T21:30:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b4a00700 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b006bad0 0x7f70b019c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5400700 1 -- 172.21.15.1:0/3721218388 learned_addr learned my addr 172.21.15.1:0/3721218388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5e00700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70b01094a0 0x7f70b01a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5400700 1 -- 172.21.15.1:0/3721218388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70b01094a0 msgr2=0x7f70b01a05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5400700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70b01094a0 0x7f70b01a05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5400700 1 -- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b006bad0 msgr2=0x7f70b019c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5400700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b006bad0 0x7f70b019c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5400700 1 -- 172.21.15.1:0/3721218388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b01a0d20 con 0x7f70b006b130 2024-04-26T21:30:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5e00700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70b01094a0 0x7f70b01a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.917+0000 7f70b5400700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70b006b130 0x7f70b019bdb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f70a400e9a0 tx=0x7f70a400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.918+0000 7f70ae000700 1 -- 172.21.15.1:0/3721218388 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a400ca10 con 0x7f70b006b130 2024-04-26T21:30:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.918+0000 7f70b7e00700 1 -- 172.21.15.1:0/3721218388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b01a1010 con 0x7f70b006b130 2024-04-26T21:30:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.918+0000 7f70ae000700 1 -- 172.21.15.1:0/3721218388 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70a400cbb0 con 0x7f70b006b130 2024-04-26T21:30:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.918+0000 7f70ae000700 1 -- 172.21.15.1:0/3721218388 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4018680 con 0x7f70b006b130 2024-04-26T21:30:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.918+0000 7f70b7e00700 1 -- 172.21.15.1:0/3721218388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b01a8890 con 0x7f70b006b130 2024-04-26T21:30:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.919+0000 7f70b7e00700 1 -- 172.21.15.1:0/3721218388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70b006a110 con 0x7f70b006b130 2024-04-26T21:30:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.921+0000 7f70ae000700 1 -- 172.21.15.1:0/3721218388 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 138713+0+0 (secure 0 0 0) 0x7f70a4010430 con 0x7f70b006b130 2024-04-26T21:30:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.922+0000 7f70ae000700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70980a3d20 0x7f70980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.922+0000 7f70b4a00700 1 -- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70980a3d20 msgr2=0x7f70980a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.922+0000 7f70ae000700 1 -- 172.21.15.1:0/3721218388 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f70a40d0b30 con 0x7f70b006b130 2024-04-26T21:30:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.922+0000 7f70b4a00700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70980a3d20 0x7f70980a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:30:12.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:12.927+0000 7f70ae000700 1 -- 172.21.15.1:0/3721218388 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f70a409cf50 con 0x7f70b006b130 2024-04-26T21:30:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.123+0000 7f70b4a00700 1 -- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70980a3d20 msgr2=0x7f70980a61e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:13.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.123+0000 7f70b4a00700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70980a3d20 0x7f70980a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:30:13.158 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:12.740+0000 7fb50e311200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:30:13.170 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:12.902+0000 7fdb82009200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T21:30:13.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.203+0000 7f70b7e00700 1 -- 172.21.15.1:0/3721218388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f70b01a8ac0 con 0x7f70b006b130 2024-04-26T21:30:13.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.204+0000 7f70ae000700 1 -- 172.21.15.1:0/3721218388 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+88 (secure 0 0 0) 0x7f70a409c850 con 0x7f70b006b130 2024-04-26T21:30:13.205 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-26T21:30:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 -- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70980a3d20 msgr2=0x7f70980a61e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:30:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70980a3d20 0x7f70980a61e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 -- 172.21.15.1:0/3721218388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70b006b130 msgr2=0x7f70b019bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70b006b130 0x7f70b019bdb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f70a400e9a0 tx=0x7f70a400ecb0 comp rx=0 tx=0).stop 2024-04-26T21:30:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 -- 172.21.15.1:0/3721218388 shutdown_connections 2024-04-26T21:30:13.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f70980a3d20 0x7f70980a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:13.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70b006b130 0x7f70b019bdb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:13.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f70b01094a0 0x7f70b01a05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:13.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 --2- 172.21.15.1:0/3721218388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70b006bad0 0x7f70b019c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:13.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 -- 172.21.15.1:0/3721218388 >> 172.21.15.1:0/3721218388 conn(0x7f70b00792d0 msgr2=0x7f70b0100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:13.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 -- 172.21.15.1:0/3721218388 shutdown_connections 2024-04-26T21:30:13.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:13.207+0000 7f70b7e00700 1 -- 172.21.15.1:0/3721218388 wait complete. 2024-04-26T21:30:13.261 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:12.883+0000 7f8e651fb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:30:13.450 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:13 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3721218388' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T21:30:13.450 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:13.169+0000 7fdb82009200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T21:30:13.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:13 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3721218388' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T21:30:13.454 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:13.157+0000 7fb50e311200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:30:13.454 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:13.254+0000 7fb50e311200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:30:13.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:13 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3721218388' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-26T21:30:13.577 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:13 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:13.321+0000 7f8e651fb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:30:13.577 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:13 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:13.421+0000 7f8e651fb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:30:13.817 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:13.449+0000 7fdb82009200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T21:30:13.954 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:13.454+0000 7fb50e311200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:30:14.052 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-26T21:30:14.052 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-26T21:30:14.052 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-04-26T21:30:14.052 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-26T21:30:14.052 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-26T21:30:14.052 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-26T21:30:14.052 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-26T21:30:14.052 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-26T21:30:14.052 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-26T21:30:14.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:30:14.053 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-04-26T21:30:14.077 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:13 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:13.629+0000 7f8e651fb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:30:14.108 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-26T21:30:14.108 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-26T21:30:14.108 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-04-26T21:30:14.108 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-04-26T21:30:14.108 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-04-26T21:30:14.108 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-04-26T21:30:14.108 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T21:30:14.108 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 41898sec preferred_lft 41898sec 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-04-26T21:30:14.109 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-04-26T21:30:14.110 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-26T21:30:14.110 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:30:14.110 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T21:30:14.110 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add name ceph-brx type bridge 2024-04-26T21:30:14.110 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr flush dev ceph-brx 2024-04-26T21:30:14.110 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set ceph-brx up 2024-04-26T21:30:14.110 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-26T21:30:14.110 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T21:30:14.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:30:14.304 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:13.974+0000 7fb50e311200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:30:14.305 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:14.064+0000 7fb50e311200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:30:14.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:30:14.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:30:14.362 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-26T21:30:14.424 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-04-26T21:30:14.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:30:14.428 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-04-26T21:30:14.474 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-04-26T21:30:14.474 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-26T21:30:14.474 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-26T21:30:14.474 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-26T21:30:14.474 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-26T21:30:14.475 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:30:14.475 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T21:30:14.475 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-26T21:30:14.475 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-26T21:30:14.475 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-26T21:30:14.475 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T21:30:14.523 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:14 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:14.175+0000 7f8e651fb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:30:14.523 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:14 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:14.270+0000 7f8e651fb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:30:14.570 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:14.309+0000 7fdb82009200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T21:30:14.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:30:14.643 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:14.304+0000 7fb50e311200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:30:14.643 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:14.436+0000 7fb50e311200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:30:14.643 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:14.529+0000 7fb50e311200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:30:14.643 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:14.642+0000 7fb50e311200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:30:14.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:30:14.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:30:14.674 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-04-26T21:30:14.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:30:14.716 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-04-26T21:30:14.805 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:30:14.806 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T21:30:14.806 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T21:30:14.806 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-26T21:30:14.806 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T21:30:14.827 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:14 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:14.522+0000 7f8e651fb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:30:14.827 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:14 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:14.660+0000 7f8e651fb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:30:14.827 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:14 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:14.758+0000 7f8e651fb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:30:14.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:30:14.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:30:14.982 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-26T21:30:14.982 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:30:14.982 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T21:30:14.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-26T21:30:14.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-26T21:30:14.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-26T21:30:14.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-26T21:30:14.983 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-26T21:30:14.983 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T21:30:15.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:30:15.170 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:14.919+0000 7fdb82009200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T21:30:15.170 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:15 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:15.063+0000 7fdb82009200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T21:30:15.192 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:14 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:14.879+0000 7f8e651fb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:30:15.204 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:14.946+0000 7fb50e311200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:30:15.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:30:15.244 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-26T21:30:15.244 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-04-26T21:30:15.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-04-26T21:30:15.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.0 master ceph-brx 2024-04-26T21:30:15.244 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-04-26T21:30:15.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:30:15.424 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:15 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:15.355+0000 7fdb82009200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T21:30:15.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:30:15.437 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T21:30:15.437 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-26T21:30:15.479 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-26T21:30:15.479 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-26T21:30:15.479 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-26T21:30:15.570 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-26T21:30:15.571 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-26T21:30:15.571 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/298b6b07-2bc1-45c5-bc65-b7e2df0eafa2 2024-04-26T21:30:15.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:30:15.571 DEBUG:teuthology.orchestra.run.smithi001:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/298b6b07-2bc1-45c5-bc65-b7e2df0eafa2 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-26T21:30:15.576 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:15 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:15.192+0000 7f8e651fb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:30:15.694 INFO:teuthology.orchestra.run.smithi001.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-26T21:30:15.694 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-04-26T21:30:15.694 INFO:teuthology.orchestra.run.smithi001.stdout:invalid new device string format 2024-04-26T21:30:15.841 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:15 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:15.543+0000 7fb50e311200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:30:15.842 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:30:15 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z[34859]: 2024-04-26T21:30:15.761+0000 7fb50e311200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:30:16.077 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:15 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:15.820+0000 7f8e651fb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:30:16.077 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:30:16 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y[34116]: 2024-04-26T21:30:16.046+0000 7f8e651fb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:30:16.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:15 smithi038 ceph-mon[32724]: Standby manager daemon z restarted 2024-04-26T21:30:16.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:15 smithi038 ceph-mon[32724]: Standby manager daemon z started 2024-04-26T21:30:16.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:15 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T21:30:16.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:15 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:30:16.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:15 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T21:30:16.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:15 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:30:16.126 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:16.056+0000 7fdb82009200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T21:30:16.127 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:15 smithi001 ceph-mon[28054]: Standby manager daemon z restarted 2024-04-26T21:30:16.127 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:15 smithi001 ceph-mon[28054]: Standby manager daemon z started 2024-04-26T21:30:16.127 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:15 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T21:30:16.127 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:15 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:30:16.127 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:15 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T21:30:16.127 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:15 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:30:16.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:15 smithi186 ceph-mon[32991]: Standby manager daemon z restarted 2024-04-26T21:30:16.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:15 smithi186 ceph-mon[32991]: Standby manager daemon z started 2024-04-26T21:30:16.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:15 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-26T21:30:16.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:15 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:30:16.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:15 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-26T21:30:16.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:15 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.186:0/1872006397' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:30:16.411 INFO:teuthology.orchestra.run.smithi001.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-04-26T21:30:16.411 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-04-26T21:30:16.411 INFO:teuthology.orchestra.run.smithi001.stdout:invalid new device string format 2024-04-26T21:30:16.411 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: resolved to: "172.21.15.1:3300,172.21.15.38:3300,172.21.15.186:3300" 2024-04-26T21:30:16.411 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.1:3300,172.21.15.38:3300,172.21.15.186:3300:/volumes/qa/sv_1/298b6b07-2bc1-45c5-bc65-b7e2df0eafa2 2024-04-26T21:30:16.411 INFO:teuthology.orchestra.run.smithi001.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=44ef3d6e-0412-11ef-bc93-c7b262605968" will pass to kernel 2024-04-26T21:30:16.414 INFO:teuthology.orchestra.run.smithi001.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-26T21:30:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: You just mounted a file system that supports labels which does not 2024-04-26T21:30:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-26T21:30:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: applications will generate AVC messages and not be allowed access to 2024-04-26T21:30:16.414 INFO:teuthology.orchestra.run.smithi001.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-26T21:30:16.417 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-26T21:30:16.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:30:16.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-26T21:30:16.472 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:16.193+0000 7fdb82009200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T21:30:16.483 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T21:30:16.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T21:30:16.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:30:16.594 INFO:teuthology.orchestra.run.smithi001.stdout:client25448 2024-04-26T21:30:16.595 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:30:16.595 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T21:30:16.648 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T21:30:16.727 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:16.534+0000 7fdb82009200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T21:30:16.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T21:30:16.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:30:16.764 INFO:teuthology.orchestra.run.smithi001.stdout:client25448 2024-04-26T21:30:16.764 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-26T21:30:16.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-26T21:30:16.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:30:16.885 INFO:teuthology.orchestra.run.smithi001.stdout:client25448 2024-04-26T21:30:16.885 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968.client25448/status') 2024-04-26T21:30:16.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-26T21:30:16.996 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:16.726+0000 7fdb82009200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T21:30:16.996 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:16.865+0000 7fdb82009200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T21:30:16.996 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-mon[28054]: mgrmap e19: x(active, since 11m), standbys: y, z 2024-04-26T21:30:16.996 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-mon[28054]: Standby manager daemon y restarted 2024-04-26T21:30:16.996 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-mon[28054]: Standby manager daemon y started 2024-04-26T21:30:16.996 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T21:30:16.996 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:30:16.996 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T21:30:16.997 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:16 smithi001 ceph-mon[28054]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:30:17.003 INFO:teuthology.orchestra.run.smithi001.stdout:instance: client.25448 (3)172.21.15.1:0/778470472 2024-04-26T21:30:17.003 INFO:teuthology.orchestra.run.smithi001.stdout:blocklisted: false 2024-04-26T21:30:17.003 INFO:teuthology.orchestra.run.smithi001.stderr:0+1 records in 2024-04-26T21:30:17.003 INFO:teuthology.orchestra.run.smithi001.stderr:0+1 records out 2024-04-26T21:30:17.003 INFO:teuthology.orchestra.run.smithi001.stderr:69 bytes copied, 8.0285e-05 s, 859 kB/s 2024-04-26T21:30:17.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-26T21:30:17.015 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T21:30:17.027 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T21:30:17.027 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:30:17.028 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-26T21:30:17.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:16 smithi186 ceph-mon[32991]: mgrmap e19: x(active, since 11m), standbys: y, z 2024-04-26T21:30:17.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:16 smithi186 ceph-mon[32991]: Standby manager daemon y restarted 2024-04-26T21:30:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:16 smithi186 ceph-mon[32991]: Standby manager daemon y started 2024-04-26T21:30:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:16 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T21:30:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:16 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:30:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:16 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T21:30:17.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:16 smithi186 ceph-mon[32991]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:30:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.223+0000 7f72a0a00640 1 -- 172.21.15.1:0/1165139748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f729c101250 msgr2=0x7f729c103680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.223+0000 7f72a0a00640 1 --2- 172.21.15.1:0/1165139748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f729c101250 0x7f729c103680 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f728c009900 tx=0x7f728c02a410 comp rx=0 tx=0).stop 2024-04-26T21:30:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.224+0000 7f72a0a00640 1 -- 172.21.15.1:0/1165139748 shutdown_connections 2024-04-26T21:30:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.224+0000 7f72a0a00640 1 --2- 172.21.15.1:0/1165139748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f729c103bc0 0x7f729c106030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.224+0000 7f72a0a00640 1 --2- 172.21.15.1:0/1165139748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f729c106570 0x7f729c1069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.224+0000 7f72a0a00640 1 --2- 172.21.15.1:0/1165139748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f729c101250 0x7f729c103680 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.224+0000 7f72a0a00640 1 -- 172.21.15.1:0/1165139748 >> 172.21.15.1:0/1165139748 conn(0x7f729c0fce30 msgr2=0x7f729c0ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.224+0000 7f72a0a00640 1 -- 172.21.15.1:0/1165139748 shutdown_connections 2024-04-26T21:30:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.224+0000 7f72a0a00640 1 -- 172.21.15.1:0/1165139748 wait complete. 2024-04-26T21:30:17.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.225+0000 7f72a0a00640 1 Processor -- start 2024-04-26T21:30:17.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.225+0000 7f72a0a00640 1 -- start start 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f72a0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f729c101250 0x7f729c19f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f72a0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f729c103bc0 0x7f729c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f72a0a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f729c106570 0x7f729c1a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f72a0a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f729c114b80 con 0x7f729c106570 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f72a0a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f729c114a00 con 0x7f729c101250 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f72a0a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f729c114d00 con 0x7f729c103bc0 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f7299e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f729c101250 0x7f729c19f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f7299e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f729c101250 0x7f729c19f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37156/0 (socket says 172.21.15.1:37156) 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f7293e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f729c103bc0 0x7f729c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f729a800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f729c106570 0x7f729c1a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:17.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.226+0000 7f7299e00640 1 -- 172.21.15.1:0/763859910 learned_addr learned my addr 172.21.15.1:0/763859910 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:17.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f729a800640 1 -- 172.21.15.1:0/763859910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f729c103bc0 msgr2=0x7f729c19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:17.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f729a800640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f729c103bc0 0x7f729c19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:17.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f729a800640 1 -- 172.21.15.1:0/763859910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f729c101250 msgr2=0x7f729c19f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:17.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f729a800640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f729c101250 0x7f729c19f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:17.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f729a800640 1 -- 172.21.15.1:0/763859910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729c1a6e00 con 0x7f729c106570 2024-04-26T21:30:17.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f7299e00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f729c101250 0x7f729c19f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:17.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f729a800640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f729c106570 0x7f729c1a6790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7294009fd0 tx=0x7f729400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:17.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f7292a00640 1 -- 172.21.15.1:0/763859910 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729400ce20 con 0x7f729c106570 2024-04-26T21:30:17.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f7292a00640 1 -- 172.21.15.1:0/763859910 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7294005020 con 0x7f729c106570 2024-04-26T21:30:17.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f72a0a00640 1 -- 172.21.15.1:0/763859910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729c1a70f0 con 0x7f729c106570 2024-04-26T21:30:17.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.227+0000 7f7292a00640 1 -- 172.21.15.1:0/763859910 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7294010690 con 0x7f729c106570 2024-04-26T21:30:17.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.228+0000 7f72a0a00640 1 -- 172.21.15.1:0/763859910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729c1a75f0 con 0x7f729c106570 2024-04-26T21:30:17.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.229+0000 7f72a0a00640 1 -- 172.21.15.1:0/763859910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f729c0fe920 con 0x7f729c106570 2024-04-26T21:30:17.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.232+0000 7f7292a00640 1 -- 172.21.15.1:0/763859910 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 20) ==== 138713+0+0 (secure 0 0 0) 0x7f72940054d0 con 0x7f729c106570 2024-04-26T21:30:17.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.234+0000 7f7292a00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f72800a3c80 0x7f72800a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:17.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.234+0000 7f7299e00640 1 -- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f72800a3c80 msgr2=0x7f72800a6140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:17.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.234+0000 7f7292a00640 1 -- 172.21.15.1:0/763859910 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f72940d1110 con 0x7f729c106570 2024-04-26T21:30:17.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.234+0000 7f7299e00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f72800a3c80 0x7f72800a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:30:17.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.238+0000 7f7292a00640 1 -- 172.21.15.1:0/763859910 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f729409d530 con 0x7f729c106570 2024-04-26T21:30:17.318 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:17.031+0000 7fdb82009200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T21:30:17.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:16 smithi038 ceph-mon[32724]: mgrmap e19: x(active, since 11m), standbys: y, z 2024-04-26T21:30:17.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:16 smithi038 ceph-mon[32724]: Standby manager daemon y restarted 2024-04-26T21:30:17.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:16 smithi038 ceph-mon[32724]: Standby manager daemon y started 2024-04-26T21:30:17.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:16 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-26T21:30:17.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:16 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T21:30:17.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:16 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-26T21:30:17.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:16 smithi038 ceph-mon[32724]: from='mgr.? 172.21.15.38:0/3085350506' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T21:30:17.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.435+0000 7f7299e00640 1 -- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f72800a3c80 msgr2=0x7f72800a6140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:17.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.435+0000 7f7299e00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f72800a3c80 0x7f72800a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:30:17.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.486+0000 7f72a0a00640 1 -- 172.21.15.1:0/763859910 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f729c063530 con 0x7f729c106570 2024-04-26T21:30:17.817 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:17.423+0000 7fdb82009200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T21:30:17.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.836+0000 7f7299e00640 1 -- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f72800a3c80 msgr2=0x7f72800a6140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:17.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.836+0000 7f7299e00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f72800a3c80 0x7f72800a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T21:30:17.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.861+0000 7f7292a00640 1 -- 172.21.15.1:0/763859910 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v24) ==== 117+0+0 (secure 0 0 0) 0x7f729409ce30 con 0x7f729c106570 2024-04-26T21:30:17.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.865+0000 7f72a0a00640 1 -- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f72800a3c80 msgr2=0x7f72800a6140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:30:17.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.865+0000 7f72a0a00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f72800a3c80 0x7f72800a6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:17.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.865+0000 7f72a0a00640 1 -- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f729c106570 msgr2=0x7f729c1a6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:17.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.865+0000 7f72a0a00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f729c106570 0x7f729c1a6790 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7294009fd0 tx=0x7f729400ef10 comp rx=0 tx=0).stop 2024-04-26T21:30:17.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.866+0000 7f72a0a00640 1 -- 172.21.15.1:0/763859910 shutdown_connections 2024-04-26T21:30:17.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.866+0000 7f72a0a00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f72800a3c80 0x7f72800a6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:17.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.866+0000 7f72a0a00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f729c103bc0 0x7f729c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:17.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.866+0000 7f72a0a00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f729c106570 0x7f729c1a6790 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:17.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.866+0000 7f72a0a00640 1 --2- 172.21.15.1:0/763859910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f729c101250 0x7f729c19f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:17.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.866+0000 7f72a0a00640 1 -- 172.21.15.1:0/763859910 >> 172.21.15.1:0/763859910 conn(0x7f729c0fce30 msgr2=0x7f729c1044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:17.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.866+0000 7f72a0a00640 1 -- 172.21.15.1:0/763859910 shutdown_connections 2024-04-26T21:30:17.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:17.866+0000 7f72a0a00640 1 -- 172.21.15.1:0/763859910 wait complete. 2024-04-26T21:30:17.885 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-26T21:30:17.896 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T21:30:17.958 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T21:30:17.958 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:30:17.959 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-26T21:30:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.157+0000 7f7add400640 1 -- 172.21.15.1:0/1350124742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ad81027f0 msgr2=0x7f7ad8102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.157+0000 7f7add400640 1 --2- 172.21.15.1:0/1350124742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ad81027f0 0x7f7ad8102c10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8009900 tx=0x7f7ac802a410 comp rx=0 tx=0).stop 2024-04-26T21:30:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.157+0000 7f7add400640 1 -- 172.21.15.1:0/1350124742 shutdown_connections 2024-04-26T21:30:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.157+0000 7f7add400640 1 --2- 172.21.15.1:0/1350124742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad81043a0 0x7f7ad810ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.157+0000 7f7add400640 1 --2- 172.21.15.1:0/1350124742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ad81039e0 0x7f7ad8103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.157+0000 7f7add400640 1 --2- 172.21.15.1:0/1350124742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ad81027f0 0x7f7ad8102c10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:18.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.157+0000 7f7add400640 1 -- 172.21.15.1:0/1350124742 >> 172.21.15.1:0/1350124742 conn(0x7f7ad80fdd60 msgr2=0x7f7ad81001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:18.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.157+0000 7f7add400640 1 -- 172.21.15.1:0/1350124742 shutdown_connections 2024-04-26T21:30:18.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.157+0000 7f7add400640 1 -- 172.21.15.1:0/1350124742 wait complete. 2024-04-26T21:30:18.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.158+0000 7f7add400640 1 Processor -- start 2024-04-26T21:30:18.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.158+0000 7f7add400640 1 -- start start 2024-04-26T21:30:18.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.158+0000 7f7add400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad81027f0 0x7f7ad819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:18.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.158+0000 7f7add400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ad81039e0 0x7f7ad819b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:18.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.158+0000 7f7add400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ad81043a0 0x7f7ad81a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:18.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.158+0000 7f7add400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad810e370 con 0x7f7ad81043a0 2024-04-26T21:30:18.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.158+0000 7f7add400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad810e1f0 con 0x7f7ad81039e0 2024-04-26T21:30:18.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.158+0000 7f7add400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad810e4f0 con 0x7f7ad81027f0 2024-04-26T21:30:18.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7acfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ad81039e0 0x7f7ad819b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:18.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7acfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ad81039e0 0x7f7ad819b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:55954/0 (socket says 172.21.15.1:55954) 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7acfe00640 1 -- 172.21.15.1:0/223169936 learned_addr learned my addr 172.21.15.1:0/223169936 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7adca00640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ad81043a0 0x7f7ad81a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7ad7e00640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad81027f0 0x7f7ad819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7acfe00640 1 -- 172.21.15.1:0/223169936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad81027f0 msgr2=0x7f7ad819ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7acfe00640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ad81027f0 0x7f7ad819ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7acfe00640 1 -- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ad81043a0 msgr2=0x7f7ad81a21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7acfe00640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ad81043a0 0x7f7ad81a21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7acfe00640 1 -- 172.21.15.1:0/223169936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad81a2860 con 0x7f7ad81039e0 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.159+0000 7f7acfe00640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ad81039e0 0x7f7ad819b190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e9f0 tx=0x7f7ac000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.160+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac000cd80 con 0x7f7ad81039e0 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.160+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ac00108e0 con 0x7f7ad81039e0 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.160+0000 7f7add400640 1 -- 172.21.15.1:0/223169936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad81a2b50 con 0x7f7ad81039e0 2024-04-26T21:30:18.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.160+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0018980 con 0x7f7ad81039e0 2024-04-26T21:30:18.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.160+0000 7f7add400640 1 -- 172.21.15.1:0/223169936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad81a30f0 con 0x7f7ad81039e0 2024-04-26T21:30:18.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.161+0000 7f7add400640 1 -- 172.21.15.1:0/223169936 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ad81a32c0 con 0x7f7ad81039e0 2024-04-26T21:30:18.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.163+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 20) ==== 138713+0+0 (secure 0 0 0) 0x7f7ac0010430 con 0x7f7ad81039e0 2024-04-26T21:30:18.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.164+0000 7f7ad5600640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7abc0a3cf0 0x7f7abc0a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:18.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.165+0000 7f7ad7e00640 1 -- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7abc0a3cf0 msgr2=0x7f7abc0a61b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:18.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.165+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9253+0+0 (secure 0 0 0) 0x7f7ac00d0e50 con 0x7f7ad81039e0 2024-04-26T21:30:18.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.165+0000 7f7ad7e00640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7abc0a3cf0 0x7f7abc0a61b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T21:30:18.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.168+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7ac009d270 con 0x7f7ad81039e0 2024-04-26T21:30:18.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:17 smithi186 ceph-mon[32991]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-04-26T21:30:18.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:17 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/763859910' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T21:30:18.228 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:17 smithi038 ceph-mon[32724]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-04-26T21:30:18.228 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:17 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/763859910' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T21:30:18.251 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:17 smithi001 ceph-mon[28054]: mgrmap e20: x(active, since 11m), standbys: z, y 2024-04-26T21:30:18.251 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:17 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/763859910' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-26T21:30:18.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.366+0000 7f7ad7e00640 1 -- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7abc0a3cf0 msgr2=0x7f7abc0a61b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/2786715792 2024-04-26T21:30:18.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.366+0000 7f7ad7e00640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7abc0a3cf0 0x7f7abc0a61b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T21:30:18.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.420+0000 7f7add400640 1 -- 172.21.15.1:0/223169936 --> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f7ad81a3490 con 0x7f7abc0a3cf0 2024-04-26T21:30:18.552 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.250+0000 7fdb82009200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T21:30:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.576+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 21) ==== 138331+0+0 (secure 0 0 0) 0x7f7ac0099150 con 0x7f7ad81039e0 2024-04-26T21:30:18.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.578+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7abc0a3cf0 msgr2=0x7f7abc0a61b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:30:18.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:18.578+0000 7f7ad5600640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/2786715792,v1:172.21.15.1:6801/2786715792] conn(0x7f7abc0a3cf0 0x7f7abc0a61b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:18.810 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.550+0000 7fdb82009200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T21:30:19.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.908+0000 7fdad3400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:30:19.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.909+0000 7fdad3400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:30:19.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.909+0000 7fdad3400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:30:19.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.909+0000 7fdad3400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:30:19.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.909+0000 7fdad3400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:30:19.069 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.909+0000 7fdad5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:30:19.069 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.909+0000 7fdad5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:30:19.069 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.909+0000 7fdad5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:30:19.069 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.909+0000 7fdad5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:30:19.069 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:30:18.909+0000 7fdad5c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:30:19.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/763859910' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: Active manager daemon x restarted 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: Activating manager daemon x 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: mgrmap e21: x(active, starting, since 0.0185766s), standbys: z, y 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T21:30:19.070 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:30:19.071 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: Manager daemon x is now available 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T21:30:19.072 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T21:30:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/763859910' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T21:30:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:30:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: Active manager daemon x restarted 2024-04-26T21:30:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: Activating manager daemon x 2024-04-26T21:30:19.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: mgrmap e21: x(active, starting, since 0.0185766s), standbys: z, y 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:30:19.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:30:19.079 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: Manager daemon x is now available 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T21:30:19.080 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T21:30:19.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/763859910' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-26T21:30:19.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:30:19.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: Active manager daemon x restarted 2024-04-26T21:30:19.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: Activating manager daemon x 2024-04-26T21:30:19.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: osdmap e96: 12 total, 12 up, 12 in 2024-04-26T21:30:19.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: mgrmap e21: x(active, starting, since 0.0185766s), standbys: z, y 2024-04-26T21:30:19.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T21:30:19.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T21:30:19.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T21:30:19.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-26T21:30:19.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T21:30:19.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T21:30:19.208 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: Manager daemon x is now available 2024-04-26T21:30:19.208 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:19.208 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-26T21:30:19.208 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-26T21:30:19.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.589+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 <== mon.2 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f7ac00ce5f0 con 0x7f7ad81039e0 2024-04-26T21:30:19.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.590+0000 7f7ad5600640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7abc0528a0 0x7f7abc052c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:19.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.590+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 --> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f7ac001ae50 con 0x7f7abc0528a0 2024-04-26T21:30:19.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.595+0000 7f7ad7e00640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7abc0528a0 0x7f7abc052c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:19.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.600+0000 7f7ad7e00640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7abc0528a0 0x7f7abc052c80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7ad819c120 tx=0x7f7ac8000bc0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:19.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.690+0000 7f7ad5600640 1 -- 172.21.15.1:0/223169936 <== mgr.15291 v2:172.21.15.1:6800/3236768718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7ac001ae50 con 0x7f7abc0528a0 2024-04-26T21:30:19.691 INFO:teuthology.orchestra.run.smithi001.stdout:Schedule set for path / 2024-04-26T21:30:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.693+0000 7f7add400640 1 -- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7abc0528a0 msgr2=0x7f7abc052c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:19.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.693+0000 7f7add400640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7abc0528a0 0x7f7abc052c80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7ad819c120 tx=0x7f7ac8000bc0 comp rx=0 tx=0).stop 2024-04-26T21:30:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.693+0000 7f7add400640 1 -- 172.21.15.1:0/223169936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ad81039e0 msgr2=0x7f7ad819b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.693+0000 7f7add400640 1 --2- 172.21.15.1:0/223169936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ad81039e0 0x7f7ad819b190 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000e9f0 tx=0x7f7ac000eeb0 comp rx=0 tx=0).stop 2024-04-26T21:30:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.693+0000 7f7adca00640 1 -- 172.21.15.1:0/223169936 reap_dead start 2024-04-26T21:30:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.693+0000 7f7add400640 1 -- 172.21.15.1:0/223169936 shutdown_connections 2024-04-26T21:30:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.693+0000 7f7add400640 1 -- 172.21.15.1:0/223169936 >> 172.21.15.1:0/223169936 conn(0x7f7ad80fdd60 msgr2=0x7f7ad8074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.693+0000 7f7add400640 1 -- 172.21.15.1:0/223169936 shutdown_connections 2024-04-26T21:30:19.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.693+0000 7f7add400640 1 -- 172.21.15.1:0/223169936 wait complete. 2024-04-26T21:30:19.711 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-26T21:30:19.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.898+0000 7f9939600640 1 -- 172.21.15.1:0/219073135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9934103b90 msgr2=0x7f9934104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:19.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.898+0000 7f9939600640 1 --2- 172.21.15.1:0/219073135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9934103b90 0x7f9934104010 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9920009900 tx=0x7f992002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:30:19.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.898+0000 7f9939600640 1 -- 172.21.15.1:0/219073135 shutdown_connections 2024-04-26T21:30:19.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.898+0000 7f9939600640 1 --2- 172.21.15.1:0/219073135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9934104550 0x7f993410ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:19.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.898+0000 7f9939600640 1 --2- 172.21.15.1:0/219073135 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9934103b90 0x7f9934104010 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:19.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.898+0000 7f9939600640 1 --2- 172.21.15.1:0/219073135 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99341029a0 0x7f9934102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:19.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.898+0000 7f9939600640 1 -- 172.21.15.1:0/219073135 >> 172.21.15.1:0/219073135 conn(0x7f99340fdf10 msgr2=0x7f9934100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:19.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.898+0000 7f9939600640 1 -- 172.21.15.1:0/219073135 shutdown_connections 2024-04-26T21:30:19.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.898+0000 7f9939600640 1 -- 172.21.15.1:0/219073135 wait complete. 2024-04-26T21:30:19.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.900+0000 7f9939600640 1 Processor -- start 2024-04-26T21:30:19.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.900+0000 7f9939600640 1 -- start start 2024-04-26T21:30:19.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.900+0000 7f9939600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99341029a0 0x7f993419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:19.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.900+0000 7f9939600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9934103b90 0x7f993419b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:19.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.900+0000 7f9939600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9934104550 0x7f99341a2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:19.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.900+0000 7f9939600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f993410e520 con 0x7f9934104550 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.900+0000 7f9939600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f993410e3a0 con 0x7f9934103b90 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.900+0000 7f9939600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f993410e6a0 con 0x7f99341029a0 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9932000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9934103b90 0x7f993419b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9932000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9934103b90 0x7f993419b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:56114/0 (socket says 172.21.15.1:56114) 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9932a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99341029a0 0x7f993419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9932000640 1 -- 172.21.15.1:0/477426596 learned_addr learned my addr 172.21.15.1:0/477426596 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9933400640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9934104550 0x7f99341a2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9932000640 1 -- 172.21.15.1:0/477426596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99341029a0 msgr2=0x7f993419add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9932000640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99341029a0 0x7f993419add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9932000640 1 -- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9934104550 msgr2=0x7f99341a2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9932000640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9934104550 0x7f99341a2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9932000640 1 -- 172.21.15.1:0/477426596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99341a29f0 con 0x7f9934103b90 2024-04-26T21:30:19.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9933400640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9934104550 0x7f99341a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:30:19.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.901+0000 7f9932000640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9934103b90 0x7f993419b310 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9920005ab0 tx=0x7f9920003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:19.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.902+0000 7f992b400640 1 -- 172.21.15.1:0/477426596 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992003f070 con 0x7f9934103b90 2024-04-26T21:30:19.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.902+0000 7f992b400640 1 -- 172.21.15.1:0/477426596 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9920031c50 con 0x7f9934103b90 2024-04-26T21:30:19.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.902+0000 7f992b400640 1 -- 172.21.15.1:0/477426596 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920035030 con 0x7f9934103b90 2024-04-26T21:30:19.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.902+0000 7f9939600640 1 -- 172.21.15.1:0/477426596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99341a2c80 con 0x7f9934103b90 2024-04-26T21:30:19.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.902+0000 7f9939600640 1 -- 172.21.15.1:0/477426596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99341a31c0 con 0x7f9934103b90 2024-04-26T21:30:19.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.903+0000 7f9939600640 1 -- 172.21.15.1:0/477426596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99341a3390 con 0x7f9934103b90 2024-04-26T21:30:19.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.906+0000 7f992b400640 1 -- 172.21.15.1:0/477426596 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7f9920042070 con 0x7f9934103b90 2024-04-26T21:30:19.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.909+0000 7f992b400640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f99180a3a10 0x7f99180a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:19.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.909+0000 7f992b400640 1 -- 172.21.15.1:0/477426596 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f99200f11d0 con 0x7f9934103b90 2024-04-26T21:30:19.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.909+0000 7f9932a00640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f99180a3a10 0x7f99180a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:19.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.910+0000 7f9932a00640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f99180a3a10 0x7f99180a5ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f99341039f0 tx=0x7f9924005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:19.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:19.913+0000 7f992b400640 1 -- 172.21.15.1:0/477426596 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f99200bd930 con 0x7f9934103b90 2024-04-26T21:30:20.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.143+0000 7f9939600640 1 -- 172.21.15.1:0/477426596 --> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7f9934076280 con 0x7f99180a3a10 2024-04-26T21:30:20.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.222+0000 7f992b400640 1 -- 172.21.15.1:0/477426596 <== mgr.15291 v2:172.21.15.1:6800/3236768718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7f9934076280 con 0x7f99180a3a10 2024-04-26T21:30:20.223 INFO:teuthology.orchestra.run.smithi001.stdout:Retention added to path / 2024-04-26T21:30:20.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.224+0000 7f9939600640 1 -- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f99180a3a10 msgr2=0x7f99180a5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:20.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.224+0000 7f9939600640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f99180a3a10 0x7f99180a5ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f99341039f0 tx=0x7f9924005f90 comp rx=0 tx=0).stop 2024-04-26T21:30:20.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.225+0000 7f9939600640 1 -- 172.21.15.1:0/477426596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9934103b90 msgr2=0x7f993419b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:20.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.225+0000 7f9939600640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9934103b90 0x7f993419b310 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9920005ab0 tx=0x7f9920003b90 comp rx=0 tx=0).stop 2024-04-26T21:30:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.225+0000 7f9939600640 1 -- 172.21.15.1:0/477426596 shutdown_connections 2024-04-26T21:30:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.225+0000 7f9939600640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f99180a3a10 0x7f99180a5ed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.225+0000 7f9939600640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9934104550 0x7f99341a2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.225+0000 7f9939600640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9934103b90 0x7f993419b310 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.225+0000 7f9939600640 1 --2- 172.21.15.1:0/477426596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99341029a0 0x7f993419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.225+0000 7f9939600640 1 -- 172.21.15.1:0/477426596 >> 172.21.15.1:0/477426596 conn(0x7f99340fdf10 msgr2=0x7f9934074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.225+0000 7f9939600640 1 -- 172.21.15.1:0/477426596 shutdown_connections 2024-04-26T21:30:20.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.225+0000 7f9939600640 1 -- 172.21.15.1:0/477426596 wait complete. 2024-04-26T21:30:20.241 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T21:30:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.422+0000 7fb242000640 1 -- 172.21.15.1:0/1904813981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c103420 msgr2=0x7fb23c105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.422+0000 7fb242000640 1 --2- 172.21.15.1:0/1904813981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c103420 0x7fb23c105810 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009900 tx=0x7fb22c02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.423+0000 7fb242000640 1 -- 172.21.15.1:0/1904813981 shutdown_connections 2024-04-26T21:30:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.423+0000 7fb242000640 1 --2- 172.21.15.1:0/1904813981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb23c108680 0x7fb23c10aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.423+0000 7fb242000640 1 --2- 172.21.15.1:0/1904813981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c105d50 0x7fb23c108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.423+0000 7fb242000640 1 --2- 172.21.15.1:0/1904813981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c103420 0x7fb23c105810 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.423+0000 7fb242000640 1 -- 172.21.15.1:0/1904813981 >> 172.21.15.1:0/1904813981 conn(0x7fb23c0fce10 msgr2=0x7fb23c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.423+0000 7fb242000640 1 -- 172.21.15.1:0/1904813981 shutdown_connections 2024-04-26T21:30:20.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.423+0000 7fb242000640 1 -- 172.21.15.1:0/1904813981 wait complete. 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.424+0000 7fb242000640 1 Processor -- start 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.424+0000 7fb242000640 1 -- start start 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb242000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb23c103420 0x7fb23c102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb242000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c105d50 0x7fb23c1013d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb242000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c108680 0x7fb23c101910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb242000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c10e540 con 0x7fb23c108680 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb242000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c10e3c0 con 0x7fb23c105d50 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb242000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb23c10e6c0 con 0x7fb23c103420 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb23b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb23c103420 0x7fb23c102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb23b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb23c103420 0x7fb23c102d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51706/0 (socket says 172.21.15.1:51706) 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb23b400640 1 -- 172.21.15.1:0/3676654205 learned_addr learned my addr 172.21.15.1:0/3676654205 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb233e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c105d50 0x7fb23c1013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:20.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb23be00640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c108680 0x7fb23c101910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb233e00640 1 -- 172.21.15.1:0/3676654205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb23c103420 msgr2=0x7fb23c102d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb233e00640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb23c103420 0x7fb23c102d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb233e00640 1 -- 172.21.15.1:0/3676654205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c108680 msgr2=0x7fb23c101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb233e00640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c108680 0x7fb23c101910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.425+0000 7fb233e00640 1 -- 172.21.15.1:0/3676654205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb23c1021a0 con 0x7fb23c105d50 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.426+0000 7fb23b400640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb23c103420 0x7fb23c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.426+0000 7fb233e00640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c105d50 0x7fb23c1013d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb22400e9c0 tx=0x7fb22400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.426+0000 7fb238c00640 1 -- 172.21.15.1:0/3676654205 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb22400ccc0 con 0x7fb23c105d50 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.426+0000 7fb238c00640 1 -- 172.21.15.1:0/3676654205 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2240108e0 con 0x7fb23c105d50 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.426+0000 7fb242000640 1 -- 172.21.15.1:0/3676654205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb23c1a2cd0 con 0x7fb23c105d50 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.426+0000 7fb238c00640 1 -- 172.21.15.1:0/3676654205 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2240187a0 con 0x7fb23c105d50 2024-04-26T21:30:20.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.426+0000 7fb242000640 1 -- 172.21.15.1:0/3676654205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb23c1a31c0 con 0x7fb23c105d50 2024-04-26T21:30:20.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.427+0000 7fb242000640 1 -- 172.21.15.1:0/3676654205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb23c0fe900 con 0x7fb23c105d50 2024-04-26T21:30:20.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.429+0000 7fb238c00640 1 -- 172.21.15.1:0/3676654205 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 22) ==== 138458+0+0 (secure 0 0 0) 0x7fb224010430 con 0x7fb23c105d50 2024-04-26T21:30:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.431+0000 7fb238c00640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb2200a3a10 0x7fb2200a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:20.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.431+0000 7fb238c00640 1 -- 172.21.15.1:0/3676654205 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fb2240d07b0 con 0x7fb23c105d50 2024-04-26T21:30:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.431+0000 7fb23b400640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb2200a3a10 0x7fb2200a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:20.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.432+0000 7fb23b400640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb2200a3a10 0x7fb2200a5ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0096c0 tx=0x7fb22c005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:20.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.435+0000 7fb238c00640 1 -- 172.21.15.1:0/3676654205 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb22409c8c0 con 0x7fb23c105d50 2024-04-26T21:30:20.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.601+0000 7fb238c00640 1 -- 172.21.15.1:0/3676654205 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 23) ==== 138765+0+0 (secure 0 0 0) 0x7fb224098f30 con 0x7fb23c105d50 2024-04-26T21:30:20.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.684+0000 7fb242000640 1 -- 172.21.15.1:0/3676654205 --> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fb23c062d70 con 0x7fb2200a3a10 2024-04-26T21:30:20.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.698+0000 7fb238c00640 1 -- 172.21.15.1:0/3676654205 <== mgr.15291 v2:172.21.15.1:6800/3236768718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fb23c062d70 con 0x7fb2200a3a10 2024-04-26T21:30:20.699 INFO:teuthology.orchestra.run.smithi001.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T21:30:19", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-26T21:30:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.701+0000 7fb242000640 1 -- 172.21.15.1:0/3676654205 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb2200a3a10 msgr2=0x7fb2200a5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.701+0000 7fb242000640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb2200a3a10 0x7fb2200a5ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb22c0096c0 tx=0x7fb22c005a20 comp rx=0 tx=0).stop 2024-04-26T21:30:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.701+0000 7fb242000640 1 -- 172.21.15.1:0/3676654205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c105d50 msgr2=0x7fb23c1013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:20.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.701+0000 7fb242000640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c105d50 0x7fb23c1013d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb22400e9c0 tx=0x7fb22400ee80 comp rx=0 tx=0).stop 2024-04-26T21:30:20.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.702+0000 7fb242000640 1 -- 172.21.15.1:0/3676654205 shutdown_connections 2024-04-26T21:30:20.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.702+0000 7fb242000640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb2200a3a10 0x7fb2200a5ed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.702+0000 7fb242000640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb23c108680 0x7fb23c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.702+0000 7fb242000640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb23c105d50 0x7fb23c1013d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.702+0000 7fb242000640 1 --2- 172.21.15.1:0/3676654205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb23c103420 0x7fb23c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.702+0000 7fb242000640 1 -- 172.21.15.1:0/3676654205 >> 172.21.15.1:0/3676654205 conn(0x7fb23c0fce10 msgr2=0x7fb23c103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:20.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.702+0000 7fb242000640 1 -- 172.21.15.1:0/3676654205 shutdown_connections 2024-04-26T21:30:20.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.702+0000 7fb242000640 1 -- 172.21.15.1:0/3676654205 wait complete. 2024-04-26T21:30:20.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T21:30:20.780 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:20 smithi001 ceph-mon[28054]: mgrmap e22: x(active, since 1.03587s), standbys: z, y 2024-04-26T21:30:20.781 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:20 smithi001 ceph-mon[28054]: [26/Apr/2024:21:30:20] ENGINE Bus STARTING 2024-04-26T21:30:20.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.908+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2546327627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc0772f0 msgr2=0x7fd3bc077710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:20.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.908+0000 7fd3c0c00640 1 --2- 172.21.15.1:0/2546327627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc0772f0 0x7fd3bc077710 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac009900 tx=0x7fd3ac02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.908+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2546327627 shutdown_connections 2024-04-26T21:30:20.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.908+0000 7fd3c0c00640 1 --2- 172.21.15.1:0/2546327627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3bc078ed0 0x7fd3bc079770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.908+0000 7fd3c0c00640 1 --2- 172.21.15.1:0/2546327627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3bc078530 0x7fd3bc078990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.908+0000 7fd3c0c00640 1 --2- 172.21.15.1:0/2546327627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc0772f0 0x7fd3bc077710 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.908+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2546327627 >> 172.21.15.1:0/2546327627 conn(0x7fd3bc100150 msgr2=0x7fd3bc102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:20.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.908+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2546327627 shutdown_connections 2024-04-26T21:30:20.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.908+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2546327627 wait complete. 2024-04-26T21:30:20.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.910+0000 7fd3c0c00640 1 Processor -- start 2024-04-26T21:30:20.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.910+0000 7fd3c0c00640 1 -- start start 2024-04-26T21:30:20.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.910+0000 7fd3c0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc0772f0 0x7fd3bc19f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.910+0000 7fd3c0c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3bc078530 0x7fd3bc19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.910+0000 7fd3c0c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3bc078ed0 0x7fd3bc1a6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.910+0000 7fd3c0c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd3bc114b00 con 0x7fd3bc078530 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.910+0000 7fd3c0c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd3bc114980 con 0x7fd3bc078ed0 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.910+0000 7fd3c0c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd3bc114c80 con 0x7fd3bc0772f0 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3b9e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc0772f0 0x7fd3bc19f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3b9e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc0772f0 0x7fd3bc19f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51746/0 (socket says 172.21.15.1:51746) 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3b9e00640 1 -- 172.21.15.1:0/2988365045 learned_addr learned my addr 172.21.15.1:0/2988365045 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3ba800640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3bc078ed0 0x7fd3bc1a6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3b9400640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3bc078530 0x7fd3bc19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3b9e00640 1 -- 172.21.15.1:0/2988365045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3bc078ed0 msgr2=0x7fd3bc1a6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3b9e00640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3bc078ed0 0x7fd3bc1a6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3b9e00640 1 -- 172.21.15.1:0/2988365045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3bc078530 msgr2=0x7fd3bc19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3b9e00640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3bc078530 0x7fd3bc19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:20.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3b9e00640 1 -- 172.21.15.1:0/2988365045 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3bc1a6dd0 con 0x7fd3bc0772f0 2024-04-26T21:30:20.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.911+0000 7fd3b9e00640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc0772f0 0x7fd3bc19f1b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac005ab0 tx=0x7fd3ac003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:20.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.912+0000 7fd3b2a00640 1 -- 172.21.15.1:0/2988365045 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac03f070 con 0x7fd3bc0772f0 2024-04-26T21:30:20.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.912+0000 7fd3b2a00640 1 -- 172.21.15.1:0/2988365045 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd3ac031c50 con 0x7fd3bc0772f0 2024-04-26T21:30:20.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.912+0000 7fd3b2a00640 1 -- 172.21.15.1:0/2988365045 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac035030 con 0x7fd3bc0772f0 2024-04-26T21:30:20.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.912+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2988365045 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3bc1a7060 con 0x7fd3bc0772f0 2024-04-26T21:30:20.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.912+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2988365045 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3bc1a75a0 con 0x7fd3bc0772f0 2024-04-26T21:30:20.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.914+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2988365045 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3bc1116b0 con 0x7fd3bc0772f0 2024-04-26T21:30:20.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.915+0000 7fd3b2a00640 1 -- 172.21.15.1:0/2988365045 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 23) ==== 138765+0+0 (secure 0 0 0) 0x7fd3ac042070 con 0x7fd3bc0772f0 2024-04-26T21:30:20.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.918+0000 7fd3b2a00640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd3a00a3d90 0x7fd3a00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:20.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.918+0000 7fd3b2a00640 1 -- 172.21.15.1:0/2988365045 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fd3ac0f12d0 con 0x7fd3bc0772f0 2024-04-26T21:30:20.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.918+0000 7fd3b9400640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd3a00a3d90 0x7fd3a00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:20.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.919+0000 7fd3b9400640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd3a00a3d90 0x7fd3a00a6250 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc1a06c0 tx=0x7fd3a8005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:20.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:20.921+0000 7fd3b2a00640 1 -- 172.21.15.1:0/2988365045 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd3ac0bceb0 con 0x7fd3bc0772f0 2024-04-26T21:30:20.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:20 smithi186 ceph-mon[32991]: mgrmap e22: x(active, since 1.03587s), standbys: z, y 2024-04-26T21:30:20.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:20 smithi186 ceph-mon[32991]: [26/Apr/2024:21:30:20] ENGINE Bus STARTING 2024-04-26T21:30:21.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:20 smithi038 ceph-mon[32724]: mgrmap e22: x(active, since 1.03587s), standbys: z, y 2024-04-26T21:30:21.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:20 smithi038 ceph-mon[32724]: [26/Apr/2024:21:30:20] ENGINE Bus STARTING 2024-04-26T21:30:21.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.169+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2988365045 --> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fd3bc062d70 con 0x7fd3a00a3d90 2024-04-26T21:30:21.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.183+0000 7fd3b2a00640 1 -- 172.21.15.1:0/2988365045 <== mgr.15291 v2:172.21.15.1:6800/3236768718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fd3bc062d70 con 0x7fd3a00a3d90 2024-04-26T21:30:21.184 INFO:teuthology.orchestra.run.smithi001.stdout:/ 1m 6m3h 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2988365045 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd3a00a3d90 msgr2=0x7fd3a00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd3a00a3d90 0x7fd3a00a6250 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc1a06c0 tx=0x7fd3a8005f90 comp rx=0 tx=0).stop 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2988365045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc0772f0 msgr2=0x7fd3bc19f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc0772f0 0x7fd3bc19f1b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac005ab0 tx=0x7fd3ac003b90 comp rx=0 tx=0).stop 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2988365045 shutdown_connections 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd3a00a3d90 0x7fd3a00a6250 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3bc078ed0 0x7fd3bc1a6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3bc078530 0x7fd3bc19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 --2- 172.21.15.1:0/2988365045 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3bc0772f0 0x7fd3bc19f1b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2988365045 >> 172.21.15.1:0/2988365045 conn(0x7fd3bc100150 msgr2=0x7fd3bc101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2988365045 shutdown_connections 2024-04-26T21:30:21.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:30:21.187+0000 7fd3c0c00640 1 -- 172.21.15.1:0/2988365045 wait complete. 2024-04-26T21:30:21.214 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-26T21:30:21.283 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-26T21:30:21.295 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-26T21:30:21.345 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:30:21.345 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-26T21:30:21.352 INFO:teuthology.run_tasks:Running task workunit... 2024-04-26T21:30:21.440 INFO:tasks.workunit:Pulling workunits from ref bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:30:21.441 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-26T21:30:21.441 INFO:tasks.workunit:timeout=3h 2024-04-26T21:30:21.441 INFO:tasks.workunit:cleanup=True 2024-04-26T21:30:21.441 DEBUG:teuthology.orchestra.run.smithi001:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-26T21:30:21.485 INFO:teuthology.orchestra.run.smithi001.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-26T21:30:21.485 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-26T21:30:21.485 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-04-26T21:30:21.485 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T21:30:21.485 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-26T21:30:21.485 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-04-26 21:27:31.722549746 +0000 2024-04-26T21:30:21.485 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-04-26 21:27:31.722549746 +0000 2024-04-26T21:30:21.485 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-04-26 21:30:16.478395698 +0000 2024-04-26T21:30:21.485 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-04-26 21:27:31.722549746 +0000 2024-04-26T21:30:21.486 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-26T21:30:21.486 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-26T21:30:21.605 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bc612ffb9c1d5bc27536bc917a705262a36d4387 2024-04-26T21:30:21.649 INFO:tasks.workunit.client.0.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-26T21:30:21.921 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: from='client.25502 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:21.921 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: [26/Apr/2024:21:30:20] ENGINE Serving on http://172.21.15.1:8765 2024-04-26T21:30:21.921 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: [26/Apr/2024:21:30:20] ENGINE Serving on https://172.21.15.1:7150 2024-04-26T21:30:21.921 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: [26/Apr/2024:21:30:20] ENGINE Bus STARTED 2024-04-26T21:30:21.921 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-26T21:30:21.921 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:30:21.921 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: from='client.25508 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:21.921 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:21.921 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:21.921 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:21.922 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:21 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:22.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: from='client.25502 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:22.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: [26/Apr/2024:21:30:20] ENGINE Serving on http://172.21.15.1:8765 2024-04-26T21:30:22.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: [26/Apr/2024:21:30:20] ENGINE Serving on https://172.21.15.1:7150 2024-04-26T21:30:22.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: [26/Apr/2024:21:30:20] ENGINE Bus STARTED 2024-04-26T21:30:22.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-26T21:30:22.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:30:22.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: from='client.25508 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:22.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:22.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:22.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:22.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:21 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: from='client.25502 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: [26/Apr/2024:21:30:20] ENGINE Serving on http://172.21.15.1:8765 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: [26/Apr/2024:21:30:20] ENGINE Serving on https://172.21.15.1:7150 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: [26/Apr/2024:21:30:20] ENGINE Bus STARTED 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: from='client.25508 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:21 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='client.25501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T21:30:23.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:30:23.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='client.25501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.206 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='client.25501 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:23.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:23.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:24.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:23 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T21:30:24.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:23 smithi038 ceph-mon[32724]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:30:24.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:23 smithi038 ceph-mon[32724]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:30:24.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:23 smithi038 ceph-mon[32724]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-26T21:30:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:23 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T21:30:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:23 smithi186 ceph-mon[32991]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:30:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:23 smithi186 ceph-mon[32991]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:30:24.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:23 smithi186 ceph-mon[32991]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-26T21:30:24.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:23 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T21:30:24.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:23 smithi001 ceph-mon[28054]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:30:24.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:23 smithi001 ceph-mon[28054]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:30:24.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:23 smithi001 ceph-mon[28054]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-04-26T21:30:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi001 to 177.5M 2024-04-26T21:30:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: Unable to set osd_memory_target on smithi001 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:30:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:25.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:25.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T21:30:25.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T21:30:25.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:24 smithi001 ceph-mon[28054]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi001 to 177.5M 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: Unable to set osd_memory_target on smithi001 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T21:30:25.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:24 smithi038 ceph-mon[32724]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T21:30:25.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi001 to 177.5M 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: Unable to set osd_memory_target on smithi001 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: Updating smithi001:/etc/ceph/ceph.conf 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: Updating smithi038:/etc/ceph/ceph.conf 2024-04-26T21:30:25.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:24 smithi186 ceph-mon[32991]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-26T21:30:26.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:26 smithi001 ceph-mon[28054]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:30:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:26 smithi001 ceph-mon[28054]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:30:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:26 smithi001 ceph-mon[28054]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:30:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:26 smithi001 ceph-mon[28054]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:30:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:26 smithi001 ceph-mon[28054]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:30:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:26 smithi001 ceph-mon[28054]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:30:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:26 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:26 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:26 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:26 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:26 smithi038 ceph-mon[32724]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:30:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:26 smithi038 ceph-mon[32724]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:30:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:26 smithi038 ceph-mon[32724]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:30:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:26 smithi038 ceph-mon[32724]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:30:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:26 smithi038 ceph-mon[32724]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:30:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:26 smithi038 ceph-mon[32724]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:30:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:26 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:26 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:26 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:26 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:26 smithi186 ceph-mon[32991]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:30:26.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:26 smithi186 ceph-mon[32991]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:30:26.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:26 smithi186 ceph-mon[32991]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:30:26.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:26 smithi186 ceph-mon[32991]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:30:26.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:26 smithi186 ceph-mon[32991]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:30:26.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:26 smithi186 ceph-mon[32991]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T21:30:26.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:26 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:26 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:26 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:26.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:26 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:27.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:27 smithi001 ceph-mon[28054]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:30:27.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:27 smithi001 ceph-mon[28054]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:30:27.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:27 smithi001 ceph-mon[28054]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:30:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:27 smithi038 ceph-mon[32724]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:30:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:27 smithi038 ceph-mon[32724]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:30:27.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:27 smithi038 ceph-mon[32724]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:30:27.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:27 smithi186 ceph-mon[32991]: Updating smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:30:27.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:27 smithi186 ceph-mon[32991]: Updating smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:30:27.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:27 smithi186 ceph-mon[32991]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-04-26T21:30:28.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:28 smithi001 ceph-mon[28054]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:30:28.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:28 smithi001 ceph-mon[28054]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 132 KiB/s wr, 75 op/s 2024-04-26T21:30:28.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:28 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:28 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:28 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:28 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:28.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:28 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:28.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:28 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:28.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:28 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:28 smithi038 ceph-mon[32724]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:30:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:28 smithi038 ceph-mon[32724]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 132 KiB/s wr, 75 op/s 2024-04-26T21:30:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:28 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:28 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:28 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:28 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:28 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:28 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:28 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:28 smithi186 ceph-mon[32991]: Updating smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T21:30:28.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:28 smithi186 ceph-mon[32991]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 132 KiB/s wr, 75 op/s 2024-04-26T21:30:28.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:28 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:28 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:28 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:28.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:28 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:30:28.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:28 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:30:28.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:28 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:30:28.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:28 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:30:30.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:30 smithi001 ceph-mon[28054]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 58 op/s 2024-04-26T21:30:30.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:30 smithi038 ceph-mon[32724]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 58 op/s 2024-04-26T21:30:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:30 smithi186 ceph-mon[32991]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 103 KiB/s wr, 58 op/s 2024-04-26T21:30:32.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:32 smithi001 ceph-mon[28054]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 50 op/s 2024-04-26T21:30:32.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:32 smithi038 ceph-mon[32724]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 50 op/s 2024-04-26T21:30:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:32 smithi186 ceph-mon[32991]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 84 KiB/s wr, 50 op/s 2024-04-26T21:30:34.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:34 smithi001 ceph-mon[28054]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-26T21:30:34.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:34 smithi038 ceph-mon[32724]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-26T21:30:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:34 smithi186 ceph-mon[32991]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-26T21:30:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.599+0000 7fa615200640 1 -- 172.21.15.1:0/2178207362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa61010a970 msgr2=0x7fa61010ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.599+0000 7fa615200640 1 --2- 172.21.15.1:0/2178207362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa61010a970 0x7fa61010ad50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa600009890 tx=0x7fa60002a360 comp rx=0 tx=0).stop 2024-04-26T21:30:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.600+0000 7fa615200640 1 -- 172.21.15.1:0/2178207362 shutdown_connections 2024-04-26T21:30:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.600+0000 7fa615200640 1 --2- 172.21.15.1:0/2178207362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa610105320 0x7fa610111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.600+0000 7fa615200640 1 --2- 172.21.15.1:0/2178207362 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa610104960 0x7fa610104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.600+0000 7fa615200640 1 --2- 172.21.15.1:0/2178207362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa61010a970 0x7fa61010ad50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.600+0000 7fa615200640 1 -- 172.21.15.1:0/2178207362 >> 172.21.15.1:0/2178207362 conn(0x7fa610100470 msgr2=0x7fa610102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.600+0000 7fa615200640 1 -- 172.21.15.1:0/2178207362 shutdown_connections 2024-04-26T21:30:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.600+0000 7fa615200640 1 -- 172.21.15.1:0/2178207362 wait complete. 2024-04-26T21:30:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.601+0000 7fa615200640 1 Processor -- start 2024-04-26T21:30:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.601+0000 7fa615200640 1 -- start start 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.601+0000 7fa615200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa610104960 0x7fa61019b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.601+0000 7fa615200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa610105320 0x7fa61019b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.601+0000 7fa615200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61010a970 0x7fa61019fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.601+0000 7fa615200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa610114c00 con 0x7fa610105320 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.601+0000 7fa615200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa610114a80 con 0x7fa61010a970 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.601+0000 7fa615200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa610114d80 con 0x7fa610104960 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa610104960 0x7fa61019b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa610105320 0x7fa61019b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61010a970 0x7fa61019fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa610104960 0x7fa61019b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33526/0 (socket says 172.21.15.1:33526) 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60f400640 1 -- 172.21.15.1:0/1658029741 learned_addr learned my addr 172.21.15.1:0/1658029741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60f400640 1 -- 172.21.15.1:0/1658029741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61010a970 msgr2=0x7fa61019fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60f400640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61010a970 0x7fa61019fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60f400640 1 -- 172.21.15.1:0/1658029741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa610105320 msgr2=0x7fa61019b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60f400640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa610105320 0x7fa61019b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60f400640 1 -- 172.21.15.1:0/1658029741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6101a0180 con 0x7fa610104960 2024-04-26T21:30:34.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.602+0000 7fa60ea00640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa610105320 0x7fa61019b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.603+0000 7fa60f400640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa610104960 0x7fa61019b190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa600032d80 tx=0x7fa600032db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:34.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.603+0000 7fa607e00640 1 -- 172.21.15.1:0/1658029741 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa600037040 con 0x7fa610104960 2024-04-26T21:30:34.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.603+0000 7fa607e00640 1 -- 172.21.15.1:0/1658029741 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa600041070 con 0x7fa610104960 2024-04-26T21:30:34.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.603+0000 7fa615200640 1 -- 172.21.15.1:0/1658029741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6101a0410 con 0x7fa610104960 2024-04-26T21:30:34.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.603+0000 7fa607e00640 1 -- 172.21.15.1:0/1658029741 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa600003850 con 0x7fa610104960 2024-04-26T21:30:34.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.603+0000 7fa615200640 1 -- 172.21.15.1:0/1658029741 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6101a7d00 con 0x7fa610104960 2024-04-26T21:30:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.607+0000 7fa607e00640 1 -- 172.21.15.1:0/1658029741 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7fa600004050 con 0x7fa610104960 2024-04-26T21:30:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.607+0000 7fa61959a740 1 Processor -- start 2024-04-26T21:30:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.607+0000 7fa61959a740 1 -- start start 2024-04-26T21:30:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.607+0000 7fa61959a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7dfe3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.607+0000 7fa61959a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e08c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.607+0000 7fa61959a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7e091c0 0x55f3a7e0b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.607+0000 7fa61959a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55f3a7d63380 con 0x55f3a7dfe900 2024-04-26T21:30:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.607+0000 7fa61959a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f3a7cffc30 con 0x55f3a7dfdfe0 2024-04-26T21:30:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.607+0000 7fa61959a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f3a7d5f340 con 0x55f3a7e091c0 2024-04-26T21:30:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.608+0000 7fa60f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e08c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.608+0000 7fa60ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7e091c0 0x55f3a7e0b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.608+0000 7fa60fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7dfe3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.608+0000 7fa60ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7e091c0 0x55f3a7e0b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33528/0 (socket says 172.21.15.1:33528) 2024-04-26T21:30:34.609 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.608+0000 7fa60ea00640 1 -- 172.21.15.1:0/2238720714 learned_addr learned my addr 172.21.15.1:0/2238720714 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:34.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.608+0000 7fa60ea00640 1 -- 172.21.15.1:0/2238720714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfdfe0 msgr2=0x55f3a7dfe3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:34.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.608+0000 7fa60ea00640 1 --2- 172.21.15.1:0/2238720714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7dfe3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.608+0000 7fa60ea00640 1 -- 172.21.15.1:0/2238720714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 msgr2=0x55f3a7e08c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:34.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.608+0000 7fa60ea00640 1 --2- 172.21.15.1:0/2238720714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e08c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.608+0000 7fa60ea00640 1 -- 172.21.15.1:0/2238720714 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3a7d21940 con 0x55f3a7e091c0 2024-04-26T21:30:34.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.609+0000 7fa60ea00640 1 --2- 172.21.15.1:0/2238720714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7e091c0 0x55f3a7e0b5b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0114f0 tx=0x7fa5fc0119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d06cedaf7d20e983 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.609+0000 7fa607e00640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa5f40a3d90 0x7fa5f40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.609+0000 7fa604c00640 1 -- 172.21.15.1:0/2238720714 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc01e070 con 0x55f3a7e091c0 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.609+0000 7fa604c00640 1 -- 172.21.15.1:0/2238720714 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5fc00fa70 con 0x55f3a7e091c0 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.609+0000 7fa604c00640 1 -- 172.21.15.1:0/2238720714 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc019b60 con 0x55f3a7e091c0 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.609+0000 7fa607e00640 1 -- 172.21.15.1:0/1658029741 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fa6000f0800 con 0x7fa610104960 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.609+0000 7fa60fe00640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa5f40a3d90 0x7fa5f40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.610+0000 7fa61959a740 1 -- 172.21.15.1:0/2238720714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7e091c0 msgr2=0x55f3a7e0b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.610+0000 7fa61959a740 1 --2- 172.21.15.1:0/2238720714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7e091c0 0x55f3a7e0b5b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0114f0 tx=0x7fa5fc0119b0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.610+0000 7fa60fe00640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa5f40a3d90 0x7fa5f40a6250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa608005fd0 tx=0x7fa608010040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.610+0000 7fa61959a740 1 -- 172.21.15.1:0/2238720714 shutdown_connections 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.610+0000 7fa61959a740 1 --2- 172.21.15.1:0/2238720714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7e091c0 0x55f3a7e0b5b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.610+0000 7fa61959a740 1 --2- 172.21.15.1:0/2238720714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e08c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.610+0000 7fa61959a740 1 --2- 172.21.15.1:0/2238720714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7dfe3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.610+0000 7fa61959a740 1 -- 172.21.15.1:0/2238720714 >> 172.21.15.1:0/2238720714 conn(0x55f3a7dfa4f0 msgr2=0x55f3a7dfa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.610+0000 7fa61959a740 1 -- 172.21.15.1:0/2238720714 shutdown_connections 2024-04-26T21:30:34.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.610+0000 7fa61959a740 1 -- 172.21.15.1:0/2238720714 wait complete. 2024-04-26T21:30:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.611+0000 7fa61959a740 1 -- start start 2024-04-26T21:30:34.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.611+0000 7fa61959a740 1 Processor -- start 2024-04-26T21:30:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.611+0000 7fa61959a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e060d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa61959a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e06610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa61959a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7e091c0 0x55f3a7e06b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa61959a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55f3a7e0bdf0 con 0x55f3a7dfe900 2024-04-26T21:30:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa61959a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f3a7ceac60 con 0x55f3a7e091c0 2024-04-26T21:30:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa61959a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f3a7e0bf70 con 0x55f3a7dfdfe0 2024-04-26T21:30:34.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa61959a740 10 client.0 _ll_register_callbacks cb 0x55f3a7d16e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e06610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e060d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33530/0 (socket says 172.21.15.1:33530) 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60f400640 1 -- 172.21.15.1:0/204024458 learned_addr learned my addr 172.21.15.1:0/204024458 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60fe00640 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7e091c0 0x55f3a7e06b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60ea00640 1 -- 172.21.15.1:0/204024458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 msgr2=0x55f3a7e060d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60ea00640 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e060d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60ea00640 1 -- 172.21.15.1:0/204024458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7e091c0 msgr2=0x55f3a7e06b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60ea00640 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7e091c0 0x55f3a7e06b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60ea00640 1 -- 172.21.15.1:0/204024458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3a7cf9510 con 0x55f3a7dfe900 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.612+0000 7fa60f400640 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e060d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.613+0000 7fa60ea00640 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e06610 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc01c040 tx=0x7fa5fc019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f363d46c7092b8a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.613+0000 7fa5f9600640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.613+0000 7fa5f9600640 1 -- 172.21.15.1:0/204024458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc01e040 con 0x55f3a7dfe900 2024-04-26T21:30:34.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.613+0000 7fa5f9600640 1 -- 172.21.15.1:0/204024458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fa5fc00f5c0 con 0x55f3a7dfe900 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.613+0000 7fa61959a740 1 -- 172.21.15.1:0/204024458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5fc00fa70 con 0x55f3a7dfe900 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.613+0000 7fa5f9600640 1 -- 172.21.15.1:0/204024458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa5fc00f7c0 con 0x55f3a7dfe900 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.613+0000 7fa61959a740 10 client.15384 fetch_fsmap learned FSMap version 24 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.613+0000 7fa61959a740 1 -- 172.21.15.1:0/204024458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f3a7a75210 con 0x55f3a7dfe900 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.614+0000 7fa5f9600640 1 -- 172.21.15.1:0/204024458 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5fc00fa70 con 0x55f3a7dfe900 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.614+0000 7fa5f9600640 1 -- 172.21.15.1:0/204024458 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa5fc005020 con 0x55f3a7dfe900 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.614+0000 7fa61959a740 10 client.15384 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.614+0000 7fa61959a740 10 client.15384 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.614+0000 7fa61959a740 20 client.15384 populate_metadata read hostname 'smithi001' 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.614+0000 7fa61959a740 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f3a7e18d90 0x55f3a7e1b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.614+0000 7fa61959a740 4 client.15384 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:30:34.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.614+0000 7fa61959a740 1 -- 172.21.15.1:0/204024458 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3a7d70f90 con 0x55f3a7e18d90 2024-04-26T21:30:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.615+0000 7fa60f400640 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f3a7e18d90 0x55f3a7e1b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.615+0000 7fa60f400640 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f3a7e18d90 0x55f3a7e1b180 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c711736b79350bb server_cookie=65cbe773c462b28c in_seq=0 out_seq=0 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.615+0000 7fa5f9600640 10 client.15384 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa5f9600640 1 -- 172.21.15.1:0/204024458 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f3a7d70f90 con 0x55f3a7e18d90 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa5f9600640 10 client.15384 handle_command_reply: tid=0 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa61959a740 1 client.15384 shutdown 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa61959a740 1 -- 172.21.15.1:0/204024458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 msgr2=0x55f3a7e06610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa61959a740 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e06610 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc01c040 tx=0x7fa5fc019640 comp rx=0 tx=0).stop 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa61959a740 1 -- 172.21.15.1:0/204024458 shutdown_connections 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa61959a740 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f3a7e18d90 0x55f3a7e1b180 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa61959a740 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7e091c0 0x55f3a7e06b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa61959a740 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e06610 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa61959a740 1 --2- 172.21.15.1:0/204024458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e060d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa5f9600640 0 client.15384 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa61959a740 1 -- 172.21.15.1:0/204024458 >> 172.21.15.1:0/204024458 conn(0x55f3a7dfa4f0 msgr2=0x55f3a7e00270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.616+0000 7fa61959a740 1 -- 172.21.15.1:0/204024458 shutdown_connections 2024-04-26T21:30:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.617+0000 7fa61959a740 1 -- 172.21.15.1:0/204024458 wait complete. 2024-04-26T21:30:34.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.617+0000 7fa61959a740 20 client.15384 trim_cache size 0 max 16384 2024-04-26T21:30:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.633+0000 7fa61959a740 1 Processor -- start 2024-04-26T21:30:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.633+0000 7fa61959a740 1 -- start start 2024-04-26T21:30:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa61959a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e14cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa61959a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e050b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa61959a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7e091c0 0x55f3a7e208c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa61959a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa6101b47a0 con 0x55f3a7dfe900 2024-04-26T21:30:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa61959a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa6101b2070 con 0x55f3a7e091c0 2024-04-26T21:30:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa61959a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6101afec0 con 0x55f3a7dfdfe0 2024-04-26T21:30:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa60ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e050b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa60ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e050b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43526/0 (socket says 172.21.15.1:43526) 2024-04-26T21:30:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa60ea00640 1 -- 172.21.15.1:0/1169799657 learned_addr learned my addr 172.21.15.1:0/1169799657 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:34.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa60f400640 1 --2- 172.21.15.1:0/1169799657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e14cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa60fe00640 1 --2- 172.21.15.1:0/1169799657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7e091c0 0x55f3a7e208c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa60ea00640 1 -- 172.21.15.1:0/1169799657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 msgr2=0x55f3a7e14cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa60ea00640 1 --2- 172.21.15.1:0/1169799657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e14cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa60ea00640 1 -- 172.21.15.1:0/1169799657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7e091c0 msgr2=0x55f3a7e208c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.634+0000 7fa60ea00640 1 --2- 172.21.15.1:0/1169799657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7e091c0 0x55f3a7e208c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.635+0000 7fa60ea00640 1 -- 172.21.15.1:0/1169799657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3a7b41620 con 0x55f3a7dfe900 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.635+0000 7fa60ea00640 1 --2- 172.21.15.1:0/1169799657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e050b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc005590 tx=0x7fa5fc012f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ddec32f007c67a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.635+0000 7fa5fbe00640 1 -- 172.21.15.1:0/1169799657 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc00f5c0 con 0x55f3a7dfe900 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.635+0000 7fa5fbe00640 1 -- 172.21.15.1:0/1169799657 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5fc030020 con 0x55f3a7dfe900 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.635+0000 7fa5fbe00640 1 -- 172.21.15.1:0/1169799657 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0043b0 con 0x55f3a7dfe900 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 -- 172.21.15.1:0/1169799657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 msgr2=0x55f3a7e050b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 --2- 172.21.15.1:0/1169799657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e050b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc005590 tx=0x7fa5fc012f70 comp rx=0 tx=0).stop 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 -- 172.21.15.1:0/1169799657 shutdown_connections 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 --2- 172.21.15.1:0/1169799657 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7e091c0 0x55f3a7e208c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 --2- 172.21.15.1:0/1169799657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7dfe900 0x55f3a7e050b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 --2- 172.21.15.1:0/1169799657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e14cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 -- 172.21.15.1:0/1169799657 >> 172.21.15.1:0/1169799657 conn(0x55f3a7dfa4f0 msgr2=0x55f3a7e049a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 -- 172.21.15.1:0/1169799657 shutdown_connections 2024-04-26T21:30:34.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 -- 172.21.15.1:0/1169799657 wait complete. 2024-04-26T21:30:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 -- start start 2024-04-26T21:30:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.636+0000 7fa61959a740 1 Processor -- start 2024-04-26T21:30:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa61959a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e111c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa61959a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfe900 0x55f3a7e11700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa61959a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7e091c0 0x55f3a7e024e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa61959a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa6101b0530 con 0x55f3a7e091c0 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa61959a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa6101b19a0 con 0x55f3a7dfe900 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa61959a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6101b0c00 con 0x55f3a7dfdfe0 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa61959a740 10 client.0 _ll_register_callbacks cb 0x7fa5fc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa60fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7e091c0 0x55f3a7e024e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa60fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7e091c0 0x55f3a7e024e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43538/0 (socket says 172.21.15.1:43538) 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa60fe00640 1 -- 172.21.15.1:0/472261109 learned_addr learned my addr 172.21.15.1:0/472261109 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa60f400640 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.637+0000 7fa60ea00640 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfe900 0x55f3a7e11700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa60fe00640 1 -- 172.21.15.1:0/472261109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 msgr2=0x55f3a7e111c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa60fe00640 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e111c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa60fe00640 1 -- 172.21.15.1:0/472261109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfe900 msgr2=0x55f3a7e11700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa60fe00640 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfe900 0x55f3a7e11700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa60fe00640 1 -- 172.21.15.1:0/472261109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3a7d1dcb0 con 0x55f3a7e091c0 2024-04-26T21:30:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa60ea00640 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfe900 0x55f3a7e11700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa60fe00640 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7e091c0 0x55f3a7e024e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa608004670 tx=0x7fa608004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae0e722bb4acae0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa5f9600640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:30:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa5f9600640 1 -- 172.21.15.1:0/472261109 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608028680 con 0x55f3a7e091c0 2024-04-26T21:30:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa61959a740 1 -- 172.21.15.1:0/472261109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5fc030020 con 0x55f3a7e091c0 2024-04-26T21:30:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.638+0000 7fa5f9600640 1 -- 172.21.15.1:0/472261109 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fa608028820 con 0x55f3a7e091c0 2024-04-26T21:30:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa5f9600640 1 -- 172.21.15.1:0/472261109 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa608036050 con 0x55f3a7e091c0 2024-04-26T21:30:34.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa61959a740 10 client.15390 fetch_fsmap learned FSMap version 24 2024-04-26T21:30:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa61959a740 1 -- 172.21.15.1:0/472261109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55f3a7e7ea10 con 0x55f3a7e091c0 2024-04-26T21:30:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa5f9600640 1 -- 172.21.15.1:0/472261109 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5fc030020 con 0x55f3a7e091c0 2024-04-26T21:30:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa5f9600640 1 -- 172.21.15.1:0/472261109 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa60800b040 con 0x55f3a7e091c0 2024-04-26T21:30:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa61959a740 10 client.15390 fetch_fsmap finished waiting for FSMap version 24 2024-04-26T21:30:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa61959a740 10 client.15390 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:30:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa61959a740 20 client.15390 populate_metadata read hostname 'smithi001' 2024-04-26T21:30:34.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa61959a740 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f3a7e7f1c0 0x55f3a7e815e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa61959a740 4 client.15390 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:30:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.639+0000 7fa61959a740 1 -- 172.21.15.1:0/472261109 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f3a7d6ae00 con 0x55f3a7e7f1c0 2024-04-26T21:30:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.640+0000 7fa60f400640 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f3a7e7f1c0 0x55f3a7e815e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.640+0000 7fa60f400640 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f3a7e7f1c0 0x55f3a7e815e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da6c116c702aff66 server_cookie=cb9f7b93f859d623 in_seq=0 out_seq=0 2024-04-26T21:30:34.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.640+0000 7fa5f9600640 10 client.15390 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:34.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.654+0000 7fa5f9600640 1 -- 172.21.15.1:0/472261109 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f3a7d6ae00 con 0x55f3a7e7f1c0 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa5f9600640 10 client.15390 handle_command_reply: tid=0 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 client.15390 shutdown 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 -- 172.21.15.1:0/472261109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7e091c0 msgr2=0x55f3a7e024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7e091c0 0x55f3a7e024e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa608004670 tx=0x7fa608004ab0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 -- 172.21.15.1:0/472261109 shutdown_connections 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f3a7e7f1c0 0x55f3a7e815e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f3a7e091c0 0x55f3a7e024e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f3a7dfe900 0x55f3a7e11700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 --2- 172.21.15.1:0/472261109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3a7dfdfe0 0x55f3a7e111c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa5f9600640 0 client.15390 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 -- 172.21.15.1:0/472261109 >> 172.21.15.1:0/472261109 conn(0x55f3a7dfa4f0 msgr2=0x55f3a7e05230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 -- 172.21.15.1:0/472261109 shutdown_connections 2024-04-26T21:30:34.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 1 -- 172.21.15.1:0/472261109 wait complete. 2024-04-26T21:30:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.655+0000 7fa61959a740 20 client.15390 trim_cache size 0 max 16384 2024-04-26T21:30:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:30:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:30:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:30:34.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:30:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.656+0000 7fa615200640 1 -- 172.21.15.1:0/1658029741 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa5f40a3d90 msgr2=0x7fa5f40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.656+0000 7fa615200640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa5f40a3d90 0x7fa5f40a6250 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa608005fd0 tx=0x7fa608010040 comp rx=0 tx=0).stop 2024-04-26T21:30:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.656+0000 7fa615200640 1 -- 172.21.15.1:0/1658029741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa610104960 msgr2=0x7fa61019b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.656+0000 7fa615200640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa610104960 0x7fa61019b190 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa600032d80 tx=0x7fa600032db0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.656+0000 7fa615200640 1 -- 172.21.15.1:0/1658029741 shutdown_connections 2024-04-26T21:30:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.656+0000 7fa615200640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa5f40a3d90 0x7fa5f40a6250 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.656+0000 7fa615200640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa61010a970 0x7fa61019fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.656+0000 7fa615200640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa610105320 0x7fa61019b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.656+0000 7fa615200640 1 --2- 172.21.15.1:0/1658029741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa610104960 0x7fa61019b190 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.656+0000 7fa615200640 1 -- 172.21.15.1:0/1658029741 >> 172.21.15.1:0/1658029741 conn(0x7fa610100470 msgr2=0x7fa610100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.657+0000 7fa615200640 1 -- 172.21.15.1:0/1658029741 shutdown_connections 2024-04-26T21:30:34.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:34.657+0000 7fa615200640 1 -- 172.21.15.1:0/1658029741 wait complete. 2024-04-26T21:30:36.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:36 smithi001 ceph-mon[28054]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-26T21:30:36.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:36 smithi038 ceph-mon[32724]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-26T21:30:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:36 smithi186 ceph-mon[32991]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 46 op/s 2024-04-26T21:30:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:38 smithi001 ceph-mon[28054]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-26T21:30:38.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:38 smithi038 ceph-mon[32724]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-26T21:30:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:38 smithi186 ceph-mon[32991]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-26T21:30:40.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:40 smithi001 ceph-mon[28054]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:30:40.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:40 smithi038 ceph-mon[32724]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:30:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:40 smithi186 ceph-mon[32991]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-26T21:30:42.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:42 smithi001 ceph-mon[28054]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-26T21:30:42.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:42 smithi038 ceph-mon[32724]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-26T21:30:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:42 smithi186 ceph-mon[32991]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-26T21:30:43.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:43 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:30:43.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:43 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:30:43.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:43 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:30:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:44 smithi186 ceph-mon[32991]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T21:30:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:44 smithi001 ceph-mon[28054]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T21:30:44.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:44 smithi038 ceph-mon[32724]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T21:30:46.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:46 smithi186 ceph-mon[32991]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-26T21:30:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:46 smithi001 ceph-mon[28054]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-26T21:30:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:46 smithi038 ceph-mon[32724]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-26T21:30:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.805+0000 7f51c9a00640 1 -- 172.21.15.1:0/3820985543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4104970 msgr2=0x7f51c4104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.805+0000 7f51c9a00640 1 --2- 172.21.15.1:0/3820985543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4104970 0x7f51c4104df0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f51b0009930 tx=0x7f51b002a420 comp rx=0 tx=0).stop 2024-04-26T21:30:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.805+0000 7f51c9a00640 1 -- 172.21.15.1:0/3820985543 shutdown_connections 2024-04-26T21:30:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.805+0000 7f51c9a00640 1 --2- 172.21.15.1:0/3820985543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51c4105330 0x7f51c4111840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.805+0000 7f51c9a00640 1 --2- 172.21.15.1:0/3820985543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4104970 0x7f51c4104df0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.805+0000 7f51c9a00640 1 --2- 172.21.15.1:0/3820985543 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c410a980 0x7f51c410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.805+0000 7f51c9a00640 1 -- 172.21.15.1:0/3820985543 >> 172.21.15.1:0/3820985543 conn(0x7f51c4100440 msgr2=0x7f51c4102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.806+0000 7f51c9a00640 1 -- 172.21.15.1:0/3820985543 shutdown_connections 2024-04-26T21:30:47.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.806+0000 7f51c9a00640 1 -- 172.21.15.1:0/3820985543 wait complete. 2024-04-26T21:30:47.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c9a00640 1 Processor -- start 2024-04-26T21:30:47.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c9a00640 1 -- start start 2024-04-26T21:30:47.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c9a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c4104970 0x7f51c419b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c9a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4105330 0x7f51c419ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c9a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51c410a980 0x7f51c419fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c9a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4114c90 con 0x7f51c4105330 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c9a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4114b10 con 0x7f51c410a980 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c9a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51c4114e10 con 0x7f51c4104970 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c3400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4105330 0x7f51c419ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c3400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4105330 0x7f51c419ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40600/0 (socket says 172.21.15.1:40600) 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.807+0000 7f51c3400640 1 -- 172.21.15.1:0/496914481 learned_addr learned my addr 172.21.15.1:0/496914481 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.808+0000 7f51c3e00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c4104970 0x7f51c419b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.808+0000 7f51c9000640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51c410a980 0x7f51c419fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.808+0000 7f51c3e00640 1 -- 172.21.15.1:0/496914481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51c410a980 msgr2=0x7f51c419fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.808+0000 7f51c3e00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51c410a980 0x7f51c419fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.808+0000 7f51c3e00640 1 -- 172.21.15.1:0/496914481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4105330 msgr2=0x7f51c419ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.808+0000 7f51c3e00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4105330 0x7f51c419ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.808+0000 7f51c3e00640 1 -- 172.21.15.1:0/496914481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c41a04f0 con 0x7f51c4104970 2024-04-26T21:30:47.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.808+0000 7f51c3400640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4105330 0x7f51c419ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.808+0000 7f51c3e00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c4104970 0x7f51c419b500 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f51b400e9f0 tx=0x7f51b400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.809+0000 7f51c0c00640 1 -- 172.21.15.1:0/496914481 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b400ccc0 con 0x7f51c4104970 2024-04-26T21:30:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.809+0000 7f51c0c00640 1 -- 172.21.15.1:0/496914481 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b4005020 con 0x7f51c4104970 2024-04-26T21:30:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.809+0000 7f51c0c00640 1 -- 172.21.15.1:0/496914481 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b400ce60 con 0x7f51c4104970 2024-04-26T21:30:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.809+0000 7f51c9a00640 1 -- 172.21.15.1:0/496914481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c41a07e0 con 0x7f51c4104970 2024-04-26T21:30:47.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.809+0000 7f51c9a00640 1 -- 172.21.15.1:0/496914481 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c41a8070 con 0x7f51c4104970 2024-04-26T21:30:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.812+0000 7f51cde5b740 1 Processor -- start 2024-04-26T21:30:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.812+0000 7f51c0c00640 1 -- 172.21.15.1:0/496914481 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 24) ==== 138821+0+0 (secure 0 0 0) 0x7f51b40051d0 con 0x7f51c4104970 2024-04-26T21:30:47.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.812+0000 7f51cde5b740 1 -- start start 2024-04-26T21:30:47.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd4fe0 0x55c18abd53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd5900 0x55c18abdfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abe25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51cde5b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c18ab3a380 con 0x55c18abd5900 2024-04-26T21:30:47.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51cde5b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c18aad6c30 con 0x55c18abd4fe0 2024-04-26T21:30:47.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51cde5b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c18ab36340 con 0x55c18abe01c0 2024-04-26T21:30:47.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51c3e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd5900 0x55c18abdfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51c9000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd4fe0 0x55c18abd53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51c3e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd5900 0x55c18abdfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40616/0 (socket says 172.21.15.1:40616) 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51c3e00640 1 -- 172.21.15.1:0/3419218651 learned_addr learned my addr 172.21.15.1:0/3419218651 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.813+0000 7f51c3400640 1 --2- 172.21.15.1:0/3419218651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abe25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.814+0000 7f51c9000640 1 -- 172.21.15.1:0/3419218651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 msgr2=0x55c18abe25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.814+0000 7f51c9000640 1 --2- 172.21.15.1:0/3419218651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abe25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.814+0000 7f51c9000640 1 -- 172.21.15.1:0/3419218651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd5900 msgr2=0x55c18abdfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.814+0000 7f51c9000640 1 --2- 172.21.15.1:0/3419218651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd5900 0x55c18abdfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.814+0000 7f51c9000640 1 -- 172.21.15.1:0/3419218651 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c18aaf8940 con 0x55c18abd4fe0 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.814+0000 7f51c9000640 1 --2- 172.21.15.1:0/3419218651 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd4fe0 0x55c18abd53c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009f10 tx=0x7f51bc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb755d469eabcd89 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.814+0000 7f51c0c00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51a80a3e10 0x7f51a80a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.814+0000 7f51b9600640 1 -- 172.21.15.1:0/3419218651 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc004d20 con 0x55c18abd4fe0 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.814+0000 7f51c0c00640 1 -- 172.21.15.1:0/496914481 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f51b40d0c80 con 0x7f51c4104970 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.815+0000 7f51b9600640 1 -- 172.21.15.1:0/3419218651 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51bc004030 con 0x55c18abd4fe0 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.815+0000 7f51b9600640 1 -- 172.21.15.1:0/3419218651 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc005280 con 0x55c18abd4fe0 2024-04-26T21:30:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.815+0000 7f51c3e00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51a80a3e10 0x7f51a80a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.815+0000 7f51cde5b740 1 -- 172.21.15.1:0/3419218651 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd4fe0 msgr2=0x55c18abd53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.815+0000 7f51cde5b740 1 --2- 172.21.15.1:0/3419218651 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd4fe0 0x55c18abd53c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009f10 tx=0x7f51bc011810 comp rx=0 tx=0).stop 2024-04-26T21:30:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.815+0000 7f51c3e00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51a80a3e10 0x7f51a80a62d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f51b4099010 tx=0x7f51b40d4680 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.816+0000 7f51cde5b740 1 -- 172.21.15.1:0/3419218651 shutdown_connections 2024-04-26T21:30:47.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.816+0000 7f51cde5b740 1 --2- 172.21.15.1:0/3419218651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abe25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.816+0000 7f51cde5b740 1 --2- 172.21.15.1:0/3419218651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd5900 0x55c18abdfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.816+0000 7f51cde5b740 1 --2- 172.21.15.1:0/3419218651 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd4fe0 0x55c18abd53c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.816+0000 7f51cde5b740 1 -- 172.21.15.1:0/3419218651 >> 172.21.15.1:0/3419218651 conn(0x55c18abd14f0 msgr2=0x55c18abd18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.816+0000 7f51cde5b740 1 -- 172.21.15.1:0/3419218651 shutdown_connections 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.816+0000 7f51cde5b740 1 -- 172.21.15.1:0/3419218651 wait complete. 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.816+0000 7f51cde5b740 1 -- start start 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.816+0000 7f51cde5b740 1 Processor -- start 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abdd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abdd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abddb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51cde5b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c18abe2df0 con 0x55c18abd4fe0 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51cde5b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c18aac1c60 con 0x55c18abd5900 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51cde5b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c18abe2f70 con 0x55c18abe01c0 2024-04-26T21:30:47.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51cde5b740 10 client.0 _ll_register_callbacks cb 0x55c18aaede50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51c9000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abdd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51c3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abdd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51c3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abdd650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:48990/0 (socket says 172.21.15.1:48990) 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51c3400640 1 -- 172.21.15.1:0/205339694 learned_addr learned my addr 172.21.15.1:0/205339694 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.817+0000 7f51c9000640 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abddb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51c3400640 1 -- 172.21.15.1:0/205339694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 msgr2=0x55c18abddb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51c3400640 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abddb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51c3400640 1 -- 172.21.15.1:0/205339694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 msgr2=0x55c18abdd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51c3400640 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abdd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51c3400640 1 -- 172.21.15.1:0/205339694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c18aad0510 con 0x55c18abd5900 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51c9000640 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abdd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51c3400640 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abdd650 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f51b0009900 tx=0x7f51b0003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dd4ade9547a186e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:47.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51ae000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:30:47.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51ae000640 1 -- 172.21.15.1:0/205339694 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0035040 con 0x55c18abd5900 2024-04-26T21:30:47.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51cde5b740 1 -- 172.21.15.1:0/205339694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51bc004030 con 0x55c18abd5900 2024-04-26T21:30:47.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.818+0000 7f51ae000640 1 -- 172.21.15.1:0/205339694 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f51b0003e20 con 0x55c18abd5900 2024-04-26T21:30:47.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.819+0000 7f51ae000640 1 -- 172.21.15.1:0/205339694 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f51b004e050 con 0x55c18abd5900 2024-04-26T21:30:47.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.819+0000 7f51cde5b740 10 client.25550 fetch_fsmap learned FSMap version 26 2024-04-26T21:30:47.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.819+0000 7f51cde5b740 1 -- 172.21.15.1:0/205339694 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c18a84c210 con 0x55c18abd5900 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.819+0000 7f51ae000640 1 -- 172.21.15.1:0/205339694 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51bc004030 con 0x55c18abd5900 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.819+0000 7f51ae000640 1 -- 172.21.15.1:0/205339694 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f51b00314a0 con 0x55c18abd5900 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.819+0000 7f51cde5b740 10 client.25550 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.819+0000 7f51cde5b740 10 client.25550 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.820+0000 7f51cde5b740 20 client.25550 populate_metadata read hostname 'smithi001' 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.820+0000 7f51cde5b740 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c18abeff20 0x55c18abf2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.820+0000 7f51cde5b740 4 client.25550 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.820+0000 7f51cde5b740 1 -- 172.21.15.1:0/205339694 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c18ab47f90 con 0x55c18abeff20 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.820+0000 7f51c9000640 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c18abeff20 0x55c18abf2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.821+0000 7f51ae000640 10 client.25550 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:47.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.821+0000 7f51c9000640 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c18abeff20 0x55c18abf2350 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94690b222712019a server_cookie=55755e5c4337fa7f in_seq=0 out_seq=0 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.821+0000 7f51ae000640 1 -- 172.21.15.1:0/205339694 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c18ab47f90 con 0x55c18abeff20 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.821+0000 7f51ae000640 10 client.25550 handle_command_reply: tid=0 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.821+0000 7f51cde5b740 1 client.25550 shutdown 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 1 -- 172.21.15.1:0/205339694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 msgr2=0x55c18abdd650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abdd650 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f51b0009900 tx=0x7f51b0003820 comp rx=0 tx=0).stop 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 1 -- 172.21.15.1:0/205339694 shutdown_connections 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c18abeff20 0x55c18abf2350 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abddb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abdd650 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51ae000640 0 client.25550 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 1 --2- 172.21.15.1:0/205339694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abdd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 1 -- 172.21.15.1:0/205339694 >> 172.21.15.1:0/205339694 conn(0x55c18abd14f0 msgr2=0x55c18abd7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 1 -- 172.21.15.1:0/205339694 shutdown_connections 2024-04-26T21:30:47.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 1 -- 172.21.15.1:0/205339694 wait complete. 2024-04-26T21:30:47.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.822+0000 7f51cde5b740 20 client.25550 trim_cache size 0 max 16384 2024-04-26T21:30:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.839+0000 7f51cde5b740 1 Processor -- start 2024-04-26T21:30:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.839+0000 7f51cde5b740 1 -- start start 2024-04-26T21:30:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.839+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abebcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.839+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abd8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.840+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abd79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.840+0000 7f51cde5b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41b4790 con 0x55c18abd4fe0 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.840+0000 7f51cde5b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41b2060 con 0x55c18abd5900 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.840+0000 7f51cde5b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41afeb0 con 0x55c18abe01c0 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.840+0000 7f51c9000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abebcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.840+0000 7f51c9000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abebcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40632/0 (socket says 172.21.15.1:40632) 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.840+0000 7f51c9000640 1 -- 172.21.15.1:0/4055939940 learned_addr learned my addr 172.21.15.1:0/4055939940 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.840+0000 7f51c3400640 1 --2- 172.21.15.1:0/4055939940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abd8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.840+0000 7f51c9000640 1 --2- 172.21.15.1:0/4055939940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abd79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.840+0000 7f51c3400640 1 -- 172.21.15.1:0/4055939940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 msgr2=0x55c18abd79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.841+0000 7f51c3400640 1 --2- 172.21.15.1:0/4055939940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abd79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.841+0000 7f51c3400640 1 -- 172.21.15.1:0/4055939940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 msgr2=0x55c18abebcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:47.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.841+0000 7f51c3400640 1 --2- 172.21.15.1:0/4055939940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abebcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.841+0000 7f51c3400640 1 -- 172.21.15.1:0/4055939940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c18a918620 con 0x55c18abd5900 2024-04-26T21:30:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.841+0000 7f51c9000640 1 --2- 172.21.15.1:0/4055939940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abebcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:30:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.841+0000 7f51c3400640 1 --2- 172.21.15.1:0/4055939940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abd8860 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f51b00318a0 tx=0x7f51b002a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50d21c04e11401f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.841+0000 7f51b8c00640 1 -- 172.21.15.1:0/4055939940 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0057070 con 0x55c18abd5900 2024-04-26T21:30:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.841+0000 7f51b8c00640 1 -- 172.21.15.1:0/4055939940 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b0005850 con 0x55c18abd5900 2024-04-26T21:30:47.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.841+0000 7f51b8c00640 1 -- 172.21.15.1:0/4055939940 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0035070 con 0x55c18abd5900 2024-04-26T21:30:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.842+0000 7f51cde5b740 1 -- 172.21.15.1:0/4055939940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 msgr2=0x55c18abd8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.842+0000 7f51cde5b740 1 --2- 172.21.15.1:0/4055939940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abd8860 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f51b00318a0 tx=0x7f51b002a420 comp rx=0 tx=0).stop 2024-04-26T21:30:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.842+0000 7f51cde5b740 1 -- 172.21.15.1:0/4055939940 shutdown_connections 2024-04-26T21:30:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.842+0000 7f51cde5b740 1 --2- 172.21.15.1:0/4055939940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abe01c0 0x55c18abd79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.842+0000 7f51cde5b740 1 --2- 172.21.15.1:0/4055939940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abd5900 0x55c18abd8860 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.842+0000 7f51cde5b740 1 --2- 172.21.15.1:0/4055939940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abebcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.842+0000 7f51cde5b740 1 -- 172.21.15.1:0/4055939940 >> 172.21.15.1:0/4055939940 conn(0x55c18abd14f0 msgr2=0x55c18abde830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:47.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.842+0000 7f51cde5b740 1 -- 172.21.15.1:0/4055939940 shutdown_connections 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.842+0000 7f51cde5b740 1 -- 172.21.15.1:0/4055939940 wait complete. 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.843+0000 7f51cde5b740 1 -- start start 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.843+0000 7f51cde5b740 1 Processor -- start 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.843+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abe85e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.843+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abd5900 0x55c18abe3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.843+0000 7f51cde5b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abe01c0 0x55c18abe3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.843+0000 7f51cde5b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41b0520 con 0x55c18abd4fe0 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.843+0000 7f51cde5b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41b1990 con 0x55c18abe01c0 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.843+0000 7f51cde5b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51c41b0bf0 con 0x55c18abd5900 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.843+0000 7f51cde5b740 10 client.0 _ll_register_callbacks cb 0x7f51b0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.844+0000 7f51c9000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abe85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.844+0000 7f51c3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abd5900 0x55c18abe3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.844+0000 7f51c9000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abe85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40642/0 (socket says 172.21.15.1:40642) 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.844+0000 7f51c9000640 1 -- 172.21.15.1:0/1181701277 learned_addr learned my addr 172.21.15.1:0/1181701277 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.844+0000 7f51c3400640 1 -- 172.21.15.1:0/1181701277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abe01c0 msgr2=0x55c18abe3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:47.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.844+0000 7f51c3400640 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abe01c0 0x55c18abe3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.844+0000 7f51c3400640 1 -- 172.21.15.1:0/1181701277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 msgr2=0x55c18abe85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:30:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.844+0000 7f51c3400640 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abe85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.844+0000 7f51c3400640 1 -- 172.21.15.1:0/1181701277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c18aaf4cb0 con 0x55c18abd5900 2024-04-26T21:30:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.845+0000 7f51c3400640 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abd5900 0x55c18abe3950 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55c18abdcce0 tx=0x7f51b0031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8de02592b60730eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:30:47.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.845+0000 7f51ae000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:30:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.845+0000 7f51ae000640 1 -- 172.21.15.1:0/1181701277 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0057070 con 0x55c18abd5900 2024-04-26T21:30:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.845+0000 7f51cde5b740 1 -- 172.21.15.1:0/1181701277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51b0005850 con 0x55c18abd5900 2024-04-26T21:30:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.845+0000 7f51ae000640 1 -- 172.21.15.1:0/1181701277 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f51b0005250 con 0x55c18abd5900 2024-04-26T21:30:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.845+0000 7f51ae000640 1 -- 172.21.15.1:0/1181701277 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f51b0064050 con 0x55c18abd5900 2024-04-26T21:30:47.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.845+0000 7f51cde5b740 10 client.25537 fetch_fsmap learned FSMap version 26 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.845+0000 7f51cde5b740 1 -- 172.21.15.1:0/1181701277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c18ac56350 con 0x55c18abd5900 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.846+0000 7f51ae000640 1 -- 172.21.15.1:0/1181701277 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b0005850 con 0x55c18abd5900 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.846+0000 7f51ae000640 1 -- 172.21.15.1:0/1181701277 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f51b0004cb0 con 0x55c18abd5900 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.846+0000 7f51cde5b740 10 client.25537 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.846+0000 7f51cde5b740 10 client.25537 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.846+0000 7f51cde5b740 20 client.25537 populate_metadata read hostname 'smithi001' 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.846+0000 7f51cde5b740 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c18ac56960 0x55c18ac58da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.846+0000 7f51cde5b740 4 client.25537 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.846+0000 7f51cde5b740 1 -- 172.21.15.1:0/1181701277 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c18ab41e00 con 0x55c18ac56960 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.846+0000 7f51c9000640 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c18ac56960 0x55c18ac58da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.847+0000 7f51ae000640 10 client.25537 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:47.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.847+0000 7f51c9000640 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c18ac56960 0x55c18ac58da0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2f2d6fe7e3a2c68 server_cookie=b204a40fd54e606b in_seq=0 out_seq=0 2024-04-26T21:30:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.851+0000 7f51ae000640 1 -- 172.21.15.1:0/1181701277 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c18ab41e00 con 0x55c18ac56960 2024-04-26T21:30:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.851+0000 7f51ae000640 10 client.25537 handle_command_reply: tid=0 2024-04-26T21:30:47.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.851+0000 7f51cde5b740 1 client.25537 shutdown 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.851+0000 7f51cde5b740 1 -- 172.21.15.1:0/1181701277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abd5900 msgr2=0x55c18abe3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.851+0000 7f51cde5b740 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abd5900 0x55c18abe3950 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x55c18abdcce0 tx=0x7f51b0031400 comp rx=0 tx=0).stop 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.851+0000 7f51cde5b740 1 -- 172.21.15.1:0/1181701277 shutdown_connections 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.851+0000 7f51cde5b740 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c18ac56960 0x55c18ac58da0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.851+0000 7f51cde5b740 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c18abd5900 0x55c18abe3950 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.851+0000 7f51cde5b740 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18abe01c0 0x55c18abe3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.852+0000 7f51cde5b740 1 --2- 172.21.15.1:0/1181701277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c18abd4fe0 0x55c18abe85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.852+0000 7f51cde5b740 1 -- 172.21.15.1:0/1181701277 >> 172.21.15.1:0/1181701277 conn(0x55c18abd14f0 msgr2=0x55c18abdd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.852+0000 7f51ae000640 0 client.25537 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.852+0000 7f51cde5b740 1 -- 172.21.15.1:0/1181701277 shutdown_connections 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.852+0000 7f51cde5b740 1 -- 172.21.15.1:0/1181701277 wait complete. 2024-04-26T21:30:47.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.852+0000 7f51cde5b740 20 client.25537 trim_cache size 0 max 16384 2024-04-26T21:30:47.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:30:47.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:30:47.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:30:47.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:30:47.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 -- 172.21.15.1:0/496914481 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51a80a3e10 msgr2=0x7f51a80a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:47.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51a80a3e10 0x7f51a80a62d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f51b4099010 tx=0x7f51b40d4680 comp rx=0 tx=0).stop 2024-04-26T21:30:47.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 -- 172.21.15.1:0/496914481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c4104970 msgr2=0x7f51c419b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:30:47.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c4104970 0x7f51c419b500 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f51b400e9f0 tx=0x7f51b400edf0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 -- 172.21.15.1:0/496914481 shutdown_connections 2024-04-26T21:30:47.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51a80a3e10 0x7f51a80a62d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51c410a980 0x7f51c419fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51c4105330 0x7f51c419ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 --2- 172.21.15.1:0/496914481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c4104970 0x7f51c419b500 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:30:47.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 -- 172.21.15.1:0/496914481 >> 172.21.15.1:0/496914481 conn(0x7f51c4100440 msgr2=0x7f51c41012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:30:47.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 -- 172.21.15.1:0/496914481 shutdown_connections 2024-04-26T21:30:47.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:30:47.853+0000 7f51c9a00640 1 -- 172.21.15.1:0/496914481 wait complete. 2024-04-26T21:30:48.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:48 smithi001 ceph-mon[28054]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-26T21:30:48.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:48 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:30:48.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:48 smithi038 ceph-mon[32724]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-26T21:30:48.379 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:48 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:30:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:48 smithi186 ceph-mon[32991]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-04-26T21:30:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:48 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:30:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:50 smithi186 ceph-mon[32991]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-04-26T21:30:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:50 smithi001 ceph-mon[28054]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-04-26T21:30:50.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:50 smithi038 ceph-mon[32724]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-04-26T21:30:51.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:51 smithi186 ceph-mon[32991]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-04-26T21:30:51.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:51 smithi001 ceph-mon[28054]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-04-26T21:30:51.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:51 smithi038 ceph-mon[32724]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-04-26T21:30:53.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:53 smithi186 ceph-mon[32991]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-26T21:30:54.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:53 smithi001 ceph-mon[28054]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-26T21:30:54.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:53 smithi038 ceph-mon[32724]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-04-26T21:30:55.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:55 smithi186 ceph-mon[32991]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T21:30:56.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:55 smithi001 ceph-mon[28054]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T21:30:56.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:55 smithi038 ceph-mon[32724]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-04-26T21:30:57.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:57 smithi186 ceph-mon[32991]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-04-26T21:30:58.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:57 smithi001 ceph-mon[28054]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-04-26T21:30:58.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:57 smithi038 ceph-mon[32724]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-04-26T21:30:59.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:30:59 smithi186 ceph-mon[32991]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-26T21:31:00.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:30:59 smithi001 ceph-mon[28054]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-26T21:31:00.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:30:59 smithi038 ceph-mon[32724]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-26T21:31:00.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:31:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:31:00.248+0000 7fdaad200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:31:00.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:31:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:31:00.248+0000 7fdaad200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:31:00.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:31:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:31:00.248+0000 7fdaad200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:31:00.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:31:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:31:00.248+0000 7fdaad200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:31:00.568 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:31:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:31:00.248+0000 7fdaad200700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-26T21:31:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.003+0000 7fd207000640 1 -- 172.21.15.1:0/356886196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd20010e2a0 msgr2=0x7fd200115e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.003+0000 7fd207000640 1 --2- 172.21.15.1:0/356886196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd20010e2a0 0x7fd200115e70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009ca0 tx=0x7fd1f802a8c0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.003+0000 7fd207000640 1 -- 172.21.15.1:0/356886196 shutdown_connections 2024-04-26T21:31:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.003+0000 7fd207000640 1 --2- 172.21.15.1:0/356886196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd200109b80 0x7fd20010dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.003+0000 7fd207000640 1 --2- 172.21.15.1:0/356886196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd20010e2a0 0x7fd200115e70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.003+0000 7fd207000640 1 --2- 172.21.15.1:0/356886196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd2001091d0 0x7fd2001095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.003+0000 7fd207000640 1 -- 172.21.15.1:0/356886196 >> 172.21.15.1:0/356886196 conn(0x7fd200076e40 msgr2=0x7fd200079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:01.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.003+0000 7fd207000640 1 -- 172.21.15.1:0/356886196 shutdown_connections 2024-04-26T21:31:01.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.003+0000 7fd207000640 1 -- 172.21.15.1:0/356886196 wait complete. 2024-04-26T21:31:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.004+0000 7fd207000640 1 Processor -- start 2024-04-26T21:31:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.004+0000 7fd207000640 1 -- start start 2024-04-26T21:31:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd207000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2001091d0 0x7fd2001a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd207000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd20010e2a0 0x7fd2001a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd207000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20019f590 0x7fd20019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd207000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd200119210 con 0x7fd20010e2a0 2024-04-26T21:31:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd207000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd200119090 con 0x7fd20019f590 2024-04-26T21:31:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd207000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd200119390 con 0x7fd2001091d0 2024-04-26T21:31:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd1f7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd20010e2a0 0x7fd2001a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd1f7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd20010e2a0 0x7fd2001a5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35746/0 (socket says 172.21.15.1:35746) 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd1f7e00640 1 -- 172.21.15.1:0/1822620514 learned_addr learned my addr 172.21.15.1:0/1822620514 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2001091d0 0x7fd2001a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd205000640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20019f590 0x7fd20019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd1f7e00640 1 -- 172.21.15.1:0/1822620514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2001091d0 msgr2=0x7fd2001a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd1f7e00640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2001091d0 0x7fd2001a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd1f7e00640 1 -- 172.21.15.1:0/1822620514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20019f590 msgr2=0x7fd20019fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd1f7e00640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20019f590 0x7fd20019fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd1f7e00640 1 -- 172.21.15.1:0/1822620514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2001a00b0 con 0x7fd20010e2a0 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.005+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2001091d0 0x7fd2001a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.006+0000 7fd1f7e00640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd20010e2a0 0x7fd2001a5920 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd1e800e9f0 tx=0x7fd1e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.006+0000 7fd1fd600640 1 -- 172.21.15.1:0/1822620514 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e800ccc0 con 0x7fd20010e2a0 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.006+0000 7fd1fd600640 1 -- 172.21.15.1:0/1822620514 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1e80054d0 con 0x7fd20010e2a0 2024-04-26T21:31:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.006+0000 7fd207000640 1 -- 172.21.15.1:0/1822620514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2001a03a0 con 0x7fd20010e2a0 2024-04-26T21:31:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.006+0000 7fd1fd600640 1 -- 172.21.15.1:0/1822620514 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e8010430 con 0x7fd20010e2a0 2024-04-26T21:31:01.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.006+0000 7fd207000640 1 -- 172.21.15.1:0/1822620514 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2001ac5a0 con 0x7fd20010e2a0 2024-04-26T21:31:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.009+0000 7fd209e17740 1 Processor -- start 2024-04-26T21:31:01.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.010+0000 7fd1fd600640 1 -- 172.21.15.1:0/1822620514 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7fd1e80046c0 con 0x7fd20010e2a0 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.010+0000 7fd209e17740 1 -- start start 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.010+0000 7fd209e17740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d534dd0 0x55a71d5351b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.010+0000 7fd209e17740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d5356f0 0x55a71d53fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.010+0000 7fd209e17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d5423a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.010+0000 7fd209e17740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a71d49a380 con 0x55a71d534dd0 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.010+0000 7fd209e17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a71d436c30 con 0x55a71d53ffb0 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.010+0000 7fd209e17740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a71d496340 con 0x55a71d5356f0 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd205000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d534dd0 0x55a71d5351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd205000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d534dd0 0x55a71d5351b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35762/0 (socket says 172.21.15.1:35762) 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd205000640 1 -- 172.21.15.1:0/4119630558 learned_addr learned my addr 172.21.15.1:0/4119630558 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd1f7e00640 1 --2- 172.21.15.1:0/4119630558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d5423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/4119630558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d5356f0 0x55a71d53fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd205000640 1 -- 172.21.15.1:0/4119630558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d5356f0 msgr2=0x55a71d53fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:01.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd205000640 1 --2- 172.21.15.1:0/4119630558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d5356f0 0x55a71d53fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd205000640 1 -- 172.21.15.1:0/4119630558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 msgr2=0x55a71d5423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd205000640 1 --2- 172.21.15.1:0/4119630558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d5423a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd205000640 1 -- 172.21.15.1:0/4119630558 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a71d458940 con 0x55a71d534dd0 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.011+0000 7fd1f7e00640 1 --2- 172.21.15.1:0/4119630558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d5423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.012+0000 7fd205000640 1 --2- 172.21.15.1:0/4119630558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d534dd0 0x55a71d5351b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009410 tx=0x7fd1f8005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e11aa40bb3ada49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.012+0000 7fd1fd600640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1e40a3e30 0x7fd1e40a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.012+0000 7fd1f5600640 1 -- 172.21.15.1:0/4119630558 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8031cf0 con 0x55a71d534dd0 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.012+0000 7fd1f5600640 1 -- 172.21.15.1:0/4119630558 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1f8036030 con 0x55a71d534dd0 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.012+0000 7fd1f5600640 1 -- 172.21.15.1:0/4119630558 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f80042a0 con 0x55a71d534dd0 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.012+0000 7fd1fd600640 1 -- 172.21.15.1:0/1822620514 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fd1e80d1250 con 0x7fd20010e2a0 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.012+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1e40a3e30 0x7fd1e40a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd209e17740 1 -- 172.21.15.1:0/4119630558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d534dd0 msgr2=0x55a71d5351b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd209e17740 1 --2- 172.21.15.1:0/4119630558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d534dd0 0x55a71d5351b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009410 tx=0x7fd1f8005ef0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1e40a3e30 0x7fd1e40a62f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x55a71d536370 tx=0x7fd1f0005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:01.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd209e17740 1 -- 172.21.15.1:0/4119630558 shutdown_connections 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd209e17740 1 --2- 172.21.15.1:0/4119630558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d5356f0 0x55a71d53fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd209e17740 1 --2- 172.21.15.1:0/4119630558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d5423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd209e17740 1 --2- 172.21.15.1:0/4119630558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d534dd0 0x55a71d5351b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd209e17740 1 -- 172.21.15.1:0/4119630558 >> 172.21.15.1:0/4119630558 conn(0x55a71d5312e0 msgr2=0x55a71d5316c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd209e17740 1 -- 172.21.15.1:0/4119630558 shutdown_connections 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd209e17740 1 -- 172.21.15.1:0/4119630558 wait complete. 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.013+0000 7fd209e17740 1 -- start start 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.014+0000 7fd209e17740 1 Processor -- start 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.014+0000 7fd209e17740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 0x55a71d53ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.014+0000 7fd209e17740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d53d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.014+0000 7fd209e17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d53d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.014+0000 7fd209e17740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a71d542a00 con 0x55a71d5356f0 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.014+0000 7fd209e17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a71d421c60 con 0x55a71d53ffb0 2024-04-26T21:31:01.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.014+0000 7fd209e17740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a71d542b80 con 0x55a71d534dd0 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.014+0000 7fd209e17740 10 client.0 _ll_register_callbacks cb 0x55a71d44de50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.014+0000 7fd1f7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d53d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd1f7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d53d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35764/0 (socket says 172.21.15.1:35764) 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd1f7e00640 1 -- 172.21.15.1:0/74798266 learned_addr learned my addr 172.21.15.1:0/74798266 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd205000640 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d53d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 0x55a71d53ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd205000640 1 -- 172.21.15.1:0/74798266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 msgr2=0x55a71d53ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd205000640 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 0x55a71d53ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd205000640 1 -- 172.21.15.1:0/74798266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 msgr2=0x55a71d53d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd205000640 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d53d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd205000640 1 -- 172.21.15.1:0/74798266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a71d430510 con 0x55a71d53ffb0 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd1f7e00640 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d53d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd205000640 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d53d930 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8006010 tx=0x7fd1f8005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=168466d3c2e48e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.015+0000 7fd1ee000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.016+0000 7fd1ee000640 1 -- 172.21.15.1:0/74798266 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f804a040 con 0x55a71d53ffb0 2024-04-26T21:31:01.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.016+0000 7fd209e17740 1 -- 172.21.15.1:0/74798266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1f8036030 con 0x55a71d53ffb0 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.016+0000 7fd1ee000640 1 -- 172.21.15.1:0/74798266 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fd1f8031650 con 0x55a71d53ffb0 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.016+0000 7fd1ee000640 1 -- 172.21.15.1:0/74798266 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd1f805e050 con 0x55a71d53ffb0 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.016+0000 7fd209e17740 10 client.25571 fetch_fsmap learned FSMap version 26 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.016+0000 7fd209e17740 1 -- 172.21.15.1:0/74798266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a71d1ac210 con 0x55a71d53ffb0 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.016+0000 7fd1ee000640 1 -- 172.21.15.1:0/74798266 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1f8036030 con 0x55a71d53ffb0 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.017+0000 7fd1ee000640 1 -- 172.21.15.1:0/74798266 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd1f804d030 con 0x55a71d53ffb0 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.017+0000 7fd209e17740 10 client.25571 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.017+0000 7fd209e17740 10 client.25571 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.017+0000 7fd209e17740 20 client.25571 populate_metadata read hostname 'smithi001' 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.017+0000 7fd209e17740 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a71d54f920 0x55a71d551d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.017+0000 7fd209e17740 4 client.25571 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.017+0000 7fd209e17740 1 -- 172.21.15.1:0/74798266 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a71d4a7f90 con 0x55a71d54f920 2024-04-26T21:31:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.017+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a71d54f920 0x55a71d551d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.018+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a71d54f920 0x55a71d551d10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca181619be48cfa8 server_cookie=ca93d1dc33219de0 in_seq=0 out_seq=0 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.018+0000 7fd1ee000640 10 client.25571 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd1ee000640 1 -- 172.21.15.1:0/74798266 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a71d4a7f90 con 0x55a71d54f920 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd1ee000640 10 client.25571 handle_command_reply: tid=0 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd209e17740 1 client.25571 shutdown 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd209e17740 1 -- 172.21.15.1:0/74798266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 msgr2=0x55a71d53d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd209e17740 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d53d930 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8006010 tx=0x7fd1f8005c50 comp rx=0 tx=0).stop 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd209e17740 1 -- 172.21.15.1:0/74798266 shutdown_connections 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd209e17740 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a71d54f920 0x55a71d551d10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd209e17740 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d53d930 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd209e17740 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d53d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd1ee000640 0 client.25571 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:01.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd209e17740 1 --2- 172.21.15.1:0/74798266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 0x55a71d53ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd209e17740 1 -- 172.21.15.1:0/74798266 >> 172.21.15.1:0/74798266 conn(0x55a71d5312e0 msgr2=0x55a71d537010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:01.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.019+0000 7fd209e17740 1 -- 172.21.15.1:0/74798266 shutdown_connections 2024-04-26T21:31:01.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.020+0000 7fd209e17740 1 -- 172.21.15.1:0/74798266 wait complete. 2024-04-26T21:31:01.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.020+0000 7fd209e17740 20 client.25571 trim_cache size 0 max 16384 2024-04-26T21:31:01.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.036+0000 7fd209e17740 1 Processor -- start 2024-04-26T21:31:01.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.036+0000 7fd209e17740 1 -- start start 2024-04-26T21:31:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.036+0000 7fd209e17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d534dd0 0x55a71d53f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.036+0000 7fd209e17740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d54b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.036+0000 7fd209e17740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d53ffb0 0x55a71d53c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd209e17740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd2001b8bf0 con 0x55a71d5356f0 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd209e17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd2001b64c0 con 0x55a71d534dd0 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd209e17740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd2001b4310 con 0x55a71d53ffb0 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd1ffe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d534dd0 0x55a71d53f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd205000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d53ffb0 0x55a71d53c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd1ffe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d534dd0 0x55a71d53f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:56974/0 (socket says 172.21.15.1:56974) 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd1ffe00640 1 -- 172.21.15.1:0/696872582 learned_addr learned my addr 172.21.15.1:0/696872582 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd1f7e00640 1 --2- 172.21.15.1:0/696872582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d54b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd205000640 1 -- 172.21.15.1:0/696872582 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d534dd0 msgr2=0x55a71d53f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd205000640 1 --2- 172.21.15.1:0/696872582 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d534dd0 0x55a71d53f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd205000640 1 -- 172.21.15.1:0/696872582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 msgr2=0x55a71d54b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd205000640 1 --2- 172.21.15.1:0/696872582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d54b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.037+0000 7fd205000640 1 -- 172.21.15.1:0/696872582 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a71d535520 con 0x55a71d53ffb0 2024-04-26T21:31:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.038+0000 7fd205000640 1 --2- 172.21.15.1:0/696872582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d53ffb0 0x55a71d53c050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009620 tx=0x7fd1f80093c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ea7633d242ca07 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.038+0000 7fd1f4c00640 1 -- 172.21.15.1:0/696872582 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f8030750 con 0x55a71d53ffb0 2024-04-26T21:31:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.038+0000 7fd1f4c00640 1 -- 172.21.15.1:0/696872582 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1f8036040 con 0x55a71d53ffb0 2024-04-26T21:31:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.038+0000 7fd1f4c00640 1 -- 172.21.15.1:0/696872582 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f802f660 con 0x55a71d53ffb0 2024-04-26T21:31:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.038+0000 7fd209e17740 1 -- 172.21.15.1:0/696872582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d53ffb0 msgr2=0x55a71d53c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.038+0000 7fd209e17740 1 --2- 172.21.15.1:0/696872582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d53ffb0 0x55a71d53c050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009620 tx=0x7fd1f80093c0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.038+0000 7fd209e17740 1 -- 172.21.15.1:0/696872582 shutdown_connections 2024-04-26T21:31:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.039+0000 7fd209e17740 1 --2- 172.21.15.1:0/696872582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d53ffb0 0x55a71d53c050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.039+0000 7fd209e17740 1 --2- 172.21.15.1:0/696872582 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d54b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.039+0000 7fd209e17740 1 --2- 172.21.15.1:0/696872582 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d534dd0 0x55a71d53f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.039+0000 7fd209e17740 1 -- 172.21.15.1:0/696872582 >> 172.21.15.1:0/696872582 conn(0x55a71d5312e0 msgr2=0x55a71d53ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.039+0000 7fd209e17740 1 -- 172.21.15.1:0/696872582 shutdown_connections 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.039+0000 7fd209e17740 1 -- 172.21.15.1:0/696872582 wait complete. 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.039+0000 7fd209e17740 1 -- start start 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.039+0000 7fd209e17740 1 Processor -- start 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd209e17740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 0x55a71d539c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd209e17740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d53a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd209e17740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d53a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd209e17740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd2001b4980 con 0x55a71d5356f0 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd209e17740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd2001b5df0 con 0x55a71d53ffb0 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd209e17740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd2001b5050 con 0x55a71d534dd0 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd209e17740 10 client.0 _ll_register_callbacks cb 0x7fd1e80d4bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd1ffe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 0x55a71d539c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd1ffe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 0x55a71d539c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:40354/0 (socket says 172.21.15.1:40354) 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd1ffe00640 1 -- 172.21.15.1:0/585232683 learned_addr learned my addr 172.21.15.1:0/585232683 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd1f7e00640 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d53a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd205000640 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d53a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd1ffe00640 1 -- 172.21.15.1:0/585232683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 msgr2=0x55a71d53a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d53a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd1ffe00640 1 -- 172.21.15.1:0/585232683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 msgr2=0x55a71d53a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d53a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.040+0000 7fd1ffe00640 1 -- 172.21.15.1:0/585232683 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a71d278620 con 0x55a71d534dd0 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.041+0000 7fd1ffe00640 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 0x55a71d539c90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000faa0 tx=0x7fd1f00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96941e9fd27f2040 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.041+0000 7fd1ee000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.041+0000 7fd1ee000640 1 -- 172.21.15.1:0/585232683 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0004020 con 0x55a71d534dd0 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.041+0000 7fd209e17740 1 -- 172.21.15.1:0/585232683 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1f8036040 con 0x55a71d534dd0 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.041+0000 7fd1ee000640 1 -- 172.21.15.1:0/585232683 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fd1f00041c0 con 0x55a71d534dd0 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.041+0000 7fd1ee000640 1 -- 172.21.15.1:0/585232683 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd1f0039050 con 0x55a71d534dd0 2024-04-26T21:31:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.041+0000 7fd209e17740 10 client.25561 fetch_fsmap learned FSMap version 26 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.041+0000 7fd209e17740 1 -- 172.21.15.1:0/585232683 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a71d454cb0 con 0x55a71d534dd0 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.042+0000 7fd1ee000640 1 -- 172.21.15.1:0/585232683 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1f8036040 con 0x55a71d534dd0 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.042+0000 7fd1ee000640 1 -- 172.21.15.1:0/585232683 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fd1f0004b70 con 0x55a71d534dd0 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.042+0000 7fd209e17740 10 client.25561 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.042+0000 7fd209e17740 10 client.25561 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.042+0000 7fd209e17740 20 client.25561 populate_metadata read hostname 'smithi001' 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.042+0000 7fd209e17740 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a71d5b6520 0x55a71d5b8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.042+0000 7fd209e17740 4 client.25561 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.042+0000 7fd209e17740 1 -- 172.21.15.1:0/585232683 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a71d4a1e00 con 0x55a71d5b6520 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.042+0000 7fd1f7e00640 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a71d5b6520 0x55a71d5b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.043+0000 7fd1f7e00640 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a71d5b6520 0x55a71d5b8940 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbcc44fa705b8172 server_cookie=4938a204a5487481 in_seq=0 out_seq=0 2024-04-26T21:31:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.043+0000 7fd1ee000640 10 client.25561 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:01.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.271+0000 7fd1ee000640 1 -- 172.21.15.1:0/585232683 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a71d4a1e00 con 0x55a71d5b6520 2024-04-26T21:31:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.271+0000 7fd1ee000640 10 client.25561 handle_command_reply: tid=0 2024-04-26T21:31:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.271+0000 7fd209e17740 1 client.25561 shutdown 2024-04-26T21:31:01.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.271+0000 7fd209e17740 1 -- 172.21.15.1:0/585232683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 msgr2=0x55a71d539c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.271+0000 7fd209e17740 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 0x55a71d539c90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000faa0 tx=0x7fd1f00046c0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.272+0000 7fd209e17740 1 -- 172.21.15.1:0/585232683 shutdown_connections 2024-04-26T21:31:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.272+0000 7fd209e17740 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a71d5b6520 0x55a71d5b8940 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.272+0000 7fd209e17740 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a71d53ffb0 0x55a71d53a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.272+0000 7fd209e17740 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a71d5356f0 0x55a71d53a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.272+0000 7fd209e17740 1 --2- 172.21.15.1:0/585232683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a71d534dd0 0x55a71d539c90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.272+0000 7fd209e17740 1 -- 172.21.15.1:0/585232683 >> 172.21.15.1:0/585232683 conn(0x55a71d5312e0 msgr2=0x55a71d53bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.272+0000 7fd1ee000640 0 client.25561 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.272+0000 7fd209e17740 1 -- 172.21.15.1:0/585232683 shutdown_connections 2024-04-26T21:31:01.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.272+0000 7fd209e17740 1 -- 172.21.15.1:0/585232683 wait complete. 2024-04-26T21:31:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.272+0000 7fd209e17740 20 client.25561 trim_cache size 0 max 16384 2024-04-26T21:31:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:31:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:31:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:31:01.275 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:31:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.274+0000 7fd207000640 1 -- 172.21.15.1:0/1822620514 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1e40a3e30 msgr2=0x7fd1e40a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.274+0000 7fd207000640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1e40a3e30 0x7fd1e40a62f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x55a71d536370 tx=0x7fd1f0005f90 comp rx=0 tx=0).stop 2024-04-26T21:31:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.274+0000 7fd207000640 1 -- 172.21.15.1:0/1822620514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd20010e2a0 msgr2=0x7fd2001a5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.274+0000 7fd207000640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd20010e2a0 0x7fd2001a5920 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd1e800e9f0 tx=0x7fd1e800eeb0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.276 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.274+0000 7fd207000640 1 -- 172.21.15.1:0/1822620514 shutdown_connections 2024-04-26T21:31:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.274+0000 7fd207000640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1e40a3e30 0x7fd1e40a62f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.274+0000 7fd207000640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd20019f590 0x7fd20019fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.274+0000 7fd207000640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd20010e2a0 0x7fd2001a5920 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.274+0000 7fd207000640 1 --2- 172.21.15.1:0/1822620514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2001091d0 0x7fd2001a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.274+0000 7fd207000640 1 -- 172.21.15.1:0/1822620514 >> 172.21.15.1:0/1822620514 conn(0x7fd200076e40 msgr2=0x7fd200079170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.275+0000 7fd207000640 1 -- 172.21.15.1:0/1822620514 shutdown_connections 2024-04-26T21:31:01.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:01.275+0000 7fd207000640 1 -- 172.21.15.1:0/1822620514 wait complete. 2024-04-26T21:31:01.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:01 smithi186 ceph-mon[32991]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-26T21:31:01.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:01 smithi186 ceph-mon[32991]: mgrmap e25: x(active, since 42s), standbys: z, y 2024-04-26T21:31:02.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:01 smithi001 ceph-mon[28054]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-26T21:31:02.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:01 smithi001 ceph-mon[28054]: mgrmap e25: x(active, since 42s), standbys: z, y 2024-04-26T21:31:02.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:01 smithi038 ceph-mon[32724]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-26T21:31:02.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:01 smithi038 ceph-mon[32724]: mgrmap e25: x(active, since 42s), standbys: z, y 2024-04-26T21:31:03.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:03 smithi186 ceph-mon[32991]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T21:31:04.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:03 smithi001 ceph-mon[28054]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T21:31:04.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:03 smithi038 ceph-mon[32724]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 5 op/s 2024-04-26T21:31:06.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:05 smithi001 ceph-mon[28054]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-26T21:31:06.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:05 smithi038 ceph-mon[32724]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-26T21:31:06.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:05 smithi186 ceph-mon[32991]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 12 KiB/s wr, 9 op/s 2024-04-26T21:31:08.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:07 smithi001 ceph-mon[28054]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-26T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:07 smithi038 ceph-mon[32724]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-26T21:31:08.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:07 smithi186 ceph-mon[32991]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-26T21:31:10.007 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:09 smithi186 ceph-mon[32991]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-26T21:31:10.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:09 smithi001 ceph-mon[28054]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-26T21:31:10.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:09 smithi038 ceph-mon[32724]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-04-26T21:31:12.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:11 smithi001 ceph-mon[28054]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-26T21:31:12.078 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:11 smithi038 ceph-mon[32724]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-26T21:31:12.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:11 smithi186 ceph-mon[32991]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-26T21:31:14.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:13 smithi001 ceph-mon[28054]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-26T21:31:14.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:13 smithi038 ceph-mon[32724]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-26T21:31:14.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:13 smithi186 ceph-mon[32991]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-26T21:31:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.429+0000 7f3c80a00640 1 -- 172.21.15.1:0/1751042901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c106110 msgr2=0x7f3c7c10dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.429+0000 7f3c80a00640 1 --2- 172.21.15.1:0/1751042901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c106110 0x7f3c7c10dce0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009ca0 tx=0x7f3c7402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.429+0000 7f3c80a00640 1 -- 172.21.15.1:0/1751042901 shutdown_connections 2024-04-26T21:31:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.430+0000 7f3c80a00640 1 --2- 172.21.15.1:0/1751042901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c7c101c70 0x7f3c7c105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.430+0000 7f3c80a00640 1 --2- 172.21.15.1:0/1751042901 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c106110 0x7f3c7c10dce0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.430+0000 7f3c80a00640 1 --2- 172.21.15.1:0/1751042901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c7c1012c0 0x7f3c7c1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.430+0000 7f3c80a00640 1 -- 172.21.15.1:0/1751042901 >> 172.21.15.1:0/1751042901 conn(0x7f3c7c0769a0 msgr2=0x7f3c7c076db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.430+0000 7f3c80a00640 1 -- 172.21.15.1:0/1751042901 shutdown_connections 2024-04-26T21:31:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.430+0000 7f3c80a00640 1 -- 172.21.15.1:0/1751042901 wait complete. 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.431+0000 7f3c80a00640 1 Processor -- start 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.431+0000 7f3c80a00640 1 -- start start 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.431+0000 7f3c80a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c1012c0 0x7f3c7c19b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.431+0000 7f3c80a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c7c106110 0x7f3c7c19b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.431+0000 7f3c80a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c7c19fa10 0x7f3c7c19fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c80a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c110840 con 0x7f3c7c1012c0 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c80a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c1106c0 con 0x7f3c7c106110 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c80a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c1109c0 con 0x7f3c7c19fa10 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c7a800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c1012c0 0x7f3c7c19b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c7a800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c1012c0 0x7f3c7c19b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56466/0 (socket says 172.21.15.1:56466) 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c7a800640 1 -- 172.21.15.1:0/1630706969 learned_addr learned my addr 172.21.15.1:0/1630706969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c7b200640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c7c19fa10 0x7f3c7c19fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c73e00640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c7c106110 0x7f3c7c19b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c7a800640 1 -- 172.21.15.1:0/1630706969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c7c19fa10 msgr2=0x7f3c7c19fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c7a800640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c7c19fa10 0x7f3c7c19fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c7a800640 1 -- 172.21.15.1:0/1630706969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c7c106110 msgr2=0x7f3c7c19b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c7a800640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c7c106110 0x7f3c7c19b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.432+0000 7f3c7a800640 1 -- 172.21.15.1:0/1630706969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c7c1a0500 con 0x7f3c7c1012c0 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.433+0000 7f3c7a800640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c1012c0 0x7f3c7c19b150 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00e9f0 tx=0x7f3c6c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.433+0000 7f3c73400640 1 -- 172.21.15.1:0/1630706969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c00cd80 con 0x7f3c7c1012c0 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.433+0000 7f3c80a00640 1 -- 172.21.15.1:0/1630706969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c7c1a07f0 con 0x7f3c7c1012c0 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.433+0000 7f3c73400640 1 -- 172.21.15.1:0/1630706969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c6c0108e0 con 0x7f3c7c1012c0 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.433+0000 7f3c73400640 1 -- 172.21.15.1:0/1630706969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c018980 con 0x7f3c7c1012c0 2024-04-26T21:31:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.433+0000 7f3c80a00640 1 -- 172.21.15.1:0/1630706969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c7c1a7fe0 con 0x7f3c7c1012c0 2024-04-26T21:31:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.436+0000 7f3c8423f740 1 Processor -- start 2024-04-26T21:31:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.436+0000 7f3c8423f740 1 -- start start 2024-04-26T21:31:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.436+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 0x5576179b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b1900 0x5576179bbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179bc1c0 0x5576179be5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c8423f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x557617916380 con 0x5576179b0fe0 2024-04-26T21:31:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c8423f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5576178b2c30 con 0x5576179bc1c0 2024-04-26T21:31:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c8423f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557617912340 con 0x5576179b1900 2024-04-26T21:31:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c73400640 1 -- 172.21.15.1:0/1630706969 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f3c6c010430 con 0x7f3c7c1012c0 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c7b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 0x5576179b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c7b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 0x5576179b13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56474/0 (socket says 172.21.15.1:56474) 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c7b200640 1 -- 172.21.15.1:0/1830104018 learned_addr learned my addr 172.21.15.1:0/1830104018 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c73e00640 1 --2- 172.21.15.1:0/1830104018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179bc1c0 0x5576179be5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c7b200640 1 -- 172.21.15.1:0/1830104018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b1900 msgr2=0x5576179bbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c7b200640 1 --2- 172.21.15.1:0/1830104018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b1900 0x5576179bbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c7b200640 1 -- 172.21.15.1:0/1830104018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179bc1c0 msgr2=0x5576179be5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c7b200640 1 --2- 172.21.15.1:0/1830104018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179bc1c0 0x5576179be5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c7b200640 1 -- 172.21.15.1:0/1830104018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576178d4940 con 0x5576179b0fe0 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.437+0000 7f3c73e00640 1 --2- 172.21.15.1:0/1830104018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179bc1c0 0x5576179be5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:31:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.438+0000 7f3c7b200640 1 --2- 172.21.15.1:0/1830104018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 0x5576179b13c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009ca0 tx=0x7f3c740093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2075f324bd849aa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.438+0000 7f3c6b400640 1 -- 172.21.15.1:0/1830104018 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74031cf0 con 0x5576179b0fe0 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.438+0000 7f3c6b400640 1 -- 172.21.15.1:0/1830104018 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c74036030 con 0x5576179b0fe0 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.438+0000 7f3c6b400640 1 -- 172.21.15.1:0/1830104018 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c740042a0 con 0x5576179b0fe0 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c73400640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3c600a3e20 0x7f3c600a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c73400640 1 -- 172.21.15.1:0/1630706969 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f3c6c0d1200 con 0x7f3c7c1012c0 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c8423f740 1 -- 172.21.15.1:0/1830104018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 msgr2=0x5576179b13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c8423f740 1 --2- 172.21.15.1:0/1830104018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 0x5576179b13c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3c74009ca0 tx=0x7f3c740093e0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c7a800640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3c600a3e20 0x7f3c600a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c8423f740 1 -- 172.21.15.1:0/1830104018 shutdown_connections 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c8423f740 1 --2- 172.21.15.1:0/1830104018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b1900 0x5576179bbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c8423f740 1 --2- 172.21.15.1:0/1830104018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179bc1c0 0x5576179be5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c8423f740 1 --2- 172.21.15.1:0/1830104018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 0x5576179b13c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c8423f740 1 -- 172.21.15.1:0/1830104018 >> 172.21.15.1:0/1830104018 conn(0x5576179ad4f0 msgr2=0x5576179ad8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c8423f740 1 -- 172.21.15.1:0/1830104018 shutdown_connections 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.439+0000 7f3c8423f740 1 -- 172.21.15.1:0/1830104018 wait complete. 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.440+0000 7f3c7a800640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3c600a3e20 0x7f3c600a62e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0ce3c0 tx=0x7f3c6c018420 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.440+0000 7f3c8423f740 1 -- start start 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.440+0000 7f3c8423f740 1 Processor -- start 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.440+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 0x5576179b90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179bc1c0 0x5576179b9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c8423f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5576179bec10 con 0x5576179b0fe0 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c8423f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55761789dc60 con 0x5576179b1900 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c8423f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5576179bed90 con 0x5576179bc1c0 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c8423f740 10 client.0 _ll_register_callbacks cb 0x5576178c9e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c73e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c73e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:35868/0 (socket says 172.21.15.1:35868) 2024-04-26T21:31:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c73e00640 1 -- 172.21.15.1:0/3984145353 learned_addr learned my addr 172.21.15.1:0/3984145353 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c7b200640 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 0x5576179b90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c7b200640 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179bc1c0 0x5576179b9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c73e00640 1 -- 172.21.15.1:0/3984145353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179bc1c0 msgr2=0x5576179b9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c73e00640 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179bc1c0 0x5576179b9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c73e00640 1 -- 172.21.15.1:0/3984145353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 msgr2=0x5576179b90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c73e00640 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 0x5576179b90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.441+0000 7f3c73e00640 1 -- 172.21.15.1:0/3984145353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576178ac510 con 0x5576179b1900 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.442+0000 7f3c73e00640 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b95f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c6400e750 tx=0x7f3c6400eb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86e018cfa97bb05f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.442+0000 7f3c5fe00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.442+0000 7f3c5fe00640 1 -- 172.21.15.1:0/3984145353 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c6401d060 con 0x5576179b1900 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.442+0000 7f3c8423f740 1 -- 172.21.15.1:0/3984145353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c74036030 con 0x5576179b1900 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.442+0000 7f3c5fe00640 1 -- 172.21.15.1:0/3984145353 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f3c64004250 con 0x5576179b1900 2024-04-26T21:31:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.442+0000 7f3c5fe00640 1 -- 172.21.15.1:0/3984145353 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3c64024050 con 0x5576179b1900 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.442+0000 7f3c8423f740 10 client.25589 fetch_fsmap learned FSMap version 26 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.442+0000 7f3c8423f740 1 -- 172.21.15.1:0/3984145353 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557617628210 con 0x5576179b1900 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.443+0000 7f3c5fe00640 1 -- 172.21.15.1:0/3984145353 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c74036030 con 0x5576179b1900 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.443+0000 7f3c5fe00640 1 -- 172.21.15.1:0/3984145353 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3c6400b040 con 0x5576179b1900 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.443+0000 7f3c8423f740 10 client.25589 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.443+0000 7f3c8423f740 10 client.25589 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.443+0000 7f3c8423f740 20 client.25589 populate_metadata read hostname 'smithi001' 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.443+0000 7f3c8423f740 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5576179cbde0 0x5576179ce210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.443+0000 7f3c8423f740 4 client.25589 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.443+0000 7f3c8423f740 1 -- 172.21.15.1:0/3984145353 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557617923f90 con 0x5576179cbde0 2024-04-26T21:31:14.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.443+0000 7f3c7b200640 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5576179cbde0 0x5576179ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.444+0000 7f3c7b200640 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5576179cbde0 0x5576179ce210 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1306e7a608b0fcb server_cookie=97d6c0dac8c63799 in_seq=0 out_seq=0 2024-04-26T21:31:14.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.444+0000 7f3c5fe00640 10 client.25589 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:14.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c5fe00640 1 -- 172.21.15.1:0/3984145353 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557617923f90 con 0x5576179cbde0 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c5fe00640 10 client.25589 handle_command_reply: tid=0 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 client.25589 shutdown 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 -- 172.21.15.1:0/3984145353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 msgr2=0x5576179b95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b95f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c6400e750 tx=0x7f3c6400eb50 comp rx=0 tx=0).stop 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 -- 172.21.15.1:0/3984145353 shutdown_connections 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5576179cbde0 0x5576179ce210 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179bc1c0 0x5576179b9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b95f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 --2- 172.21.15.1:0/3984145353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179b0fe0 0x5576179b90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 -- 172.21.15.1:0/3984145353 >> 172.21.15.1:0/3984145353 conn(0x5576179ad4f0 msgr2=0x5576179b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c5fe00640 0 client.25589 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 -- 172.21.15.1:0/3984145353 shutdown_connections 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 1 -- 172.21.15.1:0/3984145353 wait complete. 2024-04-26T21:31:14.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.445+0000 7f3c8423f740 20 client.25589 trim_cache size 0 max 16384 2024-04-26T21:31:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.461+0000 7f3c8423f740 1 Processor -- start 2024-04-26T21:31:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.461+0000 7f3c8423f740 1 -- start start 2024-04-26T21:31:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b0fe0 0x5576179c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b1900 0x5576179acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179d3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c8423f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c1b46e0 con 0x5576179bc1c0 2024-04-26T21:31:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c8423f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c1b1fb0 con 0x5576179b0fe0 2024-04-26T21:31:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c8423f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c1afe00 con 0x5576179b1900 2024-04-26T21:31:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c7b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179d3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c7b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179d3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56488/0 (socket says 172.21.15.1:56488) 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c73e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b1900 0x5576179acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c7b200640 1 -- 172.21.15.1:0/1795666004 learned_addr learned my addr 172.21.15.1:0/1795666004 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.462+0000 7f3c7b200640 1 --2- 172.21.15.1:0/1795666004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b0fe0 0x5576179c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.463+0000 7f3c7b200640 1 -- 172.21.15.1:0/1795666004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b1900 msgr2=0x5576179acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.463+0000 7f3c7b200640 1 --2- 172.21.15.1:0/1795666004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b1900 0x5576179acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.463+0000 7f3c7b200640 1 -- 172.21.15.1:0/1795666004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b0fe0 msgr2=0x5576179c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.463+0000 7f3c7b200640 1 --2- 172.21.15.1:0/1795666004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b0fe0 0x5576179c7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.463+0000 7f3c7b200640 1 -- 172.21.15.1:0/1795666004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576176f4620 con 0x5576179bc1c0 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.463+0000 7f3c7b200640 1 --2- 172.21.15.1:0/1795666004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179d3b70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3c74003e10 tx=0x7f3c74003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10169922371a114 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.463+0000 7f3c6aa00640 1 -- 172.21.15.1:0/1795666004 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74031af0 con 0x5576179bc1c0 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.463+0000 7f3c6aa00640 1 -- 172.21.15.1:0/1795666004 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c74036070 con 0x5576179bc1c0 2024-04-26T21:31:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.463+0000 7f3c6aa00640 1 -- 172.21.15.1:0/1795666004 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c740585b0 con 0x5576179bc1c0 2024-04-26T21:31:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.464+0000 7f3c8423f740 1 -- 172.21.15.1:0/1795666004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 msgr2=0x5576179d3b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.464+0000 7f3c8423f740 1 --2- 172.21.15.1:0/1795666004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179d3b70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3c74003e10 tx=0x7f3c74003da0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.464+0000 7f3c8423f740 1 -- 172.21.15.1:0/1795666004 shutdown_connections 2024-04-26T21:31:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.464+0000 7f3c8423f740 1 --2- 172.21.15.1:0/1795666004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b1900 0x5576179acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.464+0000 7f3c8423f740 1 --2- 172.21.15.1:0/1795666004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179d3b70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.464+0000 7f3c8423f740 1 --2- 172.21.15.1:0/1795666004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b0fe0 0x5576179c7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.464+0000 7f3c8423f740 1 -- 172.21.15.1:0/1795666004 >> 172.21.15.1:0/1795666004 conn(0x5576179ad4f0 msgr2=0x5576179b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.465+0000 7f3c8423f740 1 -- 172.21.15.1:0/1795666004 shutdown_connections 2024-04-26T21:31:14.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.465+0000 7f3c8423f740 1 -- 172.21.15.1:0/1795666004 wait complete. 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.465+0000 7f3c8423f740 1 -- start start 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.465+0000 7f3c8423f740 1 Processor -- start 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b0fe0 0x5576179c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c8423f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179b6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c8423f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c1b0470 con 0x5576179bc1c0 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c8423f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c1b18e0 con 0x5576179b1900 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c8423f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c1b0b40 con 0x5576179b0fe0 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c8423f740 10 client.0 _ll_register_callbacks cb 0x7f3c6400a680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c7b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b0fe0 0x5576179c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c7b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c7b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179b6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56500/0 (socket says 172.21.15.1:56500) 2024-04-26T21:31:14.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c7b200640 1 -- 172.21.15.1:0/2519015481 learned_addr learned my addr 172.21.15.1:0/2519015481 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.466+0000 7f3c73e00640 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c7b200640 1 -- 172.21.15.1:0/2519015481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b0fe0 msgr2=0x5576179c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c7b200640 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b0fe0 0x5576179c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c7b200640 1 -- 172.21.15.1:0/2519015481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 msgr2=0x5576179b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c7b200640 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c7b200640 1 -- 172.21.15.1:0/2519015481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576178d0cb0 con 0x5576179bc1c0 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c73e00640 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c7b200640 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179b6060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3c7402aeb0 tx=0x7f3c74000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f810644a9812ed9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c5fe00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c5fe00640 1 -- 172.21.15.1:0/2519015481 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74031450 con 0x5576179bc1c0 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c5fe00640 1 -- 172.21.15.1:0/2519015481 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f3c7405e070 con 0x5576179bc1c0 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.467+0000 7f3c8423f740 1 -- 172.21.15.1:0/2519015481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c74036070 con 0x5576179bc1c0 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c5fe00640 1 -- 172.21.15.1:0/2519015481 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3c740315f0 con 0x5576179bc1c0 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c8423f740 10 client.15450 fetch_fsmap learned FSMap version 26 2024-04-26T21:31:14.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c8423f740 1 -- 172.21.15.1:0/2519015481 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x557617a32220 con 0x5576179bc1c0 2024-04-26T21:31:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c5fe00640 1 -- 172.21.15.1:0/2519015481 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c74036070 con 0x5576179bc1c0 2024-04-26T21:31:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c5fe00640 1 -- 172.21.15.1:0/2519015481 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3c74050040 con 0x5576179bc1c0 2024-04-26T21:31:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c8423f740 10 client.15450 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:31:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c8423f740 10 client.15450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:31:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c8423f740 20 client.15450 populate_metadata read hostname 'smithi001' 2024-04-26T21:31:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c8423f740 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x557617a32550 0x557617a34940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c8423f740 4 client.15450 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:31:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.468+0000 7f3c8423f740 1 -- 172.21.15.1:0/2519015481 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55761791de00 con 0x557617a32550 2024-04-26T21:31:14.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.469+0000 7f3c73e00640 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x557617a32550 0x557617a34940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.469+0000 7f3c73e00640 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x557617a32550 0x557617a34940 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf1d0c1d9b9babe6 server_cookie=40676d518ad796b8 in_seq=0 out_seq=0 2024-04-26T21:31:14.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.469+0000 7f3c5fe00640 10 client.15450 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:14.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.473+0000 7f3c5fe00640 1 -- 172.21.15.1:0/2519015481 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55761791de00 con 0x557617a32550 2024-04-26T21:31:14.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.473+0000 7f3c5fe00640 10 client.15450 handle_command_reply: tid=0 2024-04-26T21:31:14.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.473+0000 7f3c8423f740 1 client.15450 shutdown 2024-04-26T21:31:14.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.473+0000 7f3c8423f740 1 -- 172.21.15.1:0/2519015481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 msgr2=0x5576179b6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:14.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.473+0000 7f3c8423f740 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179b6060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3c7402aeb0 tx=0x7f3c74000f40 comp rx=0 tx=0).stop 2024-04-26T21:31:14.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.474+0000 7f3c8423f740 1 -- 172.21.15.1:0/2519015481 shutdown_connections 2024-04-26T21:31:14.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.474+0000 7f3c8423f740 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x557617a32550 0x557617a34940 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.474+0000 7f3c8423f740 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5576179bc1c0 0x5576179b6060 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.474+0000 7f3c8423f740 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5576179b1900 0x5576179b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.474+0000 7f3c8423f740 1 --2- 172.21.15.1:0/2519015481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5576179b0fe0 0x5576179c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.474+0000 7f3c8423f740 1 -- 172.21.15.1:0/2519015481 >> 172.21.15.1:0/2519015481 conn(0x5576179ad4f0 msgr2=0x5576179ba760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.474+0000 7f3c5fe00640 0 client.15450 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.474+0000 7f3c8423f740 1 -- 172.21.15.1:0/2519015481 shutdown_connections 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.474+0000 7f3c8423f740 1 -- 172.21.15.1:0/2519015481 wait complete. 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.474+0000 7f3c8423f740 20 client.15450 trim_cache size 0 max 16384 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:31:14.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:31:14.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:31:14.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 -- 172.21.15.1:0/1630706969 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3c600a3e20 msgr2=0x7f3c600a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:14.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3c600a3e20 0x7f3c600a62e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0ce3c0 tx=0x7f3c6c018420 comp rx=0 tx=0).stop 2024-04-26T21:31:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 -- 172.21.15.1:0/1630706969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c1012c0 msgr2=0x7f3c7c19b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c1012c0 0x7f3c7c19b150 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c00e9f0 tx=0x7f3c6c00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 -- 172.21.15.1:0/1630706969 shutdown_connections 2024-04-26T21:31:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3c600a3e20 0x7f3c600a62e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c7c19fa10 0x7f3c7c19fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3c7c106110 0x7f3c7c19b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 --2- 172.21.15.1:0/1630706969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c7c1012c0 0x7f3c7c19b150 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 -- 172.21.15.1:0/1630706969 >> 172.21.15.1:0/1630706969 conn(0x7f3c7c0769a0 msgr2=0x7f3c7c0fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 -- 172.21.15.1:0/1630706969 shutdown_connections 2024-04-26T21:31:14.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:14.475+0000 7f3c80a00640 1 -- 172.21.15.1:0/1630706969 wait complete. 2024-04-26T21:31:16.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:15 smithi001 ceph-mon[28054]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-26T21:31:16.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:15 smithi038 ceph-mon[32724]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-26T21:31:16.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:15 smithi186 ceph-mon[32991]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-26T21:31:18.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:17 smithi001 ceph-mon[28054]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 20 op/s 2024-04-26T21:31:18.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:17 smithi038 ceph-mon[32724]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 20 op/s 2024-04-26T21:31:18.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:17 smithi186 ceph-mon[32991]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 KiB/s wr, 20 op/s 2024-04-26T21:31:20.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:19 smithi001 ceph-mon[28054]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:20.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:19 smithi038 ceph-mon[32724]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:20.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:19 smithi186 ceph-mon[32991]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:22.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:21 smithi001 ceph-mon[28054]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:22.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:21 smithi038 ceph-mon[32724]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:22.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:21 smithi186 ceph-mon[32991]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:24.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:23 smithi001 ceph-mon[28054]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:24.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:23 smithi038 ceph-mon[32724]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:23 smithi186 ceph-mon[32991]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:26.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:25 smithi001 ceph-mon[28054]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:26.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:25 smithi038 ceph-mon[32724]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:26.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:25 smithi186 ceph-mon[32991]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.652+0000 7fdab1e00640 1 -- 172.21.15.1:0/1326398207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaac103910 msgr2=0x7fdaac103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.652+0000 7fdab1e00640 1 --2- 172.21.15.1:0/1326398207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaac103910 0x7fdaac103cf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0009900 tx=0x7fdaa002a480 comp rx=0 tx=0).stop 2024-04-26T21:31:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.652+0000 7fdab1e00640 1 -- 172.21.15.1:0/1326398207 shutdown_connections 2024-04-26T21:31:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.652+0000 7fdab1e00640 1 --2- 172.21.15.1:0/1326398207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaac10cd30 0x7fdaac10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.652+0000 7fdab1e00640 1 --2- 172.21.15.1:0/1326398207 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaac104230 0x7fdaac10c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.652+0000 7fdab1e00640 1 --2- 172.21.15.1:0/1326398207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaac103910 0x7fdaac103cf0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.652+0000 7fdab1e00640 1 -- 172.21.15.1:0/1326398207 >> 172.21.15.1:0/1326398207 conn(0x7fdaac0fd110 msgr2=0x7fdaac0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.652+0000 7fdab1e00640 1 -- 172.21.15.1:0/1326398207 shutdown_connections 2024-04-26T21:31:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.652+0000 7fdab1e00640 1 -- 172.21.15.1:0/1326398207 wait complete. 2024-04-26T21:31:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.654+0000 7fdab1e00640 1 Processor -- start 2024-04-26T21:31:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.654+0000 7fdab1e00640 1 -- start start 2024-04-26T21:31:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.654+0000 7fdab1e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaac103910 0x7fdaac198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.654+0000 7fdab1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaac104230 0x7fdaac1994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.654+0000 7fdab1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaac10cd30 0x7fdaac19d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.654+0000 7fdab1e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac112a30 con 0x7fdaac103910 2024-04-26T21:31:27.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.654+0000 7fdab1e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac1128b0 con 0x7fdaac104230 2024-04-26T21:31:27.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.654+0000 7fdab1e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac112bb0 con 0x7fdaac10cd30 2024-04-26T21:31:27.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdab0a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaac103910 0x7fdaac198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdab0a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaac103910 0x7fdaac198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53648/0 (socket says 172.21.15.1:53648) 2024-04-26T21:31:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdab0a00640 1 -- 172.21.15.1:0/4253466897 learned_addr learned my addr 172.21.15.1:0/4253466897 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdaabe00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaac104230 0x7fdaac1994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdab1400640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaac10cd30 0x7fdaac19d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdab0a00640 1 -- 172.21.15.1:0/4253466897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaac10cd30 msgr2=0x7fdaac19d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdab0a00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaac10cd30 0x7fdaac19d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdab0a00640 1 -- 172.21.15.1:0/4253466897 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaac104230 msgr2=0x7fdaac1994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdab0a00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaac104230 0x7fdaac1994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdab0a00640 1 -- 172.21.15.1:0/4253466897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaac19df30 con 0x7fdaac103910 2024-04-26T21:31:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.655+0000 7fdaabe00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaac104230 0x7fdaac1994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.656+0000 7fdab0a00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaac103910 0x7fdaac198f90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0005ab0 tx=0x7fdaa0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.656+0000 7fdaa9600640 1 -- 172.21.15.1:0/4253466897 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa003f070 con 0x7fdaac103910 2024-04-26T21:31:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.656+0000 7fdaa9600640 1 -- 172.21.15.1:0/4253466897 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa0031b60 con 0x7fdaac103910 2024-04-26T21:31:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.656+0000 7fdaa9600640 1 -- 172.21.15.1:0/4253466897 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0035030 con 0x7fdaac103910 2024-04-26T21:31:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.656+0000 7fdab1e00640 1 -- 172.21.15.1:0/4253466897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaac19e1c0 con 0x7fdaac103910 2024-04-26T21:31:27.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.657+0000 7fdab1e00640 1 -- 172.21.15.1:0/4253466897 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaac101490 con 0x7fdaac103910 2024-04-26T21:31:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.660+0000 7fdab6272740 1 Processor -- start 2024-04-26T21:31:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.660+0000 7fdab6272740 1 -- start start 2024-04-26T21:31:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.660+0000 7fdab6272740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b423c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab6272740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b4cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab6272740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b4f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab6272740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555fd7aa7380 con 0x555fd7b42900 2024-04-26T21:31:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab6272740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555fd7a43c30 con 0x555fd7b4d1c0 2024-04-26T21:31:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab6272740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555fd7aa3340 con 0x555fd7b41fe0 2024-04-26T21:31:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab1400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab1400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58342/0 (socket says 172.21.15.1:58342) 2024-04-26T21:31:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab1400640 1 -- 172.21.15.1:0/4083681020 learned_addr learned my addr 172.21.15.1:0/4083681020 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdaabe00640 1 --2- 172.21.15.1:0/4083681020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b4f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdaa9600640 1 -- 172.21.15.1:0/4253466897 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7fdaa0042070 con 0x7fdaac103910 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab1400640 1 -- 172.21.15.1:0/4083681020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 msgr2=0x555fd7b4f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab1400640 1 --2- 172.21.15.1:0/4083681020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b4f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab1400640 1 -- 172.21.15.1:0/4083681020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 msgr2=0x555fd7b4cc80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab1400640 1 --2- 172.21.15.1:0/4083681020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b4cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdab1400640 1 -- 172.21.15.1:0/4083681020 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fd7a65940 con 0x555fd7b41fe0 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.661+0000 7fdaabe00640 1 --2- 172.21.15.1:0/4083681020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b4f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.662+0000 7fdab1400640 1 --2- 172.21.15.1:0/4083681020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b423c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009f60 tx=0x7fdaa4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c975dd330c4668e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.662+0000 7fda9dc00640 1 -- 172.21.15.1:0/4083681020 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa401e070 con 0x555fd7b41fe0 2024-04-26T21:31:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.662+0000 7fda9dc00640 1 -- 172.21.15.1:0/4083681020 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa4003c30 con 0x555fd7b41fe0 2024-04-26T21:31:27.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.662+0000 7fda9dc00640 1 -- 172.21.15.1:0/4083681020 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa400f8a0 con 0x555fd7b41fe0 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.663+0000 7fdab6272740 1 -- 172.21.15.1:0/4083681020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 msgr2=0x555fd7b423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.663+0000 7fdab6272740 1 --2- 172.21.15.1:0/4083681020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b423c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009f60 tx=0x7fdaa4009f90 comp rx=0 tx=0).stop 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.663+0000 7fdab6272740 1 -- 172.21.15.1:0/4083681020 shutdown_connections 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.663+0000 7fdab6272740 1 --2- 172.21.15.1:0/4083681020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b4f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.663+0000 7fdab6272740 1 --2- 172.21.15.1:0/4083681020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b4cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.663+0000 7fdab6272740 1 --2- 172.21.15.1:0/4083681020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b423c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.663+0000 7fdab6272740 1 -- 172.21.15.1:0/4083681020 >> 172.21.15.1:0/4083681020 conn(0x555fd7b3e4f0 msgr2=0x555fd7b3e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.664+0000 7fdab6272740 1 -- 172.21.15.1:0/4083681020 shutdown_connections 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.664+0000 7fdab6272740 1 -- 172.21.15.1:0/4083681020 wait complete. 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.664+0000 7fdaa9600640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fda940a3db0 0x7fda940a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.664+0000 7fdaa9600640 1 -- 172.21.15.1:0/4253466897 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fdaa00f10b0 con 0x7fdaac103910 2024-04-26T21:31:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.664+0000 7fdab6272740 1 -- start start 2024-04-26T21:31:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.664+0000 7fdab1400640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fda940a3db0 0x7fda940a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.664+0000 7fdab6272740 1 Processor -- start 2024-04-26T21:31:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.665+0000 7fdab6272740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b4a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.665+0000 7fdab6272740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b4a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.665+0000 7fdab6272740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b4ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.665+0000 7fdab6272740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555fd7b4fdf0 con 0x555fd7b42900 2024-04-26T21:31:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.665+0000 7fdab6272740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555fd7a2ec60 con 0x555fd7b4d1c0 2024-04-26T21:31:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.665+0000 7fdab6272740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555fd7b4ff70 con 0x555fd7b41fe0 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.665+0000 7fdab1400640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fda940a3db0 0x7fda940a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009d70 tx=0x7fdaa4005fd0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.665+0000 7fdab6272740 10 client.0 _ll_register_callbacks cb 0x555fd7a5ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdaabe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b4a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdab0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b4a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdaabe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b4a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53654/0 (socket says 172.21.15.1:53654) 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdaabe00640 1 -- 172.21.15.1:0/3943683303 learned_addr learned my addr 172.21.15.1:0/3943683303 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdab0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b4a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58354/0 (socket says 172.21.15.1:58354) 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdab1400640 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b4ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdab1400640 1 -- 172.21.15.1:0/3943683303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 msgr2=0x555fd7b4a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdab1400640 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b4a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdab1400640 1 -- 172.21.15.1:0/3943683303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 msgr2=0x555fd7b4a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdab1400640 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b4a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdab1400640 1 -- 172.21.15.1:0/3943683303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fd7a3d510 con 0x555fd7b4d1c0 2024-04-26T21:31:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdab0a00640 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b4a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.666+0000 7fdab1400640 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b4ac00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdaa40049d0 tx=0x7fdaa4009af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fde315fa97e7a21e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.667+0000 7fda92000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:31:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.667+0000 7fda92000640 1 -- 172.21.15.1:0/3943683303 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4035060 con 0x555fd7b4d1c0 2024-04-26T21:31:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.667+0000 7fdab6272740 1 -- 172.21.15.1:0/3943683303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdaa4003c30 con 0x555fd7b4d1c0 2024-04-26T21:31:27.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.667+0000 7fda92000640 1 -- 172.21.15.1:0/3943683303 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fdaa400fce0 con 0x555fd7b4d1c0 2024-04-26T21:31:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.667+0000 7fda92000640 1 -- 172.21.15.1:0/3943683303 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdaa4004e40 con 0x555fd7b4d1c0 2024-04-26T21:31:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.667+0000 7fdab6272740 10 client.25604 fetch_fsmap learned FSMap version 26 2024-04-26T21:31:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.667+0000 7fdab6272740 1 -- 172.21.15.1:0/3943683303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555fd77b9210 con 0x555fd7b4d1c0 2024-04-26T21:31:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.667+0000 7fda92000640 1 -- 172.21.15.1:0/3943683303 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa4003c30 con 0x555fd7b4d1c0 2024-04-26T21:31:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.668+0000 7fda92000640 1 -- 172.21.15.1:0/3943683303 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fdaa4025450 con 0x555fd7b4d1c0 2024-04-26T21:31:27.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.668+0000 7fdab6272740 10 client.25604 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.668+0000 7fdab6272740 10 client.25604 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.668+0000 7fdab6272740 20 client.25604 populate_metadata read hostname 'smithi001' 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.668+0000 7fdab6272740 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555fd7b5ce20 0x555fd7b5f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.668+0000 7fdab6272740 4 client.25604 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.668+0000 7fdab6272740 1 -- 172.21.15.1:0/3943683303 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fd7ab4f90 con 0x555fd7b5ce20 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.668+0000 7fdab0a00640 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555fd7b5ce20 0x555fd7b5f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.669+0000 7fdab0a00640 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555fd7b5ce20 0x555fd7b5f210 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1203f148399c2c97 server_cookie=9091b478d97a75d0 in_seq=0 out_seq=0 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.669+0000 7fda92000640 10 client.25604 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fda92000640 1 -- 172.21.15.1:0/3943683303 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555fd7ab4f90 con 0x555fd7b5ce20 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fda92000640 10 client.25604 handle_command_reply: tid=0 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fdab6272740 1 client.25604 shutdown 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fdab6272740 1 -- 172.21.15.1:0/3943683303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 msgr2=0x555fd7b4ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fdab6272740 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b4ac00 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdaa40049d0 tx=0x7fdaa4009af0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fdab6272740 1 -- 172.21.15.1:0/3943683303 shutdown_connections 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fdab6272740 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555fd7b5ce20 0x555fd7b5f210 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fdab6272740 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b4ac00 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fdab6272740 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b4a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fdab6272740 1 --2- 172.21.15.1:0/3943683303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b4a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fdab6272740 1 -- 172.21.15.1:0/3943683303 >> 172.21.15.1:0/3943683303 conn(0x555fd7b3e4f0 msgr2=0x555fd7b44270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.670+0000 7fda92000640 0 client.25604 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.671+0000 7fdab6272740 1 -- 172.21.15.1:0/3943683303 shutdown_connections 2024-04-26T21:31:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.671+0000 7fdab6272740 1 -- 172.21.15.1:0/3943683303 wait complete. 2024-04-26T21:31:27.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.671+0000 7fdab6272740 20 client.25604 trim_cache size 0 max 16384 2024-04-26T21:31:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.687+0000 7fdab6272740 1 Processor -- start 2024-04-26T21:31:27.687 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.687+0000 7fdab6272740 1 -- start start 2024-04-26T21:31:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.687+0000 7fdab6272740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b58cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.687+0000 7fdab6272740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b440f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.687+0000 7fdab6272740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b64a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.689 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.687+0000 7fdab6272740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac1b03f0 con 0x555fd7b42900 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.687+0000 7fdab6272740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac1adcc0 con 0x555fd7b4d1c0 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.687+0000 7fdab6272740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac1abb10 con 0x555fd7b41fe0 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdaabe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdab0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b58cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdab0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b58cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58366/0 (socket says 172.21.15.1:58366) 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdab0a00640 1 -- 172.21.15.1:0/1045457978 learned_addr learned my addr 172.21.15.1:0/1045457978 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdaabe00640 1 --2- 172.21.15.1:0/1045457978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b440f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53664/0 (socket says 172.21.15.1:53664) 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdab1400640 1 --2- 172.21.15.1:0/1045457978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b64a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdab0a00640 1 -- 172.21.15.1:0/1045457978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 msgr2=0x555fd7b64a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdab0a00640 1 --2- 172.21.15.1:0/1045457978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b64a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdab0a00640 1 -- 172.21.15.1:0/1045457978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 msgr2=0x555fd7b440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:27.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdab0a00640 1 --2- 172.21.15.1:0/1045457978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdab0a00640 1 -- 172.21.15.1:0/1045457978 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fd7885620 con 0x555fd7b41fe0 2024-04-26T21:31:27.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.688+0000 7fdab1400640 1 --2- 172.21.15.1:0/1045457978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b64a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:27.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.689+0000 7fdab0a00640 1 --2- 172.21.15.1:0/1045457978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b58cb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdaa002aa40 tx=0x7fdaa002a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56347ec7480e1b31 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:27.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.689+0000 7fda9d200640 1 -- 172.21.15.1:0/1045457978 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa00d5020 con 0x555fd7b41fe0 2024-04-26T21:31:27.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.689+0000 7fda9d200640 1 -- 172.21.15.1:0/1045457978 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa00bca60 con 0x555fd7b41fe0 2024-04-26T21:31:27.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.689+0000 7fda9d200640 1 -- 172.21.15.1:0/1045457978 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0049040 con 0x555fd7b41fe0 2024-04-26T21:31:27.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.690+0000 7fdab6272740 1 -- 172.21.15.1:0/1045457978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 msgr2=0x555fd7b58cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:27.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.690+0000 7fdab6272740 1 --2- 172.21.15.1:0/1045457978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b58cb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdaa002aa40 tx=0x7fdaa002a480 comp rx=0 tx=0).stop 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.690+0000 7fdab6272740 1 -- 172.21.15.1:0/1045457978 shutdown_connections 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.690+0000 7fdab6272740 1 --2- 172.21.15.1:0/1045457978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b4d1c0 0x555fd7b64a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.690+0000 7fdab6272740 1 --2- 172.21.15.1:0/1045457978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b42900 0x555fd7b440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.690+0000 7fdab6272740 1 --2- 172.21.15.1:0/1045457978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b58cb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.690+0000 7fdab6272740 1 -- 172.21.15.1:0/1045457978 >> 172.21.15.1:0/1045457978 conn(0x555fd7b3e4f0 msgr2=0x555fd7b439e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.690+0000 7fdab6272740 1 -- 172.21.15.1:0/1045457978 shutdown_connections 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.690+0000 7fdab6272740 1 -- 172.21.15.1:0/1045457978 wait complete. 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.691+0000 7fdab6272740 1 -- start start 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.691+0000 7fdab6272740 1 Processor -- start 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.691+0000 7fdab6272740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b555b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.691+0000 7fdab6272740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b42900 0x555fd7b46bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.691+0000 7fdab6272740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b4d1c0 0x555fd7b47130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab6272740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac1ac180 con 0x555fd7b4d1c0 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab6272740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac1ad5f0 con 0x555fd7b42900 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab6272740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac1ac850 con 0x555fd7b41fe0 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab6272740 10 client.0 _ll_register_callbacks cb 0x7fdaa401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58378/0 (socket says 172.21.15.1:58378) 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdaabe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b42900 0x555fd7b46bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab0a00640 1 -- 172.21.15.1:0/3762401530 learned_addr learned my addr 172.21.15.1:0/3762401530 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab1400640 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b4d1c0 0x555fd7b47130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab0a00640 1 -- 172.21.15.1:0/3762401530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b42900 msgr2=0x555fd7b46bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab0a00640 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b42900 0x555fd7b46bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.692+0000 7fdab0a00640 1 -- 172.21.15.1:0/3762401530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b4d1c0 msgr2=0x555fd7b47130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:27.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.693+0000 7fdab0a00640 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b4d1c0 0x555fd7b47130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.693+0000 7fdab0a00640 1 -- 172.21.15.1:0/3762401530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fd7a61cb0 con 0x555fd7b41fe0 2024-04-26T21:31:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.693+0000 7fdaabe00640 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b42900 0x555fd7b46bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.693+0000 7fdab1400640 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b4d1c0 0x555fd7b47130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.693+0000 7fdab0a00640 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b555b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdaa00096c0 tx=0x7fdaa00ee1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2293bdf518e9ee96 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:27.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.693+0000 7fda92000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:31:27.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.693+0000 7fda92000640 1 -- 172.21.15.1:0/3762401530 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa00a0060 con 0x555fd7b41fe0 2024-04-26T21:31:27.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.693+0000 7fda92000640 1 -- 172.21.15.1:0/3762401530 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fdaa00bad60 con 0x555fd7b41fe0 2024-04-26T21:31:27.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.693+0000 7fdab6272740 1 -- 172.21.15.1:0/3762401530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdaa00bca60 con 0x555fd7b41fe0 2024-04-26T21:31:27.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fda92000640 1 -- 172.21.15.1:0/3762401530 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdaa00d5030 con 0x555fd7b41fe0 2024-04-26T21:31:27.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fdab6272740 10 client.25591 fetch_fsmap learned FSMap version 26 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fdab6272740 1 -- 172.21.15.1:0/3762401530 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555fd7bc30e0 con 0x555fd7b41fe0 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fda92000640 1 -- 172.21.15.1:0/3762401530 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaa00bca60 con 0x555fd7b41fe0 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fda92000640 1 -- 172.21.15.1:0/3762401530 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fdaa004c060 con 0x555fd7b41fe0 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fdab6272740 10 client.25591 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fdab6272740 10 client.25591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fdab6272740 20 client.25591 populate_metadata read hostname 'smithi001' 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fdab6272740 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555fd7bc3780 0x555fd7bc5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fdab6272740 4 client.25591 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.694+0000 7fdab6272740 1 -- 172.21.15.1:0/3762401530 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555fd7aaee00 con 0x555fd7bc3780 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.695+0000 7fdaabe00640 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555fd7bc3780 0x555fd7bc5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.695+0000 7fdaabe00640 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555fd7bc3780 0x555fd7bc5ba0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a4cb3c3fa2bbcf server_cookie=5301de6cf35d6e40 in_seq=0 out_seq=0 2024-04-26T21:31:27.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.695+0000 7fda92000640 10 client.25591 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:27.700 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.699+0000 7fda92000640 1 -- 172.21.15.1:0/3762401530 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555fd7aaee00 con 0x555fd7bc3780 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.699+0000 7fda92000640 10 client.25591 handle_command_reply: tid=0 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.699+0000 7fdab6272740 1 client.25591 shutdown 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 1 -- 172.21.15.1:0/3762401530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 msgr2=0x555fd7b555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b555b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdaa00096c0 tx=0x7fdaa00ee1e0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 1 -- 172.21.15.1:0/3762401530 shutdown_connections 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555fd7bc3780 0x555fd7bc5ba0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555fd7b4d1c0 0x555fd7b47130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555fd7b42900 0x555fd7b46bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 1 --2- 172.21.15.1:0/3762401530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555fd7b41fe0 0x555fd7b555b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 1 -- 172.21.15.1:0/3762401530 >> 172.21.15.1:0/3762401530 conn(0x555fd7b3e4f0 msgr2=0x555fd7b3d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fda92000640 0 client.25591 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 1 -- 172.21.15.1:0/3762401530 shutdown_connections 2024-04-26T21:31:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 1 -- 172.21.15.1:0/3762401530 wait complete. 2024-04-26T21:31:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.700+0000 7fdab6272740 20 client.25591 trim_cache size 0 max 16384 2024-04-26T21:31:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:31:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:31:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:31:27.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 -- 172.21.15.1:0/4253466897 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fda940a3db0 msgr2=0x7fda940a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fda940a3db0 0x7fda940a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009d70 tx=0x7fdaa4005fd0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 -- 172.21.15.1:0/4253466897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaac103910 msgr2=0x7fdaac198f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaac103910 0x7fdaac198f90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0005ab0 tx=0x7fdaa0003c10 comp rx=0 tx=0).stop 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 -- 172.21.15.1:0/4253466897 shutdown_connections 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fda940a3db0 0x7fda940a6270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaac10cd30 0x7fdaac19d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaac104230 0x7fdaac1994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 --2- 172.21.15.1:0/4253466897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaac103910 0x7fdaac198f90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 -- 172.21.15.1:0/4253466897 >> 172.21.15.1:0/4253466897 conn(0x7fdaac0fd110 msgr2=0x7fdaac0fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:27.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 -- 172.21.15.1:0/4253466897 shutdown_connections 2024-04-26T21:31:27.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:27.701+0000 7fdab1e00640 1 -- 172.21.15.1:0/4253466897 wait complete. 2024-04-26T21:31:28.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:27 smithi001 ceph-mon[28054]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T21:31:28.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:27 smithi038 ceph-mon[32724]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T21:31:28.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:27 smithi186 ceph-mon[32991]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-26T21:31:29.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:28 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:29.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:28 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:29.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:28 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:31:30.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:29 smithi001 ceph-mon[28054]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T21:31:30.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:29 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:30.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:29 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:30.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:29 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:31:30.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:29 smithi038 ceph-mon[32724]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T21:31:30.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:29 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:30.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:29 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:30.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:29 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:31:30.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:29 smithi186 ceph-mon[32991]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-26T21:31:30.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:29 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:31:30.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:29 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:31:30.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:29 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:31:32.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:31 smithi001 ceph-mon[28054]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:32.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:31 smithi038 ceph-mon[32724]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:31 smithi186 ceph-mon[32991]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:34.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:33 smithi001 ceph-mon[28054]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:34.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:33 smithi038 ceph-mon[32724]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:34.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:33 smithi186 ceph-mon[32991]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:36.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:35 smithi001 ceph-mon[28054]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:36.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:35 smithi038 ceph-mon[32724]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:36.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:35 smithi186 ceph-mon[32991]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:38.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:37 smithi001 ceph-mon[28054]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:38.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:37 smithi038 ceph-mon[32724]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:38.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:37 smithi186 ceph-mon[32991]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:39.798 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:39 smithi001 ceph-mon[28054]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:40.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:39 smithi038 ceph-mon[32724]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:40.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:39 smithi186 ceph-mon[32991]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:31:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.890+0000 7f987b000640 1 -- 172.21.15.1:0/270945038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9874105320 msgr2=0x7f9874111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.890+0000 7f987b000640 1 --2- 172.21.15.1:0/270945038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9874105320 0x7f9874111830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f986c009c10 tx=0x7f986c02a6f0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.890+0000 7f987b000640 1 -- 172.21.15.1:0/270945038 shutdown_connections 2024-04-26T21:31:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.890+0000 7f987b000640 1 --2- 172.21.15.1:0/270945038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9874105320 0x7f9874111830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.890+0000 7f987b000640 1 --2- 172.21.15.1:0/270945038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9874104960 0x7f9874104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.890+0000 7f987b000640 1 --2- 172.21.15.1:0/270945038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f987410a970 0x7f987410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.890+0000 7f987b000640 1 -- 172.21.15.1:0/270945038 >> 172.21.15.1:0/270945038 conn(0x7f9874100430 msgr2=0x7f9874102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.890+0000 7f987b000640 1 -- 172.21.15.1:0/270945038 shutdown_connections 2024-04-26T21:31:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.890+0000 7f987b000640 1 -- 172.21.15.1:0/270945038 wait complete. 2024-04-26T21:31:40.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.891+0000 7f987b000640 1 Processor -- start 2024-04-26T21:31:40.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.891+0000 7f987b000640 1 -- start start 2024-04-26T21:31:40.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.891+0000 7f987b000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9874104960 0x7f987419b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.891+0000 7f987b000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874105320 0x7f987419b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.891+0000 7f987b000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f987410a970 0x7f987419fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.891+0000 7f987b000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9874114c30 con 0x7f9874105320 2024-04-26T21:31:40.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.891+0000 7f987b000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9874114ab0 con 0x7f987410a970 2024-04-26T21:31:40.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.891+0000 7f987b000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9874114db0 con 0x7f9874104960 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f9879c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9874104960 0x7f987419b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f9879c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9874104960 0x7f987419b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42308/0 (socket says 172.21.15.1:42308) 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f9879c00640 1 -- 172.21.15.1:0/432725666 learned_addr learned my addr 172.21.15.1:0/432725666 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f987a600640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f987410a970 0x7f987419fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f9879200640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874105320 0x7f987419b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f9879c00640 1 -- 172.21.15.1:0/432725666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f987410a970 msgr2=0x7f987419fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f9879c00640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f987410a970 0x7f987419fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f9879c00640 1 -- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874105320 msgr2=0x7f987419b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f9879c00640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874105320 0x7f987419b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f9879c00640 1 -- 172.21.15.1:0/432725666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98741a0130 con 0x7f9874104960 2024-04-26T21:31:40.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.892+0000 7f9879200640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874105320 0x7f987419b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:31:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.893+0000 7f9879c00640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9874104960 0x7f987419b190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f986800e9c0 tx=0x7f986800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.893+0000 7f9872000640 1 -- 172.21.15.1:0/432725666 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986800cde0 con 0x7f9874104960 2024-04-26T21:31:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.893+0000 7f9872000640 1 -- 172.21.15.1:0/432725666 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98680035f0 con 0x7f9874104960 2024-04-26T21:31:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.893+0000 7f9872000640 1 -- 172.21.15.1:0/432725666 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9868010690 con 0x7f9874104960 2024-04-26T21:31:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.893+0000 7f987b000640 1 -- 172.21.15.1:0/432725666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98741a0420 con 0x7f9874104960 2024-04-26T21:31:40.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.894+0000 7f987b000640 1 -- 172.21.15.1:0/432725666 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98741a7cb0 con 0x7f9874104960 2024-04-26T21:31:40.897 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.897+0000 7f9872000640 1 -- 172.21.15.1:0/432725666 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 25) ==== 138878+0+0 (secure 0 0 0) 0x7f9868004000 con 0x7f9874104960 2024-04-26T21:31:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.898+0000 7f987f57b740 1 Processor -- start 2024-04-26T21:31:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.898+0000 7f987f57b740 1 -- start start 2024-04-26T21:31:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.898+0000 7f9872000640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f985c0a3e20 0x7f985c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f9872000640 1 -- 172.21.15.1:0/432725666 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f98680d1260 con 0x7f9874104960 2024-04-26T21:31:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f987f57b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d8623c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f987f57b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d862900 0x55e03d86cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f987f57b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d86d1c0 0x55e03d86f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f987f57b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e03d7c7380 con 0x55e03d861fe0 2024-04-26T21:31:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f987f57b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e03d763c30 con 0x55e03d862900 2024-04-26T21:31:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f987f57b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e03d7c3340 con 0x55e03d86d1c0 2024-04-26T21:31:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f9879c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d8623c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f9879c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d8623c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44308/0 (socket says 172.21.15.1:44308) 2024-04-26T21:31:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f9879c00640 1 -- 172.21.15.1:0/1656602020 learned_addr learned my addr 172.21.15.1:0/1656602020 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.899+0000 7f987a600640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f985c0a3e20 0x7f985c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.900+0000 7f987a600640 1 --2- 172.21.15.1:0/1656602020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d86d1c0 0x55e03d86f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.900+0000 7f9879200640 1 --2- 172.21.15.1:0/1656602020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d862900 0x55e03d86cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.900+0000 7f9879c00640 1 -- 172.21.15.1:0/1656602020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d86d1c0 msgr2=0x55e03d86f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.900+0000 7f9879c00640 1 --2- 172.21.15.1:0/1656602020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d86d1c0 0x55e03d86f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.900+0000 7f9879c00640 1 -- 172.21.15.1:0/1656602020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d862900 msgr2=0x55e03d86cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.900+0000 7f9879c00640 1 --2- 172.21.15.1:0/1656602020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d862900 0x55e03d86cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.900+0000 7f9879c00640 1 -- 172.21.15.1:0/1656602020 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e03d785940 con 0x55e03d861fe0 2024-04-26T21:31:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.900+0000 7f9879c00640 1 --2- 172.21.15.1:0/1656602020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d8623c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9868098fa0 tx=0x7f986809c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1e6ce05c85bd49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.901+0000 7f9866a00640 1 -- 172.21.15.1:0/1656602020 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98680b6070 con 0x55e03d861fe0 2024-04-26T21:31:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.901+0000 7f9866a00640 1 -- 172.21.15.1:0/1656602020 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f986809d240 con 0x55e03d861fe0 2024-04-26T21:31:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.901+0000 7f9866a00640 1 -- 172.21.15.1:0/1656602020 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98680037a0 con 0x55e03d861fe0 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.901+0000 7f987f57b740 1 -- 172.21.15.1:0/1656602020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 msgr2=0x55e03d8623c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.901+0000 7f987f57b740 1 --2- 172.21.15.1:0/1656602020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d8623c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9868098fa0 tx=0x7f986809c4a0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.901+0000 7f987a600640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f985c0a3e20 0x7f985c0a62e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f986c02ac10 tx=0x7f986c009840 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.902+0000 7f987f57b740 1 -- 172.21.15.1:0/1656602020 shutdown_connections 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.902+0000 7f987f57b740 1 --2- 172.21.15.1:0/1656602020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d86d1c0 0x55e03d86f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.902+0000 7f987f57b740 1 --2- 172.21.15.1:0/1656602020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d862900 0x55e03d86cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.902+0000 7f987f57b740 1 --2- 172.21.15.1:0/1656602020 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d8623c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.902+0000 7f987f57b740 1 -- 172.21.15.1:0/1656602020 >> 172.21.15.1:0/1656602020 conn(0x55e03d85e4f0 msgr2=0x55e03d85e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.902+0000 7f987f57b740 1 -- 172.21.15.1:0/1656602020 shutdown_connections 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.902+0000 7f987f57b740 1 -- 172.21.15.1:0/1656602020 wait complete. 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.902+0000 7f987f57b740 1 -- start start 2024-04-26T21:31:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.903+0000 7f987f57b740 1 Processor -- start 2024-04-26T21:31:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.903+0000 7f987f57b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 0x55e03d86a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.903+0000 7f987f57b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d862900 0x55e03d86a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.903+0000 7f987f57b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d86ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.903+0000 7f987f57b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e03d86fdf0 con 0x55e03d862900 2024-04-26T21:31:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.903+0000 7f987f57b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e03d74ec60 con 0x55e03d86d1c0 2024-04-26T21:31:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.903+0000 7f987f57b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e03d86ff70 con 0x55e03d861fe0 2024-04-26T21:31:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.903+0000 7f987f57b740 10 client.0 _ll_register_callbacks cb 0x55e03d77ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:31:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f987a600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d86ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f987a600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d86ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:44848/0 (socket says 172.21.15.1:44848) 2024-04-26T21:31:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f987a600640 1 -- 172.21.15.1:0/677397008 learned_addr learned my addr 172.21.15.1:0/677397008 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f9879200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d862900 0x55e03d86a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f9879c00640 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 0x55e03d86a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f987a600640 1 -- 172.21.15.1:0/677397008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 msgr2=0x55e03d86a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f987a600640 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 0x55e03d86a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f987a600640 1 -- 172.21.15.1:0/677397008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d862900 msgr2=0x55e03d86a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f987a600640 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d862900 0x55e03d86a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f987a600640 1 -- 172.21.15.1:0/677397008 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e03d75d510 con 0x55e03d86d1c0 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.904+0000 7f9879200640 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d862900 0x55e03d86a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.905+0000 7f987a600640 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d86ac10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f986c009be0 tx=0x7f986c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c5f08f935125e81 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.905+0000 7f985b400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.905+0000 7f985b400640 1 -- 172.21.15.1:0/677397008 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c04e070 con 0x55e03d86d1c0 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.905+0000 7f987f57b740 1 -- 172.21.15.1:0/677397008 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986809d240 con 0x55e03d86d1c0 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.905+0000 7f985b400640 1 -- 172.21.15.1:0/677397008 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f986c04a880 con 0x55e03d86d1c0 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.905+0000 7f985b400640 1 -- 172.21.15.1:0/677397008 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f986c05a050 con 0x55e03d86d1c0 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.905+0000 7f987f57b740 10 client.25622 fetch_fsmap learned FSMap version 26 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.905+0000 7f987f57b740 1 -- 172.21.15.1:0/677397008 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e03d4d9210 con 0x55e03d86d1c0 2024-04-26T21:31:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.906+0000 7f985b400640 1 -- 172.21.15.1:0/677397008 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f986809d240 con 0x55e03d86d1c0 2024-04-26T21:31:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.906+0000 7f985b400640 1 -- 172.21.15.1:0/677397008 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f986c048040 con 0x55e03d86d1c0 2024-04-26T21:31:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.906+0000 7f987f57b740 10 client.25622 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:31:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.906+0000 7f987f57b740 10 client.25622 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:31:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.906+0000 7f987f57b740 20 client.25622 populate_metadata read hostname 'smithi001' 2024-04-26T21:31:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.906+0000 7f987f57b740 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e03d87cfc0 0x55e03d87f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.906+0000 7f987f57b740 4 client.25622 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:31:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.906+0000 7f987f57b740 1 -- 172.21.15.1:0/677397008 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e03d7d4f90 con 0x55e03d87cfc0 2024-04-26T21:31:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.907+0000 7f9879c00640 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e03d87cfc0 0x55e03d87f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.907+0000 7f9879c00640 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e03d87cfc0 0x55e03d87f3b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d19762570d1b6d7 server_cookie=386c893fd99c29ad in_seq=0 out_seq=0 2024-04-26T21:31:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.907+0000 7f985b400640 10 client.25622 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.908+0000 7f985b400640 1 -- 172.21.15.1:0/677397008 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e03d7d4f90 con 0x55e03d87cfc0 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.908+0000 7f985b400640 10 client.25622 handle_command_reply: tid=0 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.908+0000 7f987f57b740 1 client.25622 shutdown 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.908+0000 7f987f57b740 1 -- 172.21.15.1:0/677397008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 msgr2=0x55e03d86ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.908+0000 7f987f57b740 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d86ac10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f986c009be0 tx=0x7f986c005fd0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.909+0000 7f987f57b740 1 -- 172.21.15.1:0/677397008 shutdown_connections 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.909+0000 7f987f57b740 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e03d87cfc0 0x55e03d87f3b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.909+0000 7f987f57b740 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d86ac10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.909+0000 7f987f57b740 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d862900 0x55e03d86a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.909+0000 7f987f57b740 1 --2- 172.21.15.1:0/677397008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 0x55e03d86a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.909+0000 7f985b400640 0 client.25622 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.909+0000 7f987f57b740 1 -- 172.21.15.1:0/677397008 >> 172.21.15.1:0/677397008 conn(0x55e03d85e4f0 msgr2=0x55e03d864270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.909+0000 7f987f57b740 1 -- 172.21.15.1:0/677397008 shutdown_connections 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.909+0000 7f987f57b740 1 -- 172.21.15.1:0/677397008 wait complete. 2024-04-26T21:31:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.909+0000 7f987f57b740 20 client.25622 trim_cache size 0 max 16384 2024-04-26T21:31:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.935+0000 7f987f57b740 1 Processor -- start 2024-04-26T21:31:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f987f57b740 1 -- start start 2024-04-26T21:31:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f987f57b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 0x55e03d878cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f987f57b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d862900 0x55e03d884d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f987f57b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d86d1c0 0x55e03d885250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f987f57b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b47d0 con 0x55e03d86d1c0 2024-04-26T21:31:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f987f57b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b20a0 con 0x55e03d862900 2024-04-26T21:31:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f987f57b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98741afef0 con 0x55e03d861fe0 2024-04-26T21:31:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f9879c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 0x55e03d878cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f9879c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 0x55e03d878cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42346/0 (socket says 172.21.15.1:42346) 2024-04-26T21:31:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f9879c00640 1 -- 172.21.15.1:0/321503359 learned_addr learned my addr 172.21.15.1:0/321503359 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f9879200640 1 --2- 172.21.15.1:0/321503359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d862900 0x55e03d884d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.936+0000 7f987a600640 1 --2- 172.21.15.1:0/321503359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d86d1c0 0x55e03d885250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.937+0000 7f9879c00640 1 -- 172.21.15.1:0/321503359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d862900 msgr2=0x55e03d884d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.937+0000 7f9879c00640 1 --2- 172.21.15.1:0/321503359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d862900 0x55e03d884d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.937+0000 7f9879c00640 1 -- 172.21.15.1:0/321503359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d86d1c0 msgr2=0x55e03d885250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.937+0000 7f9879c00640 1 --2- 172.21.15.1:0/321503359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d86d1c0 0x55e03d885250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.937+0000 7f9879c00640 1 -- 172.21.15.1:0/321503359 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e03d862730 con 0x55e03d861fe0 2024-04-26T21:31:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.937+0000 7f987a600640 1 --2- 172.21.15.1:0/321503359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d86d1c0 0x55e03d885250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:31:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.937+0000 7f9879c00640 1 --2- 172.21.15.1:0/321503359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 0x55e03d878cb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9868004610 tx=0x7f9868098ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c29bce3dfbba139 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.937+0000 7f9866000640 1 -- 172.21.15.1:0/321503359 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98680b6040 con 0x55e03d861fe0 2024-04-26T21:31:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.937+0000 7f9866000640 1 -- 172.21.15.1:0/321503359 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f986809d240 con 0x55e03d861fe0 2024-04-26T21:31:40.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.937+0000 7f9866000640 1 -- 172.21.15.1:0/321503359 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98680991b0 con 0x55e03d861fe0 2024-04-26T21:31:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.938+0000 7f987f57b740 1 -- 172.21.15.1:0/321503359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 msgr2=0x55e03d878cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.938+0000 7f987f57b740 1 --2- 172.21.15.1:0/321503359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 0x55e03d878cb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9868004610 tx=0x7f9868098ed0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.938+0000 7f987f57b740 1 -- 172.21.15.1:0/321503359 shutdown_connections 2024-04-26T21:31:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.938+0000 7f987f57b740 1 --2- 172.21.15.1:0/321503359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d86d1c0 0x55e03d885250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.938+0000 7f987f57b740 1 --2- 172.21.15.1:0/321503359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d862900 0x55e03d884d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.938+0000 7f987f57b740 1 --2- 172.21.15.1:0/321503359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d861fe0 0x55e03d878cb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.938+0000 7f987f57b740 1 -- 172.21.15.1:0/321503359 >> 172.21.15.1:0/321503359 conn(0x55e03d85e4f0 msgr2=0x55e03d86c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.938+0000 7f987f57b740 1 -- 172.21.15.1:0/321503359 shutdown_connections 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.938+0000 7f987f57b740 1 -- 172.21.15.1:0/321503359 wait complete. 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.939+0000 7f987f57b740 1 -- start start 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.939+0000 7f987f57b740 1 Processor -- start 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.939+0000 7f987f57b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d870cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.939+0000 7f987f57b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d862900 0x55e03d871210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.939+0000 7f987f57b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d871750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.939+0000 7f987f57b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b0560 con 0x55e03d861fe0 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.939+0000 7f987f57b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b19d0 con 0x55e03d86d1c0 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.939+0000 7f987f57b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98741b0c30 con 0x55e03d862900 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.939+0000 7f987f57b740 10 client.0 _ll_register_callbacks cb 0x7f9868005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.940+0000 7f9879c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d870cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.940+0000 7f9879200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d862900 0x55e03d871210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.940+0000 7f987a600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d871750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.940+0000 7f9879c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d870cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44322/0 (socket says 172.21.15.1:44322) 2024-04-26T21:31:40.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.940+0000 7f9879c00640 1 -- 172.21.15.1:0/3336763571 learned_addr learned my addr 172.21.15.1:0/3336763571 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:40.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.940+0000 7f987a600640 1 -- 172.21.15.1:0/3336763571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d862900 msgr2=0x55e03d871210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:40.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.940+0000 7f987a600640 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d862900 0x55e03d871210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.940+0000 7f987a600640 1 -- 172.21.15.1:0/3336763571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 msgr2=0x55e03d870cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:40.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.940+0000 7f987a600640 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d870cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.940+0000 7f987a600640 1 -- 172.21.15.1:0/3336763571 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e03d5a5620 con 0x55e03d86d1c0 2024-04-26T21:31:40.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.941+0000 7f987a600640 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d871750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f986c004af0 tx=0x7f986c031770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8d589c6133ff37 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:40.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.941+0000 7f985b400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:31:40.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.941+0000 7f985b400640 1 -- 172.21.15.1:0/3336763571 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986c057020 con 0x55e03d86d1c0 2024-04-26T21:31:40.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.941+0000 7f987f57b740 1 -- 172.21.15.1:0/3336763571 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986809d240 con 0x55e03d86d1c0 2024-04-26T21:31:40.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.941+0000 7f985b400640 1 -- 172.21.15.1:0/3336763571 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f986c04aa90 con 0x55e03d86d1c0 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.941+0000 7f985b400640 1 -- 172.21.15.1:0/3336763571 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f986c04e040 con 0x55e03d86d1c0 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.941+0000 7f987f57b740 10 client.25628 fetch_fsmap learned FSMap version 26 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.941+0000 7f987f57b740 1 -- 172.21.15.1:0/3336763571 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e03d781cb0 con 0x55e03d86d1c0 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.942+0000 7f985b400640 1 -- 172.21.15.1:0/3336763571 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f986809d240 con 0x55e03d86d1c0 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.942+0000 7f985b400640 1 -- 172.21.15.1:0/3336763571 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f986c048040 con 0x55e03d86d1c0 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.942+0000 7f987f57b740 10 client.25628 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.942+0000 7f987f57b740 10 client.25628 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.942+0000 7f987f57b740 20 client.25628 populate_metadata read hostname 'smithi001' 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.942+0000 7f987f57b740 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e03d8e3da0 0x55e03d8e61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.942+0000 7f987f57b740 4 client.25628 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.942+0000 7f987f57b740 1 -- 172.21.15.1:0/3336763571 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e03d7cee00 con 0x55e03d8e3da0 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.943+0000 7f9879c00640 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e03d8e3da0 0x55e03d8e61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.943+0000 7f9879c00640 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e03d8e3da0 0x55e03d8e61c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6849e9b6f7f7e71e server_cookie=28516266cce191c8 in_seq=0 out_seq=0 2024-04-26T21:31:40.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.943+0000 7f985b400640 10 client.25628 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:40.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f985b400640 1 -- 172.21.15.1:0/3336763571 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e03d7cee00 con 0x55e03d8e3da0 2024-04-26T21:31:40.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f985b400640 10 client.25628 handle_command_reply: tid=0 2024-04-26T21:31:40.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f987f57b740 1 client.25628 shutdown 2024-04-26T21:31:40.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f987f57b740 1 -- 172.21.15.1:0/3336763571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 msgr2=0x55e03d871750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f987f57b740 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d871750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f986c004af0 tx=0x7f986c031770 comp rx=0 tx=0).stop 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f987f57b740 1 -- 172.21.15.1:0/3336763571 shutdown_connections 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f987f57b740 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e03d8e3da0 0x55e03d8e61c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f987f57b740 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e03d862900 0x55e03d871210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f987f57b740 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e03d86d1c0 0x55e03d871750 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f987f57b740 1 --2- 172.21.15.1:0/3336763571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e03d861fe0 0x55e03d870cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f987f57b740 1 -- 172.21.15.1:0/3336763571 >> 172.21.15.1:0/3336763571 conn(0x55e03d85e4f0 msgr2=0x55e03d86c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.949+0000 7f985b400640 0 client.25628 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.950+0000 7f987f57b740 1 -- 172.21.15.1:0/3336763571 shutdown_connections 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.950+0000 7f987f57b740 1 -- 172.21.15.1:0/3336763571 wait complete. 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.950+0000 7f987f57b740 20 client.25628 trim_cache size 0 max 16384 2024-04-26T21:31:40.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:31:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:31:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:31:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:31:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.950+0000 7f987b000640 1 -- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f985c0a3e20 msgr2=0x7f985c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f985c0a3e20 0x7f985c0a62e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f986c02ac10 tx=0x7f986c009840 comp rx=0 tx=0).stop 2024-04-26T21:31:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 -- 172.21.15.1:0/432725666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9874104960 msgr2=0x7f987419b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9874104960 0x7f987419b190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f986800e9c0 tx=0x7f986800ee80 comp rx=0 tx=0).stop 2024-04-26T21:31:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 -- 172.21.15.1:0/432725666 shutdown_connections 2024-04-26T21:31:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f985c0a3e20 0x7f985c0a62e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f987410a970 0x7f987419fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9874105320 0x7f987419b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 --2- 172.21.15.1:0/432725666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9874104960 0x7f987419b190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 -- 172.21.15.1:0/432725666 >> 172.21.15.1:0/432725666 conn(0x7f9874100430 msgr2=0x7f9874100c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 -- 172.21.15.1:0/432725666 shutdown_connections 2024-04-26T21:31:40.953 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:40.951+0000 7f987b000640 1 -- 172.21.15.1:0/432725666 wait complete. 2024-04-26T21:31:42.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:41 smithi001 ceph-mon[28054]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:42.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:41 smithi038 ceph-mon[32724]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:42.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:41 smithi186 ceph-mon[32991]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-26T21:31:44.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:43 smithi001 ceph-mon[28054]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:44.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:43 smithi038 ceph-mon[32724]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:44.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:43 smithi186 ceph-mon[32991]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-26T21:31:46.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:45 smithi001 ceph-mon[28054]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-26T21:31:46.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:45 smithi038 ceph-mon[32724]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-26T21:31:46.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:45 smithi186 ceph-mon[32991]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-26T21:31:48.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:47 smithi001 ceph-mon[28054]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T21:31:48.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:47 smithi038 ceph-mon[32724]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T21:31:48.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:47 smithi186 ceph-mon[32991]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T21:31:50.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:49 smithi001 ceph-mon[28054]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T21:31:50.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:49 smithi038 ceph-mon[32724]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T21:31:50.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:49 smithi186 ceph-mon[32991]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-26T21:31:52.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:51 smithi038 ceph-mon[32724]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-26T21:31:52.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:51 smithi038 ceph-mon[32724]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-26T21:31:52.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:51 smithi186 ceph-mon[32991]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-26T21:31:52.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:51 smithi186 ceph-mon[32991]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-26T21:31:52.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:51 smithi001 ceph-mon[28054]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-04-26T21:31:52.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:51 smithi001 ceph-mon[28054]: mgrmap e26: x(active, since 92s), standbys: z, y 2024-04-26T21:31:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.134+0000 7fdbd3000640 1 -- 172.21.15.1:0/3820864932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbcc101520 msgr2=0x7fdbcc101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.134+0000 7fdbd3000640 1 --2- 172.21.15.1:0/3820864932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbcc101520 0x7fdbcc101900 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0009900 tx=0x7fdbc002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.134+0000 7fdbd3000640 1 -- 172.21.15.1:0/3820864932 shutdown_connections 2024-04-26T21:31:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.134+0000 7fdbd3000640 1 --2- 172.21.15.1:0/3820864932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbcc101ed0 0x7fdbcc10eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.134+0000 7fdbd3000640 1 --2- 172.21.15.1:0/3820864932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbcc10f0d0 0x7fdbcc1114c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.134+0000 7fdbd3000640 1 --2- 172.21.15.1:0/3820864932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbcc101520 0x7fdbcc101900 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.134+0000 7fdbd3000640 1 -- 172.21.15.1:0/3820864932 >> 172.21.15.1:0/3820864932 conn(0x7fdbcc0fd110 msgr2=0x7fdbcc0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.134+0000 7fdbd3000640 1 -- 172.21.15.1:0/3820864932 shutdown_connections 2024-04-26T21:31:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.135+0000 7fdbd3000640 1 -- 172.21.15.1:0/3820864932 wait complete. 2024-04-26T21:31:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.135+0000 7fdbd3000640 1 Processor -- start 2024-04-26T21:31:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.136+0000 7fdbd3000640 1 -- start start 2024-04-26T21:31:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.136+0000 7fdbd3000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbcc101520 0x7fdbcc19b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.136+0000 7fdbd3000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbcc101ed0 0x7fdbcc19b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.136+0000 7fdbd3000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbcc10f0d0 0x7fdbcc19fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.136+0000 7fdbd3000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc114c40 con 0x7fdbcc101520 2024-04-26T21:31:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.136+0000 7fdbd3000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc114ac0 con 0x7fdbcc10f0d0 2024-04-26T21:31:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.136+0000 7fdbd3000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc114dc0 con 0x7fdbcc101ed0 2024-04-26T21:31:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.136+0000 7fdbd1c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbcc101520 0x7fdbcc19b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.136+0000 7fdbd1c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbcc101520 0x7fdbcc19b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38974/0 (socket says 172.21.15.1:38974) 2024-04-26T21:31:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.136+0000 7fdbd1c00640 1 -- 172.21.15.1:0/4263060929 learned_addr learned my addr 172.21.15.1:0/4263060929 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:54.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.137+0000 7fdbd1200640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbcc101ed0 0x7fdbcc19b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.137+0000 7fdbd2600640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbcc10f0d0 0x7fdbcc19fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.137+0000 7fdbd1200640 1 -- 172.21.15.1:0/4263060929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbcc10f0d0 msgr2=0x7fdbcc19fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.137+0000 7fdbd1200640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbcc10f0d0 0x7fdbcc19fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.137+0000 7fdbd1200640 1 -- 172.21.15.1:0/4263060929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbcc101520 msgr2=0x7fdbcc19b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.137+0000 7fdbd1200640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbcc101520 0x7fdbcc19b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.137+0000 7fdbd1200640 1 -- 172.21.15.1:0/4263060929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbcc1a0170 con 0x7fdbcc101ed0 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.137+0000 7fdbd2600640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbcc10f0d0 0x7fdbcc19fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.137+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbcc101520 0x7fdbcc19b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.137+0000 7fdbd1200640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbcc101ed0 0x7fdbcc19b6c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800e9f0 tx=0x7fdbb800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.138+0000 7fdbca000640 1 -- 172.21.15.1:0/4263060929 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb800ccc0 con 0x7fdbcc101ed0 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.138+0000 7fdbd3000640 1 -- 172.21.15.1:0/4263060929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbcc1a0460 con 0x7fdbcc101ed0 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.138+0000 7fdbca000640 1 -- 172.21.15.1:0/4263060929 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbb80108e0 con 0x7fdbcc101ed0 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.138+0000 7fdbca000640 1 -- 172.21.15.1:0/4263060929 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb80188a0 con 0x7fdbcc101ed0 2024-04-26T21:31:54.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.138+0000 7fdbd3000640 1 -- 172.21.15.1:0/4263060929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbcc1a7cf0 con 0x7fdbcc101ed0 2024-04-26T21:31:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.141+0000 7fdbca000640 1 -- 172.21.15.1:0/4263060929 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fdbb8010a90 con 0x7fdbcc101ed0 2024-04-26T21:31:54.143 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.142+0000 7fdbd75a9740 1 Processor -- start 2024-04-26T21:31:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.143+0000 7fdbca000640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdbb40a3d00 0x7fdbb40a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.143+0000 7fdbd75a9740 1 -- start start 2024-04-26T21:31:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.143+0000 7fdbca000640 1 -- 172.21.15.1:0/4263060929 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fdbb80d0ff0 con 0x7fdbcc101ed0 2024-04-26T21:31:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.143+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b47300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.143+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 0x560f26b5a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.143+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b5cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.143+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560f26aac380 con 0x560f26b5a9d0 2024-04-26T21:31:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.143+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560f26a48c30 con 0x560f26b46f20 2024-04-26T21:31:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.143+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560f26aa8340 con 0x560f26b47840 2024-04-26T21:31:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.143+0000 7fdbd2600640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdbb40a3d00 0x7fdbb40a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b47300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 0x560f26b5a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b47300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50452/0 (socket says 172.21.15.1:50452) 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1c00640 1 -- 172.21.15.1:0/2258329041 learned_addr learned my addr 172.21.15.1:0/2258329041 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1c00640 1 -- 172.21.15.1:0/2258329041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 msgr2=0x560f26b5a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/2258329041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 0x560f26b5a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1c00640 1 -- 172.21.15.1:0/2258329041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 msgr2=0x560f26b5cdc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/2258329041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b5cdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1c00640 1 -- 172.21.15.1:0/2258329041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f26a6a940 con 0x560f26b46f20 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1200640 1 --2- 172.21.15.1:0/2258329041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 0x560f26b5a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.144+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/2258329041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b47300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0006010 tx=0x7fdbc00044b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c88415778d308b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.145+0000 7fdbbea00640 1 -- 172.21.15.1:0/2258329041 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0040070 con 0x560f26b46f20 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.145+0000 7fdbbea00640 1 -- 172.21.15.1:0/2258329041 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbc00047a0 con 0x560f26b46f20 2024-04-26T21:31:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.145+0000 7fdbbea00640 1 -- 172.21.15.1:0/2258329041 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0035040 con 0x560f26b46f20 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2258329041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 msgr2=0x560f26b47300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/2258329041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b47300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0006010 tx=0x7fdbc00044b0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd2600640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdbb40a3d00 0x7fdbb40a61c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x560f26b44680 tx=0x7fdbc4009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2258329041 shutdown_connections 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/2258329041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 0x560f26b5a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/2258329041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b5cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/2258329041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b47300 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2258329041 >> 172.21.15.1:0/2258329041 conn(0x560f26b434f0 msgr2=0x560f26b438d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2258329041 shutdown_connections 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2258329041 wait complete. 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.146+0000 7fdbd75a9740 1 -- start start 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.147+0000 7fdbd75a9740 1 Processor -- start 2024-04-26T21:31:54.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.147+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b57970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.147+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 0x560f26b57eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.147+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b583f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.147+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560f26b5d540 con 0x560f26b5a9d0 2024-04-26T21:31:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.147+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560f26a33c60 con 0x560f26b46f20 2024-04-26T21:31:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.147+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560f26b5d6c0 con 0x560f26b47840 2024-04-26T21:31:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.147+0000 7fdbd75a9740 10 client.0 _ll_register_callbacks cb 0x560f26a5fe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:31:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b57970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b57970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50462/0 (socket says 172.21.15.1:50462) 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd1c00640 1 -- 172.21.15.1:0/2916214405 learned_addr learned my addr 172.21.15.1:0/2916214405 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd1200640 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 0x560f26b57eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd2600640 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd1c00640 1 -- 172.21.15.1:0/2916214405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 msgr2=0x560f26b57eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 0x560f26b57eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd1c00640 1 -- 172.21.15.1:0/2916214405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 msgr2=0x560f26b583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd1c00640 1 -- 172.21.15.1:0/2916214405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f26a42510 con 0x560f26b46f20 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b57970 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdbc002a430 tx=0x7fdbc0004b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40675230fceddc2d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.148+0000 7fdbb3400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.149+0000 7fdbb3400640 1 -- 172.21.15.1:0/2916214405 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0035070 con 0x560f26b46f20 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.149+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2916214405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbc00047a0 con 0x560f26b46f20 2024-04-26T21:31:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.149+0000 7fdbb3400640 1 -- 172.21.15.1:0/2916214405 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fdbc0003c40 con 0x560f26b46f20 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.149+0000 7fdbb3400640 1 -- 172.21.15.1:0/2916214405 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdbc0040040 con 0x560f26b46f20 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.149+0000 7fdbd75a9740 10 client.25637 fetch_fsmap learned FSMap version 26 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.149+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2916214405 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560f267be210 con 0x560f26b46f20 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.150+0000 7fdbb3400640 1 -- 172.21.15.1:0/2916214405 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbc00047a0 con 0x560f26b46f20 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.150+0000 7fdbb3400640 1 -- 172.21.15.1:0/2916214405 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fdbc00316a0 con 0x560f26b46f20 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.150+0000 7fdbd75a9740 10 client.25637 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.150+0000 7fdbd75a9740 10 client.25637 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.150+0000 7fdbd75a9740 20 client.25637 populate_metadata read hostname 'smithi001' 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.150+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560f26b6a570 0x560f26b6c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.150+0000 7fdbd75a9740 4 client.25637 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:31:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.150+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2916214405 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f26ab9f90 con 0x560f26b6a570 2024-04-26T21:31:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.151+0000 7fdbd1200640 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560f26b6a570 0x560f26b6c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.151+0000 7fdbd1200640 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560f26b6a570 0x560f26b6c960 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ec7817d7cef0800 server_cookie=eb3d2f734efb53ac in_seq=0 out_seq=0 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.151+0000 7fdbb3400640 10 client.25637 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbb3400640 1 -- 172.21.15.1:0/2916214405 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f26ab9f90 con 0x560f26b6a570 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbb3400640 10 client.25637 handle_command_reply: tid=0 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 client.25637 shutdown 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2916214405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 msgr2=0x560f26b57970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b57970 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdbc002a430 tx=0x7fdbc0004b80 comp rx=0 tx=0).stop 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2916214405 shutdown_connections 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560f26b6a570 0x560f26b6c960 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b47840 0x560f26b57eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/2916214405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b57970 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbb3400640 0 client.25637 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2916214405 >> 172.21.15.1:0/2916214405 conn(0x560f26b434f0 msgr2=0x560f26b49270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2916214405 shutdown_connections 2024-04-26T21:31:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.152+0000 7fdbd75a9740 1 -- 172.21.15.1:0/2916214405 wait complete. 2024-04-26T21:31:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.153+0000 7fdbd75a9740 20 client.25637 trim_cache size 0 max 16384 2024-04-26T21:31:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.168+0000 7fdbd75a9740 1 Processor -- start 2024-04-26T21:31:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.168+0000 7fdbd75a9740 1 -- start start 2024-04-26T21:31:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b46f20 0x560f26b5a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b47840 0x560f26b6f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b6fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1b0410 con 0x560f26b5a9d0 2024-04-26T21:31:54.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1adce0 con 0x560f26b47840 2024-04-26T21:31:54.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1abb30 con 0x560f26b46f20 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b46f20 0x560f26b5a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b47840 0x560f26b6f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd2600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b6fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b46f20 0x560f26b5a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44164/0 (socket says 172.21.15.1:44164) 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.169+0000 7fdbd1c00640 1 -- 172.21.15.1:0/3748953017 learned_addr learned my addr 172.21.15.1:0/3748953017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.170+0000 7fdbd1200640 1 -- 172.21.15.1:0/3748953017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b46f20 msgr2=0x560f26b5a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.170+0000 7fdbd1200640 1 --2- 172.21.15.1:0/3748953017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b46f20 0x560f26b5a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.170+0000 7fdbd1200640 1 -- 172.21.15.1:0/3748953017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 msgr2=0x560f26b6fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.170+0000 7fdbd1200640 1 --2- 172.21.15.1:0/3748953017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b6fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.170+0000 7fdbd1200640 1 -- 172.21.15.1:0/3748953017 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f2688a620 con 0x560f26b47840 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.170+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/3748953017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b46f20 0x560f26b5a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:54.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.170+0000 7fdbd1200640 1 --2- 172.21.15.1:0/3748953017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b47840 0x560f26b6f980 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80ce2d0 tx=0x7fdbb801a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47d3c0b71ed4aa2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.170+0000 7fdbbe000640 1 -- 172.21.15.1:0/3748953017 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb80bd0b0 con 0x560f26b47840 2024-04-26T21:31:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.170+0000 7fdbbe000640 1 -- 172.21.15.1:0/3748953017 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbb809de50 con 0x560f26b47840 2024-04-26T21:31:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.170+0000 7fdbbe000640 1 -- 172.21.15.1:0/3748953017 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb809c1d0 con 0x560f26b47840 2024-04-26T21:31:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.171+0000 7fdbd75a9740 1 -- 172.21.15.1:0/3748953017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b47840 msgr2=0x560f26b6f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.171+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/3748953017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b47840 0x560f26b6f980 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80ce2d0 tx=0x7fdbb801a040 comp rx=0 tx=0).stop 2024-04-26T21:31:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.171+0000 7fdbd75a9740 1 -- 172.21.15.1:0/3748953017 shutdown_connections 2024-04-26T21:31:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.171+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/3748953017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b5a9d0 0x560f26b6fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.171+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/3748953017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b47840 0x560f26b6f980 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.171+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/3748953017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b46f20 0x560f26b5a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.171+0000 7fdbd75a9740 1 -- 172.21.15.1:0/3748953017 >> 172.21.15.1:0/3748953017 conn(0x560f26b434f0 msgr2=0x560f26b42650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.171+0000 7fdbd75a9740 1 -- 172.21.15.1:0/3748953017 shutdown_connections 2024-04-26T21:31:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.171+0000 7fdbd75a9740 1 -- 172.21.15.1:0/3748953017 wait complete. 2024-04-26T21:31:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.172+0000 7fdbd75a9740 1 -- start start 2024-04-26T21:31:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.172+0000 7fdbd75a9740 1 Processor -- start 2024-04-26T21:31:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.172+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b6dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.172+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b47840 0x560f26b6e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.172+0000 7fdbd75a9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b5a9d0 0x560f26bcf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.172+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1ac1a0 con 0x560f26b47840 2024-04-26T21:31:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.172+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1ad610 con 0x560f26b46f20 2024-04-26T21:31:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.172+0000 7fdbd75a9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc1ac870 con 0x560f26b5a9d0 2024-04-26T21:31:54.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.172+0000 7fdbd75a9740 10 client.0 _ll_register_callbacks cb 0x7fdbb8098dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b6dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b6dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50476/0 (socket says 172.21.15.1:50476) 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd1c00640 1 -- 172.21.15.1:0/4279222675 learned_addr learned my addr 172.21.15.1:0/4279222675 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd2600640 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b5a9d0 0x560f26bcf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd1200640 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b47840 0x560f26b6e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd1c00640 1 -- 172.21.15.1:0/4279222675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b5a9d0 msgr2=0x560f26bcf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b5a9d0 0x560f26bcf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd1c00640 1 -- 172.21.15.1:0/4279222675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b47840 msgr2=0x560f26b6e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b47840 0x560f26b6e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd1c00640 1 -- 172.21.15.1:0/4279222675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f26a66cb0 con 0x560f26b46f20 2024-04-26T21:31:54.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.173+0000 7fdbd2600640 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b5a9d0 0x560f26bcf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.174+0000 7fdbd1c00640 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b6dac0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdbc002a430 tx=0x7fdbc0031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbd10fd7c58df0e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.174+0000 7fdbb3400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.174+0000 7fdbb3400640 1 -- 172.21.15.1:0/4279222675 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbc0035040 con 0x560f26b46f20 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.174+0000 7fdbd75a9740 1 -- 172.21.15.1:0/4279222675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbb809de50 con 0x560f26b46f20 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.174+0000 7fdbb3400640 1 -- 172.21.15.1:0/4279222675 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fdbc0003a40 con 0x560f26b46f20 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.174+0000 7fdbb3400640 1 -- 172.21.15.1:0/4279222675 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdbc00047a0 con 0x560f26b46f20 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.174+0000 7fdbd75a9740 10 client.25643 fetch_fsmap learned FSMap version 26 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.174+0000 7fdbd75a9740 1 -- 172.21.15.1:0/4279222675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560f26bd0dd0 con 0x560f26b46f20 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.175+0000 7fdbb3400640 1 -- 172.21.15.1:0/4279222675 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdbb809de50 con 0x560f26b46f20 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.175+0000 7fdbb3400640 1 -- 172.21.15.1:0/4279222675 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fdbc00052e0 con 0x560f26b46f20 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.175+0000 7fdbd75a9740 10 client.25643 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.175+0000 7fdbd75a9740 10 client.25643 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.175+0000 7fdbd75a9740 20 client.25643 populate_metadata read hostname 'smithi001' 2024-04-26T21:31:54.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.175+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560f26bd1350 0x560f26bd3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:31:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.175+0000 7fdbd75a9740 4 client.25643 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:31:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.175+0000 7fdbd75a9740 1 -- 172.21.15.1:0/4279222675 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560f26ab3e00 con 0x560f26bd1350 2024-04-26T21:31:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.175+0000 7fdbd1200640 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560f26bd1350 0x560f26bd3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:31:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.176+0000 7fdbd1200640 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560f26bd1350 0x560f26bd3740 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f8b177d0b7f82 server_cookie=d3a525dcb02030d2 in_seq=0 out_seq=0 2024-04-26T21:31:54.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.176+0000 7fdbb3400640 10 client.25643 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbb3400640 1 -- 172.21.15.1:0/4279222675 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560f26ab3e00 con 0x560f26bd1350 2024-04-26T21:31:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbb3400640 10 client.25643 handle_command_reply: tid=0 2024-04-26T21:31:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbd75a9740 1 client.25643 shutdown 2024-04-26T21:31:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbd75a9740 1 -- 172.21.15.1:0/4279222675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 msgr2=0x560f26b6dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b6dac0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdbc002a430 tx=0x7fdbc0031400 comp rx=0 tx=0).stop 2024-04-26T21:31:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbd75a9740 1 -- 172.21.15.1:0/4279222675 shutdown_connections 2024-04-26T21:31:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560f26bd1350 0x560f26bd3740 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560f26b5a9d0 0x560f26bcf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560f26b47840 0x560f26b6e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbd75a9740 1 --2- 172.21.15.1:0/4279222675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560f26b46f20 0x560f26b6dac0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbd75a9740 1 -- 172.21.15.1:0/4279222675 >> 172.21.15.1:0/4279222675 conn(0x560f26b434f0 msgr2=0x560f26b65950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.182+0000 7fdbb3400640 0 client.25643 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:31:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.183+0000 7fdbd75a9740 1 -- 172.21.15.1:0/4279222675 shutdown_connections 2024-04-26T21:31:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.183+0000 7fdbd75a9740 1 -- 172.21.15.1:0/4279222675 wait complete. 2024-04-26T21:31:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.183+0000 7fdbd75a9740 20 client.25643 trim_cache size 0 max 16384 2024-04-26T21:31:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:31:54.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:31:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:31:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:31:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.183+0000 7fdbd3000640 1 -- 172.21.15.1:0/4263060929 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdbb40a3d00 msgr2=0x7fdbb40a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.183+0000 7fdbd3000640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdbb40a3d00 0x7fdbb40a61c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x560f26b44680 tx=0x7fdbc4009500 comp rx=0 tx=0).stop 2024-04-26T21:31:54.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.183+0000 7fdbd3000640 1 -- 172.21.15.1:0/4263060929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbcc101ed0 msgr2=0x7fdbcc19b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:31:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.183+0000 7fdbd3000640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbcc101ed0 0x7fdbcc19b6c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800e9f0 tx=0x7fdbb800eeb0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.184+0000 7fdbd3000640 1 -- 172.21.15.1:0/4263060929 shutdown_connections 2024-04-26T21:31:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.184+0000 7fdbd3000640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdbb40a3d00 0x7fdbb40a61c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.184+0000 7fdbd3000640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbcc101ed0 0x7fdbcc19b6c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.184+0000 7fdbd3000640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbcc10f0d0 0x7fdbcc19fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.184+0000 7fdbd3000640 1 --2- 172.21.15.1:0/4263060929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdbcc101520 0x7fdbcc19b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:31:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.184+0000 7fdbd3000640 1 -- 172.21.15.1:0/4263060929 >> 172.21.15.1:0/4263060929 conn(0x7fdbcc0fd110 msgr2=0x7fdbcc0ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:31:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.185+0000 7fdbd3000640 1 -- 172.21.15.1:0/4263060929 shutdown_connections 2024-04-26T21:31:54.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:31:54.185+0000 7fdbd3000640 1 -- 172.21.15.1:0/4263060929 wait complete. 2024-04-26T21:31:54.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:53 smithi186 ceph-mon[32991]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-26T21:31:54.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:53 smithi001 ceph-mon[28054]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-26T21:31:54.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:53 smithi038 ceph-mon[32724]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-04-26T21:31:56.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:55 smithi186 ceph-mon[32991]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T21:31:56.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:55 smithi001 ceph-mon[28054]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T21:31:56.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:55 smithi038 ceph-mon[32724]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 9 op/s 2024-04-26T21:31:58.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:57 smithi186 ceph-mon[32991]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T21:31:58.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:57 smithi001 ceph-mon[28054]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T21:31:58.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:57 smithi038 ceph-mon[32724]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T21:32:00.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:31:59 smithi186 ceph-mon[32991]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T21:32:00.274 INFO:tasks.workunit.client.0.smithi001.stderr:Updating files: 58% (7165/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 95% (11729/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-26T21:32:00.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:31:59 smithi001 ceph-mon[28054]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T21:32:00.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:31:59 smithi038 ceph-mon[32724]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1023 B/s wr, 6 op/s 2024-04-26T21:32:01.008 INFO:tasks.workunit.client.0.smithi001.stderr:Note: switching to 'bc612ffb9c1d5bc27536bc917a705262a36d4387'. 2024-04-26T21:32:01.008 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-04-26T21:32:01.008 INFO:tasks.workunit.client.0.smithi001.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-26T21:32:01.008 INFO:tasks.workunit.client.0.smithi001.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-26T21:32:01.008 INFO:tasks.workunit.client.0.smithi001.stderr:state without impacting any branches by switching back to a branch. 2024-04-26T21:32:01.008 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-04-26T21:32:01.008 INFO:tasks.workunit.client.0.smithi001.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr: git switch -c 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr:Or undo this operation with: 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr: git switch - 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-04-26T21:32:01.009 INFO:tasks.workunit.client.0.smithi001.stderr:HEAD is now at bc612ffb9c1 Merge PR #55144 into wip-rishabh-testing-20240426.111959 2024-04-26T21:32:01.022 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-26T21:32:01.128 INFO:tasks.workunit.client.0.smithi001.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-26T21:32:01.136 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T21:32:01.136 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-26T21:32:01.958 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-26T21:32:02.071 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-26T21:32:02.162 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T21:32:02.168 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T21:32:02.168 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-26T21:32:02.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:01 smithi186 ceph-mon[32991]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T21:32:02.259 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T21:32:02.270 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:32:02.270 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-26T21:32:02.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:01 smithi001 ceph-mon[28054]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T21:32:02.327 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-04-26T21:32:02.328 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-04-26T21:32:02.328 DEBUG:teuthology.orchestra.run.smithi001:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc612ffb9c1d5bc27536bc917a705262a36d4387 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-04-26T21:32:02.329 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:01 smithi038 ceph-mon[32724]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1023 B/s wr, 8 op/s 2024-04-26T21:32:02.405 INFO:tasks.workunit.client.0.smithi001.stderr:+ dbench 1 2024-04-26T21:32:02.411 INFO:tasks.workunit.client.0.smithi001.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-26T21:32:02.412 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-04-26T21:32:02.412 INFO:tasks.workunit.client.0.smithi001.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-26T21:32:02.422 INFO:tasks.workunit.client.0.smithi001.stdout:0 of 1 processes prepared for launch 0 sec 2024-04-26T21:32:02.522 INFO:tasks.workunit.client.0.smithi001.stdout:1 of 1 processes prepared for launch 0 sec 2024-04-26T21:32:02.522 INFO:tasks.workunit.client.0.smithi001.stdout:releasing clients 2024-04-26T21:32:03.522 INFO:tasks.workunit.client.0.smithi001.stdout: 1 1552 26.10 MB/sec warmup 1 sec latency 15.759 ms 2024-04-26T21:32:04.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:03 smithi186 ceph-mon[32991]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:32:04.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:03 smithi001 ceph-mon[28054]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:32:04.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:03 smithi038 ceph-mon[32724]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2024-04-26T21:32:04.523 INFO:tasks.workunit.client.0.smithi001.stdout: 1 3878 19.88 MB/sec warmup 2 sec latency 12.001 ms 2024-04-26T21:32:05.523 INFO:tasks.workunit.client.0.smithi001.stdout: 1 5474 15.19 MB/sec warmup 3 sec latency 12.858 ms 2024-04-26T21:32:06.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:05 smithi186 ceph-mon[32991]: pgmap v56: 129 pgs: 129 active+clean; 3.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 212 KiB/s wr, 46 op/s 2024-04-26T21:32:06.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:05 smithi001 ceph-mon[28054]: pgmap v56: 129 pgs: 129 active+clean; 3.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 212 KiB/s wr, 46 op/s 2024-04-26T21:32:06.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:05 smithi038 ceph-mon[32724]: pgmap v56: 129 pgs: 129 active+clean; 3.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 212 KiB/s wr, 46 op/s 2024-04-26T21:32:06.523 INFO:tasks.workunit.client.0.smithi001.stdout: 1 7534 14.79 MB/sec warmup 4 sec latency 11.955 ms 2024-04-26T21:32:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.330+0000 7f1119400640 1 -- 172.21.15.1:0/2555787532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1114102910 msgr2=0x7f1114102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.330+0000 7f1119400640 1 --2- 172.21.15.1:0/2555787532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1114102910 0x7f1114102d90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1100009890 tx=0x7f110002a380 comp rx=0 tx=0).stop 2024-04-26T21:32:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.331+0000 7f1119400640 1 -- 172.21.15.1:0/2555787532 shutdown_connections 2024-04-26T21:32:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.331+0000 7f1119400640 1 --2- 172.21.15.1:0/2555787532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11141032d0 0x7f111410f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.331+0000 7f1119400640 1 --2- 172.21.15.1:0/2555787532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1114102910 0x7f1114102d90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.331+0000 7f1119400640 1 --2- 172.21.15.1:0/2555787532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1114108920 0x7f1114108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.331+0000 7f1119400640 1 -- 172.21.15.1:0/2555787532 >> 172.21.15.1:0/2555787532 conn(0x7f11140fe3c0 msgr2=0x7f11141007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.331+0000 7f1119400640 1 -- 172.21.15.1:0/2555787532 shutdown_connections 2024-04-26T21:32:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.331+0000 7f1119400640 1 -- 172.21.15.1:0/2555787532 wait complete. 2024-04-26T21:32:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.332+0000 7f1119400640 1 Processor -- start 2024-04-26T21:32:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.332+0000 7f1119400640 1 -- start start 2024-04-26T21:32:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.332+0000 7f1119400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1114102910 0x7f11141991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.332+0000 7f1119400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11141032d0 0x7f11141996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.332+0000 7f1119400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1114108920 0x7f111419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.332+0000 7f1119400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1114112be0 con 0x7f1114108920 2024-04-26T21:32:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.332+0000 7f1119400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1114112a60 con 0x7f11141032d0 2024-04-26T21:32:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.332+0000 7f1119400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1114112d60 con 0x7f1114102910 2024-04-26T21:32:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.333+0000 7f1113200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1114108920 0x7f111419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.333+0000 7f1111e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11141032d0 0x7f11141996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.333+0000 7f1113200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1114108920 0x7f111419da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60856/0 (socket says 172.21.15.1:60856) 2024-04-26T21:32:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.333+0000 7f1113200640 1 -- 172.21.15.1:0/2673249850 learned_addr learned my addr 172.21.15.1:0/2673249850 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.333+0000 7f1112800640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1114102910 0x7f11141991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.333+0000 7f1113200640 1 -- 172.21.15.1:0/2673249850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1114102910 msgr2=0x7f11141991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.333+0000 7f1113200640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1114102910 0x7f11141991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.333+0000 7f1113200640 1 -- 172.21.15.1:0/2673249850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11141032d0 msgr2=0x7f11141996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.333+0000 7f1113200640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11141032d0 0x7f11141996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.333+0000 7f1113200640 1 -- 172.21.15.1:0/2673249850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111419e190 con 0x7f1114108920 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.334+0000 7f1113200640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1114108920 0x7f111419da60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f110c009fd0 tx=0x7f110c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.334+0000 7f110b400640 1 -- 172.21.15.1:0/2673249850 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c004b20 con 0x7f1114108920 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.334+0000 7f1119400640 1 -- 172.21.15.1:0/2673249850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f111419e480 con 0x7f1114108920 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.334+0000 7f110b400640 1 -- 172.21.15.1:0/2673249850 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f110c003e20 con 0x7f1114108920 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.334+0000 7f110b400640 1 -- 172.21.15.1:0/2673249850 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c004820 con 0x7f1114108920 2024-04-26T21:32:07.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.334+0000 7f1119400640 1 -- 172.21.15.1:0/2673249850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11141a5d10 con 0x7f1114108920 2024-04-26T21:32:07.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.337+0000 7f111c27b740 1 Processor -- start 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.338+0000 7f110b400640 1 -- 172.21.15.1:0/2673249850 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f110c01d020 con 0x7f1114108920 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.338+0000 7f111c27b740 1 -- start start 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.338+0000 7f111c27b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bccfe5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.339+0000 7f111c27b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bccfeb00 0x5557bcd08e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.339+0000 7f111c27b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd0b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.339+0000 7f111c27b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5557bcc5f380 con 0x5557bccfeb00 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.339+0000 7f111c27b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5557bcbfbc30 con 0x5557bcd093c0 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.339+0000 7f111c27b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5557bcc5b340 con 0x5557bccfe1e0 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.339+0000 7f1111e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd0b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.339+0000 7f1111e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd0b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:45716/0 (socket says 172.21.15.1:45716) 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.339+0000 7f1111e00640 1 -- 172.21.15.1:0/3452726491 learned_addr learned my addr 172.21.15.1:0/3452726491 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:07.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.339+0000 7f1112800640 1 --2- 172.21.15.1:0/3452726491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bccfeb00 0x5557bcd08e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.342 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.339+0000 7f1111e00640 1 --2- 172.21.15.1:0/3452726491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bccfe5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.340+0000 7f1111e00640 1 -- 172.21.15.1:0/3452726491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 msgr2=0x5557bccfe5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.340+0000 7f1111e00640 1 --2- 172.21.15.1:0/3452726491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bccfe5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.340+0000 7f1111e00640 1 -- 172.21.15.1:0/3452726491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bccfeb00 msgr2=0x5557bcd08e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:07.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.340+0000 7f1111e00640 1 --2- 172.21.15.1:0/3452726491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bccfeb00 0x5557bcd08e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.340+0000 7f1111e00640 1 -- 172.21.15.1:0/3452726491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557bcc1d940 con 0x5557bcd093c0 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.340+0000 7f110b400640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f10f80a3d10 0x7f10f80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.340+0000 7f110b400640 1 -- 172.21.15.1:0/2673249850 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f110c0d0c10 con 0x7f1114108920 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.340+0000 7f1111e00640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f10f80a3d10 0x7f10f80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.340+0000 7f1111e00640 1 --2- 172.21.15.1:0/3452726491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd0b7b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f110002a8a0 tx=0x7f1100005760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=789f6bc9f23f4a5d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.341+0000 7f10ff400640 1 -- 172.21.15.1:0/3452726491 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100037040 con 0x5557bcd093c0 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.341+0000 7f10ff400640 1 -- 172.21.15.1:0/3452726491 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1100047070 con 0x5557bcd093c0 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.341+0000 7f10ff400640 1 -- 172.21.15.1:0/3452726491 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1100003a90 con 0x5557bcd093c0 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.341+0000 7f111c27b740 1 -- 172.21.15.1:0/3452726491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 msgr2=0x5557bcd0b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.341+0000 7f1111e00640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f10f80a3d10 0x7f10f80a61d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1100046000 tx=0x7f1100003d10 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.341+0000 7f111c27b740 1 --2- 172.21.15.1:0/3452726491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd0b7b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f110002a8a0 tx=0x7f1100005760 comp rx=0 tx=0).stop 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.342+0000 7f111c27b740 1 -- 172.21.15.1:0/3452726491 shutdown_connections 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.342+0000 7f111c27b740 1 --2- 172.21.15.1:0/3452726491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd0b7b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.342+0000 7f111c27b740 1 --2- 172.21.15.1:0/3452726491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bccfeb00 0x5557bcd08e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.342+0000 7f111c27b740 1 --2- 172.21.15.1:0/3452726491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bccfe5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.342+0000 7f111c27b740 1 -- 172.21.15.1:0/3452726491 >> 172.21.15.1:0/3452726491 conn(0x5557bccfa6f0 msgr2=0x5557bccfaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.342+0000 7f111c27b740 1 -- 172.21.15.1:0/3452726491 shutdown_connections 2024-04-26T21:32:07.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.342+0000 7f111c27b740 1 -- 172.21.15.1:0/3452726491 wait complete. 2024-04-26T21:32:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.342+0000 7f111c27b740 1 -- start start 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.342+0000 7f111c27b740 1 Processor -- start 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f111c27b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bcd06240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f111c27b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfeb00 0x5557bcd06780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f111c27b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bcd093c0 0x5557bcd06cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f111c27b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5557bcd0bff0 con 0x5557bcd093c0 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f111c27b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5557bcbe6c60 con 0x5557bccfeb00 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f111c27b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5557bcd0c170 con 0x5557bccfe1e0 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f111c27b740 10 client.0 _ll_register_callbacks cb 0x5557bcc12e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f1112800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bcd06240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f1111e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfeb00 0x5557bcd06780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f1112800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bcd06240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44800/0 (socket says 172.21.15.1:44800) 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f1113200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bcd093c0 0x5557bcd06cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.343+0000 7f1112800640 1 -- 172.21.15.1:0/3566884883 learned_addr learned my addr 172.21.15.1:0/3566884883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:07.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.344+0000 7f1112800640 1 -- 172.21.15.1:0/3566884883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfeb00 msgr2=0x5557bcd06780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.344+0000 7f1112800640 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfeb00 0x5557bcd06780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.344+0000 7f1112800640 1 -- 172.21.15.1:0/3566884883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bcd093c0 msgr2=0x5557bcd06cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.344+0000 7f1112800640 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bcd093c0 0x5557bcd06cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.344+0000 7f1112800640 1 -- 172.21.15.1:0/3566884883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557bcbf5510 con 0x5557bccfe1e0 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.344+0000 7f10f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.344+0000 7f1112800640 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bcd06240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1104012700 tx=0x7f1104012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8db038de2247209 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.344+0000 7f10f3e00640 1 -- 172.21.15.1:0/3566884883 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110400f920 con 0x5557bccfe1e0 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.344+0000 7f10f3e00640 1 -- 172.21.15.1:0/3566884883 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f110400fac0 con 0x5557bccfe1e0 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.344+0000 7f111c27b740 1 -- 172.21.15.1:0/3566884883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1100047070 con 0x5557bccfe1e0 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.345+0000 7f10f3e00640 1 -- 172.21.15.1:0/3566884883 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f110400fd00 con 0x5557bccfe1e0 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.345+0000 7f111c27b740 10 client.25639 fetch_fsmap learned FSMap version 26 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.345+0000 7f111c27b740 1 -- 172.21.15.1:0/3566884883 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5557bc971210 con 0x5557bccfe1e0 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.345+0000 7f10f3e00640 1 -- 172.21.15.1:0/3566884883 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1100047070 con 0x5557bccfe1e0 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.346+0000 7f10f3e00640 1 -- 172.21.15.1:0/3566884883 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f1104010430 con 0x5557bccfe1e0 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.346+0000 7f111c27b740 10 client.25639 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:32:07.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.346+0000 7f111c27b740 10 client.25639 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:32:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.346+0000 7f111c27b740 20 client.25639 populate_metadata read hostname 'smithi001' 2024-04-26T21:32:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.346+0000 7f111c27b740 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5557bcd190a0 0x5557bcd1b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.346+0000 7f111c27b740 4 client.25639 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:32:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.346+0000 7f111c27b740 1 -- 172.21.15.1:0/3566884883 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557bcc6cf90 con 0x5557bcd190a0 2024-04-26T21:32:07.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.346+0000 7f1111e00640 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5557bcd190a0 0x5557bcd1b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.347+0000 7f1111e00640 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5557bcd190a0 0x5557bcd1b4d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd8cb1f649c06af8 server_cookie=208eae77ec8639bc in_seq=0 out_seq=0 2024-04-26T21:32:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.347+0000 7f10f3e00640 10 client.25639 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f10f3e00640 1 -- 172.21.15.1:0/3566884883 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557bcc6cf90 con 0x5557bcd190a0 2024-04-26T21:32:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f10f3e00640 10 client.25639 handle_command_reply: tid=0 2024-04-26T21:32:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 client.25639 shutdown 2024-04-26T21:32:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 -- 172.21.15.1:0/3566884883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 msgr2=0x5557bcd06240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bcd06240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1104012700 tx=0x7f1104012bc0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 -- 172.21.15.1:0/3566884883 shutdown_connections 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5557bcd190a0 0x5557bcd1b4d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bcd093c0 0x5557bcd06cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfeb00 0x5557bcd06780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 --2- 172.21.15.1:0/3566884883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bcd06240 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f10f3e00640 0 client.25639 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 -- 172.21.15.1:0/3566884883 >> 172.21.15.1:0/3566884883 conn(0x5557bccfa6f0 msgr2=0x5557bccf5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 -- 172.21.15.1:0/3566884883 shutdown_connections 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.348+0000 7f111c27b740 1 -- 172.21.15.1:0/3566884883 wait complete. 2024-04-26T21:32:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.349+0000 7f111c27b740 20 client.25639 trim_cache size 0 max 16384 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.368+0000 7f111c27b740 1 Processor -- start 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.368+0000 7f111c27b740 1 -- start start 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.368+0000 7f111c27b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bcd00ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.368+0000 7f111c27b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bccfeb00 0x5557bcd08070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.368+0000 7f111c27b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd20db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.368+0000 7f111c27b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f11141b0580 con 0x5557bccfeb00 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.368+0000 7f111c27b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f11141ade50 con 0x5557bcd093c0 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.368+0000 7f111c27b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f11141abca0 con 0x5557bccfe1e0 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1113200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd20db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1113200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd20db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:45732/0 (socket says 172.21.15.1:45732) 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1113200640 1 -- 172.21.15.1:0/2591939359 learned_addr learned my addr 172.21.15.1:0/2591939359 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1111e00640 1 --2- 172.21.15.1:0/2591939359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bccfeb00 0x5557bcd08070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1112800640 1 --2- 172.21.15.1:0/2591939359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bcd00ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1113200640 1 -- 172.21.15.1:0/2591939359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 msgr2=0x5557bcd00ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1113200640 1 --2- 172.21.15.1:0/2591939359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bcd00ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1113200640 1 -- 172.21.15.1:0/2591939359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bccfeb00 msgr2=0x5557bcd08070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1113200640 1 --2- 172.21.15.1:0/2591939359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bccfeb00 0x5557bcd08070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1113200640 1 -- 172.21.15.1:0/2591939359 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557bca3d620 con 0x5557bcd093c0 2024-04-26T21:32:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.369+0000 7f1113200640 1 --2- 172.21.15.1:0/2591939359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd20db0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f110c098ba0 tx=0x7f110c00ceb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2ba6fd1911f4b94 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.370+0000 7f10fea00640 1 -- 172.21.15.1:0/2591939359 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c01a040 con 0x5557bcd093c0 2024-04-26T21:32:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.370+0000 7f10fea00640 1 -- 172.21.15.1:0/2591939359 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f110c09ce40 con 0x5557bcd093c0 2024-04-26T21:32:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.370+0000 7f10fea00640 1 -- 172.21.15.1:0/2591939359 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c098db0 con 0x5557bcd093c0 2024-04-26T21:32:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.370+0000 7f111c27b740 1 -- 172.21.15.1:0/2591939359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 msgr2=0x5557bcd20db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.370+0000 7f111c27b740 1 --2- 172.21.15.1:0/2591939359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd20db0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f110c098ba0 tx=0x7f110c00ceb0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.370+0000 7f111c27b740 1 -- 172.21.15.1:0/2591939359 shutdown_connections 2024-04-26T21:32:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.370+0000 7f111c27b740 1 --2- 172.21.15.1:0/2591939359 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bcd093c0 0x5557bcd20db0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.370+0000 7f111c27b740 1 --2- 172.21.15.1:0/2591939359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bccfeb00 0x5557bcd08070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.370+0000 7f111c27b740 1 --2- 172.21.15.1:0/2591939359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfe1e0 0x5557bcd00ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.370+0000 7f111c27b740 1 -- 172.21.15.1:0/2591939359 >> 172.21.15.1:0/2591939359 conn(0x5557bccfa6f0 msgr2=0x5557bcd07960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.371+0000 7f111c27b740 1 -- 172.21.15.1:0/2591939359 shutdown_connections 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.371+0000 7f111c27b740 1 -- 172.21.15.1:0/2591939359 wait complete. 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.371+0000 7f111c27b740 1 -- start start 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.371+0000 7f111c27b740 1 Processor -- start 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f111c27b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfe1e0 0x5557bcd0cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f111c27b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfeb00 0x5557bcd0d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f111c27b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bcd093c0 0x5557bcd0d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f111c27b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f11141ac310 con 0x5557bcd093c0 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f111c27b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f11141ad780 con 0x5557bccfe1e0 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f111c27b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f11141ac9e0 con 0x5557bccfeb00 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f111c27b740 10 client.0 _ll_register_callbacks cb 0x7f1104005a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f1112800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfe1e0 0x5557bcd0cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f1113200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bcd093c0 0x5557bcd0d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f1112800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfe1e0 0x5557bcd0cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:45746/0 (socket says 172.21.15.1:45746) 2024-04-26T21:32:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f1112800640 1 -- 172.21.15.1:0/2052934784 learned_addr learned my addr 172.21.15.1:0/2052934784 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f1111e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfeb00 0x5557bcd0d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f1112800640 1 -- 172.21.15.1:0/2052934784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfeb00 msgr2=0x5557bcd0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.372+0000 7f1112800640 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfeb00 0x5557bcd0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f1112800640 1 -- 172.21.15.1:0/2052934784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bcd093c0 msgr2=0x5557bcd0d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f1112800640 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bcd093c0 0x5557bcd0d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f1112800640 1 -- 172.21.15.1:0/2052934784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557bcc19cb0 con 0x5557bccfe1e0 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f1112800640 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfe1e0 0x5557bcd0cec0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1104000c00 tx=0x7f11040123d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d697c742b2da4efa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f10f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f10f3e00640 1 -- 172.21.15.1:0/2052934784 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110402d070 con 0x5557bccfe1e0 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f111c27b740 1 -- 172.21.15.1:0/2052934784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f110c09ce40 con 0x5557bccfe1e0 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f10f3e00640 1 -- 172.21.15.1:0/2052934784 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f1104010a90 con 0x5557bccfe1e0 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f10f3e00640 1 -- 172.21.15.1:0/2052934784 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1104010e60 con 0x5557bccfe1e0 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f111c27b740 10 client.25661 fetch_fsmap learned FSMap version 26 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.373+0000 7f111c27b740 1 -- 172.21.15.1:0/2052934784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5557bcd7f830 con 0x5557bccfe1e0 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.374+0000 7f10f3e00640 1 -- 172.21.15.1:0/2052934784 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f110c09ce40 con 0x5557bccfe1e0 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.374+0000 7f10f3e00640 1 -- 172.21.15.1:0/2052934784 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f1104030020 con 0x5557bccfe1e0 2024-04-26T21:32:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.374+0000 7f111c27b740 10 client.25661 fetch_fsmap finished waiting for FSMap version 26 2024-04-26T21:32:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.374+0000 7f111c27b740 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:32:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.374+0000 7f111c27b740 20 client.25661 populate_metadata read hostname 'smithi001' 2024-04-26T21:32:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.374+0000 7f111c27b740 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5557bcd7ff70 0x5557bcd82390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.374+0000 7f111c27b740 4 client.25661 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:32:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.374+0000 7f111c27b740 1 -- 172.21.15.1:0/2052934784 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5557bcc66e00 con 0x5557bcd7ff70 2024-04-26T21:32:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.374+0000 7f1111e00640 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5557bcd7ff70 0x5557bcd82390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.375+0000 7f1111e00640 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5557bcd7ff70 0x5557bcd82390 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce66ad19b4457a13 server_cookie=b6e814ee21ce9f9a in_seq=0 out_seq=0 2024-04-26T21:32:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.375+0000 7f10f3e00640 10 client.25661 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f10f3e00640 1 -- 172.21.15.1:0/2052934784 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5557bcc66e00 con 0x5557bcd7ff70 2024-04-26T21:32:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f10f3e00640 10 client.25661 handle_command_reply: tid=0 2024-04-26T21:32:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 client.25661 shutdown 2024-04-26T21:32:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 -- 172.21.15.1:0/2052934784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfe1e0 msgr2=0x5557bcd0cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfe1e0 0x5557bcd0cec0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1104000c00 tx=0x7f11040123d0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 -- 172.21.15.1:0/2052934784 shutdown_connections 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5557bcd7ff70 0x5557bcd82390 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5557bccfeb00 0x5557bcd0d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5557bcd093c0 0x5557bcd0d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 --2- 172.21.15.1:0/2052934784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5557bccfe1e0 0x5557bcd0cec0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 -- 172.21.15.1:0/2052934784 >> 172.21.15.1:0/2052934784 conn(0x5557bccfa6f0 msgr2=0x5557bccf5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f10f3e00640 0 client.25661 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 -- 172.21.15.1:0/2052934784 shutdown_connections 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.381+0000 7f111c27b740 1 -- 172.21.15.1:0/2052934784 wait complete. 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.382+0000 7f111c27b740 20 client.25661 trim_cache size 0 max 16384 2024-04-26T21:32:07.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:32:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:32:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:32:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:32:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.382+0000 7f1119400640 1 -- 172.21.15.1:0/2673249850 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f10f80a3d10 msgr2=0x7f10f80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.382+0000 7f1119400640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f10f80a3d10 0x7f10f80a61d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1100046000 tx=0x7f1100003d10 comp rx=0 tx=0).stop 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.382+0000 7f1119400640 1 -- 172.21.15.1:0/2673249850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1114108920 msgr2=0x7f111419da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.382+0000 7f1119400640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1114108920 0x7f111419da60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f110c009fd0 tx=0x7f110c00c800 comp rx=0 tx=0).stop 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.383+0000 7f1119400640 1 -- 172.21.15.1:0/2673249850 shutdown_connections 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.383+0000 7f1119400640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f10f80a3d10 0x7f10f80a61d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.383+0000 7f1119400640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1114108920 0x7f111419da60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.383+0000 7f1119400640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f11141032d0 0x7f11141996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.383+0000 7f1119400640 1 --2- 172.21.15.1:0/2673249850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1114102910 0x7f11141991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.383+0000 7f1119400640 1 -- 172.21.15.1:0/2673249850 >> 172.21.15.1:0/2673249850 conn(0x7f11140fe3c0 msgr2=0x7f11140fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.383+0000 7f1119400640 1 -- 172.21.15.1:0/2673249850 shutdown_connections 2024-04-26T21:32:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:07.383+0000 7f1119400640 1 -- 172.21.15.1:0/2673249850 wait complete. 2024-04-26T21:32:07.523 INFO:tasks.workunit.client.0.smithi001.stdout: 1 9303 13.00 MB/sec warmup 5 sec latency 18.908 ms 2024-04-26T21:32:08.120 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:07 smithi001 ceph-mon[28054]: pgmap v57: 129 pgs: 129 active+clean; 3.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 211 KiB/s wr, 43 op/s 2024-04-26T21:32:08.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:07 smithi186 ceph-mon[32991]: pgmap v57: 129 pgs: 129 active+clean; 3.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 211 KiB/s wr, 43 op/s 2024-04-26T21:32:08.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:07 smithi038 ceph-mon[32724]: pgmap v57: 129 pgs: 129 active+clean; 3.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 211 KiB/s wr, 43 op/s 2024-04-26T21:32:08.523 INFO:tasks.workunit.client.0.smithi001.stdout: 1 10654 12.78 MB/sec warmup 6 sec latency 321.956 ms 2024-04-26T21:32:09.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:08 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:09.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:08 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:09.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:08 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:09.523 INFO:tasks.workunit.client.0.smithi001.stdout: 1 10654 10.95 MB/sec warmup 7 sec latency 1322.102 ms 2024-04-26T21:32:10.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:09 smithi186 ceph-mon[32991]: pgmap v58: 129 pgs: 129 active+clean; 3.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 211 KiB/s wr, 43 op/s 2024-04-26T21:32:10.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:09 smithi001 ceph-mon[28054]: pgmap v58: 129 pgs: 129 active+clean; 3.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 211 KiB/s wr, 43 op/s 2024-04-26T21:32:10.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:09 smithi038 ceph-mon[32724]: pgmap v58: 129 pgs: 129 active+clean; 3.3 MiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 211 KiB/s wr, 43 op/s 2024-04-26T21:32:10.523 INFO:tasks.workunit.client.0.smithi001.stdout: 1 10654 9.58 MB/sec warmup 8 sec latency 2322.244 ms 2024-04-26T21:32:11.523 INFO:tasks.workunit.client.0.smithi001.stdout: 1 10764 8.55 MB/sec warmup 9 sec latency 3080.292 ms 2024-04-26T21:32:12.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:11 smithi186 ceph-mon[32991]: pgmap v59: 129 pgs: 129 active+clean; 6.5 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 556 KiB/s wr, 87 op/s 2024-04-26T21:32:12.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:11 smithi001 ceph-mon[28054]: pgmap v59: 129 pgs: 129 active+clean; 6.5 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 556 KiB/s wr, 87 op/s 2024-04-26T21:32:12.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:11 smithi038 ceph-mon[32724]: pgmap v59: 129 pgs: 129 active+clean; 6.5 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 556 KiB/s wr, 87 op/s 2024-04-26T21:32:12.524 INFO:tasks.workunit.client.0.smithi001.stdout: 1 10786 7.70 MB/sec warmup 10 sec latency 704.747 ms 2024-04-26T21:32:13.524 INFO:tasks.workunit.client.0.smithi001.stdout: 1 10786 7.00 MB/sec warmup 11 sec latency 1422.021 ms 2024-04-26T21:32:14.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:13 smithi186 ceph-mon[32991]: pgmap v60: 129 pgs: 129 active+clean; 6.5 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 556 KiB/s wr, 84 op/s 2024-04-26T21:32:14.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:13 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:13 smithi001 ceph-mon[28054]: pgmap v60: 129 pgs: 129 active+clean; 6.5 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 556 KiB/s wr, 84 op/s 2024-04-26T21:32:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:13 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:13 smithi038 ceph-mon[32724]: pgmap v60: 129 pgs: 129 active+clean; 6.5 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 556 KiB/s wr, 84 op/s 2024-04-26T21:32:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:13 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:14.524 INFO:tasks.workunit.client.0.smithi001.stdout: 1 10786 6.41 MB/sec warmup 12 sec latency 2422.141 ms 2024-04-26T21:32:15.524 INFO:tasks.workunit.client.0.smithi001.stdout: 1 10786 5.92 MB/sec warmup 13 sec latency 3422.320 ms 2024-04-26T21:32:16.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:15 smithi186 ceph-mon[32991]: pgmap v61: 129 pgs: 129 active+clean; 6.6 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 574 KiB/s wr, 92 op/s 2024-04-26T21:32:16.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:15 smithi001 ceph-mon[28054]: pgmap v61: 129 pgs: 129 active+clean; 6.6 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 574 KiB/s wr, 92 op/s 2024-04-26T21:32:16.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:15 smithi038 ceph-mon[32724]: pgmap v61: 129 pgs: 129 active+clean; 6.6 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 574 KiB/s wr, 92 op/s 2024-04-26T21:32:16.524 INFO:tasks.workunit.client.0.smithi001.stdout: 1 10895 5.57 MB/sec warmup 14 sec latency 4181.362 ms 2024-04-26T21:32:17.524 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11237 5.22 MB/sec warmup 15 sec latency 22.955 ms 2024-04-26T21:32:18.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:17 smithi186 ceph-mon[32991]: pgmap v62: 129 pgs: 129 active+clean; 6.6 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 363 KiB/s wr, 51 op/s 2024-04-26T21:32:18.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:17 smithi001 ceph-mon[28054]: pgmap v62: 129 pgs: 129 active+clean; 6.6 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 363 KiB/s wr, 51 op/s 2024-04-26T21:32:18.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:17 smithi038 ceph-mon[32724]: pgmap v62: 129 pgs: 129 active+clean; 6.6 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 363 KiB/s wr, 51 op/s 2024-04-26T21:32:18.524 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11370 4.91 MB/sec warmup 16 sec latency 814.516 ms 2024-04-26T21:32:19.525 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11370 4.62 MB/sec warmup 17 sec latency 1814.640 ms 2024-04-26T21:32:20.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:19 smithi186 ceph-mon[32991]: pgmap v63: 129 pgs: 129 active+clean; 6.6 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 363 KiB/s wr, 51 op/s 2024-04-26T21:32:20.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:19 smithi001 ceph-mon[28054]: pgmap v63: 129 pgs: 129 active+clean; 6.6 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 363 KiB/s wr, 51 op/s 2024-04-26T21:32:20.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:19 smithi038 ceph-mon[32724]: pgmap v63: 129 pgs: 129 active+clean; 6.6 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 363 KiB/s wr, 51 op/s 2024-04-26T21:32:20.525 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11370 4.36 MB/sec warmup 18 sec latency 2814.763 ms 2024-04-26T21:32:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.531+0000 7fb8bbe00640 1 -- 172.21.15.1:0/3283559336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8bc107650 msgr2=0x7fb8bc107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.531+0000 7fb8bbe00640 1 --2- 172.21.15.1:0/3283559336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8bc107650 0x7fb8bc107a30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009c10 tx=0x7fb8b402a6f0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.532+0000 7fb8bbe00640 1 -- 172.21.15.1:0/3283559336 shutdown_connections 2024-04-26T21:32:20.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.532+0000 7fb8bbe00640 1 --2- 172.21.15.1:0/3283559336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8bc102000 0x7fb8bc10e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.532+0000 7fb8bbe00640 1 --2- 172.21.15.1:0/3283559336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc101640 0x7fb8bc101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.532+0000 7fb8bbe00640 1 --2- 172.21.15.1:0/3283559336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8bc107650 0x7fb8bc107a30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.532+0000 7fb8bbe00640 1 -- 172.21.15.1:0/3283559336 >> 172.21.15.1:0/3283559336 conn(0x7fb8bc0fd110 msgr2=0x7fb8bc0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.532+0000 7fb8bbe00640 1 -- 172.21.15.1:0/3283559336 shutdown_connections 2024-04-26T21:32:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.532+0000 7fb8bbe00640 1 -- 172.21.15.1:0/3283559336 wait complete. 2024-04-26T21:32:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.533+0000 7fb8bbe00640 1 Processor -- start 2024-04-26T21:32:20.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.533+0000 7fb8bbe00640 1 -- start start 2024-04-26T21:32:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.533+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8bc101640 0x7fb8bc19b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.533+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102000 0x7fb8bc19b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.533+0000 7fb8bbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8bc107650 0x7fb8bc19fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8bbe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc112a10 con 0x7fb8bc101640 2024-04-26T21:32:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8bbe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc112890 con 0x7fb8bc102000 2024-04-26T21:32:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8bbe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc112b90 con 0x7fb8bc107650 2024-04-26T21:32:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8baa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8bc101640 0x7fb8bc19b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8baa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8bc101640 0x7fb8bc19b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36064/0 (socket says 172.21.15.1:36064) 2024-04-26T21:32:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8ba000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102000 0x7fb8bc19b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8baa00640 1 -- 172.21.15.1:0/1054752568 learned_addr learned my addr 172.21.15.1:0/1054752568 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:20.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8bb400640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8bc107650 0x7fb8bc19fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8ba000640 1 -- 172.21.15.1:0/1054752568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8bc107650 msgr2=0x7fb8bc19fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8ba000640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8bc107650 0x7fb8bc19fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8ba000640 1 -- 172.21.15.1:0/1054752568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8bc101640 msgr2=0x7fb8bc19b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8ba000640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8bc101640 0x7fb8bc19b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.534+0000 7fb8ba000640 1 -- 172.21.15.1:0/1054752568 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8bc1a01f0 con 0x7fb8bc102000 2024-04-26T21:32:20.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.535+0000 7fb8ba000640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102000 0x7fb8bc19b740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400e9f0 tx=0x7fb8a400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.535+0000 7fb8b3400640 1 -- 172.21.15.1:0/1054752568 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400ccc0 con 0x7fb8bc102000 2024-04-26T21:32:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.535+0000 7fb8bbe00640 1 -- 172.21.15.1:0/1054752568 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8bc1a04e0 con 0x7fb8bc102000 2024-04-26T21:32:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.535+0000 7fb8b3400640 1 -- 172.21.15.1:0/1054752568 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8a40035f0 con 0x7fb8bc102000 2024-04-26T21:32:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.535+0000 7fb8b3400640 1 -- 172.21.15.1:0/1054752568 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400ce60 con 0x7fb8bc102000 2024-04-26T21:32:20.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.535+0000 7fb8bbe00640 1 -- 172.21.15.1:0/1054752568 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8bc1a7d70 con 0x7fb8bc102000 2024-04-26T21:32:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.538+0000 7fb8c47b4740 1 Processor -- start 2024-04-26T21:32:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.538+0000 7fb8c47b4740 1 -- start start 2024-04-26T21:32:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8b3400640 1 -- 172.21.15.1:0/1054752568 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb8a40037a0 con 0x7fb8bc102000 2024-04-26T21:32:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 0x561dcf7333c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf733900 0x561dcf73dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf73e1c0 0x561dcf7405b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x561dcf698380 con 0x561dcf733900 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561dcf634c30 con 0x561dcf732fe0 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561dcf694340 con 0x561dcf73e1c0 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8baa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf733900 0x561dcf73dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8bb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 0x561dcf7333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8baa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf733900 0x561dcf73dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36066/0 (socket says 172.21.15.1:36066) 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8baa00640 1 -- 172.21.15.1:0/2141365959 learned_addr learned my addr 172.21.15.1:0/2141365959 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.539+0000 7fb8ba000640 1 --2- 172.21.15.1:0/2141365959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf73e1c0 0x561dcf7405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.540+0000 7fb8bb400640 1 -- 172.21.15.1:0/2141365959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf73e1c0 msgr2=0x561dcf7405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.540+0000 7fb8bb400640 1 --2- 172.21.15.1:0/2141365959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf73e1c0 0x561dcf7405b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.540+0000 7fb8bb400640 1 -- 172.21.15.1:0/2141365959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf733900 msgr2=0x561dcf73dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.540+0000 7fb8bb400640 1 --2- 172.21.15.1:0/2141365959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf733900 0x561dcf73dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.540+0000 7fb8bb400640 1 -- 172.21.15.1:0/2141365959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dcf656940 con 0x561dcf732fe0 2024-04-26T21:32:20.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.540+0000 7fb8bb400640 1 --2- 172.21.15.1:0/2141365959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 0x561dcf7333c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009b90 tx=0x7fb8ac011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78cbf8a34b334dea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:20.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.540+0000 7fb8abe00640 1 -- 172.21.15.1:0/2141365959 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac01e070 con 0x561dcf732fe0 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.540+0000 7fb8abe00640 1 -- 172.21.15.1:0/2141365959 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8ac0035f0 con 0x561dcf732fe0 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.540+0000 7fb8abe00640 1 -- 172.21.15.1:0/2141365959 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac019670 con 0x561dcf732fe0 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8b3400640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb8a00a3d00 0x7fb8a00a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8b3400640 1 -- 172.21.15.1:0/1054752568 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fb8a40d0ea0 con 0x7fb8bc102000 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8baa00640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb8a00a3d00 0x7fb8a00a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2141365959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 msgr2=0x561dcf7333c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/2141365959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 0x561dcf7333c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009b90 tx=0x7fb8ac011810 comp rx=0 tx=0).stop 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2141365959 shutdown_connections 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/2141365959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf73e1c0 0x561dcf7405b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/2141365959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf733900 0x561dcf73dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/2141365959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 0x561dcf7333c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2141365959 >> 172.21.15.1:0/2141365959 conn(0x561dcf72f4f0 msgr2=0x561dcf72f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:20.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2141365959 shutdown_connections 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.541+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2141365959 wait complete. 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.542+0000 7fb8c47b4740 1 -- start start 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.542+0000 7fb8baa00640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb8a00a3d00 0x7fb8a00a61c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x561dcf734580 tx=0x7fb8b4009280 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.542+0000 7fb8c47b4740 1 Processor -- start 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf732fe0 0x561dcf73b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf733900 0x561dcf73b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf73bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x561dcf740df0 con 0x561dcf73e1c0 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x561dcf61fc60 con 0x561dcf733900 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561dcf740f70 con 0x561dcf732fe0 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8c47b4740 10 client.0 _ll_register_callbacks cb 0x561dcf64be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8bb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf73bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8baa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf732fe0 0x561dcf73b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8bb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf73bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36082/0 (socket says 172.21.15.1:36082) 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8baa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf732fe0 0x561dcf73b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:60760/0 (socket says 172.21.15.1:60760) 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8bb400640 1 -- 172.21.15.1:0/2407018534 learned_addr learned my addr 172.21.15.1:0/2407018534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8ba000640 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf733900 0x561dcf73b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8baa00640 1 -- 172.21.15.1:0/2407018534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf733900 msgr2=0x561dcf73b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8baa00640 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf733900 0x561dcf73b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.543+0000 7fb8baa00640 1 -- 172.21.15.1:0/2407018534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 msgr2=0x561dcf73bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.544+0000 7fb8baa00640 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf73bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.544+0000 7fb8baa00640 1 -- 172.21.15.1:0/2407018534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dcf62e510 con 0x561dcf732fe0 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.544+0000 7fb8bb400640 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf73bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.544+0000 7fb8baa00640 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf732fe0 0x561dcf73b150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009be0 tx=0x7fb8b4009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a21c517b5a05886 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.544+0000 7fb8a8c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.544+0000 7fb8a8c00640 1 -- 172.21.15.1:0/2407018534 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b404f070 con 0x561dcf732fe0 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.544+0000 7fb8a8c00640 1 -- 172.21.15.1:0/2407018534 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fb8b4030d80 con 0x561dcf732fe0 2024-04-26T21:32:20.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.544+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2407018534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8ac0035f0 con 0x561dcf732fe0 2024-04-26T21:32:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8a8c00640 1 -- 172.21.15.1:0/2407018534 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb8b405b050 con 0x561dcf732fe0 2024-04-26T21:32:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8c47b4740 10 client.25657 fetch_fsmap learned FSMap version 28 2024-04-26T21:32:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2407018534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561dcf3aa210 con 0x561dcf732fe0 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8a8c00640 1 -- 172.21.15.1:0/2407018534 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8ac0035f0 con 0x561dcf732fe0 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8a8c00640 1 -- 172.21.15.1:0/2407018534 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb8b404b720 con 0x561dcf732fe0 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8c47b4740 10 client.25657 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8c47b4740 10 client.25657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8c47b4740 20 client.25657 populate_metadata read hostname 'smithi001' 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x561dcf74e000 0x561dcf750430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8c47b4740 4 client.25657 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.545+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2407018534 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dcf6a5f90 con 0x561dcf74e000 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.546+0000 7fb8ba000640 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x561dcf74e000 0x561dcf750430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.546+0000 7fb8ba000640 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x561dcf74e000 0x561dcf750430 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9689534fc9901e0 server_cookie=b497f711775e25fe in_seq=0 out_seq=0 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.546+0000 7fb8a8c00640 10 client.25657 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8a8c00640 1 -- 172.21.15.1:0/2407018534 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561dcf6a5f90 con 0x561dcf74e000 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8a8c00640 10 client.25657 handle_command_reply: tid=0 2024-04-26T21:32:20.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 client.25657 shutdown 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2407018534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf732fe0 msgr2=0x561dcf73b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf732fe0 0x561dcf73b150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009be0 tx=0x7fb8b4009500 comp rx=0 tx=0).stop 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2407018534 shutdown_connections 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x561dcf74e000 0x561dcf750430 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf73bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf733900 0x561dcf73b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/2407018534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf732fe0 0x561dcf73b150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2407018534 >> 172.21.15.1:0/2407018534 conn(0x561dcf72f4f0 msgr2=0x561dcf735270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8a8c00640 0 client.25657 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2407018534 shutdown_connections 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.547+0000 7fb8c47b4740 1 -- 172.21.15.1:0/2407018534 wait complete. 2024-04-26T21:32:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.548+0000 7fb8c47b4740 20 client.25657 trim_cache size 0 max 16384 2024-04-26T21:32:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.565+0000 7fb8c47b4740 1 Processor -- start 2024-04-26T21:32:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.565+0000 7fb8c47b4740 1 -- start start 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf732fe0 0x561dcf749cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 0x561dcf73d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf73e1c0 0x561dcf755d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1b4740 con 0x561dcf732fe0 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1b1fb0 con 0x561dcf73e1c0 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1afe00 con 0x561dcf733900 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8baa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf732fe0 0x561dcf749cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8baa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf732fe0 0x561dcf749cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36084/0 (socket says 172.21.15.1:36084) 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8baa00640 1 -- 172.21.15.1:0/765434724 learned_addr learned my addr 172.21.15.1:0/765434724 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8bb400640 1 --2- 172.21.15.1:0/765434724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf73e1c0 0x561dcf755d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8ba000640 1 --2- 172.21.15.1:0/765434724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 0x561dcf73d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8baa00640 1 -- 172.21.15.1:0/765434724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 msgr2=0x561dcf73d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8baa00640 1 --2- 172.21.15.1:0/765434724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 0x561dcf73d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8baa00640 1 -- 172.21.15.1:0/765434724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf73e1c0 msgr2=0x561dcf755d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8baa00640 1 --2- 172.21.15.1:0/765434724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf73e1c0 0x561dcf755d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.566+0000 7fb8baa00640 1 -- 172.21.15.1:0/765434724 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dcf476620 con 0x561dcf732fe0 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.567+0000 7fb8baa00640 1 --2- 172.21.15.1:0/765434724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf732fe0 0x561dcf749cb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb8b402a730 tx=0x7fb8b4003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a51a42a82c56d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.567+0000 7fb8ab400640 1 -- 172.21.15.1:0/765434724 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b404f050 con 0x561dcf732fe0 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.567+0000 7fb8ab400640 1 -- 172.21.15.1:0/765434724 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8b404b900 con 0x561dcf732fe0 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.567+0000 7fb8ab400640 1 -- 172.21.15.1:0/765434724 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b4036040 con 0x561dcf732fe0 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.568+0000 7fb8c47b4740 1 -- 172.21.15.1:0/765434724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf732fe0 msgr2=0x561dcf749cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.568+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/765434724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf732fe0 0x561dcf749cb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb8b402a730 tx=0x7fb8b4003dc0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.568+0000 7fb8c47b4740 1 -- 172.21.15.1:0/765434724 shutdown_connections 2024-04-26T21:32:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.568+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/765434724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 0x561dcf73d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.568+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/765434724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf73e1c0 0x561dcf755d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.568+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/765434724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf732fe0 0x561dcf749cb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.568+0000 7fb8c47b4740 1 -- 172.21.15.1:0/765434724 >> 172.21.15.1:0/765434724 conn(0x561dcf72f4f0 msgr2=0x561dcf73c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.568+0000 7fb8c47b4740 1 -- 172.21.15.1:0/765434724 shutdown_connections 2024-04-26T21:32:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.568+0000 7fb8c47b4740 1 -- 172.21.15.1:0/765434724 wait complete. 2024-04-26T21:32:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.568+0000 7fb8c47b4740 1 -- start start 2024-04-26T21:32:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8c47b4740 1 Processor -- start 2024-04-26T21:32:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 0x561dcf741e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 0x561dcf742370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8c47b4740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf7428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1b0470 con 0x561dcf73e1c0 2024-04-26T21:32:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1b18e0 con 0x561dcf732fe0 2024-04-26T21:32:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8c47b4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc1b0b40 con 0x561dcf733900 2024-04-26T21:32:20.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8c47b4740 10 client.0 _ll_register_callbacks cb 0x7fb8a4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8bb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf7428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8ba000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 0x561dcf742370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8bb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf7428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36086/0 (socket says 172.21.15.1:36086) 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8bb400640 1 -- 172.21.15.1:0/1055876939 learned_addr learned my addr 172.21.15.1:0/1055876939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.569+0000 7fb8ba000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 0x561dcf742370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:60772/0 (socket says 172.21.15.1:60772) 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.570+0000 7fb8baa00640 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 0x561dcf741e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.570+0000 7fb8ba000640 1 -- 172.21.15.1:0/1055876939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 msgr2=0x561dcf741e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.570+0000 7fb8ba000640 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 0x561dcf741e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.570+0000 7fb8ba000640 1 -- 172.21.15.1:0/1055876939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 msgr2=0x561dcf7428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.570+0000 7fb8ba000640 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf7428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.570+0000 7fb8ba000640 1 -- 172.21.15.1:0/1055876939 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dcf652cb0 con 0x561dcf733900 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.570+0000 7fb8bb400640 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf7428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:20.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.570+0000 7fb8baa00640 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 0x561dcf741e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.570+0000 7fb8ba000640 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 0x561dcf742370 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb8a40ce180 tx=0x7fb8a40bd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35da02cca0742f26 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.570+0000 7fb8a8c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8a8c00640 1 -- 172.21.15.1:0/1055876939 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a409c9d0 con 0x561dcf733900 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8a8c00640 1 -- 172.21.15.1:0/1055876939 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fb8a4003ac0 con 0x561dcf733900 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8c47b4740 1 -- 172.21.15.1:0/1055876939 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8b404b900 con 0x561dcf733900 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8a8c00640 1 -- 172.21.15.1:0/1055876939 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb8a40a0d80 con 0x561dcf733900 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8c47b4740 10 client.25663 fetch_fsmap learned FSMap version 28 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8c47b4740 1 -- 172.21.15.1:0/1055876939 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561dcf7bd0f0 con 0x561dcf733900 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8a8c00640 1 -- 172.21.15.1:0/1055876939 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8b404b900 con 0x561dcf733900 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8a8c00640 1 -- 172.21.15.1:0/1055876939 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb8a40bbb90 con 0x561dcf733900 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8c47b4740 10 client.25663 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8c47b4740 10 client.25663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8c47b4740 20 client.25663 populate_metadata read hostname 'smithi001' 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.571+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x561dcf7bd8d0 0x561dcf7bfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:20.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.572+0000 7fb8c47b4740 4 client.25663 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:32:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.572+0000 7fb8c47b4740 1 -- 172.21.15.1:0/1055876939 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561dcf69fe00 con 0x561dcf7bd8d0 2024-04-26T21:32:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.572+0000 7fb8baa00640 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x561dcf7bd8d0 0x561dcf7bfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.572+0000 7fb8baa00640 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x561dcf7bd8d0 0x561dcf7bfcf0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86ee2a7a60970b36 server_cookie=e7a681a52a7ca872 in_seq=0 out_seq=0 2024-04-26T21:32:20.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.572+0000 7fb8a8c00640 10 client.25663 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8a8c00640 1 -- 172.21.15.1:0/1055876939 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561dcf69fe00 con 0x561dcf7bd8d0 2024-04-26T21:32:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8a8c00640 10 client.25663 handle_command_reply: tid=0 2024-04-26T21:32:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 client.25663 shutdown 2024-04-26T21:32:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 -- 172.21.15.1:0/1055876939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 msgr2=0x561dcf742370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 0x561dcf742370 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb8a40ce180 tx=0x7fb8a40bd040 comp rx=0 tx=0).stop 2024-04-26T21:32:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 -- 172.21.15.1:0/1055876939 shutdown_connections 2024-04-26T21:32:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x561dcf7bd8d0 0x561dcf7bfcf0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dcf733900 0x561dcf742370 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x561dcf73e1c0 0x561dcf7428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 --2- 172.21.15.1:0/1055876939 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561dcf732fe0 0x561dcf741e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 -- 172.21.15.1:0/1055876939 >> 172.21.15.1:0/1055876939 conn(0x561dcf72f4f0 msgr2=0x561dcf72e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:20.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8a8c00640 0 client.25663 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:20.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 -- 172.21.15.1:0/1055876939 shutdown_connections 2024-04-26T21:32:20.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.613+0000 7fb8c47b4740 1 -- 172.21.15.1:0/1055876939 wait complete. 2024-04-26T21:32:20.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8c47b4740 20 client.25663 trim_cache size 0 max 16384 2024-04-26T21:32:20.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:32:20.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:32:20.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:32:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:32:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8bbe00640 1 -- 172.21.15.1:0/1054752568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb8a00a3d00 msgr2=0x7fb8a00a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8bbe00640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb8a00a3d00 0x7fb8a00a61c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x561dcf734580 tx=0x7fb8b4009280 comp rx=0 tx=0).stop 2024-04-26T21:32:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8bbe00640 1 -- 172.21.15.1:0/1054752568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102000 msgr2=0x7fb8bc19b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8bbe00640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102000 0x7fb8bc19b740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400e9f0 tx=0x7fb8a400edf0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8bbe00640 1 -- 172.21.15.1:0/1054752568 shutdown_connections 2024-04-26T21:32:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8bbe00640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb8a00a3d00 0x7fb8a00a61c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8bbe00640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8bc107650 0x7fb8bc19fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8bbe00640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8bc102000 0x7fb8bc19b740 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8bbe00640 1 --2- 172.21.15.1:0/1054752568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8bc101640 0x7fb8bc19b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.614+0000 7fb8bbe00640 1 -- 172.21.15.1:0/1054752568 >> 172.21.15.1:0/1054752568 conn(0x7fb8bc0fd110 msgr2=0x7fb8bc10c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.615+0000 7fb8bbe00640 1 -- 172.21.15.1:0/1054752568 shutdown_connections 2024-04-26T21:32:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:20.615+0000 7fb8bbe00640 1 -- 172.21.15.1:0/1054752568 wait complete. 2024-04-26T21:32:21.525 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11561 4.16 MB/sec warmup 19 sec latency 2903.180 ms 2024-04-26T21:32:22.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:21 smithi186 ceph-mon[32991]: pgmap v64: 129 pgs: 129 active+clean; 7.0 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 414 KiB/s wr, 66 op/s 2024-04-26T21:32:22.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:21 smithi001 ceph-mon[28054]: pgmap v64: 129 pgs: 129 active+clean; 7.0 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 414 KiB/s wr, 66 op/s 2024-04-26T21:32:22.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:21 smithi038 ceph-mon[32724]: pgmap v64: 129 pgs: 129 active+clean; 7.0 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 414 KiB/s wr, 66 op/s 2024-04-26T21:32:22.525 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11562 3.95 MB/sec warmup 20 sec latency 680.584 ms 2024-04-26T21:32:23.525 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11562 3.76 MB/sec warmup 21 sec latency 1437.431 ms 2024-04-26T21:32:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:23 smithi186 ceph-mon[32991]: pgmap v65: 129 pgs: 129 active+clean; 7.0 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 69 KiB/s wr, 22 op/s 2024-04-26T21:32:24.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:23 smithi001 ceph-mon[28054]: pgmap v65: 129 pgs: 129 active+clean; 7.0 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 69 KiB/s wr, 22 op/s 2024-04-26T21:32:24.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:23 smithi038 ceph-mon[32724]: pgmap v65: 129 pgs: 129 active+clean; 7.0 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 69 KiB/s wr, 22 op/s 2024-04-26T21:32:24.525 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11562 3.59 MB/sec warmup 22 sec latency 2437.565 ms 2024-04-26T21:32:25.526 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11562 3.44 MB/sec warmup 23 sec latency 3437.725 ms 2024-04-26T21:32:26.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:25 smithi186 ceph-mon[32991]: pgmap v66: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.8 MiB/s wr, 48 op/s 2024-04-26T21:32:26.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:25 smithi001 ceph-mon[28054]: pgmap v66: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.8 MiB/s wr, 48 op/s 2024-04-26T21:32:26.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:25 smithi038 ceph-mon[32724]: pgmap v66: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.8 MiB/s wr, 48 op/s 2024-04-26T21:32:26.526 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11628 3.30 MB/sec warmup 24 sec latency 4197.346 ms 2024-04-26T21:32:27.526 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11628 3.17 MB/sec warmup 25 sec latency 1176.281 ms 2024-04-26T21:32:28.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:27 smithi186 ceph-mon[32991]: pgmap v67: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 40 op/s 2024-04-26T21:32:28.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:27 smithi001 ceph-mon[28054]: pgmap v67: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 40 op/s 2024-04-26T21:32:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:27 smithi038 ceph-mon[32724]: pgmap v67: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 40 op/s 2024-04-26T21:32:28.526 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11628 3.04 MB/sec warmup 26 sec latency 2176.453 ms 2024-04-26T21:32:29.526 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11628 2.93 MB/sec warmup 27 sec latency 3176.550 ms 2024-04-26T21:32:30.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:29 smithi186 ceph-mon[32991]: pgmap v68: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 40 op/s 2024-04-26T21:32:30.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:29 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:30.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:29 smithi001 ceph-mon[28054]: pgmap v68: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 40 op/s 2024-04-26T21:32:30.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:29 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:30.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:29 smithi038 ceph-mon[32724]: pgmap v68: 129 pgs: 129 active+clean; 27 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 1.7 MiB/s wr, 40 op/s 2024-04-26T21:32:30.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:29 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:32:30.526 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11628 2.83 MB/sec warmup 28 sec latency 4176.721 ms 2024-04-26T21:32:31.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:30 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:31.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:30 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:31.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:30 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:32:31.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:30 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:31.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:30 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:31.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:30 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:32:31.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:30 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:32:31.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:30 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:32:31.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:30 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:32:31.526 INFO:tasks.workunit.client.0.smithi001.stdout: 1 11828 2.77 MB/sec warmup 29 sec latency 4947.124 ms 2024-04-26T21:32:32.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:31 smithi001 ceph-mon[28054]: pgmap v69: 129 pgs: 129 active+clean; 35 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.5 MiB/s wr, 60 op/s 2024-04-26T21:32:32.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:31 smithi038 ceph-mon[32724]: pgmap v69: 129 pgs: 129 active+clean; 35 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.5 MiB/s wr, 60 op/s 2024-04-26T21:32:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:31 smithi186 ceph-mon[32991]: pgmap v69: 129 pgs: 129 active+clean; 35 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.5 MiB/s wr, 60 op/s 2024-04-26T21:32:32.526 INFO:tasks.workunit.client.0.smithi001.stdout: 1 13248 2.88 MB/sec warmup 30 sec latency 19.766 ms 2024-04-26T21:32:33.527 INFO:tasks.workunit.client.0.smithi001.stdout: 1 14444 3.13 MB/sec warmup 31 sec latency 27.840 ms 2024-04-26T21:32:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.770+0000 7f9914c00640 1 -- 172.21.15.1:0/1989609908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f991010f2d0 msgr2=0x7f99101116c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.770+0000 7f9914c00640 1 --2- 172.21.15.1:0/1989609908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f991010f2d0 0x7f99101116c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9908009ca0 tx=0x7f990802a8c0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.770+0000 7f9914c00640 1 -- 172.21.15.1:0/1989609908 shutdown_connections 2024-04-26T21:32:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.770+0000 7f9914c00640 1 --2- 172.21.15.1:0/1989609908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9910102040 0x7f991010ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.770+0000 7f9914c00640 1 --2- 172.21.15.1:0/1989609908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f991010f2d0 0x7f99101116c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.770+0000 7f9914c00640 1 --2- 172.21.15.1:0/1989609908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9910101720 0x7f9910101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.770+0000 7f9914c00640 1 -- 172.21.15.1:0/1989609908 >> 172.21.15.1:0/1989609908 conn(0x7f9910078950 msgr2=0x7f9910078d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.770+0000 7f9914c00640 1 -- 172.21.15.1:0/1989609908 shutdown_connections 2024-04-26T21:32:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.770+0000 7f9914c00640 1 -- 172.21.15.1:0/1989609908 wait complete. 2024-04-26T21:32:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.772+0000 7f9914c00640 1 Processor -- start 2024-04-26T21:32:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.772+0000 7f9914c00640 1 -- start start 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.772+0000 7f9914c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9910101720 0x7f991019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.772+0000 7f9914c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9910102040 0x7f991019b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.772+0000 7f9914c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991010f2d0 0x7f991019fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.772+0000 7f9914c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9910114e30 con 0x7f991010f2d0 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.772+0000 7f9914c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9910114cb0 con 0x7f9910102040 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.772+0000 7f9914c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9910114fb0 con 0x7f9910101720 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9910101720 0x7f991019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990d400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9910102040 0x7f991019b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9910101720 0x7f991019b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43166/0 (socket says 172.21.15.1:43166) 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990d400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9910102040 0x7f991019b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54158/0 (socket says 172.21.15.1:54158) 2024-04-26T21:32:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990de00640 1 -- 172.21.15.1:0/4291307785 learned_addr learned my addr 172.21.15.1:0/4291307785 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990e800640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991010f2d0 0x7f991019fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990d400640 1 -- 172.21.15.1:0/4291307785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9910101720 msgr2=0x7f991019b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990d400640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9910101720 0x7f991019b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990d400640 1 -- 172.21.15.1:0/4291307785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991010f2d0 msgr2=0x7f991019fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990d400640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991010f2d0 0x7f991019fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990d400640 1 -- 172.21.15.1:0/4291307785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99101a0310 con 0x7f9910102040 2024-04-26T21:32:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.773+0000 7f990de00640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9910101720 0x7f991019b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.774+0000 7f990d400640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9910102040 0x7f991019b8b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00e9f0 tx=0x7f98fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.774+0000 7f9906a00640 1 -- 172.21.15.1:0/4291307785 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00ccc0 con 0x7f9910102040 2024-04-26T21:32:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.774+0000 7f9914c00640 1 -- 172.21.15.1:0/4291307785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99101a0600 con 0x7f9910102040 2024-04-26T21:32:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.774+0000 7f9906a00640 1 -- 172.21.15.1:0/4291307785 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98fc005020 con 0x7f9910102040 2024-04-26T21:32:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.774+0000 7f9906a00640 1 -- 172.21.15.1:0/4291307785 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc004a00 con 0x7f9910102040 2024-04-26T21:32:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.774+0000 7f9914c00640 1 -- 172.21.15.1:0/4291307785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99101a7e90 con 0x7f9910102040 2024-04-26T21:32:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.779+0000 7f9917b7d740 1 Processor -- start 2024-04-26T21:32:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.779+0000 7f9917b7d740 1 -- start start 2024-04-26T21:32:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.779+0000 7f9906a00640 1 -- 172.21.15.1:0/4291307785 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f98fc004ba0 con 0x7f9910102040 2024-04-26T21:32:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.779+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 0x564d53b251b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.779+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b256f0 0x564d53b2fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.779+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b2ffb0 0x564d53b323a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.779+0000 7f9917b7d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564d53a8a380 con 0x564d53b24dd0 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.779+0000 7f9917b7d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564d53a26c30 con 0x564d53b256f0 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.779+0000 7f9917b7d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d53a86340 con 0x564d53b2ffb0 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990e800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 0x564d53b251b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b256f0 0x564d53b2fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b256f0 0x564d53b2fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54166/0 (socket says 172.21.15.1:54166) 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990de00640 1 -- 172.21.15.1:0/1096505631 learned_addr learned my addr 172.21.15.1:0/1096505631 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990d400640 1 --2- 172.21.15.1:0/1096505631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b2ffb0 0x564d53b323a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990de00640 1 -- 172.21.15.1:0/1096505631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b2ffb0 msgr2=0x564d53b323a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990de00640 1 --2- 172.21.15.1:0/1096505631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b2ffb0 0x564d53b323a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990de00640 1 -- 172.21.15.1:0/1096505631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 msgr2=0x564d53b251b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990de00640 1 --2- 172.21.15.1:0/1096505631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 0x564d53b251b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990de00640 1 -- 172.21.15.1:0/1096505631 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d53a48940 con 0x564d53b256f0 2024-04-26T21:32:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.780+0000 7f990e800640 1 --2- 172.21.15.1:0/1096505631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 0x564d53b251b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.781+0000 7f990de00640 1 --2- 172.21.15.1:0/1096505631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b256f0 0x564d53b2fa70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9900011650 tx=0x7f9900009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d1dfdaec37d47b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.781+0000 7f98fb000640 1 -- 172.21.15.1:0/1096505631 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990001e070 con 0x564d53b256f0 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.781+0000 7f98fb000640 1 -- 172.21.15.1:0/1096505631 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f990000fb70 con 0x564d53b256f0 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.781+0000 7f98fb000640 1 -- 172.21.15.1:0/1096505631 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9900019c60 con 0x564d53b256f0 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.781+0000 7f9906a00640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f98f40a3d90 0x7f98f40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.781+0000 7f9906a00640 1 -- 172.21.15.1:0/4291307785 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f98fc0d0af0 con 0x7f9910102040 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f990e800640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f98f40a3d90 0x7f98f40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f9917b7d740 1 -- 172.21.15.1:0/1096505631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b256f0 msgr2=0x564d53b2fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f9917b7d740 1 --2- 172.21.15.1:0/1096505631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b256f0 0x564d53b2fa70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9900011650 tx=0x7f9900009f90 comp rx=0 tx=0).stop 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f9917b7d740 1 -- 172.21.15.1:0/1096505631 shutdown_connections 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f9917b7d740 1 --2- 172.21.15.1:0/1096505631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b2ffb0 0x564d53b323a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f9917b7d740 1 --2- 172.21.15.1:0/1096505631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b256f0 0x564d53b2fa70 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f9917b7d740 1 --2- 172.21.15.1:0/1096505631 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 0x564d53b251b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f9917b7d740 1 -- 172.21.15.1:0/1096505631 >> 172.21.15.1:0/1096505631 conn(0x564d53b212e0 msgr2=0x564d53b216c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f9917b7d740 1 -- 172.21.15.1:0/1096505631 shutdown_connections 2024-04-26T21:32:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f9917b7d740 1 -- 172.21.15.1:0/1096505631 wait complete. 2024-04-26T21:32:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.782+0000 7f990e800640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f98f40a3d90 0x7f98f40a6250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9908005e30 tx=0x7f9908005d80 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.783+0000 7f9917b7d740 1 -- start start 2024-04-26T21:32:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.783+0000 7f9917b7d740 1 Processor -- start 2024-04-26T21:32:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b2cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 0x564d53b2d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b2ffb0 0x564d53b2d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f9917b7d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564d53b32be0 con 0x564d53b2ffb0 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f9917b7d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564d53a11c60 con 0x564d53b24dd0 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f9917b7d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d53b32d60 con 0x564d53b256f0 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f9917b7d740 10 client.0 _ll_register_callbacks cb 0x564d53a3de50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b2cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b2cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54174/0 (socket says 172.21.15.1:54174) 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990de00640 1 -- 172.21.15.1:0/2339066183 learned_addr learned my addr 172.21.15.1:0/2339066183 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990e800640 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b2ffb0 0x564d53b2d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990d400640 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 0x564d53b2d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990de00640 1 -- 172.21.15.1:0/2339066183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 msgr2=0x564d53b2d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990de00640 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 0x564d53b2d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990de00640 1 -- 172.21.15.1:0/2339066183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b2ffb0 msgr2=0x564d53b2d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990de00640 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b2ffb0 0x564d53b2d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990de00640 1 -- 172.21.15.1:0/2339066183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d53a20510 con 0x564d53b24dd0 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990d400640 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 0x564d53b2d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.784+0000 7f990e800640 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b2ffb0 0x564d53b2d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.785+0000 7f990de00640 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b2cf40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9900000c00 tx=0x7f9900009ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5638b3fb707f183d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.785+0000 7f98ef400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.785+0000 7f98ef400640 1 -- 172.21.15.1:0/2339066183 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9900027060 con 0x564d53b24dd0 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.785+0000 7f98ef400640 1 -- 172.21.15.1:0/2339066183 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f990000f6c0 con 0x564d53b24dd0 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.785+0000 7f9917b7d740 1 -- 172.21.15.1:0/2339066183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f990000fb70 con 0x564d53b24dd0 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.785+0000 7f98ef400640 1 -- 172.21.15.1:0/2339066183 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f990002e050 con 0x564d53b24dd0 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.785+0000 7f9917b7d740 10 client.25691 fetch_fsmap learned FSMap version 28 2024-04-26T21:32:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.785+0000 7f9917b7d740 1 -- 172.21.15.1:0/2339066183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564d5379c210 con 0x564d53b24dd0 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.786+0000 7f98ef400640 1 -- 172.21.15.1:0/2339066183 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f990000fb70 con 0x564d53b24dd0 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.786+0000 7f98ef400640 1 -- 172.21.15.1:0/2339066183 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9900021030 con 0x564d53b24dd0 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.786+0000 7f9917b7d740 10 client.25691 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.786+0000 7f9917b7d740 10 client.25691 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.786+0000 7f9917b7d740 20 client.25691 populate_metadata read hostname 'smithi001' 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.786+0000 7f9917b7d740 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564d53b3fc40 0x564d53b42030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.786+0000 7f9917b7d740 4 client.25691 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.786+0000 7f9917b7d740 1 -- 172.21.15.1:0/2339066183 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d53a97f90 con 0x564d53b3fc40 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.786+0000 7f990d400640 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564d53b3fc40 0x564d53b42030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.787+0000 7f990d400640 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564d53b3fc40 0x564d53b42030 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7371c36515e0e8db server_cookie=4ba8d14d42015e in_seq=0 out_seq=0 2024-04-26T21:32:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.787+0000 7f98ef400640 10 client.25691 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.787+0000 7f98ef400640 1 -- 172.21.15.1:0/2339066183 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564d53a97f90 con 0x564d53b3fc40 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.787+0000 7f98ef400640 10 client.25691 handle_command_reply: tid=0 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 client.25691 shutdown 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 -- 172.21.15.1:0/2339066183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 msgr2=0x564d53b2cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b2cf40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9900000c00 tx=0x7f9900009ef0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 -- 172.21.15.1:0/2339066183 shutdown_connections 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564d53b3fc40 0x564d53b42030 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 0x564d53b2d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b2ffb0 0x564d53b2d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 --2- 172.21.15.1:0/2339066183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b2cf40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f98ef400640 0 client.25691 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 -- 172.21.15.1:0/2339066183 >> 172.21.15.1:0/2339066183 conn(0x564d53b212e0 msgr2=0x564d53b27060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 -- 172.21.15.1:0/2339066183 shutdown_connections 2024-04-26T21:32:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 1 -- 172.21.15.1:0/2339066183 wait complete. 2024-04-26T21:32:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.788+0000 7f9917b7d740 20 client.25691 trim_cache size 0 max 16384 2024-04-26T21:32:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.810+0000 7f9917b7d740 1 Processor -- start 2024-04-26T21:32:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.810+0000 7f9917b7d740 1 -- start start 2024-04-26T21:32:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.810+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b3baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.810+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b256f0 0x564d53b27790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.810+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b2ffb0 0x564d53b477a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.810+0000 7f9917b7d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f99101b4a20 con 0x564d53b256f0 2024-04-26T21:32:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.810+0000 7f9917b7d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f99101b2290 con 0x564d53b24dd0 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.810+0000 7f9917b7d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f99101b00e0 con 0x564d53b2ffb0 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b3baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b3baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54180/0 (socket says 172.21.15.1:54180) 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990de00640 1 -- 172.21.15.1:0/2856176679 learned_addr learned my addr 172.21.15.1:0/2856176679 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990d400640 1 --2- 172.21.15.1:0/2856176679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b256f0 0x564d53b27790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990e800640 1 --2- 172.21.15.1:0/2856176679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b2ffb0 0x564d53b477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990de00640 1 -- 172.21.15.1:0/2856176679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b2ffb0 msgr2=0x564d53b477a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990de00640 1 --2- 172.21.15.1:0/2856176679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b2ffb0 0x564d53b477a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990de00640 1 -- 172.21.15.1:0/2856176679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b256f0 msgr2=0x564d53b27790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990de00640 1 --2- 172.21.15.1:0/2856176679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b256f0 0x564d53b27790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990de00640 1 -- 172.21.15.1:0/2856176679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d53868620 con 0x564d53b24dd0 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990d400640 1 --2- 172.21.15.1:0/2856176679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b256f0 0x564d53b27790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.811+0000 7f990de00640 1 --2- 172.21.15.1:0/2856176679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b3baa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9900009c90 tx=0x7f9900005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a6482722fceb9bc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.812+0000 7f98fa600640 1 -- 172.21.15.1:0/2856176679 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9900027040 con 0x564d53b24dd0 2024-04-26T21:32:33.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.812+0000 7f98fa600640 1 -- 172.21.15.1:0/2856176679 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9900005580 con 0x564d53b24dd0 2024-04-26T21:32:33.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.812+0000 7f98fa600640 1 -- 172.21.15.1:0/2856176679 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990001e040 con 0x564d53b24dd0 2024-04-26T21:32:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.813+0000 7f9917b7d740 1 -- 172.21.15.1:0/2856176679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 msgr2=0x564d53b3baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.813+0000 7f9917b7d740 1 --2- 172.21.15.1:0/2856176679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b3baa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9900009c90 tx=0x7f9900005d90 comp rx=0 tx=0).stop 2024-04-26T21:32:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.813+0000 7f9917b7d740 1 -- 172.21.15.1:0/2856176679 shutdown_connections 2024-04-26T21:32:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.813+0000 7f9917b7d740 1 --2- 172.21.15.1:0/2856176679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b2ffb0 0x564d53b477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.813+0000 7f9917b7d740 1 --2- 172.21.15.1:0/2856176679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b256f0 0x564d53b27790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.813+0000 7f9917b7d740 1 --2- 172.21.15.1:0/2856176679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b24dd0 0x564d53b3baa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.813+0000 7f9917b7d740 1 -- 172.21.15.1:0/2856176679 >> 172.21.15.1:0/2856176679 conn(0x564d53b212e0 msgr2=0x564d53b20440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.813+0000 7f9917b7d740 1 -- 172.21.15.1:0/2856176679 shutdown_connections 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.813+0000 7f9917b7d740 1 -- 172.21.15.1:0/2856176679 wait complete. 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.813+0000 7f9917b7d740 1 -- start start 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f9917b7d740 1 Processor -- start 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 0x564d53b37fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 0x564d53b384f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f9917b7d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b2ffb0 0x564d53b34f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f9917b7d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f99101b0750 con 0x564d53b24dd0 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f9917b7d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f99101b1bc0 con 0x564d53b2ffb0 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f9917b7d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f99101b0e20 con 0x564d53b256f0 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f9917b7d740 10 client.0 _ll_register_callbacks cb 0x7f990001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f990e800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b2ffb0 0x564d53b34f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f990e800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b2ffb0 0x564d53b34f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54192/0 (socket says 172.21.15.1:54192) 2024-04-26T21:32:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f990e800640 1 -- 172.21.15.1:0/1505010432 learned_addr learned my addr 172.21.15.1:0/1505010432 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f990de00640 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 0x564d53b37fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.814+0000 7f990d400640 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 0x564d53b384f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.815+0000 7f990e800640 1 -- 172.21.15.1:0/1505010432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 msgr2=0x564d53b384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.815+0000 7f990e800640 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 0x564d53b384f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.815+0000 7f990e800640 1 -- 172.21.15.1:0/1505010432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 msgr2=0x564d53b37fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.815+0000 7f990e800640 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 0x564d53b37fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.815+0000 7f990e800640 1 -- 172.21.15.1:0/1505010432 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d53a44cb0 con 0x564d53b2ffb0 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.815+0000 7f990e800640 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b2ffb0 0x564d53b34f80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9908006010 tx=0x7f99080095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4209994800377126 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.815+0000 7f98ef400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.815+0000 7f98ef400640 1 -- 172.21.15.1:0/1505010432 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9908059060 con 0x564d53b2ffb0 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.815+0000 7f9917b7d740 1 -- 172.21.15.1:0/1505010432 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9900005580 con 0x564d53b2ffb0 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f98ef400640 1 -- 172.21.15.1:0/1505010432 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f9908003d80 con 0x564d53b2ffb0 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f98ef400640 1 -- 172.21.15.1:0/1505010432 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9908060050 con 0x564d53b2ffb0 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f9917b7d740 10 client.25697 fetch_fsmap learned FSMap version 28 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f9917b7d740 1 -- 172.21.15.1:0/1505010432 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564d53ba6940 con 0x564d53b2ffb0 2024-04-26T21:32:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f98ef400640 1 -- 172.21.15.1:0/1505010432 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9900005580 con 0x564d53b2ffb0 2024-04-26T21:32:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f98ef400640 1 -- 172.21.15.1:0/1505010432 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f990802f9b0 con 0x564d53b2ffb0 2024-04-26T21:32:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f9917b7d740 10 client.25697 fetch_fsmap finished waiting for FSMap version 28 2024-04-26T21:32:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f9917b7d740 10 client.25697 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:32:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f9917b7d740 20 client.25697 populate_metadata read hostname 'smithi001' 2024-04-26T21:32:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f9917b7d740 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564d53ba7070 0x564d53ba9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f9917b7d740 4 client.25697 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:32:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.816+0000 7f9917b7d740 1 -- 172.21.15.1:0/1505010432 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564d53a91e00 con 0x564d53ba7070 2024-04-26T21:32:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.817+0000 7f990de00640 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564d53ba7070 0x564d53ba9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.817+0000 7f990de00640 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564d53ba7070 0x564d53ba9490 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f5dbdf9c9536ec server_cookie=938a86bf890154bd in_seq=0 out_seq=0 2024-04-26T21:32:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.817+0000 7f98ef400640 10 client.25697 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.849+0000 7f98ef400640 1 -- 172.21.15.1:0/1505010432 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564d53a91e00 con 0x564d53ba7070 2024-04-26T21:32:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.849+0000 7f98ef400640 10 client.25697 handle_command_reply: tid=0 2024-04-26T21:32:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.849+0000 7f9917b7d740 1 client.25697 shutdown 2024-04-26T21:32:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 1 -- 172.21.15.1:0/1505010432 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b2ffb0 msgr2=0x564d53b34f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b2ffb0 0x564d53b34f80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9908006010 tx=0x7f99080095c0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 1 -- 172.21.15.1:0/1505010432 shutdown_connections 2024-04-26T21:32:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564d53ba7070 0x564d53ba9490 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d53b256f0 0x564d53b384f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d53b2ffb0 0x564d53b34f80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 1 --2- 172.21.15.1:0/1505010432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564d53b24dd0 0x564d53b37fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 1 -- 172.21.15.1:0/1505010432 >> 172.21.15.1:0/1505010432 conn(0x564d53b212e0 msgr2=0x564d53b2e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f98ef400640 0 client.25697 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 1 -- 172.21.15.1:0/1505010432 shutdown_connections 2024-04-26T21:32:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 1 -- 172.21.15.1:0/1505010432 wait complete. 2024-04-26T21:32:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.850+0000 7f9917b7d740 20 client.25697 trim_cache size 0 max 16384 2024-04-26T21:32:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:32:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:32:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:32:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:32:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.851+0000 7f9914c00640 1 -- 172.21.15.1:0/4291307785 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f98f40a3d90 msgr2=0x7f98f40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.851+0000 7f9914c00640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f98f40a3d90 0x7f98f40a6250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9908005e30 tx=0x7f9908005d80 comp rx=0 tx=0).stop 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.851+0000 7f9914c00640 1 -- 172.21.15.1:0/4291307785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9910102040 msgr2=0x7f991019b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.851+0000 7f9914c00640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9910102040 0x7f991019b8b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00e9f0 tx=0x7f98fc00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.851+0000 7f9914c00640 1 -- 172.21.15.1:0/4291307785 shutdown_connections 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.851+0000 7f9914c00640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f98f40a3d90 0x7f98f40a6250 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.851+0000 7f9914c00640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f991010f2d0 0x7f991019fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.851+0000 7f9914c00640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9910102040 0x7f991019b8b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.851+0000 7f9914c00640 1 --2- 172.21.15.1:0/4291307785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9910101720 0x7f991019b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.852+0000 7f9914c00640 1 -- 172.21.15.1:0/4291307785 >> 172.21.15.1:0/4291307785 conn(0x7f9910078950 msgr2=0x7f99101108e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.852+0000 7f9914c00640 1 -- 172.21.15.1:0/4291307785 shutdown_connections 2024-04-26T21:32:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:33.852+0000 7f9914c00640 1 -- 172.21.15.1:0/4291307785 wait complete. 2024-04-26T21:32:34.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:33 smithi001 ceph-mon[28054]: pgmap v70: 129 pgs: 129 active+clean; 35 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 2.4 MiB/s wr, 45 op/s 2024-04-26T21:32:34.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:33 smithi038 ceph-mon[32724]: pgmap v70: 129 pgs: 129 active+clean; 35 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 2.4 MiB/s wr, 45 op/s 2024-04-26T21:32:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:33 smithi186 ceph-mon[32991]: pgmap v70: 129 pgs: 129 active+clean; 35 MiB data, 470 MiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 2.4 MiB/s wr, 45 op/s 2024-04-26T21:32:34.527 INFO:tasks.workunit.client.0.smithi001.stdout: 1 14906 3.05 MB/sec warmup 32 sec latency 25.346 ms 2024-04-26T21:32:35.527 INFO:tasks.workunit.client.0.smithi001.stdout: 1 14918 2.96 MB/sec warmup 33 sec latency 979.622 ms 2024-04-26T21:32:36.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:35 smithi001 ceph-mon[28054]: pgmap v71: 129 pgs: 129 active+clean; 27 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.6 MiB/s wr, 110 op/s 2024-04-26T21:32:36.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:35 smithi038 ceph-mon[32724]: pgmap v71: 129 pgs: 129 active+clean; 27 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.6 MiB/s wr, 110 op/s 2024-04-26T21:32:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:35 smithi186 ceph-mon[32991]: pgmap v71: 129 pgs: 129 active+clean; 27 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 2.6 MiB/s wr, 110 op/s 2024-04-26T21:32:36.527 INFO:tasks.workunit.client.0.smithi001.stdout: 1 15108 2.89 MB/sec warmup 34 sec latency 1735.794 ms 2024-04-26T21:32:37.527 INFO:tasks.workunit.client.0.smithi001.stdout: 1 16554 2.98 MB/sec warmup 35 sec latency 24.951 ms 2024-04-26T21:32:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:37 smithi001 ceph-mon[28054]: pgmap v72: 129 pgs: 129 active+clean; 27 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 910 KiB/s wr, 84 op/s 2024-04-26T21:32:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:37 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:38.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:37 smithi038 ceph-mon[32724]: pgmap v72: 129 pgs: 129 active+clean; 27 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 910 KiB/s wr, 84 op/s 2024-04-26T21:32:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:37 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:37 smithi186 ceph-mon[32991]: pgmap v72: 129 pgs: 129 active+clean; 27 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 910 KiB/s wr, 84 op/s 2024-04-26T21:32:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:37 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:38.527 INFO:tasks.workunit.client.0.smithi001.stdout: 1 17535 3.11 MB/sec warmup 36 sec latency 25.981 ms 2024-04-26T21:32:39.527 INFO:tasks.workunit.client.0.smithi001.stdout: 1 18111 3.15 MB/sec warmup 37 sec latency 25.009 ms 2024-04-26T21:32:40.256 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:39 smithi186 ceph-mon[32991]: pgmap v73: 129 pgs: 129 active+clean; 27 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 910 KiB/s wr, 84 op/s 2024-04-26T21:32:40.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:39 smithi001 ceph-mon[28054]: pgmap v73: 129 pgs: 129 active+clean; 27 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 910 KiB/s wr, 84 op/s 2024-04-26T21:32:40.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:39 smithi038 ceph-mon[32724]: pgmap v73: 129 pgs: 129 active+clean; 27 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 910 KiB/s wr, 84 op/s 2024-04-26T21:32:40.527 INFO:tasks.workunit.client.0.smithi001.stdout: 1 18466 3.07 MB/sec warmup 38 sec latency 124.222 ms 2024-04-26T21:32:41.528 INFO:tasks.workunit.client.0.smithi001.stdout: 1 18534 3.00 MB/sec warmup 39 sec latency 884.261 ms 2024-04-26T21:32:42.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:41 smithi001 ceph-mon[28054]: pgmap v74: 129 pgs: 129 active+clean; 29 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-04-26T21:32:42.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:41 smithi038 ceph-mon[32724]: pgmap v74: 129 pgs: 129 active+clean; 29 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-04-26T21:32:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:41 smithi186 ceph-mon[32991]: pgmap v74: 129 pgs: 129 active+clean; 29 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-04-26T21:32:42.528 INFO:tasks.workunit.client.0.smithi001.stdout: 1 18898 2.96 MB/sec warmup 40 sec latency 706.908 ms 2024-04-26T21:32:43.528 INFO:tasks.workunit.client.0.smithi001.stdout: 1 20274 3.03 MB/sec warmup 41 sec latency 24.966 ms 2024-04-26T21:32:44.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:43 smithi001 ceph-mon[28054]: pgmap v75: 129 pgs: 129 active+clean; 29 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 320 KiB/s wr, 98 op/s 2024-04-26T21:32:44.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:43 smithi038 ceph-mon[32724]: pgmap v75: 129 pgs: 129 active+clean; 29 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 320 KiB/s wr, 98 op/s 2024-04-26T21:32:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:43 smithi186 ceph-mon[32991]: pgmap v75: 129 pgs: 129 active+clean; 29 MiB data, 476 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 320 KiB/s wr, 98 op/s 2024-04-26T21:32:44.528 INFO:tasks.workunit.client.0.smithi001.stdout: 1 21459 3.20 MB/sec warmup 42 sec latency 21.861 ms 2024-04-26T21:32:45.528 INFO:tasks.workunit.client.0.smithi001.stdout: 1 21743 3.16 MB/sec warmup 43 sec latency 32.200 ms 2024-04-26T21:32:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:46 smithi001 ceph-mon[28054]: pgmap v76: 129 pgs: 129 active+clean; 31 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 491 KiB/s wr, 133 op/s 2024-04-26T21:32:46.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:46 smithi038 ceph-mon[32724]: pgmap v76: 129 pgs: 129 active+clean; 31 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 491 KiB/s wr, 133 op/s 2024-04-26T21:32:46.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:46 smithi186 ceph-mon[32991]: pgmap v76: 129 pgs: 129 active+clean; 31 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 491 KiB/s wr, 133 op/s 2024-04-26T21:32:46.528 INFO:tasks.workunit.client.0.smithi001.stdout: 1 22261 3.10 MB/sec warmup 44 sec latency 90.135 ms 2024-04-26T21:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.019+0000 7f2fe8800640 1 -- 172.21.15.1:0/1781125656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fe0105510 msgr2=0x7f2fe01119d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.019+0000 7f2fe8800640 1 --2- 172.21.15.1:0/1781125656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fe0105510 0x7f2fe01119d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009cd0 tx=0x7f2fdc02a8f0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.019+0000 7f2fe8800640 1 -- 172.21.15.1:0/1781125656 shutdown_connections 2024-04-26T21:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.019+0000 7f2fe8800640 1 --2- 172.21.15.1:0/1781125656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe0104b50 0x7f2fe0104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.019+0000 7f2fe8800640 1 --2- 172.21.15.1:0/1781125656 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fe0105510 0x7f2fe01119d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.019+0000 7f2fe8800640 1 --2- 172.21.15.1:0/1781125656 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe010ac10 0x7f2fe010aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.019+0000 7f2fe8800640 1 -- 172.21.15.1:0/1781125656 >> 172.21.15.1:0/1781125656 conn(0x7f2fe0100600 msgr2=0x7f2fe0102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.020+0000 7f2fe8800640 1 -- 172.21.15.1:0/1781125656 shutdown_connections 2024-04-26T21:32:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.020+0000 7f2fe8800640 1 -- 172.21.15.1:0/1781125656 wait complete. 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.021+0000 7f2fe8800640 1 Processor -- start 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.021+0000 7f2fe8800640 1 -- start start 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.021+0000 7f2fe8800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe0104b50 0x7f2fe019b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.021+0000 7f2fe8800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe0105510 0x7f2fe019b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.021+0000 7f2fe8800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fe010ac10 0x7f2fe019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.021+0000 7f2fe8800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe0114e50 con 0x7f2fe0105510 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.021+0000 7f2fe8800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe0114cd0 con 0x7f2fe010ac10 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.021+0000 7f2fe8800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe0114fd0 con 0x7f2fe0104b50 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe0105510 0x7f2fe019b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe0104b50 0x7f2fe019b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe6800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fe010ac10 0x7f2fe019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe0105510 0x7f2fe019b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38868/0 (socket says 172.21.15.1:38868) 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5400640 1 -- 172.21.15.1:0/3946302037 learned_addr learned my addr 172.21.15.1:0/3946302037 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe0104b50 0x7f2fe019b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46672/0 (socket says 172.21.15.1:46672) 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5e00640 1 -- 172.21.15.1:0/3946302037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fe010ac10 msgr2=0x7f2fe019faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fe010ac10 0x7f2fe019faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5e00640 1 -- 172.21.15.1:0/3946302037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe0105510 msgr2=0x7f2fe019b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe0105510 0x7f2fe019b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5e00640 1 -- 172.21.15.1:0/3946302037 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe01a0220 con 0x7f2fe0104b50 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.022+0000 7f2fe5400640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe0105510 0x7f2fe019b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.023+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe0104b50 0x7f2fe019b230 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400e9f0 tx=0x7f2fd400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.023+0000 7f2fdaa00640 1 -- 172.21.15.1:0/3946302037 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd400ccc0 con 0x7f2fe0104b50 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.023+0000 7f2fdaa00640 1 -- 172.21.15.1:0/3946302037 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd40108e0 con 0x7f2fe0104b50 2024-04-26T21:32:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.023+0000 7f2fe8800640 1 -- 172.21.15.1:0/3946302037 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fe01a0510 con 0x7f2fe0104b50 2024-04-26T21:32:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.023+0000 7f2fdaa00640 1 -- 172.21.15.1:0/3946302037 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4018880 con 0x7f2fe0104b50 2024-04-26T21:32:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.023+0000 7f2fe8800640 1 -- 172.21.15.1:0/3946302037 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fe01a7da0 con 0x7f2fe0104b50 2024-04-26T21:32:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.026+0000 7f2fdaa00640 1 -- 172.21.15.1:0/3946302037 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f2fd4010430 con 0x7f2fe0104b50 2024-04-26T21:32:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.027+0000 7f2feb7d1740 1 Processor -- start 2024-04-26T21:32:47.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.027+0000 7f2feb7d1740 1 -- start start 2024-04-26T21:32:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.027+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 0x55ca382c21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.027+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c26f0 0x55ca382cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.027+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382ccfb0 0x55ca382cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.027+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ca38227380 con 0x55ca382c1dd0 2024-04-26T21:32:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ca381c3c30 con 0x55ca382c26f0 2024-04-26T21:32:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ca38223340 con 0x55ca382ccfb0 2024-04-26T21:32:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382ccfb0 0x55ca382cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fdaa00640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2fc80a3d90 0x7f2fc80a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382ccfb0 0x55ca382cf3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46682/0 (socket says 172.21.15.1:46682) 2024-04-26T21:32:47.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe5400640 1 -- 172.21.15.1:0/3396541992 learned_addr learned my addr 172.21.15.1:0/3396541992 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe6800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 0x55ca382c21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/3396541992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c26f0 0x55ca382cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe5400640 1 -- 172.21.15.1:0/3396541992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c26f0 msgr2=0x55ca382cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe5400640 1 --2- 172.21.15.1:0/3396541992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c26f0 0x55ca382cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe5400640 1 -- 172.21.15.1:0/3396541992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 msgr2=0x55ca382c21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe5400640 1 --2- 172.21.15.1:0/3396541992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 0x55ca382c21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe5400640 1 -- 172.21.15.1:0/3396541992 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca381e5940 con 0x55ca382ccfb0 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/3396541992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c26f0 0x55ca382cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fdaa00640 1 -- 172.21.15.1:0/3946302037 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f2fd40d0f20 con 0x7f2fe0104b50 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.028+0000 7f2fe6800640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2fc80a3d90 0x7f2fc80a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.029+0000 7f2fe5400640 1 --2- 172.21.15.1:0/3396541992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382ccfb0 0x55ca382cf3a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc010580 tx=0x7f2fcc010a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1bbb35a6d0cf1c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.029+0000 7f2fd3000640 1 -- 172.21.15.1:0/3396541992 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc00e830 con 0x55ca382ccfb0 2024-04-26T21:32:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.029+0000 7f2fd3000640 1 -- 172.21.15.1:0/3396541992 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fcc00ee80 con 0x55ca382ccfb0 2024-04-26T21:32:47.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.029+0000 7f2fd3000640 1 -- 172.21.15.1:0/3396541992 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc011440 con 0x55ca382ccfb0 2024-04-26T21:32:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.030+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3396541992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382ccfb0 msgr2=0x55ca382cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.030+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/3396541992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382ccfb0 0x55ca382cf3a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc010580 tx=0x7f2fcc010a40 comp rx=0 tx=0).stop 2024-04-26T21:32:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.030+0000 7f2fe6800640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2fc80a3d90 0x7f2fc80a6250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc004af0 tx=0x7f2fdc03f040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.030+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3396541992 shutdown_connections 2024-04-26T21:32:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.030+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/3396541992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382ccfb0 0x55ca382cf3a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.030+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/3396541992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c26f0 0x55ca382cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.030+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/3396541992 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 0x55ca382c21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.030+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3396541992 >> 172.21.15.1:0/3396541992 conn(0x55ca382be2e0 msgr2=0x55ca382be6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.030+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3396541992 shutdown_connections 2024-04-26T21:32:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.030+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3396541992 wait complete. 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.031+0000 7f2feb7d1740 1 -- start start 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.031+0000 7f2feb7d1740 1 Processor -- start 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 0x55ca382c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382ccfb0 0x55ca382caa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ca382cfbe0 con 0x55ca382c1dd0 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ca381aec60 con 0x55ca382ccfb0 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ca382cfd60 con 0x55ca382c26f0 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2feb7d1740 10 client.0 _ll_register_callbacks cb 0x55ca381dae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2fe5e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 0x55ca382c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2fe6800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382ccfb0 0x55ca382caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2fe5e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 0x55ca382c9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38878/0 (socket says 172.21.15.1:38878) 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2fe5e00640 1 -- 172.21.15.1:0/3952996874 learned_addr learned my addr 172.21.15.1:0/3952996874 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.032+0000 7f2fe5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2fe5400640 1 -- 172.21.15.1:0/3952996874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382ccfb0 msgr2=0x55ca382caa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2fe5400640 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382ccfb0 0x55ca382caa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2fe5400640 1 -- 172.21.15.1:0/3952996874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 msgr2=0x55ca382c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2fe5400640 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 0x55ca382c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2fe5400640 1 -- 172.21.15.1:0/3952996874 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca381bd510 con 0x55ca382c26f0 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 0x55ca382c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2fe5400640 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382ca4f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc010550 tx=0x7f2fcc004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a91e43655f43829 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2fc7400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2fc7400640 1 -- 172.21.15.1:0/3952996874 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc01c480 con 0x55ca382c26f0 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3952996874 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ca38234f90 con 0x55ca382c26f0 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.033+0000 7f2fc7400640 1 -- 172.21.15.1:0/3952996874 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f2fcc01c620 con 0x55ca382c26f0 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.034+0000 7f2fc7400640 1 -- 172.21.15.1:0/3952996874 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2fcc01c820 con 0x55ca382c26f0 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.034+0000 7f2feb7d1740 10 client.25693 fetch_fsmap learned FSMap version 30 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.034+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3952996874 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ca37f39210 con 0x55ca382c26f0 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.034+0000 7f2fc7400640 1 -- 172.21.15.1:0/3952996874 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ca38234f90 con 0x55ca382c26f0 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.034+0000 7f2fc7400640 1 -- 172.21.15.1:0/3952996874 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f2fcc003960 con 0x55ca382c26f0 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.035+0000 7f2feb7d1740 10 client.25693 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.035+0000 7f2feb7d1740 10 client.25693 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.035+0000 7f2feb7d1740 20 client.25693 populate_metadata read hostname 'smithi001' 2024-04-26T21:32:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.035+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ca382dcd60 0x55ca382df150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.035+0000 7f2feb7d1740 4 client.25693 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:32:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.035+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3952996874 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca3822ee00 con 0x55ca382dcd60 2024-04-26T21:32:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.035+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ca382dcd60 0x55ca382df150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.035+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ca382dcd60 0x55ca382df150 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8beab7db8e27b3d server_cookie=c7d7dba0b9cbfb5f in_seq=0 out_seq=0 2024-04-26T21:32:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.036+0000 7f2fc7400640 10 client.25693 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.036+0000 7f2fc7400640 1 -- 172.21.15.1:0/3952996874 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ca3822ee00 con 0x55ca382dcd60 2024-04-26T21:32:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.036+0000 7f2fc7400640 10 client.25693 handle_command_reply: tid=0 2024-04-26T21:32:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.036+0000 7f2feb7d1740 1 client.25693 shutdown 2024-04-26T21:32:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3952996874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 msgr2=0x55ca382ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382ca4f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc010550 tx=0x7f2fcc004270 comp rx=0 tx=0).stop 2024-04-26T21:32:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3952996874 shutdown_connections 2024-04-26T21:32:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ca382dcd60 0x55ca382df150 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382ca4f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382ccfb0 0x55ca382caa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/3952996874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382c1dd0 0x55ca382c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2fc7400640 0 client.25693 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3952996874 >> 172.21.15.1:0/3952996874 conn(0x55ca382be2e0 msgr2=0x55ca382c4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3952996874 shutdown_connections 2024-04-26T21:32:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 1 -- 172.21.15.1:0/3952996874 wait complete. 2024-04-26T21:32:47.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.037+0000 7f2feb7d1740 20 client.25693 trim_cache size 0 max 16384 2024-04-26T21:32:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.056+0000 7f2feb7d1740 1 Processor -- start 2024-04-26T21:32:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.056+0000 7f2feb7d1740 1 -- start start 2024-04-26T21:32:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 0x55ca382e4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe01b4910 con 0x55ca382ccfb0 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe01b2180 con 0x55ca382c1dd0 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe01affd0 con 0x55ca382c26f0 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2fe5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2fe5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47678/0 (socket says 172.21.15.1:47678) 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2fe5e00640 1 -- 172.21.15.1:0/1175672783 learned_addr learned my addr 172.21.15.1:0/1175672783 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2fe6800640 1 --2- 172.21.15.1:0/1175672783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 0x55ca382e4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2fe5400640 1 --2- 172.21.15.1:0/1175672783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2fe5e00640 1 -- 172.21.15.1:0/1175672783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 msgr2=0x55ca382cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/1175672783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2fe5e00640 1 -- 172.21.15.1:0/1175672783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 msgr2=0x55ca382e4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/1175672783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 0x55ca382e4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.057+0000 7f2fe5e00640 1 -- 172.21.15.1:0/1175672783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca38005620 con 0x55ca382c1dd0 2024-04-26T21:32:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.058+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/1175672783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d8aa0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40a5190 tx=0x7f2fd40d0040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb8add3cc55dfeca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.058+0000 7f2fd2600640 1 -- 172.21.15.1:0/1175672783 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40a56c0 con 0x55ca382c1dd0 2024-04-26T21:32:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.058+0000 7f2fd2600640 1 -- 172.21.15.1:0/1175672783 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd409aba0 con 0x55ca382c1dd0 2024-04-26T21:32:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.058+0000 7f2fd2600640 1 -- 172.21.15.1:0/1175672783 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4098850 con 0x55ca382c1dd0 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.059+0000 7f2feb7d1740 1 -- 172.21.15.1:0/1175672783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 msgr2=0x55ca382d8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.059+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/1175672783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d8aa0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40a5190 tx=0x7f2fd40d0040 comp rx=0 tx=0).stop 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.059+0000 7f2feb7d1740 1 -- 172.21.15.1:0/1175672783 shutdown_connections 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.059+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/1175672783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.059+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/1175672783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 0x55ca382e4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.059+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/1175672783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d8aa0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.059+0000 7f2feb7d1740 1 -- 172.21.15.1:0/1175672783 >> 172.21.15.1:0/1175672783 conn(0x55ca382be2e0 msgr2=0x55ca382cb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.059+0000 7f2feb7d1740 1 -- 172.21.15.1:0/1175672783 shutdown_connections 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.059+0000 7f2feb7d1740 1 -- 172.21.15.1:0/1175672783 wait complete. 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.059+0000 7f2feb7d1740 1 -- start start 2024-04-26T21:32:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2feb7d1740 1 Processor -- start 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2feb7d1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 0x55ca382d1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe01b0640 con 0x55ca382ccfb0 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe01b1ab0 con 0x55ca382c1dd0 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2feb7d1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe01b0d10 con 0x55ca382c26f0 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2feb7d1740 10 client.0 _ll_register_callbacks cb 0x7f2fcc0103d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2fe5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2fe5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47694/0 (socket says 172.21.15.1:47694) 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2fe5e00640 1 -- 172.21.15.1:0/990021002 learned_addr learned my addr 172.21.15.1:0/990021002 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2fe5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.060+0000 7f2fe6800640 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 0x55ca382d1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fe5e00640 1 -- 172.21.15.1:0/990021002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 msgr2=0x55ca382d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fe5e00640 1 -- 172.21.15.1:0/990021002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 msgr2=0x55ca382d1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 0x55ca382d1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fe5e00640 1 -- 172.21.15.1:0/990021002 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca381e1cb0 con 0x55ca382c1dd0 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fe5400640 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fe6800640 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 0x55ca382d1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fe5e00640 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d08d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4098f60 tx=0x7f2fd4009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=318a55cd9be5ac04 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fc7400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fc7400640 1 -- 172.21.15.1:0/990021002 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd409cb00 con 0x55ca382c1dd0 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2fc7400640 1 -- 172.21.15.1:0/990021002 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f2fd4010d90 con 0x55ca382c1dd0 2024-04-26T21:32:47.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.061+0000 7f2feb7d1740 1 -- 172.21.15.1:0/990021002 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fd409aba0 con 0x55ca382c1dd0 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2fc7400640 1 -- 172.21.15.1:0/990021002 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2fd40a56c0 con 0x55ca382c1dd0 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2feb7d1740 10 client.25715 fetch_fsmap learned FSMap version 30 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2feb7d1740 1 -- 172.21.15.1:0/990021002 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ca38343200 con 0x55ca382c1dd0 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2fc7400640 1 -- 172.21.15.1:0/990021002 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd409aba0 con 0x55ca382c1dd0 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2fc7400640 1 -- 172.21.15.1:0/990021002 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f2fd40cbce0 con 0x55ca382c1dd0 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2feb7d1740 10 client.25715 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2feb7d1740 10 client.25715 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2feb7d1740 20 client.25715 populate_metadata read hostname 'smithi001' 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ca383438a0 0x55ca38345cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2feb7d1740 4 client.25715 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2feb7d1740 1 -- 172.21.15.1:0/990021002 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ca381aa4b0 con 0x55ca383438a0 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.062+0000 7f2fe5400640 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ca383438a0 0x55ca38345cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:32:47.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.063+0000 7f2fe5400640 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ca383438a0 0x55ca38345cc0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e95653799faf0f1a server_cookie=57db8a7cbd555fda in_seq=0 out_seq=0 2024-04-26T21:32:47.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.063+0000 7f2fc7400640 10 client.25715 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.104+0000 7f2fc7400640 1 -- 172.21.15.1:0/990021002 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ca381aa4b0 con 0x55ca383438a0 2024-04-26T21:32:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.104+0000 7f2fc7400640 10 client.25715 handle_command_reply: tid=0 2024-04-26T21:32:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.104+0000 7f2feb7d1740 1 client.25715 shutdown 2024-04-26T21:32:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 1 -- 172.21.15.1:0/990021002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 msgr2=0x55ca382d08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d08d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4098f60 tx=0x7f2fd4009700 comp rx=0 tx=0).stop 2024-04-26T21:32:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 1 -- 172.21.15.1:0/990021002 shutdown_connections 2024-04-26T21:32:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ca383438a0 0x55ca38345cc0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ca382c26f0 0x55ca382d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ca382ccfb0 0x55ca382d1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2fc7400640 0 client.25715 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:32:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 1 --2- 172.21.15.1:0/990021002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ca382c1dd0 0x55ca382d08d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 1 -- 172.21.15.1:0/990021002 >> 172.21.15.1:0/990021002 conn(0x55ca382be2e0 msgr2=0x55ca382ca7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 1 -- 172.21.15.1:0/990021002 shutdown_connections 2024-04-26T21:32:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 1 -- 172.21.15.1:0/990021002 wait complete. 2024-04-26T21:32:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.105+0000 7f2feb7d1740 20 client.25715 trim_cache size 0 max 16384 2024-04-26T21:32:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:32:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:32:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:32:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 -- 172.21.15.1:0/3946302037 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2fc80a3d90 msgr2=0x7f2fc80a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2fc80a3d90 0x7f2fc80a6250 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc004af0 tx=0x7f2fdc03f040 comp rx=0 tx=0).stop 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 -- 172.21.15.1:0/3946302037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe0104b50 msgr2=0x7f2fe019b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe0104b50 0x7f2fe019b230 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400e9f0 tx=0x7f2fd400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 -- 172.21.15.1:0/3946302037 shutdown_connections 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2fc80a3d90 0x7f2fc80a6250 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2fe010ac10 0x7f2fe019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fe0105510 0x7f2fe019b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 --2- 172.21.15.1:0/3946302037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe0104b50 0x7f2fe019b230 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 -- 172.21.15.1:0/3946302037 >> 172.21.15.1:0/3946302037 conn(0x7f2fe0100600 msgr2=0x7f2fe0101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:32:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 -- 172.21.15.1:0/3946302037 shutdown_connections 2024-04-26T21:32:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:32:47.106+0000 7f2fe8800640 1 -- 172.21.15.1:0/3946302037 wait complete. 2024-04-26T21:32:47.528 INFO:tasks.workunit.client.0.smithi001.stdout: 1 23726 3.17 MB/sec warmup 45 sec latency 24.585 ms 2024-04-26T21:32:48.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:48 smithi001 ceph-mon[28054]: pgmap v77: 129 pgs: 129 active+clean; 31 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 327 KiB/s wr, 69 op/s 2024-04-26T21:32:48.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:48 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:48.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:48 smithi038 ceph-mon[32724]: pgmap v77: 129 pgs: 129 active+clean; 31 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 327 KiB/s wr, 69 op/s 2024-04-26T21:32:48.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:48 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:48 smithi186 ceph-mon[32991]: pgmap v77: 129 pgs: 129 active+clean; 31 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 327 KiB/s wr, 69 op/s 2024-04-26T21:32:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:48 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:32:48.529 INFO:tasks.workunit.client.0.smithi001.stdout: 1 24701 3.27 MB/sec warmup 46 sec latency 26.313 ms 2024-04-26T21:32:49.529 INFO:tasks.workunit.client.0.smithi001.stdout: 1 25247 3.29 MB/sec warmup 47 sec latency 29.541 ms 2024-04-26T21:32:50.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:50 smithi001 ceph-mon[28054]: pgmap v78: 129 pgs: 129 active+clean; 31 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 327 KiB/s wr, 69 op/s 2024-04-26T21:32:50.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:50 smithi038 ceph-mon[32724]: pgmap v78: 129 pgs: 129 active+clean; 31 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 327 KiB/s wr, 69 op/s 2024-04-26T21:32:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:50 smithi186 ceph-mon[32991]: pgmap v78: 129 pgs: 129 active+clean; 31 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 327 KiB/s wr, 69 op/s 2024-04-26T21:32:50.529 INFO:tasks.workunit.client.0.smithi001.stdout: 1 25541 3.23 MB/sec warmup 48 sec latency 314.439 ms 2024-04-26T21:32:51.529 INFO:tasks.workunit.client.0.smithi001.stdout: 1 25597 3.17 MB/sec warmup 49 sec latency 1079.840 ms 2024-04-26T21:32:52.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:52 smithi001 ceph-mon[28054]: pgmap v79: 129 pgs: 129 active+clean; 32 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 533 KiB/s wr, 128 op/s 2024-04-26T21:32:52.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:52 smithi038 ceph-mon[32724]: pgmap v79: 129 pgs: 129 active+clean; 32 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 533 KiB/s wr, 128 op/s 2024-04-26T21:32:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:52 smithi186 ceph-mon[32991]: pgmap v79: 129 pgs: 129 active+clean; 32 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 533 KiB/s wr, 128 op/s 2024-04-26T21:32:52.529 INFO:tasks.workunit.client.0.smithi001.stdout: 1 25665 3.10 MB/sec warmup 50 sec latency 704.305 ms 2024-04-26T21:32:53.529 INFO:tasks.workunit.client.0.smithi001.stdout: 1 25665 3.04 MB/sec warmup 51 sec latency 1358.385 ms 2024-04-26T21:32:54.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:54 smithi001 ceph-mon[28054]: pgmap v80: 129 pgs: 129 active+clean; 32 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 376 KiB/s wr, 94 op/s 2024-04-26T21:32:54.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:54 smithi038 ceph-mon[32724]: pgmap v80: 129 pgs: 129 active+clean; 32 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 376 KiB/s wr, 94 op/s 2024-04-26T21:32:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:54 smithi186 ceph-mon[32991]: pgmap v80: 129 pgs: 129 active+clean; 32 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 376 KiB/s wr, 94 op/s 2024-04-26T21:32:54.529 INFO:tasks.workunit.client.0.smithi001.stdout: 1 25748 2.99 MB/sec warmup 52 sec latency 2285.794 ms 2024-04-26T21:32:55.529 INFO:tasks.workunit.client.0.smithi001.stdout: 1 27283 3.05 MB/sec warmup 53 sec latency 34.278 ms 2024-04-26T21:32:56.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:56 smithi001 ceph-mon[28054]: pgmap v81: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-04-26T21:32:56.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:56 smithi038 ceph-mon[32724]: pgmap v81: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-04-26T21:32:56.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:56 smithi186 ceph-mon[32991]: pgmap v81: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 1.5 MiB/s wr, 136 op/s 2024-04-26T21:32:56.530 INFO:tasks.workunit.client.0.smithi001.stdout: 1 28381 3.19 MB/sec warmup 54 sec latency 29.833 ms 2024-04-26T21:32:57.529 INFO:tasks.workunit.client.0.smithi001.stdout: 1 28818 3.16 MB/sec warmup 55 sec latency 26.642 ms 2024-04-26T21:32:58.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:32:58 smithi001 ceph-mon[28054]: pgmap v82: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-04-26T21:32:58.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:32:58 smithi038 ceph-mon[32724]: pgmap v82: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-04-26T21:32:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:32:58 smithi186 ceph-mon[32991]: pgmap v82: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-04-26T21:32:58.530 INFO:tasks.workunit.client.0.smithi001.stdout: 1 29343 3.12 MB/sec warmup 56 sec latency 22.600 ms 2024-04-26T21:32:59.530 INFO:tasks.workunit.client.0.smithi001.stdout: 1 30790 3.17 MB/sec warmup 57 sec latency 18.664 ms 2024-04-26T21:33:00.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.260+0000 7f063d400640 1 -- 172.21.15.1:0/3788094288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0638109b80 msgr2=0x7f063810dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.260+0000 7f063d400640 1 --2- 172.21.15.1:0/3788094288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0638109b80 0x7f063810dbc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0624009900 tx=0x7f062402a400 comp rx=0 tx=0).stop 2024-04-26T21:33:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.261+0000 7f063d400640 1 -- 172.21.15.1:0/3788094288 shutdown_connections 2024-04-26T21:33:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.261+0000 7f063d400640 1 --2- 172.21.15.1:0/3788094288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f063810e2a0 0x7f0638115e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.261+0000 7f063d400640 1 --2- 172.21.15.1:0/3788094288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0638109b80 0x7f063810dbc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.261+0000 7f063d400640 1 --2- 172.21.15.1:0/3788094288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06381091d0 0x7f06381095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.261+0000 7f063d400640 1 -- 172.21.15.1:0/3788094288 >> 172.21.15.1:0/3788094288 conn(0x7f0638076e40 msgr2=0x7f0638079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.261+0000 7f063d400640 1 -- 172.21.15.1:0/3788094288 shutdown_connections 2024-04-26T21:33:00.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.261+0000 7f063d400640 1 -- 172.21.15.1:0/3788094288 wait complete. 2024-04-26T21:33:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.262+0000 7f063d400640 1 Processor -- start 2024-04-26T21:33:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.262+0000 7f063d400640 1 -- start start 2024-04-26T21:33:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.262+0000 7f063d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06381091d0 0x7f06381a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.262+0000 7f063d400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063810e2a0 0x7f06381a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.262+0000 7f063d400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f063819f5f0 0x7f063819faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.262+0000 7f063d400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0638119230 con 0x7f063810e2a0 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.262+0000 7f063d400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f06381190b0 con 0x7f063819f5f0 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.262+0000 7f063d400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f06381193b0 con 0x7f06381091d0 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0636800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06381091d0 0x7f06381a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0636800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06381091d0 0x7f06381a5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:35634/0 (socket says 172.21.15.1:35634) 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0636800640 1 -- 172.21.15.1:0/1932133638 learned_addr learned my addr 172.21.15.1:0/1932133638 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0637200640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f063819f5f0 0x7f063819faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0635e00640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063810e2a0 0x7f06381a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0636800640 1 -- 172.21.15.1:0/1932133638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f063819f5f0 msgr2=0x7f063819faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0636800640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f063819f5f0 0x7f063819faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0636800640 1 -- 172.21.15.1:0/1932133638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063810e2a0 msgr2=0x7f06381a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0636800640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063810e2a0 0x7f06381a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0636800640 1 -- 172.21.15.1:0/1932133638 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06381a0110 con 0x7f06381091d0 2024-04-26T21:33:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.263+0000 7f0636800640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06381091d0 0x7f06381a5440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f062800e9f0 tx=0x7f062800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:00.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.264+0000 7f062f400640 1 -- 172.21.15.1:0/1932133638 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062800cd80 con 0x7f06381091d0 2024-04-26T21:33:00.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.264+0000 7f063d400640 1 -- 172.21.15.1:0/1932133638 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06381a0400 con 0x7f06381091d0 2024-04-26T21:33:00.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.264+0000 7f062f400640 1 -- 172.21.15.1:0/1932133638 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06280108e0 con 0x7f06381091d0 2024-04-26T21:33:00.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.264+0000 7f062f400640 1 -- 172.21.15.1:0/1932133638 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0628018980 con 0x7f06381091d0 2024-04-26T21:33:00.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.264+0000 7f063d400640 1 -- 172.21.15.1:0/1932133638 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06381ac6a0 con 0x7f06381091d0 2024-04-26T21:33:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.268+0000 7f064027f740 1 Processor -- start 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.268+0000 7f064027f740 1 -- start start 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.268+0000 7f062f400640 1 -- 172.21.15.1:0/1932133638 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0628010430 con 0x7f06381091d0 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f064027f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38bc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f064027f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f064027f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f064027f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55fde3821380 con 0x55fde38c6fb0 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f064027f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fde37bdc30 con 0x55fde38bc6f0 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f064027f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fde381d340 con 0x55fde38bbdd0 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f0636800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f0636800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43696/0 (socket says 172.21.15.1:43696) 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f0636800640 1 -- 172.21.15.1:0/850890084 learned_addr learned my addr 172.21.15.1:0/850890084 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:00.270 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f0637200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38bc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.269+0000 7f0635e00640 1 --2- 172.21.15.1:0/850890084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.270+0000 7f0636800640 1 -- 172.21.15.1:0/850890084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 msgr2=0x55fde38bc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.270+0000 7f0636800640 1 --2- 172.21.15.1:0/850890084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38bc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.270+0000 7f0636800640 1 -- 172.21.15.1:0/850890084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 msgr2=0x55fde38c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.270+0000 7f0636800640 1 --2- 172.21.15.1:0/850890084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.270+0000 7f0636800640 1 -- 172.21.15.1:0/850890084 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fde37df940 con 0x55fde38bc6f0 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.270+0000 7f0636800640 1 --2- 172.21.15.1:0/850890084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c6a70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f06280ce190 tx=0x7f06280d95f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56fed8f90eb382cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.270+0000 7f062f400640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f061c0a3d90 0x7f061c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.270+0000 7f0623400640 1 -- 172.21.15.1:0/850890084 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06280e4070 con 0x55fde38bc6f0 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.270+0000 7f0623400640 1 -- 172.21.15.1:0/850890084 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0628097410 con 0x55fde38bc6f0 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.270+0000 7f0623400640 1 -- 172.21.15.1:0/850890084 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06280914d0 con 0x55fde38bc6f0 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.271+0000 7f062f400640 1 -- 172.21.15.1:0/1932133638 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f06280d0ee0 con 0x7f06381091d0 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.271+0000 7f0637200640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f061c0a3d90 0x7f061c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.271+0000 7f064027f740 1 -- 172.21.15.1:0/850890084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 msgr2=0x55fde38c6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.271+0000 7f064027f740 1 --2- 172.21.15.1:0/850890084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c6a70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f06280ce190 tx=0x7f06280d95f0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.271+0000 7f0637200640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f061c0a3d90 0x7f061c0a6250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0630005fd0 tx=0x7f0630009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.271+0000 7f064027f740 1 -- 172.21.15.1:0/850890084 shutdown_connections 2024-04-26T21:33:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.272+0000 7f064027f740 1 --2- 172.21.15.1:0/850890084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.272+0000 7f064027f740 1 --2- 172.21.15.1:0/850890084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c6a70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.272+0000 7f064027f740 1 --2- 172.21.15.1:0/850890084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38bc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.272+0000 7f064027f740 1 -- 172.21.15.1:0/850890084 >> 172.21.15.1:0/850890084 conn(0x55fde38b82e0 msgr2=0x55fde38b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.272+0000 7f064027f740 1 -- 172.21.15.1:0/850890084 shutdown_connections 2024-04-26T21:33:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.272+0000 7f064027f740 1 -- 172.21.15.1:0/850890084 wait complete. 2024-04-26T21:33:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.272+0000 7f064027f740 1 -- start start 2024-04-26T21:33:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.272+0000 7f064027f740 1 Processor -- start 2024-04-26T21:33:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f064027f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38c3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f064027f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f064027f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f064027f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55fde38c9be0 con 0x55fde38c6fb0 2024-04-26T21:33:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f064027f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55fde37a8c60 con 0x55fde38bc6f0 2024-04-26T21:33:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f064027f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fde38c9d60 con 0x55fde38bbdd0 2024-04-26T21:33:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f064027f740 10 client.0 _ll_register_callbacks cb 0x55fde37d4e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:33:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f0635e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f0635e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43708/0 (socket says 172.21.15.1:43708) 2024-04-26T21:33:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f0635e00640 1 -- 172.21.15.1:0/3783265643 learned_addr learned my addr 172.21.15.1:0/3783265643 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f0636800640 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38c3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.273+0000 7f0637200640 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.274+0000 7f0635e00640 1 -- 172.21.15.1:0/3783265643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 msgr2=0x55fde38c3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.274+0000 7f0635e00640 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38c3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.274+0000 7f0635e00640 1 -- 172.21.15.1:0/3783265643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 msgr2=0x55fde38c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.274+0000 7f0635e00640 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.274+0000 7f0635e00640 1 -- 172.21.15.1:0/3783265643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fde37b7510 con 0x55fde38bc6f0 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.274+0000 7f0635e00640 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c4430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0624000c00 tx=0x7f0624004440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f1cd149bd6ec58d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.274+0000 7f0617e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.274+0000 7f0617e00640 1 -- 172.21.15.1:0/3783265643 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624035040 con 0x55fde38bc6f0 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.274+0000 7f0617e00640 1 -- 172.21.15.1:0/3783265643 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f0624005450 con 0x55fde38bc6f0 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.274+0000 7f064027f740 1 -- 172.21.15.1:0/3783265643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0628097410 con 0x55fde38bc6f0 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f0617e00640 1 -- 172.21.15.1:0/3783265643 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f062404f050 con 0x55fde38bc6f0 2024-04-26T21:33:00.278 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f064027f740 10 client.25727 fetch_fsmap learned FSMap version 30 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f064027f740 1 -- 172.21.15.1:0/3783265643 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fde3533210 con 0x55fde38bc6f0 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f0617e00640 1 -- 172.21.15.1:0/3783265643 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0628097410 con 0x55fde38bc6f0 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f0617e00640 1 -- 172.21.15.1:0/3783265643 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f0624003a00 con 0x55fde38bc6f0 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f064027f740 10 client.25727 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f064027f740 10 client.25727 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f064027f740 20 client.25727 populate_metadata read hostname 'smithi001' 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f064027f740 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55fde38d6cc0 0x55fde38d9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f064027f740 4 client.25727 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.275+0000 7f064027f740 1 -- 172.21.15.1:0/3783265643 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fde382ef90 con 0x55fde38d6cc0 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.276+0000 7f0636800640 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55fde38d6cc0 0x55fde38d9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.276+0000 7f0636800640 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55fde38d6cc0 0x55fde38d9110 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac120088737cbea6 server_cookie=7a9adf0e30eaabb3 in_seq=0 out_seq=0 2024-04-26T21:33:00.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.276+0000 7f0617e00640 10 client.25727 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:00.280 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f0617e00640 1 -- 172.21.15.1:0/3783265643 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fde382ef90 con 0x55fde38d6cc0 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f0617e00640 10 client.25727 handle_command_reply: tid=0 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f064027f740 1 client.25727 shutdown 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f064027f740 1 -- 172.21.15.1:0/3783265643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 msgr2=0x55fde38c4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f064027f740 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c4430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0624000c00 tx=0x7f0624004440 comp rx=0 tx=0).stop 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f064027f740 1 -- 172.21.15.1:0/3783265643 shutdown_connections 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f064027f740 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55fde38d6cc0 0x55fde38d9110 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f064027f740 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f064027f740 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c4430 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f0617e00640 0 client.25727 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f064027f740 1 --2- 172.21.15.1:0/3783265643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38c3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.278+0000 7f064027f740 1 -- 172.21.15.1:0/3783265643 >> 172.21.15.1:0/3783265643 conn(0x55fde38b82e0 msgr2=0x55fde38be010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.279+0000 7f064027f740 1 -- 172.21.15.1:0/3783265643 shutdown_connections 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.279+0000 7f064027f740 1 -- 172.21.15.1:0/3783265643 wait complete. 2024-04-26T21:33:00.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.279+0000 7f064027f740 20 client.25727 trim_cache size 0 max 16384 2024-04-26T21:33:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.296+0000 7f064027f740 1 Processor -- start 2024-04-26T21:33:00.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.296+0000 7f064027f740 1 -- start start 2024-04-26T21:33:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f064027f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38bbdd0 0x55fde38d2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f064027f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f064027f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38c6fb0 0x55fde38dea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f064027f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f06381b8e00 con 0x55fde38bbdd0 2024-04-26T21:33:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f064027f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f06381b6670 con 0x55fde38bc6f0 2024-04-26T21:33:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f064027f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f06381b44c0 con 0x55fde38c6fb0 2024-04-26T21:33:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0635e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0637200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38c6fb0 0x55fde38dea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0635e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43722/0 (socket says 172.21.15.1:43722) 2024-04-26T21:33:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0635e00640 1 -- 172.21.15.1:0/83170760 learned_addr learned my addr 172.21.15.1:0/83170760 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0636800640 1 --2- 172.21.15.1:0/83170760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38bbdd0 0x55fde38d2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0635e00640 1 -- 172.21.15.1:0/83170760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38c6fb0 msgr2=0x55fde38dea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0635e00640 1 --2- 172.21.15.1:0/83170760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38c6fb0 0x55fde38dea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0635e00640 1 -- 172.21.15.1:0/83170760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38bbdd0 msgr2=0x55fde38d2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0635e00640 1 --2- 172.21.15.1:0/83170760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38bbdd0 0x55fde38d2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0635e00640 1 -- 172.21.15.1:0/83170760 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fde35ff620 con 0x55fde38bc6f0 2024-04-26T21:33:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0636800640 1 --2- 172.21.15.1:0/83170760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38bbdd0 0x55fde38d2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.297+0000 7f0637200640 1 --2- 172.21.15.1:0/83170760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38c6fb0 0x55fde38dea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:00.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.298+0000 7f0635e00640 1 --2- 172.21.15.1:0/83170760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c2ed0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0624031440 tx=0x7f0624031650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6559a1112955f21e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.298+0000 7f0622a00640 1 -- 172.21.15.1:0/83170760 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624035070 con 0x55fde38bc6f0 2024-04-26T21:33:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.298+0000 7f0622a00640 1 -- 172.21.15.1:0/83170760 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0624041070 con 0x55fde38bc6f0 2024-04-26T21:33:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.298+0000 7f0622a00640 1 -- 172.21.15.1:0/83170760 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624003b70 con 0x55fde38bc6f0 2024-04-26T21:33:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.299+0000 7f064027f740 1 -- 172.21.15.1:0/83170760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 msgr2=0x55fde38c2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.299+0000 7f064027f740 1 --2- 172.21.15.1:0/83170760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c2ed0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0624031440 tx=0x7f0624031650 comp rx=0 tx=0).stop 2024-04-26T21:33:00.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.299+0000 7f064027f740 1 -- 172.21.15.1:0/83170760 shutdown_connections 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.299+0000 7f064027f740 1 --2- 172.21.15.1:0/83170760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38c6fb0 0x55fde38dea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.299+0000 7f064027f740 1 --2- 172.21.15.1:0/83170760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38c2ed0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.299+0000 7f064027f740 1 --2- 172.21.15.1:0/83170760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38bbdd0 0x55fde38d2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.299+0000 7f064027f740 1 -- 172.21.15.1:0/83170760 >> 172.21.15.1:0/83170760 conn(0x55fde38b82e0 msgr2=0x55fde38c27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.299+0000 7f064027f740 1 -- 172.21.15.1:0/83170760 shutdown_connections 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.299+0000 7f064027f740 1 -- 172.21.15.1:0/83170760 wait complete. 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.299+0000 7f064027f740 1 -- start start 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.300+0000 7f064027f740 1 Processor -- start 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.300+0000 7f064027f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.300+0000 7f064027f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.300+0000 7f064027f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.300+0000 7f064027f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f06381b4b30 con 0x55fde38c6fb0 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.300+0000 7f064027f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f06381b5fa0 con 0x55fde38bc6f0 2024-04-26T21:33:00.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.300+0000 7f064027f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f06381b5200 con 0x55fde38bbdd0 2024-04-26T21:33:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.300+0000 7f064027f740 10 client.0 _ll_register_callbacks cb 0x7f0624009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:33:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.300+0000 7f0635e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.300+0000 7f0636800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0635e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38cf4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43726/0 (socket says 172.21.15.1:43726) 2024-04-26T21:33:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0636800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:35678/0 (socket says 172.21.15.1:35678) 2024-04-26T21:33:00.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0635e00640 1 -- 172.21.15.1:0/753664885 learned_addr learned my addr 172.21.15.1:0/753664885 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0637200640 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0635e00640 1 -- 172.21.15.1:0/753664885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 msgr2=0x55fde38cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0635e00640 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0635e00640 1 -- 172.21.15.1:0/753664885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 msgr2=0x55fde38c0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0635e00640 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0635e00640 1 -- 172.21.15.1:0/753664885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fde37dbcb0 con 0x55fde38bc6f0 2024-04-26T21:33:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0637200640 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:33:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0635e00640 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38cf4f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0624003f20 tx=0x7f0624004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fd67ae534e3bb9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.301+0000 7f0617e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:33:00.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f0617e00640 1 -- 172.21.15.1:0/753664885 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624035070 con 0x55fde38bc6f0 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f0617e00640 1 -- 172.21.15.1:0/753664885 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f062404c040 con 0x55fde38bc6f0 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f064027f740 1 -- 172.21.15.1:0/753664885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0624041070 con 0x55fde38bc6f0 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f0617e00640 1 -- 172.21.15.1:0/753664885 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0624064050 con 0x55fde38bc6f0 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f064027f740 10 client.25733 fetch_fsmap learned FSMap version 30 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f064027f740 1 -- 172.21.15.1:0/753664885 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fde393cc40 con 0x55fde38bc6f0 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f0617e00640 1 -- 172.21.15.1:0/753664885 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0624041070 con 0x55fde38bc6f0 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f0617e00640 1 -- 172.21.15.1:0/753664885 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f0624031a80 con 0x55fde38bc6f0 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f064027f740 10 client.25733 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f064027f740 10 client.25733 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f064027f740 20 client.25733 populate_metadata read hostname 'smithi001' 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f064027f740 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55fde393d290 0x55fde393f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f064027f740 4 client.25733 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.302+0000 7f064027f740 1 -- 172.21.15.1:0/753664885 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fde3828e00 con 0x55fde393d290 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.303+0000 7f0636800640 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55fde393d290 0x55fde393f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:00.306 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.303+0000 7f0617e00640 10 client.25733 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:00.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.303+0000 7f0636800640 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55fde393d290 0x55fde393f6b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98039d5dca98542 server_cookie=acf197ac1e759a4e in_seq=0 out_seq=0 2024-04-26T21:33:00.319 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:00 smithi001 ceph-mon[28054]: pgmap v83: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-04-26T21:33:00.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:00 smithi038 ceph-mon[32724]: pgmap v83: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-04-26T21:33:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f0617e00640 1 -- 172.21.15.1:0/753664885 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fde3828e00 con 0x55fde393d290 2024-04-26T21:33:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f0617e00640 10 client.25733 handle_command_reply: tid=0 2024-04-26T21:33:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 client.25733 shutdown 2024-04-26T21:33:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 -- 172.21.15.1:0/753664885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 msgr2=0x55fde38cf4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38cf4f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0624003f20 tx=0x7f0624004040 comp rx=0 tx=0).stop 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 -- 172.21.15.1:0/753664885 shutdown_connections 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55fde393d290 0x55fde393f6b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55fde38c6fb0 0x55fde38c0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fde38bc6f0 0x55fde38cf4f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 --2- 172.21.15.1:0/753664885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde38bbdd0 0x55fde38cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f0617e00640 0 client.25733 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 -- 172.21.15.1:0/753664885 >> 172.21.15.1:0/753664885 conn(0x55fde38b82e0 msgr2=0x55fde38be010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 -- 172.21.15.1:0/753664885 shutdown_connections 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.351+0000 7f064027f740 1 -- 172.21.15.1:0/753664885 wait complete. 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.352+0000 7f064027f740 20 client.25733 trim_cache size 0 max 16384 2024-04-26T21:33:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:33:00.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:33:00.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:33:00.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:33:00.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.352+0000 7f063d400640 1 -- 172.21.15.1:0/1932133638 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f061c0a3d90 msgr2=0x7f061c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.352+0000 7f063d400640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f061c0a3d90 0x7f061c0a6250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0630005fd0 tx=0x7f0630009500 comp rx=0 tx=0).stop 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.352+0000 7f063d400640 1 -- 172.21.15.1:0/1932133638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06381091d0 msgr2=0x7f06381a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.352+0000 7f063d400640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06381091d0 0x7f06381a5440 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f062800e9f0 tx=0x7f062800eeb0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.353+0000 7f063d400640 1 -- 172.21.15.1:0/1932133638 shutdown_connections 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.353+0000 7f063d400640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f061c0a3d90 0x7f061c0a6250 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.353+0000 7f063d400640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f063819f5f0 0x7f063819faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.353+0000 7f063d400640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f063810e2a0 0x7f06381a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.353+0000 7f063d400640 1 --2- 172.21.15.1:0/1932133638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06381091d0 0x7f06381a5440 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.353+0000 7f063d400640 1 -- 172.21.15.1:0/1932133638 >> 172.21.15.1:0/1932133638 conn(0x7f0638076e40 msgr2=0x7f0638078160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.353+0000 7f063d400640 1 -- 172.21.15.1:0/1932133638 shutdown_connections 2024-04-26T21:33:00.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:00.353+0000 7f063d400640 1 -- 172.21.15.1:0/1932133638 wait complete. 2024-04-26T21:33:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:00 smithi186 ceph-mon[32991]: pgmap v83: 129 pgs: 129 active+clean; 39 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 1.3 MiB/s wr, 100 op/s 2024-04-26T21:33:00.530 INFO:tasks.workunit.client.0.smithi001.stdout: 1 31927 3.30 MB/sec warmup 58 sec latency 21.905 ms 2024-04-26T21:33:01.530 INFO:tasks.workunit.client.0.smithi001.stdout: 1 32157 3.26 MB/sec warmup 59 sec latency 88.871 ms 2024-04-26T21:33:02.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:02 smithi001 ceph-mon[28054]: pgmap v84: 129 pgs: 129 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-04-26T21:33:02.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:02 smithi038 ceph-mon[32724]: pgmap v84: 129 pgs: 129 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-04-26T21:33:02.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:02 smithi186 ceph-mon[32991]: pgmap v84: 129 pgs: 129 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-04-26T21:33:02.530 INFO:tasks.workunit.client.0.smithi001.stdout: 1 32337 3.22 MB/sec warmup 60 sec latency 72.673 ms 2024-04-26T21:33:03.530 INFO:tasks.workunit.client.0.smithi001.stdout: 1 32755 3.17 MB/sec warmup 61 sec latency 154.396 ms 2024-04-26T21:33:04.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:04 smithi038 ceph-mon[32724]: pgmap v85: 129 pgs: 129 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.3 MiB/s wr, 106 op/s 2024-04-26T21:33:04.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:04 smithi186 ceph-mon[32991]: pgmap v85: 129 pgs: 129 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.3 MiB/s wr, 106 op/s 2024-04-26T21:33:04.530 INFO:tasks.workunit.client.0.smithi001.stdout: 1 32755 3.12 MB/sec warmup 62 sec latency 1154.554 ms 2024-04-26T21:33:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:04 smithi001 ceph-mon[28054]: pgmap v85: 129 pgs: 129 active+clean; 34 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 1.3 MiB/s wr, 106 op/s 2024-04-26T21:33:05.530 INFO:tasks.workunit.client.0.smithi001.stdout: 1 32755 3.07 MB/sec warmup 63 sec latency 2154.706 ms 2024-04-26T21:33:06.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:06 smithi186 ceph-mon[32991]: pgmap v86: 129 pgs: 129 active+clean; 35 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.6 MiB/s wr, 187 op/s 2024-04-26T21:33:06.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 32964 3.03 MB/sec warmup 64 sec latency 2916.382 ms 2024-04-26T21:33:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:06 smithi001 ceph-mon[28054]: pgmap v86: 129 pgs: 129 active+clean; 35 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.6 MiB/s wr, 187 op/s 2024-04-26T21:33:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:06 smithi038 ceph-mon[32724]: pgmap v86: 129 pgs: 129 active+clean; 35 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 1.6 MiB/s wr, 187 op/s 2024-04-26T21:33:07.530 INFO:tasks.workunit.client.0.smithi001.stdout: 1 34333 3.08 MB/sec warmup 65 sec latency 35.816 ms 2024-04-26T21:33:08.370 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:08 smithi001 ceph-mon[28054]: pgmap v87: 129 pgs: 129 active+clean; 35 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 490 KiB/s wr, 145 op/s 2024-04-26T21:33:08.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:08 smithi186 ceph-mon[32991]: pgmap v87: 129 pgs: 129 active+clean; 35 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 490 KiB/s wr, 145 op/s 2024-04-26T21:33:08.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 35307 3.14 MB/sec warmup 66 sec latency 38.370 ms 2024-04-26T21:33:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:08 smithi038 ceph-mon[32724]: pgmap v87: 129 pgs: 129 active+clean; 35 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 490 KiB/s wr, 145 op/s 2024-04-26T21:33:09.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 35831 3.16 MB/sec warmup 67 sec latency 26.737 ms 2024-04-26T21:33:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:10 smithi186 ceph-mon[32991]: pgmap v88: 129 pgs: 129 active+clean; 35 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 490 KiB/s wr, 145 op/s 2024-04-26T21:33:10.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 36174 3.12 MB/sec warmup 68 sec latency 155.950 ms 2024-04-26T21:33:10.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:10 smithi001 ceph-mon[28054]: pgmap v88: 129 pgs: 129 active+clean; 35 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 490 KiB/s wr, 145 op/s 2024-04-26T21:33:10.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:10 smithi038 ceph-mon[32724]: pgmap v88: 129 pgs: 129 active+clean; 35 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 490 KiB/s wr, 145 op/s 2024-04-26T21:33:11.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 36298 3.08 MB/sec warmup 69 sec latency 920.137 ms 2024-04-26T21:33:12.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:12 smithi186 ceph-mon[32991]: pgmap v89: 129 pgs: 129 active+clean; 36 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 660 KiB/s wr, 179 op/s 2024-04-26T21:33:12.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 36644 3.06 MB/sec warmup 70 sec latency 620.806 ms 2024-04-26T21:33:12.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:12 smithi001 ceph-mon[28054]: pgmap v89: 129 pgs: 129 active+clean; 36 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 660 KiB/s wr, 179 op/s 2024-04-26T21:33:12.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:12 smithi038 ceph-mon[32724]: pgmap v89: 129 pgs: 129 active+clean; 36 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 660 KiB/s wr, 179 op/s 2024-04-26T21:33:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.504+0000 7f096b000640 1 -- 172.21.15.1:0/553781589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0964103ba0 msgr2=0x7f0964103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.504+0000 7f096b000640 1 --2- 172.21.15.1:0/553781589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0964103ba0 0x7f0964103f80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0954009900 tx=0x7f095402a480 comp rx=0 tx=0).stop 2024-04-26T21:33:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.504+0000 7f096b000640 1 -- 172.21.15.1:0/553781589 shutdown_connections 2024-04-26T21:33:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.504+0000 7f096b000640 1 --2- 172.21.15.1:0/553781589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f096410cf70 0x7f096410f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.504+0000 7f096b000640 1 --2- 172.21.15.1:0/553781589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09641044c0 0x7f096410c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.504+0000 7f096b000640 1 --2- 172.21.15.1:0/553781589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0964103ba0 0x7f0964103f80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.504+0000 7f096b000640 1 -- 172.21.15.1:0/553781589 >> 172.21.15.1:0/553781589 conn(0x7f0964078980 msgr2=0x7f0964078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.504+0000 7f096b000640 1 -- 172.21.15.1:0/553781589 shutdown_connections 2024-04-26T21:33:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.504+0000 7f096b000640 1 -- 172.21.15.1:0/553781589 wait complete. 2024-04-26T21:33:13.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.506+0000 7f096b000640 1 Processor -- start 2024-04-26T21:33:13.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.506+0000 7f096b000640 1 -- start start 2024-04-26T21:33:13.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.506+0000 7f096b000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0964103ba0 0x7f096419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.506+0000 7f096b000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09641044c0 0x7f096419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.509 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.506+0000 7f096b000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f096410cf70 0x7f096419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.506+0000 7f096b000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0964112be0 con 0x7f096410cf70 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.506+0000 7f096b000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0964112a60 con 0x7f0964103ba0 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.506+0000 7f096b000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0964112d60 con 0x7f09641044c0 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.507+0000 7f0963400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09641044c0 0x7f096419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.507+0000 7f0963400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09641044c0 0x7f096419b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54220/0 (socket says 172.21.15.1:54220) 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.507+0000 7f0963400640 1 -- 172.21.15.1:0/866612655 learned_addr learned my addr 172.21.15.1:0/866612655 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.507+0000 7f0969000640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f096410cf70 0x7f096419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.507+0000 7f0963e00640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0964103ba0 0x7f096419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.507+0000 7f0963400640 1 -- 172.21.15.1:0/866612655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0964103ba0 msgr2=0x7f096419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.507+0000 7f0963400640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0964103ba0 0x7f096419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.507+0000 7f0963400640 1 -- 172.21.15.1:0/866612655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f096410cf70 msgr2=0x7f096419fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.507+0000 7f0963400640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f096410cf70 0x7f096419fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.507+0000 7f0963400640 1 -- 172.21.15.1:0/866612655 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09641a0310 con 0x7f09641044c0 2024-04-26T21:33:13.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.508+0000 7f0963400640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09641044c0 0x7f096419b860 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f095000d950 tx=0x7f095000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:13.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.508+0000 7f0960c00640 1 -- 172.21.15.1:0/866612655 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0950017030 con 0x7f09641044c0 2024-04-26T21:33:13.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.508+0000 7f0960c00640 1 -- 172.21.15.1:0/866612655 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0950004020 con 0x7f09641044c0 2024-04-26T21:33:13.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.508+0000 7f0960c00640 1 -- 172.21.15.1:0/866612655 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09500047a0 con 0x7f09641044c0 2024-04-26T21:33:13.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.508+0000 7f096b000640 1 -- 172.21.15.1:0/866612655 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09641a0600 con 0x7f09641044c0 2024-04-26T21:33:13.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.508+0000 7f096b000640 1 -- 172.21.15.1:0/866612655 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09641a7e90 con 0x7f09641044c0 2024-04-26T21:33:13.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.512+0000 7f0960c00640 1 -- 172.21.15.1:0/866612655 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0950009a30 con 0x7f09641044c0 2024-04-26T21:33:13.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.512+0000 7f096de3d740 1 Processor -- start 2024-04-26T21:33:13.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.513+0000 7f096de3d740 1 -- start start 2024-04-26T21:33:13.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.513+0000 7f096de3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f69a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.513+0000 7f096de3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 0x555a5f6a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.513+0000 7f096de3d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.513+0000 7f096de3d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555a5f5ff380 con 0x555a5f6a4fb0 2024-04-26T21:33:13.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.513+0000 7f096de3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555a5f59bc30 con 0x555a5f699dd0 2024-04-26T21:33:13.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.513+0000 7f096de3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555a5f5fb340 con 0x555a5f69a6f0 2024-04-26T21:33:13.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0963e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 0x555a5f6a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0963e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 0x555a5f6a4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54222/0 (socket says 172.21.15.1:54222) 2024-04-26T21:33:13.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0963e00640 1 -- 172.21.15.1:0/1339868597 learned_addr learned my addr 172.21.15.1:0/1339868597 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:13.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0969000640 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f69a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0963400640 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0963e00640 1 -- 172.21.15.1:0/1339868597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 msgr2=0x555a5f69a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0963e00640 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f69a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0963e00640 1 -- 172.21.15.1:0/1339868597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 msgr2=0x555a5f6a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0963e00640 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0963e00640 1 -- 172.21.15.1:0/1339868597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a5f5bd940 con 0x555a5f69a6f0 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0963400640 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0969000640 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f69a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.514+0000 7f0960c00640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09480a3d10 0x7f09480a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.515+0000 7f0963e00640 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 0x555a5f6a4a70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0954000c00 tx=0x7f0954004170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e9b8577ce4253eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.515+0000 7f0960c00640 1 -- 172.21.15.1:0/866612655 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f09500d0640 con 0x7f09641044c0 2024-04-26T21:33:13.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.515+0000 7f0969000640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09480a3d10 0x7f09480a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.515+0000 7f0959600640 1 -- 172.21.15.1:0/1339868597 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954035040 con 0x555a5f69a6f0 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.515+0000 7f0959600640 1 -- 172.21.15.1:0/1339868597 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09540054f0 con 0x555a5f69a6f0 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.515+0000 7f0959600640 1 -- 172.21.15.1:0/1339868597 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09540035f0 con 0x555a5f69a6f0 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.516+0000 7f096de3d740 1 -- 172.21.15.1:0/1339868597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 msgr2=0x555a5f6a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.516+0000 7f0969000640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09480a3d10 0x7f09480a61d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f095c009fd0 tx=0x7f095c009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.516+0000 7f096de3d740 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 0x555a5f6a4a70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0954000c00 tx=0x7f0954004170 comp rx=0 tx=0).stop 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.516+0000 7f096de3d740 1 -- 172.21.15.1:0/1339868597 shutdown_connections 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.516+0000 7f096de3d740 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 0x555a5f6a4a70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.516+0000 7f096de3d740 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.516+0000 7f096de3d740 1 --2- 172.21.15.1:0/1339868597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f69a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.516+0000 7f096de3d740 1 -- 172.21.15.1:0/1339868597 >> 172.21.15.1:0/1339868597 conn(0x555a5f6962e0 msgr2=0x555a5f6966c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.516+0000 7f096de3d740 1 -- 172.21.15.1:0/1339868597 shutdown_connections 2024-04-26T21:33:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.516+0000 7f096de3d740 1 -- 172.21.15.1:0/1339868597 wait complete. 2024-04-26T21:33:13.522 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.517+0000 7f096de3d740 1 -- start start 2024-04-26T21:33:13.522 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.517+0000 7f096de3d740 1 Processor -- start 2024-04-26T21:33:13.522 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f096de3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f6a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f096de3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 0x555a5f6a2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f096de3d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f096de3d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555a5f6a7b80 con 0x555a5f6a4fb0 2024-04-26T21:33:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f096de3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555a5f586c60 con 0x555a5f699dd0 2024-04-26T21:33:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f096de3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555a5f6a7d00 con 0x555a5f69a6f0 2024-04-26T21:33:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f096de3d740 10 client.0 _ll_register_callbacks cb 0x555a5f5b2e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:33:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f0963e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f6a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f0963e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f6a1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50798/0 (socket says 172.21.15.1:50798) 2024-04-26T21:33:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f0963e00640 1 -- 172.21.15.1:0/862343963 learned_addr learned my addr 172.21.15.1:0/862343963 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:13.523 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f0963400640 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 0x555a5f6a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f0969000640 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f0963e00640 1 -- 172.21.15.1:0/862343963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 msgr2=0x555a5f6a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f0963e00640 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 0x555a5f6a2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f0963e00640 1 -- 172.21.15.1:0/862343963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 msgr2=0x555a5f6a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f0963e00640 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.518+0000 7f0963e00640 1 -- 172.21.15.1:0/862343963 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a5f595510 con 0x555a5f699dd0 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.519+0000 7f0963e00640 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f6a1f00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f095402a9a0 tx=0x7f095402a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d7edf1752d6d962 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.519+0000 7f094e000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.519+0000 7f094e000640 1 -- 172.21.15.1:0/862343963 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954053060 con 0x555a5f699dd0 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.519+0000 7f096de3d740 1 -- 172.21.15.1:0/862343963 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09540054f0 con 0x555a5f699dd0 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.519+0000 7f094e000640 1 -- 172.21.15.1:0/862343963 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f0954004d30 con 0x555a5f699dd0 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.519+0000 7f094e000640 1 -- 172.21.15.1:0/862343963 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f095405a050 con 0x555a5f699dd0 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.519+0000 7f096de3d740 10 client.25745 fetch_fsmap learned FSMap version 30 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.519+0000 7f096de3d740 1 -- 172.21.15.1:0/862343963 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555a5f311210 con 0x555a5f699dd0 2024-04-26T21:33:13.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.520+0000 7f094e000640 1 -- 172.21.15.1:0/862343963 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09540054f0 con 0x555a5f699dd0 2024-04-26T21:33:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.520+0000 7f094e000640 1 -- 172.21.15.1:0/862343963 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f0954003af0 con 0x555a5f699dd0 2024-04-26T21:33:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.520+0000 7f096de3d740 10 client.25745 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T21:33:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.520+0000 7f096de3d740 10 client.25745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:33:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.520+0000 7f096de3d740 20 client.25745 populate_metadata read hostname 'smithi001' 2024-04-26T21:33:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.520+0000 7f096de3d740 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555a5f6b4b30 0x555a5f6b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.520+0000 7f096de3d740 4 client.25745 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:33:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.520+0000 7f096de3d740 1 -- 172.21.15.1:0/862343963 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a5f60cf90 con 0x555a5f6b4b30 2024-04-26T21:33:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.521+0000 7f0963400640 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555a5f6b4b30 0x555a5f6b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.521+0000 7f0963400640 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555a5f6b4b30 0x555a5f6b6f20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e9f576cc3ce28c server_cookie=dc3130ded23dafd0 in_seq=0 out_seq=0 2024-04-26T21:33:13.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.521+0000 7f094e000640 10 client.25745 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f094e000640 1 -- 172.21.15.1:0/862343963 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a5f60cf90 con 0x555a5f6b4b30 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f094e000640 10 client.25745 handle_command_reply: tid=0 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 client.25745 shutdown 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 -- 172.21.15.1:0/862343963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 msgr2=0x555a5f6a1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f6a1f00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f095402a9a0 tx=0x7f095402a480 comp rx=0 tx=0).stop 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 -- 172.21.15.1:0/862343963 shutdown_connections 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555a5f6b4b30 0x555a5f6b6f20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f69a6f0 0x555a5f6a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 --2- 172.21.15.1:0/862343963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f699dd0 0x555a5f6a1f00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 -- 172.21.15.1:0/862343963 >> 172.21.15.1:0/862343963 conn(0x555a5f6962e0 msgr2=0x555a5f69c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:13.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f094e000640 0 client.25745 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:13.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 -- 172.21.15.1:0/862343963 shutdown_connections 2024-04-26T21:33:13.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.522+0000 7f096de3d740 1 -- 172.21.15.1:0/862343963 wait complete. 2024-04-26T21:33:13.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.523+0000 7f096de3d740 20 client.25745 trim_cache size 0 max 16384 2024-04-26T21:33:13.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 37975 3.10 MB/sec warmup 71 sec latency 29.015 ms 2024-04-26T21:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.541+0000 7f096de3d740 1 Processor -- start 2024-04-26T21:33:13.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.541+0000 7f096de3d740 1 -- start start 2024-04-26T21:33:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f096de3d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f699dd0 0x555a5f6b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f096de3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f69a6f0 0x555a5f69c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f096de3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f096de3d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f09641b4930 con 0x555a5f699dd0 2024-04-26T21:33:13.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f096de3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09641b21a0 con 0x555a5f69a6f0 2024-04-26T21:33:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f096de3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09641afff0 con 0x555a5f6a4fb0 2024-04-26T21:33:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f0969000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f0969000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54246/0 (socket says 172.21.15.1:54246) 2024-04-26T21:33:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f0969000640 1 -- 172.21.15.1:0/1013450973 learned_addr learned my addr 172.21.15.1:0/1013450973 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f0963e00640 1 --2- 172.21.15.1:0/1013450973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f699dd0 0x555a5f6b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.542+0000 7f0963400640 1 --2- 172.21.15.1:0/1013450973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f69a6f0 0x555a5f69c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.543+0000 7f0969000640 1 -- 172.21.15.1:0/1013450973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f69a6f0 msgr2=0x555a5f69c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:13.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.543+0000 7f0969000640 1 --2- 172.21.15.1:0/1013450973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f69a6f0 0x555a5f69c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.543+0000 7f0969000640 1 -- 172.21.15.1:0/1013450973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f699dd0 msgr2=0x555a5f6b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:13.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.543+0000 7f0969000640 1 --2- 172.21.15.1:0/1013450973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f699dd0 0x555a5f6b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.543+0000 7f0969000640 1 -- 172.21.15.1:0/1013450973 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a5f691940 con 0x555a5f6a4fb0 2024-04-26T21:33:13.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.543+0000 7f0963400640 1 --2- 172.21.15.1:0/1013450973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f69a6f0 0x555a5f69c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:13.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.543+0000 7f0969000640 1 --2- 172.21.15.1:0/1013450973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a25d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f095c0113f0 tx=0x7f095c016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57d29ee12a30b25e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:13.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.543+0000 7f0958c00640 1 -- 172.21.15.1:0/1013450973 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c004b20 con 0x555a5f6a4fb0 2024-04-26T21:33:13.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.543+0000 7f0958c00640 1 -- 172.21.15.1:0/1013450973 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f095c004cc0 con 0x555a5f6a4fb0 2024-04-26T21:33:13.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.543+0000 7f0958c00640 1 -- 172.21.15.1:0/1013450973 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c029470 con 0x555a5f6a4fb0 2024-04-26T21:33:13.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.544+0000 7f096de3d740 1 -- 172.21.15.1:0/1013450973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f6a4fb0 msgr2=0x555a5f6a25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:13.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.544+0000 7f096de3d740 1 --2- 172.21.15.1:0/1013450973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a25d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f095c0113f0 tx=0x7f095c016000 comp rx=0 tx=0).stop 2024-04-26T21:33:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.544+0000 7f096de3d740 1 -- 172.21.15.1:0/1013450973 shutdown_connections 2024-04-26T21:33:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.544+0000 7f096de3d740 1 --2- 172.21.15.1:0/1013450973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f6a4fb0 0x555a5f6a25d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.544+0000 7f096de3d740 1 --2- 172.21.15.1:0/1013450973 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f69a6f0 0x555a5f69c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.544+0000 7f096de3d740 1 --2- 172.21.15.1:0/1013450973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f699dd0 0x555a5f6b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.544+0000 7f096de3d740 1 -- 172.21.15.1:0/1013450973 >> 172.21.15.1:0/1013450973 conn(0x555a5f6962e0 msgr2=0x555a5f6a1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:13.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.544+0000 7f096de3d740 1 -- 172.21.15.1:0/1013450973 shutdown_connections 2024-04-26T21:33:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.544+0000 7f096de3d740 1 -- 172.21.15.1:0/1013450973 wait complete. 2024-04-26T21:33:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.545+0000 7f096de3d740 1 -- start start 2024-04-26T21:33:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.545+0000 7f096de3d740 1 Processor -- start 2024-04-26T21:33:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.545+0000 7f096de3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f699dd0 0x555a5f6ad340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.545+0000 7f096de3d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f69a6f0 0x555a5f69e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.545+0000 7f096de3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f6a4fb0 0x555a5f69ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.545+0000 7f096de3d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f09641b0660 con 0x555a5f69a6f0 2024-04-26T21:33:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.545+0000 7f096de3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09641b1ad0 con 0x555a5f6a4fb0 2024-04-26T21:33:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.545+0000 7f096de3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09641b0d30 con 0x555a5f699dd0 2024-04-26T21:33:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.545+0000 7f096de3d740 10 client.0 _ll_register_callbacks cb 0x7f095c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:33:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0969000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f6a4fb0 0x555a5f69ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0963e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f699dd0 0x555a5f6ad340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0969000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f6a4fb0 0x555a5f69ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50816/0 (socket says 172.21.15.1:50816) 2024-04-26T21:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0969000640 1 -- 172.21.15.1:0/2490363197 learned_addr learned my addr 172.21.15.1:0/2490363197 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0963e00640 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f699dd0 0x555a5f6ad340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54248/0 (socket says 172.21.15.1:54248) 2024-04-26T21:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0963400640 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f69a6f0 0x555a5f69e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0963e00640 1 -- 172.21.15.1:0/2490363197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f6a4fb0 msgr2=0x555a5f69ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0963e00640 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f6a4fb0 0x555a5f69ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.560 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0963e00640 1 -- 172.21.15.1:0/2490363197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f69a6f0 msgr2=0x555a5f69e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0963e00640 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f69a6f0 0x555a5f69e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0963e00640 1 -- 172.21.15.1:0/2490363197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a5f3dd620 con 0x555a5f699dd0 2024-04-26T21:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f0963e00640 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f699dd0 0x555a5f6ad340 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f095402aa70 tx=0x7f0954004980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de030b83bbc95460 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.546+0000 7f094e000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.547+0000 7f094e000640 1 -- 172.21.15.1:0/2490363197 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954053070 con 0x555a5f699dd0 2024-04-26T21:33:13.561 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.547+0000 7f094e000640 1 -- 172.21.15.1:0/2490363197 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f0954031840 con 0x555a5f699dd0 2024-04-26T21:33:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.547+0000 7f096de3d740 1 -- 172.21.15.1:0/2490363197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f095c004cc0 con 0x555a5f699dd0 2024-04-26T21:33:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.547+0000 7f094e000640 1 -- 172.21.15.1:0/2490363197 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0954035050 con 0x555a5f699dd0 2024-04-26T21:33:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.547+0000 7f096de3d740 10 client.25735 fetch_fsmap learned FSMap version 30 2024-04-26T21:33:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.547+0000 7f096de3d740 1 -- 172.21.15.1:0/2490363197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555a5f5b9cb0 con 0x555a5f699dd0 2024-04-26T21:33:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.548+0000 7f094e000640 1 -- 172.21.15.1:0/2490363197 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f095c004cc0 con 0x555a5f699dd0 2024-04-26T21:33:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.548+0000 7f094e000640 1 -- 172.21.15.1:0/2490363197 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f09540039b0 con 0x555a5f699dd0 2024-04-26T21:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.548+0000 7f096de3d740 10 client.25735 fetch_fsmap finished waiting for FSMap version 30 2024-04-26T21:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.548+0000 7f096de3d740 10 client.25735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.548+0000 7f096de3d740 20 client.25735 populate_metadata read hostname 'smithi001' 2024-04-26T21:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.548+0000 7f096de3d740 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555a5f71af20 0x555a5f71d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.548+0000 7f096de3d740 4 client.25735 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:33:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.548+0000 7f096de3d740 1 -- 172.21.15.1:0/2490363197 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a5f606e00 con 0x555a5f71af20 2024-04-26T21:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.548+0000 7f0963400640 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555a5f71af20 0x555a5f71d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.549+0000 7f0963400640 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555a5f71af20 0x555a5f71d310 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b88cc2d86067e693 server_cookie=cbe87d94e97aedb8 in_seq=0 out_seq=0 2024-04-26T21:33:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.549+0000 7f094e000640 10 client.25735 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.586+0000 7f094e000640 1 -- 172.21.15.1:0/2490363197 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555a5f606e00 con 0x555a5f71af20 2024-04-26T21:33:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.586+0000 7f094e000640 10 client.25735 handle_command_reply: tid=0 2024-04-26T21:33:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.586+0000 7f096de3d740 1 client.25735 shutdown 2024-04-26T21:33:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.586+0000 7f096de3d740 1 -- 172.21.15.1:0/2490363197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f699dd0 msgr2=0x555a5f6ad340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.586+0000 7f096de3d740 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f699dd0 0x555a5f6ad340 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f095402aa70 tx=0x7f0954004980 comp rx=0 tx=0).stop 2024-04-26T21:33:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.587+0000 7f096de3d740 1 -- 172.21.15.1:0/2490363197 shutdown_connections 2024-04-26T21:33:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.587+0000 7f096de3d740 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555a5f71af20 0x555a5f71d310 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.587+0000 7f096de3d740 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555a5f6a4fb0 0x555a5f69ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.587+0000 7f096de3d740 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555a5f69a6f0 0x555a5f69e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.587+0000 7f096de3d740 1 --2- 172.21.15.1:0/2490363197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555a5f699dd0 0x555a5f6ad340 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.587+0000 7f094e000640 0 client.25735 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.587+0000 7f096de3d740 1 -- 172.21.15.1:0/2490363197 >> 172.21.15.1:0/2490363197 conn(0x555a5f6962e0 msgr2=0x555a5f6a26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.587+0000 7f096de3d740 1 -- 172.21.15.1:0/2490363197 shutdown_connections 2024-04-26T21:33:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.587+0000 7f096de3d740 1 -- 172.21.15.1:0/2490363197 wait complete. 2024-04-26T21:33:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.587+0000 7f096de3d740 20 client.25735 trim_cache size 0 max 16384 2024-04-26T21:33:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:33:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:33:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:33:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:33:13.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 -- 172.21.15.1:0/866612655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09480a3d10 msgr2=0x7f09480a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:13.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09480a3d10 0x7f09480a61d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f095c009fd0 tx=0x7f095c009500 comp rx=0 tx=0).stop 2024-04-26T21:33:13.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 -- 172.21.15.1:0/866612655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09641044c0 msgr2=0x7f096419b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:13.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09641044c0 0x7f096419b860 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f095000d950 tx=0x7f095000de10 comp rx=0 tx=0).stop 2024-04-26T21:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 -- 172.21.15.1:0/866612655 shutdown_connections 2024-04-26T21:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09480a3d10 0x7f09480a61d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09641044c0 0x7f096419b860 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f096410cf70 0x7f096419fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 --2- 172.21.15.1:0/866612655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0964103ba0 0x7f096419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 -- 172.21.15.1:0/866612655 >> 172.21.15.1:0/866612655 conn(0x7f0964078980 msgr2=0x7f0964108df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 -- 172.21.15.1:0/866612655 shutdown_connections 2024-04-26T21:33:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:13.588+0000 7f096b000640 1 -- 172.21.15.1:0/866612655 wait complete. 2024-04-26T21:33:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:14 smithi186 ceph-mon[32991]: pgmap v90: 129 pgs: 129 active+clean; 36 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 408 KiB/s wr, 114 op/s 2024-04-26T21:33:14.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 39191 3.19 MB/sec warmup 72 sec latency 30.033 ms 2024-04-26T21:33:14.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:14 smithi001 ceph-mon[28054]: pgmap v90: 129 pgs: 129 active+clean; 36 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 408 KiB/s wr, 114 op/s 2024-04-26T21:33:14.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:14 smithi038 ceph-mon[32724]: pgmap v90: 129 pgs: 129 active+clean; 36 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 408 KiB/s wr, 114 op/s 2024-04-26T21:33:15.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 39497 3.17 MB/sec warmup 73 sec latency 28.517 ms 2024-04-26T21:33:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:16 smithi186 ceph-mon[32991]: pgmap v91: 129 pgs: 129 active+clean; 39 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 689 KiB/s wr, 163 op/s 2024-04-26T21:33:16.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 39882 3.13 MB/sec warmup 74 sec latency 315.955 ms 2024-04-26T21:33:16.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:16 smithi001 ceph-mon[28054]: pgmap v91: 129 pgs: 129 active+clean; 39 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 689 KiB/s wr, 163 op/s 2024-04-26T21:33:16.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:16 smithi038 ceph-mon[32724]: pgmap v91: 129 pgs: 129 active+clean; 39 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 689 KiB/s wr, 163 op/s 2024-04-26T21:33:17.531 INFO:tasks.workunit.client.0.smithi001.stdout: 1 41427 3.18 MB/sec warmup 75 sec latency 34.219 ms 2024-04-26T21:33:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:18 smithi186 ceph-mon[32991]: pgmap v92: 129 pgs: 129 active+clean; 39 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 451 KiB/s wr, 82 op/s 2024-04-26T21:33:18.532 INFO:tasks.workunit.client.0.smithi001.stdout: 1 42430 3.23 MB/sec warmup 76 sec latency 24.955 ms 2024-04-26T21:33:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:18 smithi001 ceph-mon[28054]: pgmap v92: 129 pgs: 129 active+clean; 39 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 451 KiB/s wr, 82 op/s 2024-04-26T21:33:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:18 smithi038 ceph-mon[32724]: pgmap v92: 129 pgs: 129 active+clean; 39 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 451 KiB/s wr, 82 op/s 2024-04-26T21:33:19.532 INFO:tasks.workunit.client.0.smithi001.stdout: 1 43009 3.25 MB/sec warmup 77 sec latency 32.094 ms 2024-04-26T21:33:20.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:20 smithi038 ceph-mon[32724]: pgmap v93: 129 pgs: 129 active+clean; 39 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 451 KiB/s wr, 82 op/s 2024-04-26T21:33:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:20 smithi186 ceph-mon[32991]: pgmap v93: 129 pgs: 129 active+clean; 39 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 451 KiB/s wr, 82 op/s 2024-04-26T21:33:20.532 INFO:tasks.workunit.client.0.smithi001.stdout: 1 43264 3.21 MB/sec warmup 78 sec latency 428.375 ms 2024-04-26T21:33:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:20 smithi001 ceph-mon[28054]: pgmap v93: 129 pgs: 129 active+clean; 39 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 451 KiB/s wr, 82 op/s 2024-04-26T21:33:21.532 INFO:tasks.workunit.client.0.smithi001.stdout: 1 43388 3.18 MB/sec warmup 79 sec latency 1192.745 ms 2024-04-26T21:33:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:22 smithi186 ceph-mon[32991]: pgmap v94: 129 pgs: 129 active+clean; 43 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 816 KiB/s wr, 133 op/s 2024-04-26T21:33:22.532 INFO:tasks.workunit.client.0.smithi001.stdout: 1 43775 3.16 MB/sec warmup 80 sec latency 595.832 ms 2024-04-26T21:33:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:22 smithi001 ceph-mon[28054]: pgmap v94: 129 pgs: 129 active+clean; 43 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 816 KiB/s wr, 133 op/s 2024-04-26T21:33:22.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:22 smithi038 ceph-mon[32724]: pgmap v94: 129 pgs: 129 active+clean; 43 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 816 KiB/s wr, 133 op/s 2024-04-26T21:33:23.532 INFO:tasks.workunit.client.0.smithi001.stdout: 1 45153 3.19 MB/sec warmup 81 sec latency 25.414 ms 2024-04-26T21:33:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:24 smithi186 ceph-mon[32991]: pgmap v95: 129 pgs: 129 active+clean; 43 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 646 KiB/s wr, 99 op/s 2024-04-26T21:33:24.532 INFO:tasks.workunit.client.0.smithi001.stdout: 1 46380 3.28 MB/sec warmup 82 sec latency 20.709 ms 2024-04-26T21:33:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:24 smithi001 ceph-mon[28054]: pgmap v95: 129 pgs: 129 active+clean; 43 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 646 KiB/s wr, 99 op/s 2024-04-26T21:33:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:24 smithi038 ceph-mon[32724]: pgmap v95: 129 pgs: 129 active+clean; 43 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 646 KiB/s wr, 99 op/s 2024-04-26T21:33:25.533 INFO:tasks.workunit.client.0.smithi001.stdout: 1 46743 3.25 MB/sec warmup 83 sec latency 22.880 ms 2024-04-26T21:33:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:26 smithi186 ceph-mon[32991]: pgmap v96: 129 pgs: 129 active+clean; 45 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 841 KiB/s wr, 135 op/s 2024-04-26T21:33:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:26 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:33:26.533 INFO:tasks.workunit.client.0.smithi001.stdout: 1 46870 3.21 MB/sec warmup 84 sec latency 676.374 ms 2024-04-26T21:33:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:26 smithi001 ceph-mon[28054]: pgmap v96: 129 pgs: 129 active+clean; 45 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 841 KiB/s wr, 135 op/s 2024-04-26T21:33:26.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:26 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:33:26.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:26 smithi038 ceph-mon[32724]: pgmap v96: 129 pgs: 129 active+clean; 45 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 841 KiB/s wr, 135 op/s 2024-04-26T21:33:26.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:26 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:33:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.738+0000 7f4ffb200640 1 -- 172.21.15.1:0/4211143222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ffc104990 msgr2=0x7f4ffc104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.738+0000 7f4ffb200640 1 --2- 172.21.15.1:0/4211143222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ffc104990 0x7f4ffc104e10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009890 tx=0x7f4fe802a370 comp rx=0 tx=0).stop 2024-04-26T21:33:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.738+0000 7f4ffb200640 1 -- 172.21.15.1:0/4211143222 shutdown_connections 2024-04-26T21:33:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.738+0000 7f4ffb200640 1 --2- 172.21.15.1:0/4211143222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ffc104990 0x7f4ffc104e10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.738+0000 7f4ffb200640 1 --2- 172.21.15.1:0/4211143222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ffc105350 0x7f4ffc111860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.738+0000 7f4ffb200640 1 --2- 172.21.15.1:0/4211143222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc10a9a0 0x7f4ffc10ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.738+0000 7f4ffb200640 1 -- 172.21.15.1:0/4211143222 >> 172.21.15.1:0/4211143222 conn(0x7f4ffc100440 msgr2=0x7f4ffc102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.738+0000 7f4ffb200640 1 -- 172.21.15.1:0/4211143222 shutdown_connections 2024-04-26T21:33:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.738+0000 7f4ffb200640 1 -- 172.21.15.1:0/4211143222 wait complete. 2024-04-26T21:33:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.739+0000 7f4ffb200640 1 Processor -- start 2024-04-26T21:33:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.739+0000 7f4ffb200640 1 -- start start 2024-04-26T21:33:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ffb200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ffc104990 0x7f4ffc19b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ffb200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105350 0x7f4ffc19b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ffb200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ffc10a9a0 0x7f4ffc19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ffb200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc114c30 con 0x7f4ffc104990 2024-04-26T21:33:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ffb200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc114ab0 con 0x7f4ffc105350 2024-04-26T21:33:26.742 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ffb200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc114db0 con 0x7f4ffc10a9a0 2024-04-26T21:33:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ff9e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ffc104990 0x7f4ffc19b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ffa800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ffc10a9a0 0x7f4ffc19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ff9e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ffc104990 0x7f4ffc19b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33804/0 (socket says 172.21.15.1:33804) 2024-04-26T21:33:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ff9e00640 1 -- 172.21.15.1:0/1149586251 learned_addr learned my addr 172.21.15.1:0/1149586251 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.740+0000 7f4ff9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105350 0x7f4ffc19b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.743 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ffa800640 1 -- 172.21.15.1:0/1149586251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105350 msgr2=0x7f4ffc19b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ffa800640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105350 0x7f4ffc19b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ffa800640 1 -- 172.21.15.1:0/1149586251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ffc104990 msgr2=0x7f4ffc19b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ffa800640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ffc104990 0x7f4ffc19b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ffa800640 1 -- 172.21.15.1:0/1149586251 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ffc1a01d0 con 0x7f4ffc10a9a0 2024-04-26T21:33:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ffa800640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ffc10a9a0 0x7f4ffc19faa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009fd0 tx=0x7f4ff400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ff2a00640 1 -- 172.21.15.1:0/1149586251 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff400ce20 con 0x7f4ffc10a9a0 2024-04-26T21:33:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ff2a00640 1 -- 172.21.15.1:0/1149586251 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ff4005020 con 0x7f4ffc10a9a0 2024-04-26T21:33:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ff2a00640 1 -- 172.21.15.1:0/1149586251 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4010690 con 0x7f4ffc10a9a0 2024-04-26T21:33:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ffb200640 1 -- 172.21.15.1:0/1149586251 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ffc1a04c0 con 0x7f4ffc10a9a0 2024-04-26T21:33:26.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.741+0000 7f4ffb200640 1 -- 172.21.15.1:0/1149586251 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ffc1a7d50 con 0x7f4ffc10a9a0 2024-04-26T21:33:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f4ff2a00640 1 -- 172.21.15.1:0/1149586251 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f4ff40054d0 con 0x7f4ffc10a9a0 2024-04-26T21:33:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f500379b740 1 Processor -- start 2024-04-26T21:33:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f500379b740 1 -- start start 2024-04-26T21:33:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f500379b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e71794cfe0 0x55e71794d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f500379b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 0x55e717957c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f500379b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7179581c0 0x55e71795a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f500379b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e7178b2380 con 0x55e71794d900 2024-04-26T21:33:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f500379b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e71784ec30 con 0x55e71794cfe0 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f500379b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e7178ae340 con 0x55e7179581c0 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f4ff9e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 0x55e717957c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f4ff9400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7179581c0 0x55e71795a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff9400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7179581c0 0x55e71795a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:47698/0 (socket says 172.21.15.1:47698) 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff9400640 1 -- 172.21.15.1:0/3842060556 learned_addr learned my addr 172.21.15.1:0/3842060556 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.745+0000 7f4ff9e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 0x55e717957c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33820/0 (socket says 172.21.15.1:33820) 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff9400640 1 --2- 172.21.15.1:0/3842060556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e71794cfe0 0x55e71794d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff9e00640 1 -- 172.21.15.1:0/3842060556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7179581c0 msgr2=0x55e71795a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff9e00640 1 --2- 172.21.15.1:0/3842060556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7179581c0 0x55e71795a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff9e00640 1 -- 172.21.15.1:0/3842060556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e71794cfe0 msgr2=0x55e71794d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff9e00640 1 --2- 172.21.15.1:0/3842060556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e71794cfe0 0x55e71794d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff9e00640 1 -- 172.21.15.1:0/3842060556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e717870940 con 0x55e71794d900 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff9400640 1 --2- 172.21.15.1:0/3842060556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7179581c0 0x55e71795a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff9e00640 1 --2- 172.21.15.1:0/3842060556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 0x55e717957c80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009c20 tx=0x7f4fec011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc370bc29328e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.746+0000 7f4ff2a00640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4fe00a3d00 0x7f4fe00a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.747+0000 7f4fe7000640 1 -- 172.21.15.1:0/3842060556 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec004230 con 0x55e71794d900 2024-04-26T21:33:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.747+0000 7f4fe7000640 1 -- 172.21.15.1:0/3842060556 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fec0043d0 con 0x55e71794d900 2024-04-26T21:33:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.747+0000 7f4fe7000640 1 -- 172.21.15.1:0/3842060556 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec0196f0 con 0x55e71794d900 2024-04-26T21:33:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.747+0000 7f4ff2a00640 1 -- 172.21.15.1:0/1149586251 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f4ff40d1340 con 0x7f4ffc10a9a0 2024-04-26T21:33:26.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.747+0000 7f4ff9400640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4fe00a3d00 0x7f4fe00a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f500379b740 1 -- 172.21.15.1:0/3842060556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 msgr2=0x55e717957c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f500379b740 1 --2- 172.21.15.1:0/3842060556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 0x55e717957c80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009c20 tx=0x7f4fec011810 comp rx=0 tx=0).stop 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f4ff9400640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4fe00a3d00 0x7f4fe00a61c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x55e71794a910 tx=0x7f4fe8030340 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f500379b740 1 -- 172.21.15.1:0/3842060556 shutdown_connections 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f500379b740 1 --2- 172.21.15.1:0/3842060556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7179581c0 0x55e71795a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f500379b740 1 --2- 172.21.15.1:0/3842060556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 0x55e717957c80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f500379b740 1 --2- 172.21.15.1:0/3842060556 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e71794cfe0 0x55e71794d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f500379b740 1 -- 172.21.15.1:0/3842060556 >> 172.21.15.1:0/3842060556 conn(0x55e7179494f0 msgr2=0x55e7179498d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f500379b740 1 -- 172.21.15.1:0/3842060556 shutdown_connections 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f500379b740 1 -- 172.21.15.1:0/3842060556 wait complete. 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.748+0000 7f500379b740 1 -- start start 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f500379b740 1 Processor -- start 2024-04-26T21:33:26.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f500379b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e717957c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f500379b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 0x55e717955000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f500379b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717955740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f500379b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e71795adf0 con 0x55e71794cfe0 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f500379b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e717839c60 con 0x55e7179581c0 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f500379b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e71795af70 con 0x55e71794d900 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f500379b740 10 client.0 _ll_register_callbacks cb 0x55e717865e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f4ff9e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e717957c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f4ff9e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e717957c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33836/0 (socket says 172.21.15.1:33836) 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f4ff9e00640 1 -- 172.21.15.1:0/1419512935 learned_addr learned my addr 172.21.15.1:0/1419512935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f4ffa800640 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717955740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.749+0000 7f4ff9400640 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 0x55e717955000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.750+0000 7f4ff9e00640 1 -- 172.21.15.1:0/1419512935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 msgr2=0x55e717955000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.750+0000 7f4ff9e00640 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 0x55e717955000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.750+0000 7f4ff9e00640 1 -- 172.21.15.1:0/1419512935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 msgr2=0x55e717955740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.750+0000 7f4ff9e00640 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717955740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.750+0000 7f4ff9e00640 1 -- 172.21.15.1:0/1419512935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e717848510 con 0x55e71794cfe0 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.750+0000 7f4ffa800640 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717955740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.750+0000 7f4ff9e00640 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e717957c70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4fec003e70 tx=0x7f4fec003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89055b4b59461cc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.750+0000 7f4fdb400640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.750+0000 7f4fdb400640 1 -- 172.21.15.1:0/1419512935 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec028020 con 0x55e71794cfe0 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.751+0000 7f4fdb400640 1 -- 172.21.15.1:0/1419512935 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f4fec005040 con 0x55e71794cfe0 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.751+0000 7f500379b740 1 -- 172.21.15.1:0/1419512935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fec0043d0 con 0x55e71794cfe0 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.751+0000 7f4fdb400640 1 -- 172.21.15.1:0/1419512935 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4fec033050 con 0x55e71794cfe0 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.751+0000 7f500379b740 10 client.15612 fetch_fsmap learned FSMap version 31 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.751+0000 7f500379b740 1 -- 172.21.15.1:0/1419512935 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e7175c4210 con 0x55e71794cfe0 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.751+0000 7f4fdb400640 1 -- 172.21.15.1:0/1419512935 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fec0043d0 con 0x55e71794cfe0 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.751+0000 7f4fdb400640 1 -- 172.21.15.1:0/1419512935 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 31) ==== 8358+0+0 (secure 0 0 0) 0x7f4fec018070 con 0x55e71794cfe0 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.751+0000 7f500379b740 10 client.15612 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T21:33:26.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.751+0000 7f500379b740 10 client.15612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:33:26.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.752+0000 7f500379b740 20 client.15612 populate_metadata read hostname 'smithi001' 2024-04-26T21:33:26.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.752+0000 7f500379b740 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e717967d50 0x55e71796a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.752+0000 7f500379b740 4 client.15612 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:33:26.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.752+0000 7f500379b740 1 -- 172.21.15.1:0/1419512935 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7178bff90 con 0x55e717967d50 2024-04-26T21:33:26.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.752+0000 7f4ff9400640 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e717967d50 0x55e71796a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.753+0000 7f4ff9400640 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e717967d50 0x55e71796a140 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc96da3275465cdb server_cookie=af9ee77714638ebb in_seq=0 out_seq=0 2024-04-26T21:33:26.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.753+0000 7f4fdb400640 10 client.15612 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:26.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f4fdb400640 1 -- 172.21.15.1:0/1419512935 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e7178bff90 con 0x55e717967d50 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f4fdb400640 10 client.15612 handle_command_reply: tid=0 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 client.15612 shutdown 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 -- 172.21.15.1:0/1419512935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 msgr2=0x55e717957c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e717957c70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4fec003e70 tx=0x7f4fec003820 comp rx=0 tx=0).stop 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 -- 172.21.15.1:0/1419512935 shutdown_connections 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e717967d50 0x55e71796a140 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 0x55e717955000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717955740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f4fdb400640 0 client.15612 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 --2- 172.21.15.1:0/1419512935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e717957c70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 -- 172.21.15.1:0/1419512935 >> 172.21.15.1:0/1419512935 conn(0x55e7179494f0 msgr2=0x55e71794f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 -- 172.21.15.1:0/1419512935 shutdown_connections 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 1 -- 172.21.15.1:0/1419512935 wait complete. 2024-04-26T21:33:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.754+0000 7f500379b740 20 client.15612 trim_cache size 0 max 16384 2024-04-26T21:33:26.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.771+0000 7f500379b740 1 Processor -- start 2024-04-26T21:33:26.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.771+0000 7f500379b740 1 -- start start 2024-04-26T21:33:26.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.771+0000 7f500379b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e71796f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.771+0000 7f500379b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 0x55e71796fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.771+0000 7f500379b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717963d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.771+0000 7f500379b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b48e0 con 0x55e71794cfe0 2024-04-26T21:33:26.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.771+0000 7f500379b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b2120 con 0x55e7179581c0 2024-04-26T21:33:26.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.771+0000 7f500379b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1aff70 con 0x55e71794d900 2024-04-26T21:33:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ff9e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e71796f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ff9e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e71796f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33846/0 (socket says 172.21.15.1:33846) 2024-04-26T21:33:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ff9e00640 1 -- 172.21.15.1:0/3839320815 learned_addr learned my addr 172.21.15.1:0/3839320815 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ff9400640 1 --2- 172.21.15.1:0/3839320815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 0x55e71796fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ffa800640 1 --2- 172.21.15.1:0/3839320815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717963d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ff9400640 1 -- 172.21.15.1:0/3839320815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 msgr2=0x55e717963d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ff9400640 1 --2- 172.21.15.1:0/3839320815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717963d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ff9400640 1 -- 172.21.15.1:0/3839320815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 msgr2=0x55e71796f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ff9400640 1 --2- 172.21.15.1:0/3839320815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e71796f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ff9400640 1 -- 172.21.15.1:0/3839320815 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e717690620 con 0x55e71794d900 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.772+0000 7f4ff9e00640 1 --2- 172.21.15.1:0/3839320815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e71796f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.773+0000 7f4ff9400640 1 --2- 172.21.15.1:0/3839320815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 0x55e71796fe10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8000c00 tx=0x7f4fe8005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b478e312073e373 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.773+0000 7f4fe6600640 1 -- 172.21.15.1:0/3839320815 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8037040 con 0x55e71794d900 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.773+0000 7f4fe6600640 1 -- 172.21.15.1:0/3839320815 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fe8047040 con 0x55e71794d900 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.773+0000 7f4fe6600640 1 -- 172.21.15.1:0/3839320815 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8003a50 con 0x55e71794d900 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.773+0000 7f500379b740 1 -- 172.21.15.1:0/3839320815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 msgr2=0x55e71796fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.773+0000 7f500379b740 1 --2- 172.21.15.1:0/3839320815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 0x55e71796fe10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8000c00 tx=0x7f4fe8005f90 comp rx=0 tx=0).stop 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.774+0000 7f500379b740 1 -- 172.21.15.1:0/3839320815 shutdown_connections 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.774+0000 7f500379b740 1 --2- 172.21.15.1:0/3839320815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794d900 0x55e71796fe10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.774+0000 7f500379b740 1 --2- 172.21.15.1:0/3839320815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717963d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.774+0000 7f500379b740 1 --2- 172.21.15.1:0/3839320815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794cfe0 0x55e71796f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.774+0000 7f500379b740 1 -- 172.21.15.1:0/3839320815 >> 172.21.15.1:0/3839320815 conn(0x55e7179494f0 msgr2=0x55e717950300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.774+0000 7f500379b740 1 -- 172.21.15.1:0/3839320815 shutdown_connections 2024-04-26T21:33:26.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.774+0000 7f500379b740 1 -- 172.21.15.1:0/3839320815 wait complete. 2024-04-26T21:33:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.774+0000 7f500379b740 1 -- start start 2024-04-26T21:33:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.774+0000 7f500379b740 1 Processor -- start 2024-04-26T21:33:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f500379b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794cfe0 0x55e717960770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f500379b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 0x55e717951b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f500379b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717952270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f500379b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b05e0 con 0x55e71794d900 2024-04-26T21:33:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f500379b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b1a50 con 0x55e7179581c0 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f500379b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4ffc1b0cb0 con 0x55e71794cfe0 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f500379b740 10 client.0 _ll_register_callbacks cb 0x7f4fec005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f4ff9e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794cfe0 0x55e717960770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f4ff9400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 0x55e717951b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f4ff9e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794cfe0 0x55e717960770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:47720/0 (socket says 172.21.15.1:47720) 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f4ff9e00640 1 -- 172.21.15.1:0/4046825828 learned_addr learned my addr 172.21.15.1:0/4046825828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f4ffa800640 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717952270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f4ff9e00640 1 -- 172.21.15.1:0/4046825828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 msgr2=0x55e717952270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f4ff9e00640 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717952270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f4ff9e00640 1 -- 172.21.15.1:0/4046825828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 msgr2=0x55e717951b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f4ff9e00640 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 0x55e717951b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.775+0000 7f4ff9e00640 1 -- 172.21.15.1:0/4046825828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e71786ccb0 con 0x55e71794cfe0 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.776+0000 7f4ff9e00640 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794cfe0 0x55e717960770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4fec004a80 tx=0x7f4fec009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fc1bb2610c59154 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:26.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.776+0000 7f4fdb400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.776+0000 7f4fdb400640 1 -- 172.21.15.1:0/4046825828 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec028040 con 0x55e71794cfe0 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.776+0000 7f4fdb400640 1 -- 172.21.15.1:0/4046825828 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f4fec003890 con 0x55e71794cfe0 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.776+0000 7f500379b740 1 -- 172.21.15.1:0/4046825828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fe8047040 con 0x55e71794cfe0 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.776+0000 7f4fdb400640 1 -- 172.21.15.1:0/4046825828 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4fec03e050 con 0x55e71794cfe0 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.776+0000 7f500379b740 10 client.25753 fetch_fsmap learned FSMap version 31 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.776+0000 7f500379b740 1 -- 172.21.15.1:0/4046825828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e7179ce0a0 con 0x55e71794cfe0 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.777+0000 7f4fdb400640 1 -- 172.21.15.1:0/4046825828 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fe8047040 con 0x55e71794cfe0 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.777+0000 7f4fdb400640 1 -- 172.21.15.1:0/4046825828 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 31) ==== 8358+0+0 (secure 0 0 0) 0x7f4fec018040 con 0x55e71794cfe0 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.777+0000 7f500379b740 10 client.25753 fetch_fsmap finished waiting for FSMap version 31 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.777+0000 7f500379b740 10 client.25753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.777+0000 7f500379b740 20 client.25753 populate_metadata read hostname 'smithi001' 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.777+0000 7f500379b740 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e7179ce700 0x55e7179d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.777+0000 7f500379b740 4 client.25753 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.777+0000 7f500379b740 1 -- 172.21.15.1:0/4046825828 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e7178b9e00 con 0x55e7179ce700 2024-04-26T21:33:26.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.777+0000 7f4ff9400640 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e7179ce700 0x55e7179d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:26.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.778+0000 7f4ff9400640 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e7179ce700 0x55e7179d0af0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db1590f35d6bb08 server_cookie=bed90b30bb7a201e in_seq=0 out_seq=0 2024-04-26T21:33:26.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.778+0000 7f4fdb400640 10 client.25753 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f4fdb400640 1 -- 172.21.15.1:0/4046825828 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e7178b9e00 con 0x55e7179ce700 2024-04-26T21:33:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f4fdb400640 10 client.25753 handle_command_reply: tid=0 2024-04-26T21:33:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f500379b740 1 client.25753 shutdown 2024-04-26T21:33:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f500379b740 1 -- 172.21.15.1:0/4046825828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794cfe0 msgr2=0x55e717960770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f500379b740 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794cfe0 0x55e717960770 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4fec004a80 tx=0x7f4fec009b20 comp rx=0 tx=0).stop 2024-04-26T21:33:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f500379b740 1 -- 172.21.15.1:0/4046825828 shutdown_connections 2024-04-26T21:33:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f500379b740 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e7179ce700 0x55e7179d0af0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f500379b740 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e7179581c0 0x55e717952270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f500379b740 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e71794d900 0x55e717951b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f500379b740 1 --2- 172.21.15.1:0/4046825828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e71794cfe0 0x55e717960770 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f500379b740 1 -- 172.21.15.1:0/4046825828 >> 172.21.15.1:0/4046825828 conn(0x55e7179494f0 msgr2=0x55e717953870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.840+0000 7f4fdb400640 0 client.25753 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.841+0000 7f500379b740 1 -- 172.21.15.1:0/4046825828 shutdown_connections 2024-04-26T21:33:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.841+0000 7f500379b740 1 -- 172.21.15.1:0/4046825828 wait complete. 2024-04-26T21:33:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.841+0000 7f500379b740 20 client.25753 trim_cache size 0 max 16384 2024-04-26T21:33:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:33:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:33:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:33:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.842+0000 7f4ffb200640 1 -- 172.21.15.1:0/1149586251 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4fe00a3d00 msgr2=0x7f4fe00a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.842+0000 7f4ffb200640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4fe00a3d00 0x7f4fe00a61c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x55e71794a910 tx=0x7f4fe8030340 comp rx=0 tx=0).stop 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.842+0000 7f4ffb200640 1 -- 172.21.15.1:0/1149586251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ffc10a9a0 msgr2=0x7f4ffc19faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.842+0000 7f4ffb200640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ffc10a9a0 0x7f4ffc19faa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009fd0 tx=0x7f4ff400ef10 comp rx=0 tx=0).stop 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.843+0000 7f4ffb200640 1 -- 172.21.15.1:0/1149586251 shutdown_connections 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.843+0000 7f4ffb200640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4fe00a3d00 0x7f4fe00a61c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.843+0000 7f4ffb200640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ffc10a9a0 0x7f4ffc19faa0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.843+0000 7f4ffb200640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ffc105350 0x7f4ffc19b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.843+0000 7f4ffb200640 1 --2- 172.21.15.1:0/1149586251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ffc104990 0x7f4ffc19b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.843+0000 7f4ffb200640 1 -- 172.21.15.1:0/1149586251 >> 172.21.15.1:0/1149586251 conn(0x7f4ffc100440 msgr2=0x7f4ffc100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.843+0000 7f4ffb200640 1 -- 172.21.15.1:0/1149586251 shutdown_connections 2024-04-26T21:33:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:26.843+0000 7f4ffb200640 1 -- 172.21.15.1:0/1149586251 wait complete. 2024-04-26T21:33:27.533 INFO:tasks.workunit.client.0.smithi001.stdout: 1 47582 3.23 MB/sec warmup 85 sec latency 425.699 ms 2024-04-26T21:33:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:28 smithi186 ceph-mon[32991]: pgmap v97: 129 pgs: 129 active+clean; 45 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 560 KiB/s wr, 86 op/s 2024-04-26T21:33:28.533 INFO:tasks.workunit.client.0.smithi001.stdout: 1 48914 3.26 MB/sec warmup 86 sec latency 25.379 ms 2024-04-26T21:33:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:28 smithi001 ceph-mon[28054]: pgmap v97: 129 pgs: 129 active+clean; 45 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 560 KiB/s wr, 86 op/s 2024-04-26T21:33:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:28 smithi038 ceph-mon[32724]: pgmap v97: 129 pgs: 129 active+clean; 45 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 560 KiB/s wr, 86 op/s 2024-04-26T21:33:29.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:29 smithi186 ceph-mon[32991]: pgmap v98: 129 pgs: 129 active+clean; 45 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 560 KiB/s wr, 86 op/s 2024-04-26T21:33:29.533 INFO:tasks.workunit.client.0.smithi001.stdout: 1 49974 3.31 MB/sec warmup 87 sec latency 21.039 ms 2024-04-26T21:33:29.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:29 smithi001 ceph-mon[28054]: pgmap v98: 129 pgs: 129 active+clean; 45 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 560 KiB/s wr, 86 op/s 2024-04-26T21:33:29.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:29 smithi038 ceph-mon[32724]: pgmap v98: 129 pgs: 129 active+clean; 45 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 560 KiB/s wr, 86 op/s 2024-04-26T21:33:30.533 INFO:tasks.workunit.client.0.smithi001.stdout: 1 50410 3.28 MB/sec warmup 88 sec latency 19.656 ms 2024-04-26T21:33:31.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:31 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:33:31.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:31 smithi186 ceph-mon[32991]: pgmap v99: 129 pgs: 129 active+clean; 47 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 887 KiB/s wr, 150 op/s 2024-04-26T21:33:31.533 INFO:tasks.workunit.client.0.smithi001.stdout: 1 51891 3.31 MB/sec warmup 89 sec latency 71.841 ms 2024-04-26T21:33:31.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:31 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:33:31.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:31 smithi001 ceph-mon[28054]: pgmap v99: 129 pgs: 129 active+clean; 47 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 887 KiB/s wr, 150 op/s 2024-04-26T21:33:31.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:31 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:33:31.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:31 smithi038 ceph-mon[32724]: pgmap v99: 129 pgs: 129 active+clean; 47 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 887 KiB/s wr, 150 op/s 2024-04-26T21:33:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:32 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:33:32.533 INFO:tasks.workunit.client.0.smithi001.stdout: 1 53281 3.40 MB/sec warmup 90 sec latency 24.144 ms 2024-04-26T21:33:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:32 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:33:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:32 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:33:33.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:33 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:33:33.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:33 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:33:33.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:33 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:33:33.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:33 smithi186 ceph-mon[32991]: pgmap v100: 129 pgs: 129 active+clean; 47 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 522 KiB/s wr, 99 op/s 2024-04-26T21:33:33.533 INFO:tasks.workunit.client.0.smithi001.stdout: 1 53680 3.38 MB/sec warmup 91 sec latency 20.428 ms 2024-04-26T21:33:33.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:33 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:33:33.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:33 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:33:33.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:33 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:33:33.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:33 smithi001 ceph-mon[28054]: pgmap v100: 129 pgs: 129 active+clean; 47 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 522 KiB/s wr, 99 op/s 2024-04-26T21:33:33.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:33 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:33:33.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:33 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:33:33.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:33 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:33:33.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:33 smithi038 ceph-mon[32724]: pgmap v100: 129 pgs: 129 active+clean; 47 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 522 KiB/s wr, 99 op/s 2024-04-26T21:33:34.534 INFO:tasks.workunit.client.0.smithi001.stdout: 1 53903 3.35 MB/sec warmup 92 sec latency 582.360 ms 2024-04-26T21:33:35.534 INFO:tasks.workunit.client.0.smithi001.stdout: 1 53903 3.31 MB/sec warmup 93 sec latency 1582.508 ms 2024-04-26T21:33:36.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:35 smithi001 ceph-mon[28054]: pgmap v101: 129 pgs: 129 active+clean; 49 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1010 KiB/s wr, 155 op/s 2024-04-26T21:33:36.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:35 smithi038 ceph-mon[32724]: pgmap v101: 129 pgs: 129 active+clean; 49 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1010 KiB/s wr, 155 op/s 2024-04-26T21:33:36.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:35 smithi186 ceph-mon[32991]: pgmap v101: 129 pgs: 129 active+clean; 49 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 1010 KiB/s wr, 155 op/s 2024-04-26T21:33:36.534 INFO:tasks.workunit.client.0.smithi001.stdout: 1 54063 3.28 MB/sec warmup 94 sec latency 2348.137 ms 2024-04-26T21:33:37.534 INFO:tasks.workunit.client.0.smithi001.stdout: 1 55596 3.31 MB/sec warmup 95 sec latency 23.568 ms 2024-04-26T21:33:38.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:37 smithi001 ceph-mon[28054]: pgmap v102: 129 pgs: 129 active+clean; 49 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 815 KiB/s wr, 119 op/s 2024-04-26T21:33:38.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:37 smithi038 ceph-mon[32724]: pgmap v102: 129 pgs: 129 active+clean; 49 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 815 KiB/s wr, 119 op/s 2024-04-26T21:33:38.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:37 smithi186 ceph-mon[32991]: pgmap v102: 129 pgs: 129 active+clean; 49 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 815 KiB/s wr, 119 op/s 2024-04-26T21:33:38.534 INFO:tasks.workunit.client.0.smithi001.stdout: 1 56599 3.36 MB/sec warmup 96 sec latency 25.597 ms 2024-04-26T21:33:39.534 INFO:tasks.workunit.client.0.smithi001.stdout: 1 57179 3.37 MB/sec warmup 97 sec latency 24.173 ms 2024-04-26T21:33:40.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.001+0000 7fcefa200640 1 -- 172.21.15.1:0/3106341528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcef4108920 msgr2=0x7fcef4108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.001+0000 7fcefa200640 1 --2- 172.21.15.1:0/3106341528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcef4108920 0x7fcef4108d00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009900 tx=0x7fcee402a410 comp rx=0 tx=0).stop 2024-04-26T21:33:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.002+0000 7fcefa200640 1 -- 172.21.15.1:0/3106341528 shutdown_connections 2024-04-26T21:33:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.002+0000 7fcefa200640 1 --2- 172.21.15.1:0/3106341528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcef41032d0 0x7fcef410f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.002+0000 7fcefa200640 1 --2- 172.21.15.1:0/3106341528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcef4102910 0x7fcef4102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.002+0000 7fcefa200640 1 --2- 172.21.15.1:0/3106341528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcef4108920 0x7fcef4108d00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.002+0000 7fcefa200640 1 -- 172.21.15.1:0/3106341528 >> 172.21.15.1:0/3106341528 conn(0x7fcef40fe3c0 msgr2=0x7fcef41007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.002+0000 7fcefa200640 1 -- 172.21.15.1:0/3106341528 shutdown_connections 2024-04-26T21:33:40.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.002+0000 7fcefa200640 1 -- 172.21.15.1:0/3106341528 wait complete. 2024-04-26T21:33:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.003+0000 7fcefa200640 1 Processor -- start 2024-04-26T21:33:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.003+0000 7fcefa200640 1 -- start start 2024-04-26T21:33:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.003+0000 7fcefa200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcef4102910 0x7fcef4199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.003+0000 7fcefa200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcef41032d0 0x7fcef4199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.003+0000 7fcefa200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcef4108920 0x7fcef419da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.003+0000 7fcefa200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcef4112c00 con 0x7fcef4108920 2024-04-26T21:33:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.003+0000 7fcefa200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcef4112a80 con 0x7fcef4102910 2024-04-26T21:33:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.003+0000 7fcefa200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcef4112d80 con 0x7fcef41032d0 2024-04-26T21:33:40.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fcef3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcef4102910 0x7fcef4199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fcef3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcef4102910 0x7fcef4199140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51142/0 (socket says 172.21.15.1:51142) 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fcef3400640 1 -- 172.21.15.1:0/3914893997 learned_addr learned my addr 172.21.15.1:0/3914893997 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fcef3e00640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcef4108920 0x7fcef419da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fceebe00640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcef41032d0 0x7fcef4199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fcef3400640 1 -- 172.21.15.1:0/3914893997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcef41032d0 msgr2=0x7fcef4199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fcef3400640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcef41032d0 0x7fcef4199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fcef3400640 1 -- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcef4108920 msgr2=0x7fcef419da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fcef3400640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcef4108920 0x7fcef419da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fcef3400640 1 -- 172.21.15.1:0/3914893997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef419e130 con 0x7fcef4102910 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.004+0000 7fcef3e00640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcef4108920 0x7fcef419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:40.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.005+0000 7fcef3400640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcef4102910 0x7fcef4199140 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcee402aa00 tx=0x7fcee4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:40.007 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:39 smithi186 ceph-mon[32991]: pgmap v103: 129 pgs: 129 active+clean; 49 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 815 KiB/s wr, 119 op/s 2024-04-26T21:33:40.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.005+0000 7fcef0c00640 1 -- 172.21.15.1:0/3914893997 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee403f070 con 0x7fcef4102910 2024-04-26T21:33:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.005+0000 7fcef0c00640 1 -- 172.21.15.1:0/3914893997 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcee403bb20 con 0x7fcef4102910 2024-04-26T21:33:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.005+0000 7fcefa200640 1 -- 172.21.15.1:0/3914893997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef419e3c0 con 0x7fcef4102910 2024-04-26T21:33:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.005+0000 7fcef0c00640 1 -- 172.21.15.1:0/3914893997 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee4035030 con 0x7fcef4102910 2024-04-26T21:33:40.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.005+0000 7fcefa200640 1 -- 172.21.15.1:0/3914893997 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef41a5cb0 con 0x7fcef4102910 2024-04-26T21:33:40.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.009+0000 7fcef0c00640 1 -- 172.21.15.1:0/3914893997 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fcee4042070 con 0x7fcef4102910 2024-04-26T21:33:40.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.009+0000 7fcefd004740 1 Processor -- start 2024-04-26T21:33:40.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.009+0000 7fcefd004740 1 -- start start 2024-04-26T21:33:40.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcefd004740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcefd004740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb015e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcefd004740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 0x55aaeb0187b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcefd004740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55aaeaf6c380 con 0x55aaeb00bb00 2024-04-26T21:33:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcefd004740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55aaeaf08c30 con 0x55aaeb00b1e0 2024-04-26T21:33:40.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcefd004740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aaeaf68340 con 0x55aaeb0163c0 2024-04-26T21:33:40.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51144/0 (socket says 172.21.15.1:51144) 2024-04-26T21:33:40.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcef3e00640 1 -- 172.21.15.1:0/2135227812 learned_addr learned my addr 172.21.15.1:0/2135227812 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:40.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fceebe00640 1 --2- 172.21.15.1:0/2135227812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 0x55aaeb0187b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcef3400640 1 --2- 172.21.15.1:0/2135227812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb015e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.010+0000 7fcef0c00640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fced80a3d10 0x7fced80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.011+0000 7fcef0c00640 1 -- 172.21.15.1:0/3914893997 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fcee40f12b0 con 0x7fcef4102910 2024-04-26T21:33:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.011+0000 7fcef3e00640 1 -- 172.21.15.1:0/2135227812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 msgr2=0x55aaeb0187b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.011+0000 7fcef3e00640 1 --2- 172.21.15.1:0/2135227812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 0x55aaeb0187b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.011+0000 7fcef3e00640 1 -- 172.21.15.1:0/2135227812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 msgr2=0x55aaeb015e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.011+0000 7fcef3e00640 1 --2- 172.21.15.1:0/2135227812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb015e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.011+0000 7fcef3e00640 1 -- 172.21.15.1:0/2135227812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aaeaf2a940 con 0x55aaeb00b1e0 2024-04-26T21:33:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.011+0000 7fceebe00640 1 --2- 172.21.15.1:0/2135227812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 0x55aaeb0187b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.011+0000 7fcef3e00640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fced80a3d10 0x7fced80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.011+0000 7fcef3e00640 1 --2- 172.21.15.1:0/2135227812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00b5c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fceec00f850 tx=0x7fceec00fd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55af133e5b75ea0e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.012+0000 7fcee3e00640 1 -- 172.21.15.1:0/2135227812 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec01f050 con 0x55aaeb00b1e0 2024-04-26T21:33:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.012+0000 7fcee3e00640 1 -- 172.21.15.1:0/2135227812 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceec02e440 con 0x55aaeb00b1e0 2024-04-26T21:33:40.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.012+0000 7fcee3e00640 1 -- 172.21.15.1:0/2135227812 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec022580 con 0x55aaeb00b1e0 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.013+0000 7fcefd004740 1 -- 172.21.15.1:0/2135227812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 msgr2=0x55aaeb00b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.013+0000 7fcefd004740 1 --2- 172.21.15.1:0/2135227812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00b5c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fceec00f850 tx=0x7fceec00fd10 comp rx=0 tx=0).stop 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.013+0000 7fcef3e00640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fced80a3d10 0x7fced80a61d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fceec02b000 tx=0x7fceec02c000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.013+0000 7fcefd004740 1 -- 172.21.15.1:0/2135227812 shutdown_connections 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.013+0000 7fcefd004740 1 --2- 172.21.15.1:0/2135227812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 0x55aaeb0187b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.013+0000 7fcefd004740 1 --2- 172.21.15.1:0/2135227812 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb015e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.013+0000 7fcefd004740 1 --2- 172.21.15.1:0/2135227812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00b5c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.013+0000 7fcefd004740 1 -- 172.21.15.1:0/2135227812 >> 172.21.15.1:0/2135227812 conn(0x55aaeb0076f0 msgr2=0x55aaeb007ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.013+0000 7fcefd004740 1 -- 172.21.15.1:0/2135227812 shutdown_connections 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.013+0000 7fcefd004740 1 -- 172.21.15.1:0/2135227812 wait complete. 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.014+0000 7fcefd004740 1 -- start start 2024-04-26T21:33:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.014+0000 7fcefd004740 1 Processor -- start 2024-04-26T21:33:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.014+0000 7fcefd004740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.014+0000 7fcefd004740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb00bb00 0x55aaeb00fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.014+0000 7fcefd004740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb0163c0 0x55aaeb00ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.014+0000 7fcefd004740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55aaeb002ef0 con 0x55aaeb0163c0 2024-04-26T21:33:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.014+0000 7fcefd004740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55aaeaef3c60 con 0x55aaeb00b1e0 2024-04-26T21:33:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.014+0000 7fcefd004740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aaeb003070 con 0x55aaeb00bb00 2024-04-26T21:33:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fcefd004740 10 client.0 _ll_register_callbacks cb 0x55aaeaf1fe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fcef3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb0163c0 0x55aaeb00ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fcef3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51152/0 (socket says 172.21.15.1:51152) 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fcef3400640 1 -- 172.21.15.1:0/2770259531 learned_addr learned my addr 172.21.15.1:0/2770259531 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fceebe00640 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb00bb00 0x55aaeb00fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fcef3400640 1 -- 172.21.15.1:0/2770259531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb00bb00 msgr2=0x55aaeb00fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fcef3400640 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb00bb00 0x55aaeb00fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fcef3400640 1 -- 172.21.15.1:0/2770259531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb0163c0 msgr2=0x55aaeb00ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fcef3400640 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb0163c0 0x55aaeb00ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.015+0000 7fcef3400640 1 -- 172.21.15.1:0/2770259531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aaeaf02510 con 0x55aaeb00b1e0 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.016+0000 7fcef3400640 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00f550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcee40ee650 tx=0x7fcee4031610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a2c8eb27de745a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.016+0000 7fcee0c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.016+0000 7fcee0c00640 1 -- 172.21.15.1:0/2770259531 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee40bf060 con 0x55aaeb00b1e0 2024-04-26T21:33:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.016+0000 7fcee0c00640 1 -- 172.21.15.1:0/2770259531 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fcee40b9560 con 0x55aaeb00b1e0 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.016+0000 7fcefd004740 1 -- 172.21.15.1:0/2770259531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceec02e440 con 0x55aaeb00b1e0 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.016+0000 7fcee0c00640 1 -- 172.21.15.1:0/2770259531 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcee4049050 con 0x55aaeb00b1e0 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.016+0000 7fcefd004740 10 client.25781 fetch_fsmap learned FSMap version 32 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.016+0000 7fcefd004740 1 -- 172.21.15.1:0/2770259531 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55aaeac7e210 con 0x55aaeb00b1e0 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.016+0000 7fcee0c00640 1 -- 172.21.15.1:0/2770259531 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceec02e440 con 0x55aaeb00b1e0 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.016+0000 7fcee0c00640 1 -- 172.21.15.1:0/2770259531 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcee40d6210 con 0x55aaeb00b1e0 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.017+0000 7fcefd004740 10 client.25781 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.017+0000 7fcefd004740 10 client.25781 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.017+0000 7fcefd004740 20 client.25781 populate_metadata read hostname 'smithi001' 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.017+0000 7fcefd004740 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aaeb021cd0 0x55aaeb024150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.017+0000 7fcefd004740 4 client.25781 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.017+0000 7fcefd004740 1 -- 172.21.15.1:0/2770259531 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aaeaf79f90 con 0x55aaeb021cd0 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.017+0000 7fceebe00640 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aaeb021cd0 0x55aaeb024150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.017+0000 7fceebe00640 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aaeb021cd0 0x55aaeb024150 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=378e80217ac2ea78 server_cookie=8c99c05651e03845 in_seq=0 out_seq=0 2024-04-26T21:33:40.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.017+0000 7fcee0c00640 10 client.25781 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcee0c00640 1 -- 172.21.15.1:0/2770259531 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aaeaf79f90 con 0x55aaeb021cd0 2024-04-26T21:33:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcee0c00640 10 client.25781 handle_command_reply: tid=0 2024-04-26T21:33:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 client.25781 shutdown 2024-04-26T21:33:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 -- 172.21.15.1:0/2770259531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 msgr2=0x55aaeb00f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00f550 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcee40ee650 tx=0x7fcee4031610 comp rx=0 tx=0).stop 2024-04-26T21:33:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 -- 172.21.15.1:0/2770259531 shutdown_connections 2024-04-26T21:33:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aaeb021cd0 0x55aaeb024150 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb00bb00 0x55aaeb00fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb0163c0 0x55aaeb00ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 --2- 172.21.15.1:0/2770259531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00f550 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcee0c00640 0 client.25781 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 -- 172.21.15.1:0/2770259531 >> 172.21.15.1:0/2770259531 conn(0x55aaeb0076f0 msgr2=0x55aaeb00d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 -- 172.21.15.1:0/2770259531 shutdown_connections 2024-04-26T21:33:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.019+0000 7fcefd004740 1 -- 172.21.15.1:0/2770259531 wait complete. 2024-04-26T21:33:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.020+0000 7fcefd004740 20 client.25781 trim_cache size 0 max 16384 2024-04-26T21:33:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.037+0000 7fcefd004740 1 Processor -- start 2024-04-26T21:33:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcefd004740 1 -- start start 2024-04-26T21:33:40.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcefd004740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcefd004740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb029a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcefd004740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb0163c0 0x55aaeb029f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcefd004740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41b05d0 con 0x55aaeb00bb00 2024-04-26T21:33:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcefd004740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41adde0 con 0x55aaeb0163c0 2024-04-26T21:33:40.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcefd004740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41abc30 con 0x55aaeb00b1e0 2024-04-26T21:33:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb0163c0 0x55aaeb029f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb0163c0 0x55aaeb029f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51158/0 (socket says 172.21.15.1:51158) 2024-04-26T21:33:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcef3e00640 1 -- 172.21.15.1:0/324349071 learned_addr learned my addr 172.21.15.1:0/324349071 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fceebe00640 1 --2- 172.21.15.1:0/324349071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb029a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.038+0000 7fcef3400640 1 --2- 172.21.15.1:0/324349071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.039+0000 7fcef3e00640 1 -- 172.21.15.1:0/324349071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb00b1e0 msgr2=0x55aaeb00fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.039+0000 7fcef3e00640 1 --2- 172.21.15.1:0/324349071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.039+0000 7fcef3e00640 1 -- 172.21.15.1:0/324349071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 msgr2=0x55aaeb029a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.039+0000 7fcef3e00640 1 --2- 172.21.15.1:0/324349071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb029a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.039+0000 7fcef3e00640 1 -- 172.21.15.1:0/324349071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aaead4a620 con 0x55aaeb0163c0 2024-04-26T21:33:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.039+0000 7fceebe00640 1 --2- 172.21.15.1:0/324349071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb029a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.039+0000 7fcef3e00640 1 --2- 172.21.15.1:0/324349071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb0163c0 0x55aaeb029f60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fceec00f010 tx=0x7fceec01c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b448dfe267a46517 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.039+0000 7fcee3400640 1 -- 172.21.15.1:0/324349071 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec004980 con 0x55aaeb0163c0 2024-04-26T21:33:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.039+0000 7fcee3400640 1 -- 172.21.15.1:0/324349071 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceec034040 con 0x55aaeb0163c0 2024-04-26T21:33:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.039+0000 7fcee3400640 1 -- 172.21.15.1:0/324349071 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec02e440 con 0x55aaeb0163c0 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.040+0000 7fcefd004740 1 -- 172.21.15.1:0/324349071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb0163c0 msgr2=0x55aaeb029f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.040+0000 7fcefd004740 1 --2- 172.21.15.1:0/324349071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb0163c0 0x55aaeb029f60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fceec00f010 tx=0x7fceec01c000 comp rx=0 tx=0).stop 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.040+0000 7fcefd004740 1 -- 172.21.15.1:0/324349071 shutdown_connections 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.040+0000 7fcefd004740 1 --2- 172.21.15.1:0/324349071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb0163c0 0x55aaeb029f60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.040+0000 7fcefd004740 1 --2- 172.21.15.1:0/324349071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb029a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.040+0000 7fcefd004740 1 --2- 172.21.15.1:0/324349071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb00b1e0 0x55aaeb00fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.040+0000 7fcefd004740 1 -- 172.21.15.1:0/324349071 >> 172.21.15.1:0/324349071 conn(0x55aaeb0076f0 msgr2=0x55aaeb002440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.040+0000 7fcefd004740 1 -- 172.21.15.1:0/324349071 shutdown_connections 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.040+0000 7fcefd004740 1 -- 172.21.15.1:0/324349071 wait complete. 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.041+0000 7fcefd004740 1 -- start start 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.041+0000 7fcefd004740 1 Processor -- start 2024-04-26T21:33:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.041+0000 7fcefd004740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb01a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.041+0000 7fcefd004740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb01a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.041+0000 7fcefd004740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 0x55aaeb087290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.041+0000 7fcefd004740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41ac2a0 con 0x55aaeb00bb00 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.041+0000 7fcefd004740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41ad710 con 0x55aaeb00b1e0 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.041+0000 7fcefd004740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcef41ac970 con 0x55aaeb0163c0 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.041+0000 7fcefd004740 10 client.0 _ll_register_callbacks cb 0x7fcee4005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb01a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb01a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51166/0 (socket says 172.21.15.1:51166) 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fceebe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb01a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3400640 1 -- 172.21.15.1:0/3309292459 learned_addr learned my addr 172.21.15.1:0/3309292459 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3e00640 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 0x55aaeb087290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3400640 1 -- 172.21.15.1:0/3309292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 msgr2=0x55aaeb087290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3400640 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 0x55aaeb087290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3400640 1 -- 172.21.15.1:0/3309292459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 msgr2=0x55aaeb01a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3400640 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb01a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3400640 1 -- 172.21.15.1:0/3309292459 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aaeaf26cb0 con 0x55aaeb00b1e0 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3e00640 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 0x55aaeb087290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fceebe00640 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb01a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcef3400640 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb01a290 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcee40bd1f0 tx=0x7fcee4005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae48cf1f230b7bec server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcee0c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.042+0000 7fcee0c00640 1 -- 172.21.15.1:0/3309292459 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee40b9190 con 0x55aaeb00b1e0 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcee0c00640 1 -- 172.21.15.1:0/3309292459 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fcee40bd270 con 0x55aaeb00b1e0 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcefd004740 1 -- 172.21.15.1:0/3309292459 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceec034040 con 0x55aaeb00b1e0 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcee0c00640 1 -- 172.21.15.1:0/3309292459 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcee40cb050 con 0x55aaeb00b1e0 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcefd004740 10 client.25787 fetch_fsmap learned FSMap version 32 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcefd004740 1 -- 172.21.15.1:0/3309292459 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55aaeb0888e0 con 0x55aaeb00b1e0 2024-04-26T21:33:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcee0c00640 1 -- 172.21.15.1:0/3309292459 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fceec034040 con 0x55aaeb00b1e0 2024-04-26T21:33:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcee0c00640 1 -- 172.21.15.1:0/3309292459 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fcee4031b60 con 0x55aaeb00b1e0 2024-04-26T21:33:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcefd004740 10 client.25787 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T21:33:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcefd004740 10 client.25787 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:33:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcefd004740 20 client.25787 populate_metadata read hostname 'smithi001' 2024-04-26T21:33:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcefd004740 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aaeb088ef0 0x55aaeb08b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcefd004740 4 client.25787 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:33:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.043+0000 7fcefd004740 1 -- 172.21.15.1:0/3309292459 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aaeb01ec30 con 0x55aaeb088ef0 2024-04-26T21:33:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.044+0000 7fceebe00640 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aaeb088ef0 0x55aaeb08b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.044+0000 7fceebe00640 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aaeb088ef0 0x55aaeb08b310 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=936aa19aad8728ea server_cookie=e62396353c2a4885 in_seq=0 out_seq=0 2024-04-26T21:33:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.044+0000 7fcee0c00640 10 client.25787 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:40.069 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:39 smithi001 ceph-mon[28054]: pgmap v103: 129 pgs: 129 active+clean; 49 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 815 KiB/s wr, 119 op/s 2024-04-26T21:33:40.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:39 smithi038 ceph-mon[32724]: pgmap v103: 129 pgs: 129 active+clean; 49 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 815 KiB/s wr, 119 op/s 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.213+0000 7fcee0c00640 1 -- 172.21.15.1:0/3309292459 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aaeb01ec30 con 0x55aaeb088ef0 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.213+0000 7fcee0c00640 10 client.25787 handle_command_reply: tid=0 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.213+0000 7fcefd004740 1 client.25787 shutdown 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 1 -- 172.21.15.1:0/3309292459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 msgr2=0x55aaeb01a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb01a290 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcee40bd1f0 tx=0x7fcee4005c00 comp rx=0 tx=0).stop 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 1 -- 172.21.15.1:0/3309292459 shutdown_connections 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aaeb088ef0 0x55aaeb08b310 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aaeb0163c0 0x55aaeb087290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aaeb00bb00 0x55aaeb01a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 1 --2- 172.21.15.1:0/3309292459 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aaeb00b1e0 0x55aaeb01a290 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcee0c00640 0 client.25787 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 1 -- 172.21.15.1:0/3309292459 >> 172.21.15.1:0/3309292459 conn(0x55aaeb0076f0 msgr2=0x55aaeb00cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 1 -- 172.21.15.1:0/3309292459 shutdown_connections 2024-04-26T21:33:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 1 -- 172.21.15.1:0/3309292459 wait complete. 2024-04-26T21:33:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.214+0000 7fcefd004740 20 client.25787 trim_cache size 0 max 16384 2024-04-26T21:33:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:33:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:33:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:33:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:33:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.215+0000 7fcefa200640 1 -- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fced80a3d10 msgr2=0x7fced80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.215+0000 7fcefa200640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fced80a3d10 0x7fced80a61d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fceec02b000 tx=0x7fceec02c000 comp rx=0 tx=0).stop 2024-04-26T21:33:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.215+0000 7fcefa200640 1 -- 172.21.15.1:0/3914893997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcef4102910 msgr2=0x7fcef4199140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.215+0000 7fcefa200640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcef4102910 0x7fcef4199140 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcee402aa00 tx=0x7fcee4003c10 comp rx=0 tx=0).stop 2024-04-26T21:33:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.215+0000 7fcefa200640 1 -- 172.21.15.1:0/3914893997 shutdown_connections 2024-04-26T21:33:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.215+0000 7fcefa200640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fced80a3d10 0x7fced80a61d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.215+0000 7fcefa200640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcef41032d0 0x7fcef4199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.215+0000 7fcefa200640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcef4108920 0x7fcef419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.215+0000 7fcefa200640 1 --2- 172.21.15.1:0/3914893997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcef4102910 0x7fcef4199140 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.215+0000 7fcefa200640 1 -- 172.21.15.1:0/3914893997 >> 172.21.15.1:0/3914893997 conn(0x7fcef40fe3c0 msgr2=0x7fcef40feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.216+0000 7fcefa200640 1 -- 172.21.15.1:0/3914893997 shutdown_connections 2024-04-26T21:33:40.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:40.216+0000 7fcefa200640 1 -- 172.21.15.1:0/3914893997 wait complete. 2024-04-26T21:33:40.535 INFO:tasks.workunit.client.0.smithi001.stdout: 1 57503 3.34 MB/sec warmup 98 sec latency 263.414 ms 2024-04-26T21:33:41.535 INFO:tasks.workunit.client.0.smithi001.stdout: 1 57571 3.31 MB/sec warmup 99 sec latency 1016.327 ms 2024-04-26T21:33:42.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:41 smithi001 ceph-mon[28054]: pgmap v104: 129 pgs: 129 active+clean; 51 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-04-26T21:33:42.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:41 smithi038 ceph-mon[32724]: pgmap v104: 129 pgs: 129 active+clean; 51 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-04-26T21:33:42.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:41 smithi186 ceph-mon[32991]: pgmap v104: 129 pgs: 129 active+clean; 51 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-04-26T21:33:42.535 INFO:tasks.workunit.client.0.smithi001.stdout: 1 57936 3.29 MB/sec warmup 100 sec latency 654.142 ms 2024-04-26T21:33:43.535 INFO:tasks.workunit.client.0.smithi001.stdout: 1 59270 3.31 MB/sec warmup 101 sec latency 29.407 ms 2024-04-26T21:33:44.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:43 smithi001 ceph-mon[28054]: pgmap v105: 129 pgs: 129 active+clean; 51 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 714 KiB/s wr, 94 op/s 2024-04-26T21:33:44.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:43 smithi038 ceph-mon[32724]: pgmap v105: 129 pgs: 129 active+clean; 51 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 714 KiB/s wr, 94 op/s 2024-04-26T21:33:44.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:43 smithi186 ceph-mon[32991]: pgmap v105: 129 pgs: 129 active+clean; 51 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 714 KiB/s wr, 94 op/s 2024-04-26T21:33:44.535 INFO:tasks.workunit.client.0.smithi001.stdout: 1 60534 3.39 MB/sec warmup 102 sec latency 22.587 ms 2024-04-26T21:33:45.536 INFO:tasks.workunit.client.0.smithi001.stdout: 1 60791 3.36 MB/sec warmup 103 sec latency 22.748 ms 2024-04-26T21:33:46.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:45 smithi001 ceph-mon[28054]: pgmap v106: 129 pgs: 129 active+clean; 48 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 1.1 MiB/s wr, 166 op/s 2024-04-26T21:33:46.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:45 smithi038 ceph-mon[32724]: pgmap v106: 129 pgs: 129 active+clean; 48 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 1.1 MiB/s wr, 166 op/s 2024-04-26T21:33:46.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:45 smithi186 ceph-mon[32991]: pgmap v106: 129 pgs: 129 active+clean; 48 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 1.1 MiB/s wr, 166 op/s 2024-04-26T21:33:46.536 INFO:tasks.workunit.client.0.smithi001.stdout: 1 61051 3.33 MB/sec warmup 104 sec latency 455.449 ms 2024-04-26T21:33:47.536 INFO:tasks.workunit.client.0.smithi001.stdout: 1 61051 3.30 MB/sec warmup 105 sec latency 1166.438 ms 2024-04-26T21:33:48.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:47 smithi001 ceph-mon[28054]: pgmap v107: 129 pgs: 129 active+clean; 48 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 673 KiB/s wr, 110 op/s 2024-04-26T21:33:48.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:47 smithi038 ceph-mon[32724]: pgmap v107: 129 pgs: 129 active+clean; 48 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 673 KiB/s wr, 110 op/s 2024-04-26T21:33:48.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:47 smithi186 ceph-mon[32991]: pgmap v107: 129 pgs: 129 active+clean; 48 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 673 KiB/s wr, 110 op/s 2024-04-26T21:33:48.536 INFO:tasks.workunit.client.0.smithi001.stdout: 1 61051 3.27 MB/sec warmup 106 sec latency 2166.595 ms 2024-04-26T21:33:49.536 INFO:tasks.workunit.client.0.smithi001.stdout: 1 61051 3.24 MB/sec warmup 107 sec latency 3166.725 ms 2024-04-26T21:33:50.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:49 smithi038 ceph-mon[32724]: pgmap v108: 129 pgs: 129 active+clean; 48 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 673 KiB/s wr, 110 op/s 2024-04-26T21:33:50.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:49 smithi186 ceph-mon[32991]: pgmap v108: 129 pgs: 129 active+clean; 48 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 673 KiB/s wr, 110 op/s 2024-04-26T21:33:50.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:49 smithi001 ceph-mon[28054]: pgmap v108: 129 pgs: 129 active+clean; 48 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 673 KiB/s wr, 110 op/s 2024-04-26T21:33:50.536 INFO:tasks.workunit.client.0.smithi001.stdout: 1 61051 3.21 MB/sec warmup 108 sec latency 4166.911 ms 2024-04-26T21:33:51.536 INFO:tasks.workunit.client.0.smithi001.stdout: 1 61242 3.18 MB/sec warmup 109 sec latency 4916.562 ms 2024-04-26T21:33:52.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:51 smithi186 ceph-mon[32991]: pgmap v109: 129 pgs: 129 active+clean; 49 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 977 KiB/s wr, 133 op/s 2024-04-26T21:33:52.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:51 smithi001 ceph-mon[28054]: pgmap v109: 129 pgs: 129 active+clean; 49 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 977 KiB/s wr, 133 op/s 2024-04-26T21:33:52.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:51 smithi038 ceph-mon[32724]: pgmap v109: 129 pgs: 129 active+clean; 49 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 977 KiB/s wr, 133 op/s 2024-04-26T21:33:52.537 INFO:tasks.workunit.client.0.smithi001.stdout: 1 62731 3.21 MB/sec warmup 110 sec latency 28.164 ms 2024-04-26T21:33:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.377+0000 7f0d64c00640 1 -- 172.21.15.1:0/2953359372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d6010a970 msgr2=0x7f0d6010ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.377+0000 7f0d64c00640 1 --2- 172.21.15.1:0/2953359372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d6010a970 0x7f0d6010ad50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009900 tx=0x7f0d5002a410 comp rx=0 tx=0).stop 2024-04-26T21:33:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.378+0000 7f0d64c00640 1 -- 172.21.15.1:0/2953359372 shutdown_connections 2024-04-26T21:33:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.378+0000 7f0d64c00640 1 --2- 172.21.15.1:0/2953359372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d60104960 0x7f0d60104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.378+0000 7f0d64c00640 1 --2- 172.21.15.1:0/2953359372 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d60105320 0x7f0d60111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.378+0000 7f0d64c00640 1 --2- 172.21.15.1:0/2953359372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d6010a970 0x7f0d6010ad50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.378+0000 7f0d64c00640 1 -- 172.21.15.1:0/2953359372 >> 172.21.15.1:0/2953359372 conn(0x7f0d60100430 msgr2=0x7f0d60102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:53.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.378+0000 7f0d64c00640 1 -- 172.21.15.1:0/2953359372 shutdown_connections 2024-04-26T21:33:53.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.378+0000 7f0d64c00640 1 -- 172.21.15.1:0/2953359372 wait complete. 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.379+0000 7f0d64c00640 1 Processor -- start 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.379+0000 7f0d64c00640 1 -- start start 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d64c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d60104960 0x7f0d6019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d64c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d60105320 0x7f0d6019b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d64c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d6010a970 0x7f0d6019fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d64c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0d60114c50 con 0x7f0d6010a970 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d64c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0d60114ad0 con 0x7f0d60104960 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d64c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d60114dd0 con 0x7f0d60105320 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d5ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d60105320 0x7f0d6019b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d5ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d60105320 0x7f0d6019b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:53710/0 (socket says 172.21.15.1:53710) 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d5ea00640 1 -- 172.21.15.1:0/3373621707 learned_addr learned my addr 172.21.15.1:0/3373621707 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:53.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d5f400640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d60104960 0x7f0d6019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.380+0000 7f0d5fe00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d6010a970 0x7f0d6019fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d5ea00640 1 -- 172.21.15.1:0/3373621707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d60104960 msgr2=0x7f0d6019b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d60104960 0x7f0d6019b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d5ea00640 1 -- 172.21.15.1:0/3373621707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d6010a970 msgr2=0x7f0d6019fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d6010a970 0x7f0d6019fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d5ea00640 1 -- 172.21.15.1:0/3373621707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d601a01a0 con 0x7f0d60105320 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d5f400640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d60104960 0x7f0d6019b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d60105320 0x7f0d6019b6f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00eb30 tx=0x7f0d4c00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d57e00640 1 -- 172.21.15.1:0/3373621707 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c010430 con 0x7f0d60105320 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d57e00640 1 -- 172.21.15.1:0/3373621707 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d4c010a80 con 0x7f0d60105320 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d64c00640 1 -- 172.21.15.1:0/3373621707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d601a0490 con 0x7f0d60105320 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.381+0000 7f0d57e00640 1 -- 172.21.15.1:0/3373621707 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c018a50 con 0x7f0d60105320 2024-04-26T21:33:53.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.382+0000 7f0d64c00640 1 -- 172.21.15.1:0/3373621707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d601a7d20 con 0x7f0d60105320 2024-04-26T21:33:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.385+0000 7f0d57e00640 1 -- 172.21.15.1:0/3373621707 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0d4c018bf0 con 0x7f0d60105320 2024-04-26T21:33:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.385+0000 7f0d68ffa740 1 Processor -- start 2024-04-26T21:33:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.385+0000 7f0d68ffa740 1 -- start start 2024-04-26T21:33:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874de3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874de900 0x55f4874e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 0x55f4874eb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55f487443380 con 0x55f4874e91c0 2024-04-26T21:33:53.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f4873dfc30 con 0x55f4874ddfe0 2024-04-26T21:33:53.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f48743f340 con 0x55f4874de900 2024-04-26T21:33:53.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d5fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874de3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d5fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874de3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60616/0 (socket says 172.21.15.1:60616) 2024-04-26T21:33:53.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d5fe00640 1 -- 172.21.15.1:0/149322264 learned_addr learned my addr 172.21.15.1:0/149322264 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:53.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d5f400640 1 --2- 172.21.15.1:0/149322264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874de900 0x55f4874e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.386+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/149322264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 0x55f4874eb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d5fe00640 1 -- 172.21.15.1:0/149322264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874de900 msgr2=0x55f4874e8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d5fe00640 1 --2- 172.21.15.1:0/149322264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874de900 0x55f4874e8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d5fe00640 1 -- 172.21.15.1:0/149322264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 msgr2=0x55f4874eb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d5fe00640 1 --2- 172.21.15.1:0/149322264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 0x55f4874eb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d5fe00640 1 -- 172.21.15.1:0/149322264 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f487401940 con 0x55f4874ddfe0 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d57e00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0d440a3d80 0x7f0d440a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d5fe00640 1 --2- 172.21.15.1:0/149322264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874de3c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0d58009f40 tx=0x7f0d58010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f11779f23433df63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d57e00640 1 -- 172.21.15.1:0/3373621707 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f0d4c0d13e0 con 0x7f0d60105320 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d4be00640 1 -- 172.21.15.1:0/149322264 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d58003b70 con 0x55f4874ddfe0 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d4be00640 1 -- 172.21.15.1:0/149322264 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d58003d10 con 0x55f4874ddfe0 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d4be00640 1 -- 172.21.15.1:0/149322264 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d580119c0 con 0x55f4874ddfe0 2024-04-26T21:33:53.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.387+0000 7f0d5f400640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0d440a3d80 0x7f0d440a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.392 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.388+0000 7f0d68ffa740 1 -- 172.21.15.1:0/149322264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 msgr2=0x55f4874de3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.388+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/149322264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874de3c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0d58009f40 tx=0x7f0d58010860 comp rx=0 tx=0).stop 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.388+0000 7f0d5f400640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0d440a3d80 0x7f0d440a6240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0d50000c00 tx=0x7f0d50005b90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.389+0000 7f0d68ffa740 1 -- 172.21.15.1:0/149322264 shutdown_connections 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.389+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/149322264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874de900 0x55f4874e8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.389+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/149322264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 0x55f4874eb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.389+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/149322264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874de3c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.389+0000 7f0d68ffa740 1 -- 172.21.15.1:0/149322264 >> 172.21.15.1:0/149322264 conn(0x55f4874da4f0 msgr2=0x55f4874da8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.389+0000 7f0d68ffa740 1 -- 172.21.15.1:0/149322264 shutdown_connections 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.389+0000 7f0d68ffa740 1 -- 172.21.15.1:0/149322264 wait complete. 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.389+0000 7f0d68ffa740 1 -- start start 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.389+0000 7f0d68ffa740 1 Processor -- start 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874ddfe0 0x55f4874e6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f4874e6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874e91c0 0x55f4874e6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55f4874ebdf0 con 0x55f4874ddfe0 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f4873cac60 con 0x55f4874de900 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f4874ebf70 con 0x55f4874e91c0 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d68ffa740 10 client.0 _ll_register_callbacks cb 0x55f4873f6e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d5ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f4874e6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d5ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f4874e6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60620/0 (socket says 172.21.15.1:60620) 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d5ea00640 1 -- 172.21.15.1:0/1765526423 learned_addr learned my addr 172.21.15.1:0/1765526423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d5fe00640 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874e91c0 0x55f4874e6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d5f400640 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874ddfe0 0x55f4874e6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d5ea00640 1 -- 172.21.15.1:0/1765526423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874e91c0 msgr2=0x55f4874e6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874e91c0 0x55f4874e6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d5ea00640 1 -- 172.21.15.1:0/1765526423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874ddfe0 msgr2=0x55f4874e6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:53.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874ddfe0 0x55f4874e6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.390+0000 7f0d5ea00640 1 -- 172.21.15.1:0/1765526423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4873d9510 con 0x55f4874de900 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.391+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f4874e6660 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00c7d0 tx=0x7f0d4c099270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=470b512753d150e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.391+0000 7f0d48c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.391+0000 7f0d48c00640 1 -- 172.21.15.1:0/1765526423 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c09d610 con 0x55f4874de900 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.391+0000 7f0d48c00640 1 -- 172.21.15.1:0/1765526423 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f0d4c0ce940 con 0x55f4874de900 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.391+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1765526423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d58003d10 con 0x55f4874de900 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.391+0000 7f0d48c00640 1 -- 172.21.15.1:0/1765526423 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0d4c0ceb40 con 0x55f4874de900 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.391+0000 7f0d68ffa740 10 client.25799 fetch_fsmap learned FSMap version 32 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.391+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1765526423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f487155210 con 0x55f4874de900 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.392+0000 7f0d48c00640 1 -- 172.21.15.1:0/1765526423 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d58003d10 con 0x55f4874de900 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.392+0000 7f0d48c00640 1 -- 172.21.15.1:0/1765526423 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0d4c098c60 con 0x55f4874de900 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.392+0000 7f0d68ffa740 10 client.25799 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.392+0000 7f0d68ffa740 10 client.25799 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.392+0000 7f0d68ffa740 20 client.25799 populate_metadata read hostname 'smithi001' 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.392+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f4874f8ee0 0x55f4874fb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.392+0000 7f0d68ffa740 4 client.25799 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:33:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.392+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1765526423 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f487450f90 con 0x55f4874f8ee0 2024-04-26T21:33:53.396 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.392+0000 7f0d5f400640 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f4874f8ee0 0x55f4874fb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.393+0000 7f0d5f400640 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f4874f8ee0 0x55f4874fb2d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fbdf7b602ac70e1 server_cookie=7ed013e6a256c019 in_seq=0 out_seq=0 2024-04-26T21:33:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.393+0000 7f0d48c00640 10 client.25799 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d48c00640 1 -- 172.21.15.1:0/1765526423 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f487450f90 con 0x55f4874f8ee0 2024-04-26T21:33:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d48c00640 10 client.25799 handle_command_reply: tid=0 2024-04-26T21:33:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 client.25799 shutdown 2024-04-26T21:33:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1765526423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 msgr2=0x55f4874e6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f4874e6660 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00c7d0 tx=0x7f0d4c099270 comp rx=0 tx=0).stop 2024-04-26T21:33:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1765526423 shutdown_connections 2024-04-26T21:33:53.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f4874f8ee0 0x55f4874fb2d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874e91c0 0x55f4874e6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f4874e6660 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1765526423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874ddfe0 0x55f4874e6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1765526423 >> 172.21.15.1:0/1765526423 conn(0x55f4874da4f0 msgr2=0x55f4874e0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d48c00640 0 client.25799 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1765526423 shutdown_connections 2024-04-26T21:33:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1765526423 wait complete. 2024-04-26T21:33:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.394+0000 7f0d68ffa740 20 client.25799 trim_cache size 0 max 16384 2024-04-26T21:33:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.412+0000 7f0d68ffa740 1 Processor -- start 2024-04-26T21:33:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.412+0000 7f0d68ffa740 1 -- start start 2024-04-26T21:33:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874ddfe0 0x55f4874f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f487500b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 0x55f4875010d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0d601b48a0 con 0x55f4874e91c0 2024-04-26T21:33:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0d601b20b0 con 0x55f4874de900 2024-04-26T21:33:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d601aff00 con 0x55f4874ddfe0 2024-04-26T21:33:53.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d5ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f487500b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d5f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874ddfe0 0x55f4874f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d5ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f487500b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60622/0 (socket says 172.21.15.1:60622) 2024-04-26T21:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d5ea00640 1 -- 172.21.15.1:0/774553749 learned_addr learned my addr 172.21.15.1:0/774553749 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d5fe00640 1 --2- 172.21.15.1:0/774553749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 0x55f4875010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d5ea00640 1 -- 172.21.15.1:0/774553749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874ddfe0 msgr2=0x55f4874f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/774553749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874ddfe0 0x55f4874f4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d5ea00640 1 -- 172.21.15.1:0/774553749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 msgr2=0x55f4875010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/774553749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 0x55f4875010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.413+0000 7f0d5ea00640 1 -- 172.21.15.1:0/774553749 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f487221620 con 0x55f4874de900 2024-04-26T21:33:53.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.414+0000 7f0d5fe00640 1 --2- 172.21.15.1:0/774553749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 0x55f4875010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:33:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.414+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/774553749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f487500b90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c099370 tx=0x7f0d4c0992e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5aa895367917274 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:53.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.414+0000 7f0d4b400640 1 -- 172.21.15.1:0/774553749 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c09ca50 con 0x55f4874de900 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.414+0000 7f0d4b400640 1 -- 172.21.15.1:0/774553749 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d4c0a12c0 con 0x55f4874de900 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.414+0000 7f0d4b400640 1 -- 172.21.15.1:0/774553749 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0ce940 con 0x55f4874de900 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.415+0000 7f0d68ffa740 1 -- 172.21.15.1:0/774553749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 msgr2=0x55f487500b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.415+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/774553749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f487500b90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c099370 tx=0x7f0d4c0992e0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.415+0000 7f0d68ffa740 1 -- 172.21.15.1:0/774553749 shutdown_connections 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.415+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/774553749 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874e91c0 0x55f4875010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.415+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/774553749 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874de900 0x55f487500b90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.415+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/774553749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874ddfe0 0x55f4874f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.415+0000 7f0d68ffa740 1 -- 172.21.15.1:0/774553749 >> 172.21.15.1:0/774553749 conn(0x55f4874da4f0 msgr2=0x55f4874d9650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.415+0000 7f0d68ffa740 1 -- 172.21.15.1:0/774553749 shutdown_connections 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.415+0000 7f0d68ffa740 1 -- 172.21.15.1:0/774553749 wait complete. 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.416+0000 7f0d68ffa740 1 -- start start 2024-04-26T21:33:53.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.416+0000 7f0d68ffa740 1 Processor -- start 2024-04-26T21:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.416+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.416+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874de900 0x55f4874f1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.416+0000 7f0d68ffa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874e91c0 0x55f4874ee190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.416+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0d601b0570 con 0x55f4874de900 2024-04-26T21:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.416+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0d601b19e0 con 0x55f4874ddfe0 2024-04-26T21:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.416+0000 7f0d68ffa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d601b0c40 con 0x55f4874e91c0 2024-04-26T21:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d68ffa740 10 client.0 _ll_register_callbacks cb 0x7f0d4c009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874f11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60630/0 (socket says 172.21.15.1:60630) 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5f400640 1 -- 172.21.15.1:0/1581484408 learned_addr learned my addr 172.21.15.1:0/1581484408 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874de900 0x55f4874f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5fe00640 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874e91c0 0x55f4874ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5f400640 1 -- 172.21.15.1:0/1581484408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874e91c0 msgr2=0x55f4874ee190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5f400640 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874e91c0 0x55f4874ee190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5f400640 1 -- 172.21.15.1:0/1581484408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874de900 msgr2=0x55f4874f1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5f400640 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874de900 0x55f4874f1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5f400640 1 -- 172.21.15.1:0/1581484408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4873fdcb0 con 0x55f4874ddfe0 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.417+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874de900 0x55f4874f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.418+0000 7f0d5f400640 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874f11c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009230 tx=0x7f0d50009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e8e30cab814fda server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.418+0000 7f0d48c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.418+0000 7f0d48c00640 1 -- 172.21.15.1:0/1581484408 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d50035040 con 0x55f4874ddfe0 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.418+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1581484408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d4c0a12c0 con 0x55f4874ddfe0 2024-04-26T21:33:53.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.418+0000 7f0d48c00640 1 -- 172.21.15.1:0/1581484408 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f0d500511b0 con 0x55f4874ddfe0 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.418+0000 7f0d48c00640 1 -- 172.21.15.1:0/1581484408 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0d500513b0 con 0x55f4874ddfe0 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.418+0000 7f0d68ffa740 10 client.25805 fetch_fsmap learned FSMap version 32 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.418+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1581484408 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f48755fab0 con 0x55f4874ddfe0 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.418+0000 7f0d48c00640 1 -- 172.21.15.1:0/1581484408 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d4c0a12c0 con 0x55f4874ddfe0 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.419+0000 7f0d48c00640 1 -- 172.21.15.1:0/1581484408 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f0d50004ce0 con 0x55f4874ddfe0 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.419+0000 7f0d68ffa740 10 client.25805 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.419+0000 7f0d68ffa740 10 client.25805 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.419+0000 7f0d68ffa740 20 client.25805 populate_metadata read hostname 'smithi001' 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.419+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f4875601e0 0x55f487562600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.419+0000 7f0d68ffa740 4 client.25805 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.419+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1581484408 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f48744ae00 con 0x55f4875601e0 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.419+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f4875601e0 0x55f487562600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.419+0000 7f0d5ea00640 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f4875601e0 0x55f487562600 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6926827d19e9a4f server_cookie=777e4a94aaad90e3 in_seq=0 out_seq=0 2024-04-26T21:33:53.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.419+0000 7f0d48c00640 10 client.25805 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d48c00640 1 -- 172.21.15.1:0/1581484408 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f48744ae00 con 0x55f4875601e0 2024-04-26T21:33:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d48c00640 10 client.25805 handle_command_reply: tid=0 2024-04-26T21:33:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d68ffa740 1 client.25805 shutdown 2024-04-26T21:33:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1581484408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 msgr2=0x55f4874f11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874f11c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009230 tx=0x7f0d50009260 comp rx=0 tx=0).stop 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1581484408 shutdown_connections 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f4875601e0 0x55f487562600 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4874e91c0 0x55f4874ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f4874de900 0x55f4874f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d68ffa740 1 --2- 172.21.15.1:0/1581484408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f4874ddfe0 0x55f4874f11c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1581484408 >> 172.21.15.1:0/1581484408 conn(0x55f4874da4f0 msgr2=0x55f4874e60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.479+0000 7f0d48c00640 0 client.25805 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.480+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1581484408 shutdown_connections 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.480+0000 7f0d68ffa740 1 -- 172.21.15.1:0/1581484408 wait complete. 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.480+0000 7f0d68ffa740 20 client.25805 trim_cache size 0 max 16384 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:33:53.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:33:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:33:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:33:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 -- 172.21.15.1:0/3373621707 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0d440a3d80 msgr2=0x7f0d440a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0d440a3d80 0x7f0d440a6240 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0d50000c00 tx=0x7f0d50005b90 comp rx=0 tx=0).stop 2024-04-26T21:33:53.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 -- 172.21.15.1:0/3373621707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d60105320 msgr2=0x7f0d6019b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:33:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d60105320 0x7f0d6019b6f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00eb30 tx=0x7f0d4c00c2d0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 -- 172.21.15.1:0/3373621707 shutdown_connections 2024-04-26T21:33:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0d440a3d80 0x7f0d440a6240 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d60105320 0x7f0d6019b6f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d6010a970 0x7f0d6019fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 --2- 172.21.15.1:0/3373621707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d60104960 0x7f0d6019b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:33:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 -- 172.21.15.1:0/3373621707 >> 172.21.15.1:0/3373621707 conn(0x7f0d60100430 msgr2=0x7f0d60100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:33:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 -- 172.21.15.1:0/3373621707 shutdown_connections 2024-04-26T21:33:53.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:33:53.481+0000 7f0d64c00640 1 -- 172.21.15.1:0/3373621707 wait complete. 2024-04-26T21:33:53.537 INFO:tasks.workunit.client.0.smithi001.stdout: 1 63950 3.28 MB/sec warmup 111 sec latency 21.479 ms 2024-04-26T21:33:54.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:53 smithi186 ceph-mon[32991]: pgmap v110: 129 pgs: 129 active+clean; 49 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 752 KiB/s wr, 95 op/s 2024-04-26T21:33:54.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:53 smithi001 ceph-mon[28054]: pgmap v110: 129 pgs: 129 active+clean; 49 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 752 KiB/s wr, 95 op/s 2024-04-26T21:33:54.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:53 smithi038 ceph-mon[32724]: pgmap v110: 129 pgs: 129 active+clean; 49 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 752 KiB/s wr, 95 op/s 2024-04-26T21:33:54.537 INFO:tasks.workunit.client.0.smithi001.stdout: 1 64309 3.26 MB/sec warmup 112 sec latency 20.819 ms 2024-04-26T21:33:55.537 INFO:tasks.workunit.client.0.smithi001.stdout: 1 64592 3.23 MB/sec warmup 113 sec latency 476.353 ms 2024-04-26T21:33:56.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:55 smithi186 ceph-mon[32991]: pgmap v111: 129 pgs: 129 active+clean; 49 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.2 MiB/s wr, 162 op/s 2024-04-26T21:33:56.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:55 smithi001 ceph-mon[28054]: pgmap v111: 129 pgs: 129 active+clean; 49 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.2 MiB/s wr, 162 op/s 2024-04-26T21:33:56.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:55 smithi038 ceph-mon[32724]: pgmap v111: 129 pgs: 129 active+clean; 49 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 1.2 MiB/s wr, 162 op/s 2024-04-26T21:33:56.537 INFO:tasks.workunit.client.0.smithi001.stdout: 1 64660 3.21 MB/sec warmup 114 sec latency 1248.832 ms 2024-04-26T21:33:57.537 INFO:tasks.workunit.client.0.smithi001.stdout: 1 64660 3.18 MB/sec warmup 115 sec latency 1143.417 ms 2024-04-26T21:33:58.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:57 smithi186 ceph-mon[32991]: pgmap v112: 129 pgs: 129 active+clean; 49 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 741 KiB/s wr, 89 op/s 2024-04-26T21:33:58.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:57 smithi001 ceph-mon[28054]: pgmap v112: 129 pgs: 129 active+clean; 49 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 741 KiB/s wr, 89 op/s 2024-04-26T21:33:58.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:57 smithi038 ceph-mon[32724]: pgmap v112: 129 pgs: 129 active+clean; 49 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 741 KiB/s wr, 89 op/s 2024-04-26T21:33:58.538 INFO:tasks.workunit.client.0.smithi001.stdout: 1 64660 3.15 MB/sec warmup 116 sec latency 2143.539 ms 2024-04-26T21:33:59.538 INFO:tasks.workunit.client.0.smithi001.stdout: 1 64660 3.12 MB/sec warmup 117 sec latency 3143.683 ms 2024-04-26T21:34:00.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:33:59 smithi186 ceph-mon[32991]: pgmap v113: 129 pgs: 129 active+clean; 49 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 741 KiB/s wr, 89 op/s 2024-04-26T21:34:00.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:33:59 smithi001 ceph-mon[28054]: pgmap v113: 129 pgs: 129 active+clean; 49 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 741 KiB/s wr, 89 op/s 2024-04-26T21:34:00.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:33:59 smithi038 ceph-mon[32724]: pgmap v113: 129 pgs: 129 active+clean; 49 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 741 KiB/s wr, 89 op/s 2024-04-26T21:34:00.538 INFO:tasks.workunit.client.0.smithi001.stdout: 1 64660 3.10 MB/sec warmup 118 sec latency 4143.834 ms 2024-04-26T21:34:01.538 INFO:tasks.workunit.client.0.smithi001.stdout: 1 65033 3.09 MB/sec warmup 119 sec latency 4417.031 ms 2024-04-26T21:34:02.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:01 smithi186 ceph-mon[32991]: pgmap v114: 129 pgs: 129 active+clean; 51 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-26T21:34:02.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:01 smithi001 ceph-mon[28054]: pgmap v114: 129 pgs: 129 active+clean; 51 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-26T21:34:02.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:01 smithi038 ceph-mon[32724]: pgmap v114: 129 pgs: 129 active+clean; 51 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 1.0 MiB/s wr, 111 op/s 2024-04-26T21:34:03.538 INFO:tasks.workunit.client.0.smithi001.stdout: 1 67641 10.87 MB/sec execute 1 sec latency 41.106 ms 2024-04-26T21:34:04.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:03 smithi186 ceph-mon[32991]: pgmap v115: 129 pgs: 129 active+clean; 51 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 727 KiB/s wr, 88 op/s 2024-04-26T21:34:04.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:03 smithi001 ceph-mon[28054]: pgmap v115: 129 pgs: 129 active+clean; 51 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 727 KiB/s wr, 88 op/s 2024-04-26T21:34:04.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:03 smithi038 ceph-mon[32724]: pgmap v115: 129 pgs: 129 active+clean; 51 MiB data, 588 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 727 KiB/s wr, 88 op/s 2024-04-26T21:34:04.538 INFO:tasks.workunit.client.0.smithi001.stdout: 1 67895 5.58 MB/sec execute 2 sec latency 23.681 ms 2024-04-26T21:34:05.538 INFO:tasks.workunit.client.0.smithi001.stdout: 1 68143 3.81 MB/sec execute 3 sec latency 635.059 ms 2024-04-26T21:34:06.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:05 smithi186 ceph-mon[32991]: pgmap v116: 129 pgs: 129 active+clean; 51 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 160 op/s 2024-04-26T21:34:06.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:05 smithi001 ceph-mon[28054]: pgmap v116: 129 pgs: 129 active+clean; 51 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 160 op/s 2024-04-26T21:34:06.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:05 smithi038 ceph-mon[32724]: pgmap v116: 129 pgs: 129 active+clean; 51 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 1.3 MiB/s wr, 160 op/s 2024-04-26T21:34:06.538 INFO:tasks.workunit.client.0.smithi001.stdout: 1 68304 2.97 MB/sec execute 4 sec latency 1383.884 ms 2024-04-26T21:34:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.656+0000 7f816d000640 1 -- 172.21.15.1:0/1658261057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8168077780 msgr2=0x7f8168113290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.656+0000 7f816d000640 1 --2- 172.21.15.1:0/1658261057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8168077780 0x7f8168113290 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8154009930 tx=0x7f815402a420 comp rx=0 tx=0).stop 2024-04-26T21:34:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.656+0000 7f816d000640 1 -- 172.21.15.1:0/1658261057 shutdown_connections 2024-04-26T21:34:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.656+0000 7f816d000640 1 --2- 172.21.15.1:0/1658261057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81681137d0 0x7f8168115bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.656+0000 7f816d000640 1 --2- 172.21.15.1:0/1658261057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8168077780 0x7f8168113290 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.656+0000 7f816d000640 1 --2- 172.21.15.1:0/1658261057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168076e60 0x7f8168077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.656+0000 7f816d000640 1 -- 172.21.15.1:0/1658261057 >> 172.21.15.1:0/1658261057 conn(0x7f8168100620 msgr2=0x7f8168102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.657+0000 7f816d000640 1 -- 172.21.15.1:0/1658261057 shutdown_connections 2024-04-26T21:34:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.657+0000 7f816d000640 1 -- 172.21.15.1:0/1658261057 wait complete. 2024-04-26T21:34:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.658+0000 7f816d000640 1 Processor -- start 2024-04-26T21:34:06.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.658+0000 7f816d000640 1 -- start start 2024-04-26T21:34:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.658+0000 7f816d000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168076e60 0x7f81681a5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.658+0000 7f816d000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168077780 0x7f81681a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.658+0000 7f816d000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81681137d0 0x7f816819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.658+0000 7f816d000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8168119240 con 0x7f8168076e60 2024-04-26T21:34:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.658+0000 7f816d000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f81681190c0 con 0x7f81681137d0 2024-04-26T21:34:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.658+0000 7f816d000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f81681193c0 con 0x7f8168077780 2024-04-26T21:34:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.659+0000 7f8166800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81681137d0 0x7f816819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.659+0000 7f8165e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168076e60 0x7f81681a5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.659+0000 7f8166800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81681137d0 0x7f816819f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53768/0 (socket says 172.21.15.1:53768) 2024-04-26T21:34:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.659+0000 7f8166800640 1 -- 172.21.15.1:0/600490272 learned_addr learned my addr 172.21.15.1:0/600490272 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.659+0000 7f8165400640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168077780 0x7f81681a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.659+0000 7f8165e00640 1 -- 172.21.15.1:0/600490272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168077780 msgr2=0x7f81681a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.659+0000 7f8165e00640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168077780 0x7f81681a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.659+0000 7f8165e00640 1 -- 172.21.15.1:0/600490272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81681137d0 msgr2=0x7f816819f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.659+0000 7f8165e00640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81681137d0 0x7f816819f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.659+0000 7f8165e00640 1 -- 172.21.15.1:0/600490272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816819ff60 con 0x7f8168076e60 2024-04-26T21:34:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.660+0000 7f8165e00640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168076e60 0x7f81681a5540 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f815800e9f0 tx=0x7f815800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.660+0000 7f815ea00640 1 -- 172.21.15.1:0/600490272 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815800ccc0 con 0x7f8168076e60 2024-04-26T21:34:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.660+0000 7f815ea00640 1 -- 172.21.15.1:0/600490272 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8158005020 con 0x7f8168076e60 2024-04-26T21:34:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.660+0000 7f816d000640 1 -- 172.21.15.1:0/600490272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81681a0250 con 0x7f8168076e60 2024-04-26T21:34:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.660+0000 7f815ea00640 1 -- 172.21.15.1:0/600490272 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8158004a00 con 0x7f8168076e60 2024-04-26T21:34:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.661+0000 7f816d000640 1 -- 172.21.15.1:0/600490272 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81681ac4c0 con 0x7f8168076e60 2024-04-26T21:34:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.664+0000 7f815ea00640 1 -- 172.21.15.1:0/600490272 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f8158004ba0 con 0x7f8168076e60 2024-04-26T21:34:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.665+0000 7f816fe46740 1 Processor -- start 2024-04-26T21:34:06.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.665+0000 7f816fe46740 1 -- start start 2024-04-26T21:34:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f816fe46740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfcc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f816fe46740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f816fe46740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfd93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f816fe46740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d94cf31380 con 0x55d94cfcbdd0 2024-04-26T21:34:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f816fe46740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d94cecdc30 con 0x55d94cfcc6f0 2024-04-26T21:34:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f816fe46740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d94cf2d340 con 0x55d94cfd6fb0 2024-04-26T21:34:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f8166800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfcc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f8165e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f8165400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfd93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f8166800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfcc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51934/0 (socket says 172.21.15.1:51934) 2024-04-26T21:34:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f8165400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfd93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50928/0 (socket says 172.21.15.1:50928) 2024-04-26T21:34:06.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f8166800640 1 -- 172.21.15.1:0/3678536566 learned_addr learned my addr 172.21.15.1:0/3678536566 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.666+0000 7f815ea00640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f814c0a3d90 0x7f814c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8165e00640 1 -- 172.21.15.1:0/3678536566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 msgr2=0x55d94cfd93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8165e00640 1 --2- 172.21.15.1:0/3678536566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfd93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8165e00640 1 -- 172.21.15.1:0/3678536566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 msgr2=0x55d94cfcc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8165e00640 1 --2- 172.21.15.1:0/3678536566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfcc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8165e00640 1 -- 172.21.15.1:0/3678536566 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d94ceef940 con 0x55d94cfcc6f0 2024-04-26T21:34:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8165400640 1 --2- 172.21.15.1:0/3678536566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfd93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8166800640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f814c0a3d90 0x7f814c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f815ea00640 1 -- 172.21.15.1:0/600490272 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f81580d0b40 con 0x7f8168076e60 2024-04-26T21:34:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8165e00640 1 --2- 172.21.15.1:0/3678536566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd6a70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f81580cde20 tx=0x7f815801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13cef643a1d80ffb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8153400640 1 -- 172.21.15.1:0/3678536566 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815809bfd0 con 0x55d94cfcc6f0 2024-04-26T21:34:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8153400640 1 -- 172.21.15.1:0/3678536566 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f815809cd70 con 0x55d94cfcc6f0 2024-04-26T21:34:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.667+0000 7f8153400640 1 -- 172.21.15.1:0/3678536566 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81580139c0 con 0x55d94cfcc6f0 2024-04-26T21:34:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.668+0000 7f816fe46740 1 -- 172.21.15.1:0/3678536566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 msgr2=0x55d94cfd6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.668+0000 7f8166800640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f814c0a3d90 0x7f814c0a6250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8160003cf0 tx=0x7f8160004a40 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:06.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.668+0000 7f816fe46740 1 --2- 172.21.15.1:0/3678536566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd6a70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f81580cde20 tx=0x7f815801f000 comp rx=0 tx=0).stop 2024-04-26T21:34:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.669+0000 7f816fe46740 1 -- 172.21.15.1:0/3678536566 shutdown_connections 2024-04-26T21:34:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.669+0000 7f816fe46740 1 --2- 172.21.15.1:0/3678536566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfd93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.669+0000 7f816fe46740 1 --2- 172.21.15.1:0/3678536566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd6a70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.669+0000 7f816fe46740 1 --2- 172.21.15.1:0/3678536566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfcc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.670 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.669+0000 7f816fe46740 1 -- 172.21.15.1:0/3678536566 >> 172.21.15.1:0/3678536566 conn(0x55d94cfc82e0 msgr2=0x55d94cfc86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.669+0000 7f816fe46740 1 -- 172.21.15.1:0/3678536566 shutdown_connections 2024-04-26T21:34:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.669+0000 7f816fe46740 1 -- 172.21.15.1:0/3678536566 wait complete. 2024-04-26T21:34:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.670+0000 7f816fe46740 1 -- start start 2024-04-26T21:34:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.670+0000 7f816fe46740 1 Processor -- start 2024-04-26T21:34:06.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.671+0000 7f816fe46740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfd3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.671+0000 7f816fe46740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.671+0000 7f816fe46740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfd4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.671+0000 7f816fe46740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d94cfd9b80 con 0x55d94cfcbdd0 2024-04-26T21:34:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.671+0000 7f816fe46740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d94ceb8c60 con 0x55d94cfcc6f0 2024-04-26T21:34:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.671+0000 7f816fe46740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d94cfd9d00 con 0x55d94cfd6fb0 2024-04-26T21:34:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.671+0000 7f816fe46740 10 client.0 _ll_register_callbacks cb 0x55d94cee4e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.671+0000 7f8165400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.671+0000 7f8165e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfd3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.672+0000 7f8165e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfd3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51946/0 (socket says 172.21.15.1:51946) 2024-04-26T21:34:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.672+0000 7f8165e00640 1 -- 172.21.15.1:0/2295565442 learned_addr learned my addr 172.21.15.1:0/2295565442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:06.673 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.672+0000 7f8166800640 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfd4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.672+0000 7f8165e00640 1 -- 172.21.15.1:0/2295565442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 msgr2=0x55d94cfd4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.672+0000 7f8165e00640 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfd4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.672+0000 7f8165e00640 1 -- 172.21.15.1:0/2295565442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 msgr2=0x55d94cfd43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.672+0000 7f8165e00640 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.672+0000 7f8165e00640 1 -- 172.21.15.1:0/2295565442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d94cec7510 con 0x55d94cfcbdd0 2024-04-26T21:34:06.674 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.673+0000 7f8165e00640 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfd3e90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f815809c600 tx=0x7f8158005460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de473ab733a65409 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.673+0000 7f8147e00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:34:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.673+0000 7f8147e00640 1 -- 172.21.15.1:0/2295565442 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815800f3c0 con 0x55d94cfcbdd0 2024-04-26T21:34:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.673+0000 7f816fe46740 1 -- 172.21.15.1:0/2295565442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f815809cd70 con 0x55d94cfcbdd0 2024-04-26T21:34:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.673+0000 7f8147e00640 1 -- 172.21.15.1:0/2295565442 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f81580051d0 con 0x55d94cfcbdd0 2024-04-26T21:34:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.673+0000 7f8147e00640 1 -- 172.21.15.1:0/2295565442 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f815801f3a0 con 0x55d94cfcbdd0 2024-04-26T21:34:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.673+0000 7f816fe46740 10 client.15663 fetch_fsmap learned FSMap version 32 2024-04-26T21:34:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.673+0000 7f816fe46740 1 -- 172.21.15.1:0/2295565442 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d94cc43210 con 0x55d94cfcbdd0 2024-04-26T21:34:06.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.674+0000 7f8147e00640 1 -- 172.21.15.1:0/2295565442 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f815809cd70 con 0x55d94cfcbdd0 2024-04-26T21:34:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.675+0000 7f8147e00640 1 -- 172.21.15.1:0/2295565442 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f81580d4140 con 0x55d94cfcbdd0 2024-04-26T21:34:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.676+0000 7f816fe46740 10 client.15663 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T21:34:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.676+0000 7f816fe46740 10 client.15663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:34:06.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.676+0000 7f816fe46740 20 client.15663 populate_metadata read hostname 'smithi001' 2024-04-26T21:34:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.676+0000 7f816fe46740 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d94cfe6a80 0x55d94cfe8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.676+0000 7f816fe46740 4 client.15663 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:06.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.676+0000 7f816fe46740 1 -- 172.21.15.1:0/2295565442 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d94cf3ef90 con 0x55d94cfe6a80 2024-04-26T21:34:06.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.678+0000 7f8165400640 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d94cfe6a80 0x55d94cfe8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.682+0000 7f8165400640 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d94cfe6a80 0x55d94cfe8e70 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58fe71776d9e54ed server_cookie=7a1d2eb4bb57e4d3 in_seq=0 out_seq=0 2024-04-26T21:34:06.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.682+0000 7f8147e00640 10 client.15663 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.684+0000 7f8147e00640 1 -- 172.21.15.1:0/2295565442 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d94cf3ef90 con 0x55d94cfe6a80 2024-04-26T21:34:06.684 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.684+0000 7f8147e00640 10 client.15663 handle_command_reply: tid=0 2024-04-26T21:34:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.684+0000 7f816fe46740 1 client.15663 shutdown 2024-04-26T21:34:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f816fe46740 1 -- 172.21.15.1:0/2295565442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 msgr2=0x55d94cfd3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f816fe46740 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfd3e90 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f815809c600 tx=0x7f8158005460 comp rx=0 tx=0).stop 2024-04-26T21:34:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f816fe46740 1 -- 172.21.15.1:0/2295565442 shutdown_connections 2024-04-26T21:34:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f816fe46740 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d94cfe6a80 0x55d94cfe8e70 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f816fe46740 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfd4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f816fe46740 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f816fe46740 1 --2- 172.21.15.1:0/2295565442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfd3e90 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f816fe46740 1 -- 172.21.15.1:0/2295565442 >> 172.21.15.1:0/2295565442 conn(0x55d94cfc82e0 msgr2=0x55d94cfce010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f8147e00640 0 client.15663 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f816fe46740 1 -- 172.21.15.1:0/2295565442 shutdown_connections 2024-04-26T21:34:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.685+0000 7f816fe46740 1 -- 172.21.15.1:0/2295565442 wait complete. 2024-04-26T21:34:06.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.686+0000 7f816fe46740 20 client.15663 trim_cache size 0 max 16384 2024-04-26T21:34:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.704+0000 7f816fe46740 1 Processor -- start 2024-04-26T21:34:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.704+0000 7f816fe46740 1 -- start start 2024-04-26T21:34:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f816fe46740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfcbdd0 0x55d94cfe2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.705 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f816fe46740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f816fe46740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfd6fb0 0x55d94cfc6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f816fe46740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f81681b8ef0 con 0x55d94cfcc6f0 2024-04-26T21:34:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f816fe46740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f81681b6700 con 0x55d94cfd6fb0 2024-04-26T21:34:06.706 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f816fe46740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f81681b4550 con 0x55d94cfcbdd0 2024-04-26T21:34:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f8166800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfd6fb0 0x55d94cfc6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f8166800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfd6fb0 0x55d94cfc6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53786/0 (socket says 172.21.15.1:53786) 2024-04-26T21:34:06.707 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f8166800640 1 -- 172.21.15.1:0/1860022860 learned_addr learned my addr 172.21.15.1:0/1860022860 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f8165e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfcbdd0 0x55d94cfe2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.705+0000 7f8165400640 1 --2- 172.21.15.1:0/1860022860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.706+0000 7f8166800640 1 -- 172.21.15.1:0/1860022860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfcbdd0 msgr2=0x55d94cfe2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.706+0000 7f8166800640 1 --2- 172.21.15.1:0/1860022860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfcbdd0 0x55d94cfe2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.706+0000 7f8166800640 1 -- 172.21.15.1:0/1860022860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcc6f0 msgr2=0x55d94cfd3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.706+0000 7f8166800640 1 --2- 172.21.15.1:0/1860022860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.706+0000 7f8166800640 1 -- 172.21.15.1:0/1860022860 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d94cfc7a60 con 0x55d94cfd6fb0 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.706+0000 7f8166800640 1 --2- 172.21.15.1:0/1860022860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfd6fb0 0x55d94cfc6bb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f816001f780 tx=0x7f8160013030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a7fc10248b2d187 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.706+0000 7f8152a00640 1 -- 172.21.15.1:0/1860022860 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816001f250 con 0x55d94cfd6fb0 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.706+0000 7f8152a00640 1 -- 172.21.15.1:0/1860022860 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f816001c440 con 0x55d94cfd6fb0 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.706+0000 7f8152a00640 1 -- 172.21.15.1:0/1860022860 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160010e60 con 0x55d94cfd6fb0 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.707+0000 7f816fe46740 1 -- 172.21.15.1:0/1860022860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfd6fb0 msgr2=0x55d94cfc6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:06.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.707+0000 7f816fe46740 1 --2- 172.21.15.1:0/1860022860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfd6fb0 0x55d94cfc6bb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f816001f780 tx=0x7f8160013030 comp rx=0 tx=0).stop 2024-04-26T21:34:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.707+0000 7f816fe46740 1 -- 172.21.15.1:0/1860022860 shutdown_connections 2024-04-26T21:34:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.707+0000 7f816fe46740 1 --2- 172.21.15.1:0/1860022860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfd6fb0 0x55d94cfc6bb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.707+0000 7f816fe46740 1 --2- 172.21.15.1:0/1860022860 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcc6f0 0x55d94cfd3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.707+0000 7f816fe46740 1 --2- 172.21.15.1:0/1860022860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfcbdd0 0x55d94cfe2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.707+0000 7f816fe46740 1 -- 172.21.15.1:0/1860022860 >> 172.21.15.1:0/1860022860 conn(0x55d94cfc82e0 msgr2=0x55d94cfd46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:06.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.708+0000 7f816fe46740 1 -- 172.21.15.1:0/1860022860 shutdown_connections 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.708+0000 7f816fe46740 1 -- 172.21.15.1:0/1860022860 wait complete. 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.709+0000 7f816fe46740 1 -- start start 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.709+0000 7f816fe46740 1 Processor -- start 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.709+0000 7f816fe46740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfdaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.709+0000 7f816fe46740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfdb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.709+0000 7f816fe46740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfdb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.709+0000 7f816fe46740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f81681b4bc0 con 0x55d94cfcbdd0 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.709+0000 7f816fe46740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f81681b6030 con 0x55d94cfcc6f0 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.709+0000 7f816fe46740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f81681b5290 con 0x55d94cfd6fb0 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.709+0000 7f816fe46740 10 client.0 _ll_register_callbacks cb 0x7f8160009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:06.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfdaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfdb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfdaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51978/0 (socket says 172.21.15.1:51978) 2024-04-26T21:34:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfdb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53802/0 (socket says 172.21.15.1:53802) 2024-04-26T21:34:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165e00640 1 -- 172.21.15.1:0/3294340546 learned_addr learned my addr 172.21.15.1:0/3294340546 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8166800640 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfdb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165400640 1 -- 172.21.15.1:0/3294340546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 msgr2=0x55d94cfdb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165400640 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfdb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165400640 1 -- 172.21.15.1:0/3294340546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 msgr2=0x55d94cfdaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165400640 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfdaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165400640 1 -- 172.21.15.1:0/3294340546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d94cd0f620 con 0x55d94cfcc6f0 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165e00640 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfdaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8166800640 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfdb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8165400640 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfdb010 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f815402a940 tx=0x7f8154004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4535896fe6cf1651 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.710+0000 7f8147e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.711+0000 7f8147e00640 1 -- 172.21.15.1:0/3294340546 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154035040 con 0x55d94cfcc6f0 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.711+0000 7f816fe46740 1 -- 172.21.15.1:0/3294340546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f816001c440 con 0x55d94cfcc6f0 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.711+0000 7f8147e00640 1 -- 172.21.15.1:0/3294340546 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f8154005340 con 0x55d94cfcc6f0 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.711+0000 7f8147e00640 1 -- 172.21.15.1:0/3294340546 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8154055050 con 0x55d94cfcc6f0 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.711+0000 7f816fe46740 10 client.25823 fetch_fsmap learned FSMap version 32 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.711+0000 7f816fe46740 1 -- 172.21.15.1:0/3294340546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d94ceebcb0 con 0x55d94cfcc6f0 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.711+0000 7f8147e00640 1 -- 172.21.15.1:0/3294340546 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f816001c440 con 0x55d94cfcc6f0 2024-04-26T21:34:06.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.711+0000 7f8147e00640 1 -- 172.21.15.1:0/3294340546 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7f8154004870 con 0x55d94cfcc6f0 2024-04-26T21:34:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.712+0000 7f816fe46740 10 client.25823 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T21:34:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.712+0000 7f816fe46740 10 client.25823 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:34:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.712+0000 7f816fe46740 20 client.25823 populate_metadata read hostname 'smithi001' 2024-04-26T21:34:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.712+0000 7f816fe46740 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d94d04d590 0x55d94d04f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.712+0000 7f816fe46740 4 client.25823 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:34:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.712+0000 7f816fe46740 1 -- 172.21.15.1:0/3294340546 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d94cf38e00 con 0x55d94d04d590 2024-04-26T21:34:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.712+0000 7f8165e00640 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d94d04d590 0x55d94d04f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.712+0000 7f8165e00640 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d94d04d590 0x55d94d04f9b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b58a397e356c1b3 server_cookie=ef91812db52d6786 in_seq=0 out_seq=0 2024-04-26T21:34:06.716 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.712+0000 7f8147e00640 10 client.25823 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f8147e00640 1 -- 172.21.15.1:0/3294340546 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d94cf38e00 con 0x55d94d04d590 2024-04-26T21:34:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f8147e00640 10 client.25823 handle_command_reply: tid=0 2024-04-26T21:34:06.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f816fe46740 1 client.25823 shutdown 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f816fe46740 1 -- 172.21.15.1:0/3294340546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 msgr2=0x55d94cfdb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f816fe46740 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfdb010 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f815402a940 tx=0x7f8154004610 comp rx=0 tx=0).stop 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f816fe46740 1 -- 172.21.15.1:0/3294340546 shutdown_connections 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f816fe46740 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d94d04d590 0x55d94d04f9b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f816fe46740 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d94cfd6fb0 0x55d94cfdb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f816fe46740 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d94cfcc6f0 0x55d94cfdb010 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f816fe46740 1 --2- 172.21.15.1:0/3294340546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d94cfcbdd0 0x55d94cfdaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f8147e00640 0 client.25823 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.754+0000 7f816fe46740 1 -- 172.21.15.1:0/3294340546 >> 172.21.15.1:0/3294340546 conn(0x55d94cfc82e0 msgr2=0x55d94cfd3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.755+0000 7f816fe46740 1 -- 172.21.15.1:0/3294340546 shutdown_connections 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.755+0000 7f816fe46740 1 -- 172.21.15.1:0/3294340546 wait complete. 2024-04-26T21:34:06.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.755+0000 7f816fe46740 20 client.25823 trim_cache size 0 max 16384 2024-04-26T21:34:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:34:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:34:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:34:06.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 -- 172.21.15.1:0/600490272 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f814c0a3d90 msgr2=0x7f814c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f814c0a3d90 0x7f814c0a6250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8160003cf0 tx=0x7f8160004a40 comp rx=0 tx=0).stop 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 -- 172.21.15.1:0/600490272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168076e60 msgr2=0x7f81681a5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168076e60 0x7f81681a5540 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f815800e9f0 tx=0x7f815800eeb0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 -- 172.21.15.1:0/600490272 shutdown_connections 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f814c0a3d90 0x7f814c0a6250 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168077780 0x7f81681a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f81681137d0 0x7f816819f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 --2- 172.21.15.1:0/600490272 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168076e60 0x7f81681a5540 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 -- 172.21.15.1:0/600490272 >> 172.21.15.1:0/600490272 conn(0x7f8168100620 msgr2=0x7f8168114f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 -- 172.21.15.1:0/600490272 shutdown_connections 2024-04-26T21:34:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:06.756+0000 7f816d000640 1 -- 172.21.15.1:0/600490272 wait complete. 2024-04-26T21:34:07.539 INFO:tasks.workunit.client.0.smithi001.stdout: 1 69849 3.66 MB/sec execute 5 sec latency 49.171 ms 2024-04-26T21:34:08.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:07 smithi186 ceph-mon[32991]: pgmap v117: 129 pgs: 129 active+clean; 51 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 868 KiB/s wr, 94 op/s 2024-04-26T21:34:08.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:07 smithi001 ceph-mon[28054]: pgmap v117: 129 pgs: 129 active+clean; 51 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 868 KiB/s wr, 94 op/s 2024-04-26T21:34:08.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:07 smithi038 ceph-mon[32724]: pgmap v117: 129 pgs: 129 active+clean; 51 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 868 KiB/s wr, 94 op/s 2024-04-26T21:34:08.542 INFO:tasks.workunit.client.0.smithi001.stdout: 1 71051 4.81 MB/sec execute 6 sec latency 31.399 ms 2024-04-26T21:34:09.539 INFO:tasks.workunit.client.0.smithi001.stdout: 1 71376 4.33 MB/sec execute 7 sec latency 27.774 ms 2024-04-26T21:34:10.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:09 smithi186 ceph-mon[32991]: pgmap v118: 129 pgs: 129 active+clean; 51 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 868 KiB/s wr, 94 op/s 2024-04-26T21:34:10.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:09 smithi001 ceph-mon[28054]: pgmap v118: 129 pgs: 129 active+clean; 51 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 868 KiB/s wr, 94 op/s 2024-04-26T21:34:10.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:09 smithi038 ceph-mon[32724]: pgmap v118: 129 pgs: 129 active+clean; 51 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 868 KiB/s wr, 94 op/s 2024-04-26T21:34:10.539 INFO:tasks.workunit.client.0.smithi001.stdout: 1 71685 3.82 MB/sec execute 8 sec latency 429.676 ms 2024-04-26T21:34:11.539 INFO:tasks.workunit.client.0.smithi001.stdout: 1 71886 3.46 MB/sec execute 9 sec latency 1177.415 ms 2024-04-26T21:34:12.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:11 smithi186 ceph-mon[32991]: pgmap v119: 129 pgs: 129 active+clean; 53 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:34:12.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:11 smithi001 ceph-mon[28054]: pgmap v119: 129 pgs: 129 active+clean; 53 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:34:12.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:11 smithi038 ceph-mon[32724]: pgmap v119: 129 pgs: 129 active+clean; 53 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:34:12.540 INFO:tasks.workunit.client.0.smithi001.stdout: 1 73404 3.74 MB/sec execute 10 sec latency 31.570 ms 2024-04-26T21:34:13.540 INFO:tasks.workunit.client.0.smithi001.stdout: 1 74371 4.08 MB/sec execute 11 sec latency 76.410 ms 2024-04-26T21:34:14.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:13 smithi186 ceph-mon[32991]: pgmap v120: 129 pgs: 129 active+clean; 53 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 1.0 MiB/s wr, 133 op/s 2024-04-26T21:34:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:13 smithi001 ceph-mon[28054]: pgmap v120: 129 pgs: 129 active+clean; 53 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 1.0 MiB/s wr, 133 op/s 2024-04-26T21:34:14.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:13 smithi038 ceph-mon[32724]: pgmap v120: 129 pgs: 129 active+clean; 53 MiB data, 589 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 1.0 MiB/s wr, 133 op/s 2024-04-26T21:34:14.541 INFO:tasks.workunit.client.0.smithi001.stdout: 1 74932 4.12 MB/sec execute 12 sec latency 25.782 ms 2024-04-26T21:34:15.540 INFO:tasks.workunit.client.0.smithi001.stdout: 1 75210 3.81 MB/sec execute 13 sec latency 232.850 ms 2024-04-26T21:34:16.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:15 smithi186 ceph-mon[32991]: pgmap v121: 129 pgs: 129 active+clean; 54 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-04-26T21:34:16.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:15 smithi001 ceph-mon[28054]: pgmap v121: 129 pgs: 129 active+clean; 54 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-04-26T21:34:16.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:15 smithi038 ceph-mon[32724]: pgmap v121: 129 pgs: 129 active+clean; 54 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 1.5 MiB/s wr, 185 op/s 2024-04-26T21:34:16.541 INFO:tasks.workunit.client.0.smithi001.stdout: 1 75266 3.55 MB/sec execute 14 sec latency 993.940 ms 2024-04-26T21:34:17.541 INFO:tasks.workunit.client.0.smithi001.stdout: 1 75266 3.31 MB/sec execute 15 sec latency 1169.002 ms 2024-04-26T21:34:18.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:17 smithi186 ceph-mon[32991]: pgmap v122: 129 pgs: 129 active+clean; 54 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 952 KiB/s wr, 113 op/s 2024-04-26T21:34:18.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:17 smithi001 ceph-mon[28054]: pgmap v122: 129 pgs: 129 active+clean; 54 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 952 KiB/s wr, 113 op/s 2024-04-26T21:34:18.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:17 smithi038 ceph-mon[32724]: pgmap v122: 129 pgs: 129 active+clean; 54 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 952 KiB/s wr, 113 op/s 2024-04-26T21:34:18.541 INFO:tasks.workunit.client.0.smithi001.stdout: 1 75266 3.11 MB/sec execute 16 sec latency 2169.189 ms 2024-04-26T21:34:19.541 INFO:tasks.workunit.client.0.smithi001.stdout: 1 75266 2.92 MB/sec execute 17 sec latency 3169.358 ms 2024-04-26T21:34:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.902+0000 7feb9a400640 1 -- 172.21.15.1:0/698193526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94104940 msgr2=0x7feb94104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.902+0000 7feb9a400640 1 --2- 172.21.15.1:0/698193526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94104940 0x7feb94104dc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feb80009930 tx=0x7feb8002a440 comp rx=0 tx=0).stop 2024-04-26T21:34:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.903+0000 7feb9a400640 1 -- 172.21.15.1:0/698193526 shutdown_connections 2024-04-26T21:34:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.903+0000 7feb9a400640 1 --2- 172.21.15.1:0/698193526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb94105300 0x7feb94111810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.903+0000 7feb9a400640 1 --2- 172.21.15.1:0/698193526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94104940 0x7feb94104dc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.903+0000 7feb9a400640 1 --2- 172.21.15.1:0/698193526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb9410a950 0x7feb9410ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.903+0000 7feb9a400640 1 -- 172.21.15.1:0/698193526 >> 172.21.15.1:0/698193526 conn(0x7feb94100410 msgr2=0x7feb94102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.903+0000 7feb9a400640 1 -- 172.21.15.1:0/698193526 shutdown_connections 2024-04-26T21:34:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.903+0000 7feb9a400640 1 -- 172.21.15.1:0/698193526 wait complete. 2024-04-26T21:34:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.904+0000 7feb9a400640 1 Processor -- start 2024-04-26T21:34:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.904+0000 7feb9a400640 1 -- start start 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.904+0000 7feb9a400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94104940 0x7feb9419b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.904+0000 7feb9a400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94105300 0x7feb9419b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.904+0000 7feb9a400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb9410a950 0x7feb9419fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.904+0000 7feb9a400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7feb94114c30 con 0x7feb94104940 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.904+0000 7feb9a400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb94114ab0 con 0x7feb94105300 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.904+0000 7feb9a400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb94114db0 con 0x7feb9410a950 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb99000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94104940 0x7feb9419b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb99a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb9410a950 0x7feb9419fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb93e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94105300 0x7feb9419b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb93e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94105300 0x7feb9419b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50604/0 (socket says 172.21.15.1:50604) 2024-04-26T21:34:19.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb99000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94104940 0x7feb9419b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46886/0 (socket says 172.21.15.1:46886) 2024-04-26T21:34:19.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb99a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb9410a950 0x7feb9419fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:46544/0 (socket says 172.21.15.1:46544) 2024-04-26T21:34:19.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb93e00640 1 -- 172.21.15.1:0/3412870787 learned_addr learned my addr 172.21.15.1:0/3412870787 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:19.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb99a00640 1 -- 172.21.15.1:0/3412870787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94105300 msgr2=0x7feb9419b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:19.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb99a00640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94105300 0x7feb9419b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb99a00640 1 -- 172.21.15.1:0/3412870787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94104940 msgr2=0x7feb9419b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:19.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb99a00640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94104940 0x7feb9419b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb99a00640 1 -- 172.21.15.1:0/3412870787 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb941a0180 con 0x7feb9410a950 2024-04-26T21:34:19.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.905+0000 7feb93e00640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94105300 0x7feb9419b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:19.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.906+0000 7feb99a00640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb9410a950 0x7feb9419fa50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009fd0 tx=0x7feb8c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.907+0000 7feb91600640 1 -- 172.21.15.1:0/3412870787 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c00ce20 con 0x7feb9410a950 2024-04-26T21:34:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.907+0000 7feb91600640 1 -- 172.21.15.1:0/3412870787 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8c005020 con 0x7feb9410a950 2024-04-26T21:34:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.907+0000 7feb91600640 1 -- 172.21.15.1:0/3412870787 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c010690 con 0x7feb9410a950 2024-04-26T21:34:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.907+0000 7feb9a400640 1 -- 172.21.15.1:0/3412870787 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb941a0470 con 0x7feb9410a950 2024-04-26T21:34:19.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.907+0000 7feb9a400640 1 -- 172.21.15.1:0/3412870787 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb941a7d00 con 0x7feb9410a950 2024-04-26T21:34:19.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.910+0000 7feb91600640 1 -- 172.21.15.1:0/3412870787 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7feb8c0054d0 con 0x7feb9410a950 2024-04-26T21:34:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.910+0000 7feb9e81d740 1 Processor -- start 2024-04-26T21:34:19.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.910+0000 7feb9e81d740 1 -- start start 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da0263c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 0x55c0da030c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da0335b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb9e81d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c0d9f8b380 con 0x55c0da025fe0 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb9e81d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c0d9f27c30 con 0x55c0da026900 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb9e81d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c0d9f87340 con 0x55c0da0311c0 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb93e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da0263c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb93e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da0263c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46898/0 (socket says 172.21.15.1:46898) 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb93e00640 1 -- 172.21.15.1:0/3837559634 learned_addr learned my addr 172.21.15.1:0/3837559634 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb99000640 1 --2- 172.21.15.1:0/3837559634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 0x55c0da030c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.911+0000 7feb91600640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb7c0a3d80 0x7feb7c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.912+0000 7feb93e00640 1 --2- 172.21.15.1:0/3837559634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da0335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.912+0000 7feb91600640 1 -- 172.21.15.1:0/3412870787 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7feb8c0d12c0 con 0x7feb9410a950 2024-04-26T21:34:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.912+0000 7feb99a00640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb7c0a3d80 0x7feb7c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.912+0000 7feb99000640 1 -- 172.21.15.1:0/3837559634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 msgr2=0x55c0da0335b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.912+0000 7feb99000640 1 --2- 172.21.15.1:0/3837559634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da0335b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.912+0000 7feb99000640 1 -- 172.21.15.1:0/3837559634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 msgr2=0x55c0da0263c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.912+0000 7feb99000640 1 --2- 172.21.15.1:0/3837559634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da0263c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.912+0000 7feb99000640 1 -- 172.21.15.1:0/3837559634 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0d9f49940 con 0x55c0da026900 2024-04-26T21:34:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.912+0000 7feb93e00640 1 --2- 172.21.15.1:0/3837559634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da0263c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:19.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.912+0000 7feb99000640 1 --2- 172.21.15.1:0/3837559634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 0x55c0da030c80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feb88009c20 tx=0x7feb88011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6aa7d9dd3902c687 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.913+0000 7feb86000640 1 -- 172.21.15.1:0/3837559634 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb88004050 con 0x55c0da026900 2024-04-26T21:34:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.913+0000 7feb86000640 1 -- 172.21.15.1:0/3837559634 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb880054d0 con 0x55c0da026900 2024-04-26T21:34:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.913+0000 7feb99a00640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb7c0a3d80 0x7feb7c0a6240 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0997c0 tx=0x7feb8c0057c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.913+0000 7feb86000640 1 -- 172.21.15.1:0/3837559634 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb88004c80 con 0x55c0da026900 2024-04-26T21:34:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.913+0000 7feb9e81d740 1 -- 172.21.15.1:0/3837559634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 msgr2=0x55c0da030c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.913+0000 7feb9e81d740 1 --2- 172.21.15.1:0/3837559634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 0x55c0da030c80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feb88009c20 tx=0x7feb88011810 comp rx=0 tx=0).stop 2024-04-26T21:34:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.913+0000 7feb9e81d740 1 -- 172.21.15.1:0/3837559634 shutdown_connections 2024-04-26T21:34:19.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.913+0000 7feb9e81d740 1 --2- 172.21.15.1:0/3837559634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da0335b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.914+0000 7feb9e81d740 1 --2- 172.21.15.1:0/3837559634 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 0x55c0da030c80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.914+0000 7feb9e81d740 1 --2- 172.21.15.1:0/3837559634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da0263c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.914+0000 7feb9e81d740 1 -- 172.21.15.1:0/3837559634 >> 172.21.15.1:0/3837559634 conn(0x55c0da0224f0 msgr2=0x55c0da0228d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.914+0000 7feb9e81d740 1 -- 172.21.15.1:0/3837559634 shutdown_connections 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.914+0000 7feb9e81d740 1 -- 172.21.15.1:0/3837559634 wait complete. 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.914+0000 7feb9e81d740 1 -- start start 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.914+0000 7feb9e81d740 1 Processor -- start 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da02e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 0x55c0da02e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da02eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb9e81d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c0da033df0 con 0x55c0da025fe0 2024-04-26T21:34:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb9e81d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c0d9f12c60 con 0x55c0da026900 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb9e81d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c0da033f70 con 0x55c0da0311c0 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb9e81d740 10 client.0 _ll_register_callbacks cb 0x55c0d9f3ee50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb93e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da02e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb93e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da02e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46900/0 (socket says 172.21.15.1:46900) 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb93e00640 1 -- 172.21.15.1:0/3477048031 learned_addr learned my addr 172.21.15.1:0/3477048031 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb99000640 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 0x55c0da02e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb93e00640 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da02eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb93e00640 1 -- 172.21.15.1:0/3477048031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 msgr2=0x55c0da02eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb93e00640 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da02eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.915+0000 7feb93e00640 1 -- 172.21.15.1:0/3477048031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 msgr2=0x55c0da02e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.916+0000 7feb93e00640 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 0x55c0da02e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.916+0000 7feb93e00640 1 -- 172.21.15.1:0/3477048031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0d9f21510 con 0x55c0da025fe0 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.916+0000 7feb99000640 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 0x55c0da02e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.916+0000 7feb93e00640 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da02e0a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7feb80006010 tx=0x7feb80005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8989102ce57d8e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.916+0000 7feb7aa00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.916+0000 7feb7aa00640 1 -- 172.21.15.1:0/3477048031 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb80041070 con 0x55c0da025fe0 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.916+0000 7feb7aa00640 1 -- 172.21.15.1:0/3477048031 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7feb80004980 con 0x55c0da025fe0 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.916+0000 7feb9e81d740 1 -- 172.21.15.1:0/3477048031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb880054d0 con 0x55c0da025fe0 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb7aa00640 1 -- 172.21.15.1:0/3477048031 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feb80035040 con 0x55c0da025fe0 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb9e81d740 10 client.15681 fetch_fsmap learned FSMap version 32 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb9e81d740 1 -- 172.21.15.1:0/3477048031 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c0d9c9d210 con 0x55c0da025fe0 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb7aa00640 1 -- 172.21.15.1:0/3477048031 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb880054d0 con 0x55c0da025fe0 2024-04-26T21:34:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb7aa00640 1 -- 172.21.15.1:0/3477048031 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7feb800314a0 con 0x55c0da025fe0 2024-04-26T21:34:19.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb9e81d740 10 client.15681 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T21:34:19.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb9e81d740 10 client.15681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:34:19.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb9e81d740 20 client.15681 populate_metadata read hostname 'smithi001' 2024-04-26T21:34:19.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb9e81d740 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c0da040e80 0x55c0da043270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb9e81d740 4 client.15681 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:19.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.917+0000 7feb9e81d740 1 -- 172.21.15.1:0/3477048031 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0d9f98f90 con 0x55c0da040e80 2024-04-26T21:34:19.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.918+0000 7feb99000640 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c0da040e80 0x55c0da043270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.918+0000 7feb99000640 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c0da040e80 0x55c0da043270 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e741f7689aafbd0e server_cookie=459eaea8d3400437 in_seq=0 out_seq=0 2024-04-26T21:34:19.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.918+0000 7feb7aa00640 10 client.15681 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:19.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.919+0000 7feb7aa00640 1 -- 172.21.15.1:0/3477048031 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0d9f98f90 con 0x55c0da040e80 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.919+0000 7feb7aa00640 10 client.15681 handle_command_reply: tid=0 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 client.15681 shutdown 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 -- 172.21.15.1:0/3477048031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 msgr2=0x55c0da02e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da02e0a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7feb80006010 tx=0x7feb80005a20 comp rx=0 tx=0).stop 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 -- 172.21.15.1:0/3477048031 shutdown_connections 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c0da040e80 0x55c0da043270 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da02eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da026900 0x55c0da02e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 --2- 172.21.15.1:0/3477048031 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da025fe0 0x55c0da02e0a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb7aa00640 0 client.15681 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 -- 172.21.15.1:0/3477048031 >> 172.21.15.1:0/3477048031 conn(0x55c0da0224f0 msgr2=0x55c0da028270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 -- 172.21.15.1:0/3477048031 shutdown_connections 2024-04-26T21:34:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 1 -- 172.21.15.1:0/3477048031 wait complete. 2024-04-26T21:34:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.920+0000 7feb9e81d740 20 client.15681 trim_cache size 0 max 16384 2024-04-26T21:34:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb9e81d740 1 Processor -- start 2024-04-26T21:34:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb9e81d740 1 -- start start 2024-04-26T21:34:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 0x55c0da03cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da048b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb9e81d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b47c0 con 0x55c0da026900 2024-04-26T21:34:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb9e81d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b1fd0 con 0x55c0da025fe0 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb9e81d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb941afe20 con 0x55c0da0311c0 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb99000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb99000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46914/0 (socket says 172.21.15.1:46914) 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb99000640 1 -- 172.21.15.1:0/467569364 learned_addr learned my addr 172.21.15.1:0/467569364 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.937+0000 7feb93e00640 1 --2- 172.21.15.1:0/467569364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 0x55c0da03cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.938+0000 7feb93e00640 1 --2- 172.21.15.1:0/467569364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da048b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.938+0000 7feb99000640 1 -- 172.21.15.1:0/467569364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 msgr2=0x55c0da048b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.938+0000 7feb99000640 1 --2- 172.21.15.1:0/467569364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da048b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.938+0000 7feb99000640 1 -- 172.21.15.1:0/467569364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 msgr2=0x55c0da03cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.938+0000 7feb99000640 1 --2- 172.21.15.1:0/467569364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 0x55c0da03cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.938+0000 7feb99000640 1 -- 172.21.15.1:0/467569364 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0da026730 con 0x55c0da026900 2024-04-26T21:34:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.938+0000 7feb93e00640 1 --2- 172.21.15.1:0/467569364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da048b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.938+0000 7feb99000640 1 --2- 172.21.15.1:0/467569364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02fed0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feb88009730 tx=0x7feb88005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8a7d6df43b4f8a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.938+0000 7feb85600640 1 -- 172.21.15.1:0/467569364 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb88009820 con 0x55c0da026900 2024-04-26T21:34:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb85600640 1 -- 172.21.15.1:0/467569364 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb880054e0 con 0x55c0da026900 2024-04-26T21:34:19.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb85600640 1 -- 172.21.15.1:0/467569364 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb88004910 con 0x55c0da026900 2024-04-26T21:34:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb9e81d740 1 -- 172.21.15.1:0/467569364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 msgr2=0x55c0da02fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb9e81d740 1 --2- 172.21.15.1:0/467569364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02fed0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feb88009730 tx=0x7feb88005f90 comp rx=0 tx=0).stop 2024-04-26T21:34:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb9e81d740 1 -- 172.21.15.1:0/467569364 shutdown_connections 2024-04-26T21:34:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb9e81d740 1 --2- 172.21.15.1:0/467569364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da048b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb9e81d740 1 --2- 172.21.15.1:0/467569364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02fed0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb9e81d740 1 --2- 172.21.15.1:0/467569364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 0x55c0da03cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb9e81d740 1 -- 172.21.15.1:0/467569364 >> 172.21.15.1:0/467569364 conn(0x55c0da0224f0 msgr2=0x55c0da02f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:19.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb9e81d740 1 -- 172.21.15.1:0/467569364 shutdown_connections 2024-04-26T21:34:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.939+0000 7feb9e81d740 1 -- 172.21.15.1:0/467569364 wait complete. 2024-04-26T21:34:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.940+0000 7feb9e81d740 1 -- start start 2024-04-26T21:34:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.940+0000 7feb9e81d740 1 Processor -- start 2024-04-26T21:34:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.940+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 0x55c0da02af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.940+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.940+0000 7feb9e81d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da02b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.940+0000 7feb9e81d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b0490 con 0x55c0da026900 2024-04-26T21:34:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.940+0000 7feb9e81d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b1900 con 0x55c0da025fe0 2024-04-26T21:34:19.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.940+0000 7feb9e81d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b0b60 con 0x55c0da0311c0 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.940+0000 7feb9e81d740 10 client.0 _ll_register_callbacks cb 0x7feb80009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb99000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb99000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46918/0 (socket says 172.21.15.1:46918) 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb99000640 1 -- 172.21.15.1:0/62647679 learned_addr learned my addr 172.21.15.1:0/62647679 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb93e00640 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 0x55c0da02af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb93e00640 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da02b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb99000640 1 -- 172.21.15.1:0/62647679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 msgr2=0x55c0da02b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb99000640 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da02b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb99000640 1 -- 172.21.15.1:0/62647679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 msgr2=0x55c0da02af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb99000640 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 0x55c0da02af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb99000640 1 -- 172.21.15.1:0/62647679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0d9d69620 con 0x55c0da026900 2024-04-26T21:34:19.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb93e00640 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 0x55c0da02af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb99000640 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02b440 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7feb88009250 tx=0x7feb88009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394e45c19e532e29 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.941+0000 7feb7aa00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.942+0000 7feb7aa00640 1 -- 172.21.15.1:0/62647679 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8802e3d0 con 0x55c0da026900 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.942+0000 7feb7aa00640 1 -- 172.21.15.1:0/62647679 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7feb8802e570 con 0x55c0da026900 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.942+0000 7feb9e81d740 1 -- 172.21.15.1:0/62647679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb880054e0 con 0x55c0da026900 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.942+0000 7feb7aa00640 1 -- 172.21.15.1:0/62647679 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7feb88045050 con 0x55c0da026900 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.942+0000 7feb9e81d740 10 client.15687 fetch_fsmap learned FSMap version 32 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.942+0000 7feb9e81d740 1 -- 172.21.15.1:0/62647679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c0d9f45cb0 con 0x55c0da026900 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.942+0000 7feb7aa00640 1 -- 172.21.15.1:0/62647679 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb880054e0 con 0x55c0da026900 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.942+0000 7feb7aa00640 1 -- 172.21.15.1:0/62647679 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7feb88040030 con 0x55c0da026900 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.942+0000 7feb9e81d740 10 client.15687 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T21:34:19.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.943+0000 7feb9e81d740 10 client.15687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:34:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.943+0000 7feb9e81d740 20 client.15687 populate_metadata read hostname 'smithi001' 2024-04-26T21:34:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.943+0000 7feb9e81d740 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c0da0a7a20 0x55c0da0a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.943+0000 7feb9e81d740 4 client.15687 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:34:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.943+0000 7feb9e81d740 1 -- 172.21.15.1:0/62647679 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c0d9f92e00 con 0x55c0da0a7a20 2024-04-26T21:34:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.943+0000 7feb93e00640 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c0da0a7a20 0x55c0da0a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.943+0000 7feb93e00640 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c0da0a7a20 0x55c0da0a9e40 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35e1ec692f490d91 server_cookie=e1590660b202f2ae in_seq=0 out_seq=0 2024-04-26T21:34:19.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.943+0000 7feb7aa00640 10 client.15687 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb7aa00640 1 -- 172.21.15.1:0/62647679 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c0d9f92e00 con 0x55c0da0a7a20 2024-04-26T21:34:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb7aa00640 10 client.15687 handle_command_reply: tid=0 2024-04-26T21:34:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb9e81d740 1 client.15687 shutdown 2024-04-26T21:34:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb9e81d740 1 -- 172.21.15.1:0/62647679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 msgr2=0x55c0da02b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb9e81d740 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02b440 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7feb88009250 tx=0x7feb88009e50 comp rx=0 tx=0).stop 2024-04-26T21:34:19.999 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb9e81d740 1 -- 172.21.15.1:0/62647679 shutdown_connections 2024-04-26T21:34:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb9e81d740 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c0da0a7a20 0x55c0da0a9e40 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb9e81d740 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c0da0311c0 0x55c0da02b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb9e81d740 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c0da026900 0x55c0da02b440 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb9e81d740 1 --2- 172.21.15.1:0/62647679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c0da025fe0 0x55c0da02af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb9e81d740 1 -- 172.21.15.1:0/62647679 >> 172.21.15.1:0/62647679 conn(0x55c0da0224f0 msgr2=0x55c0da02e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb7aa00640 0 client.15687 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.998+0000 7feb9e81d740 1 -- 172.21.15.1:0/62647679 shutdown_connections 2024-04-26T21:34:20.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.999+0000 7feb9e81d740 1 -- 172.21.15.1:0/62647679 wait complete. 2024-04-26T21:34:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:19.999+0000 7feb9e81d740 20 client.15687 trim_cache size 0 max 16384 2024-04-26T21:34:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:34:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:34:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:34:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:34:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 -- 172.21.15.1:0/3412870787 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb7c0a3d80 msgr2=0x7feb7c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb7c0a3d80 0x7feb7c0a6240 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0997c0 tx=0x7feb8c0057c0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 -- 172.21.15.1:0/3412870787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb9410a950 msgr2=0x7feb9419fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb9410a950 0x7feb9419fa50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009fd0 tx=0x7feb8c00ef10 comp rx=0 tx=0).stop 2024-04-26T21:34:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 -- 172.21.15.1:0/3412870787 shutdown_connections 2024-04-26T21:34:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb7c0a3d80 0x7feb7c0a6240 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb9410a950 0x7feb9419fa50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94105300 0x7feb9419b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 --2- 172.21.15.1:0/3412870787 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94104940 0x7feb9419b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 -- 172.21.15.1:0/3412870787 >> 172.21.15.1:0/3412870787 conn(0x7feb94100410 msgr2=0x7feb94100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 -- 172.21.15.1:0/3412870787 shutdown_connections 2024-04-26T21:34:20.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:20.000+0000 7feb9a400640 1 -- 172.21.15.1:0/3412870787 wait complete. 2024-04-26T21:34:20.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:19 smithi186 ceph-mon[32991]: pgmap v123: 129 pgs: 129 active+clean; 54 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 952 KiB/s wr, 113 op/s 2024-04-26T21:34:20.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:19 smithi001 ceph-mon[28054]: pgmap v123: 129 pgs: 129 active+clean; 54 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 952 KiB/s wr, 113 op/s 2024-04-26T21:34:20.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:19 smithi038 ceph-mon[32724]: pgmap v123: 129 pgs: 129 active+clean; 54 MiB data, 641 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 952 KiB/s wr, 113 op/s 2024-04-26T21:34:20.542 INFO:tasks.workunit.client.0.smithi001.stdout: 1 75727 2.86 MB/sec execute 18 sec latency 3583.857 ms 2024-04-26T21:34:21.542 INFO:tasks.workunit.client.0.smithi001.stdout: 1 77120 3.02 MB/sec execute 19 sec latency 31.372 ms 2024-04-26T21:34:22.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:21 smithi186 ceph-mon[32991]: pgmap v124: 129 pgs: 129 active+clean; 57 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-04-26T21:34:22.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:21 smithi001 ceph-mon[28054]: pgmap v124: 129 pgs: 129 active+clean; 57 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-04-26T21:34:22.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:21 smithi038 ceph-mon[32724]: pgmap v124: 129 pgs: 129 active+clean; 57 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 1.2 MiB/s wr, 130 op/s 2024-04-26T21:34:22.542 INFO:tasks.workunit.client.0.smithi001.stdout: 1 78310 3.41 MB/sec execute 20 sec latency 25.729 ms 2024-04-26T21:34:23.542 INFO:tasks.workunit.client.0.smithi001.stdout: 1 78552 3.27 MB/sec execute 21 sec latency 21.021 ms 2024-04-26T21:34:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:23 smithi186 ceph-mon[32991]: pgmap v125: 129 pgs: 129 active+clean; 57 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 762 KiB/s wr, 69 op/s 2024-04-26T21:34:24.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:23 smithi001 ceph-mon[28054]: pgmap v125: 129 pgs: 129 active+clean; 57 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 762 KiB/s wr, 69 op/s 2024-04-26T21:34:24.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:23 smithi038 ceph-mon[32724]: pgmap v125: 129 pgs: 129 active+clean; 57 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 762 KiB/s wr, 69 op/s 2024-04-26T21:34:24.542 INFO:tasks.workunit.client.0.smithi001.stdout: 1 79270 3.21 MB/sec execute 22 sec latency 20.802 ms 2024-04-26T21:34:25.544 INFO:tasks.workunit.client.0.smithi001.stdout: 1 80657 3.33 MB/sec execute 23 sec latency 26.629 ms 2024-04-26T21:34:26.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:25 smithi186 ceph-mon[32991]: pgmap v126: 129 pgs: 129 active+clean; 57 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.0 MiB/s wr, 134 op/s 2024-04-26T21:34:26.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:25 smithi001 ceph-mon[28054]: pgmap v126: 129 pgs: 129 active+clean; 57 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.0 MiB/s wr, 134 op/s 2024-04-26T21:34:26.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:25 smithi038 ceph-mon[32724]: pgmap v126: 129 pgs: 129 active+clean; 57 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 1.0 MiB/s wr, 134 op/s 2024-04-26T21:34:26.544 INFO:tasks.workunit.client.0.smithi001.stdout: 1 81866 3.64 MB/sec execute 24 sec latency 22.810 ms 2024-04-26T21:34:27.546 INFO:tasks.workunit.client.0.smithi001.stdout: 1 82302 3.51 MB/sec execute 25 sec latency 45.874 ms 2024-04-26T21:34:28.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:27 smithi186 ceph-mon[32991]: pgmap v127: 129 pgs: 129 active+clean; 57 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 588 KiB/s wr, 82 op/s 2024-04-26T21:34:28.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:27 smithi001 ceph-mon[28054]: pgmap v127: 129 pgs: 129 active+clean; 57 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 588 KiB/s wr, 82 op/s 2024-04-26T21:34:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:27 smithi038 ceph-mon[32724]: pgmap v127: 129 pgs: 129 active+clean; 57 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 588 KiB/s wr, 82 op/s 2024-04-26T21:34:28.545 INFO:tasks.workunit.client.0.smithi001.stdout: 1 82302 3.38 MB/sec execute 26 sec latency 1027.593 ms 2024-04-26T21:34:29.546 INFO:tasks.workunit.client.0.smithi001.stdout: 1 82302 3.25 MB/sec execute 27 sec latency 2027.759 ms 2024-04-26T21:34:30.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:29 smithi186 ceph-mon[32991]: pgmap v128: 129 pgs: 129 active+clean; 57 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 588 KiB/s wr, 82 op/s 2024-04-26T21:34:30.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:29 smithi001 ceph-mon[28054]: pgmap v128: 129 pgs: 129 active+clean; 57 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 588 KiB/s wr, 82 op/s 2024-04-26T21:34:30.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:29 smithi038 ceph-mon[32724]: pgmap v128: 129 pgs: 129 active+clean; 57 MiB data, 662 MiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 588 KiB/s wr, 82 op/s 2024-04-26T21:34:30.546 INFO:tasks.workunit.client.0.smithi001.stdout: 1 82302 3.14 MB/sec execute 28 sec latency 3027.875 ms 2024-04-26T21:34:31.546 INFO:tasks.workunit.client.0.smithi001.stdout: 1 82458 3.04 MB/sec execute 29 sec latency 3782.076 ms 2024-04-26T21:34:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:31 smithi186 ceph-mon[32991]: pgmap v129: 129 pgs: 129 active+clean; 58 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 982 KiB/s wr, 122 op/s 2024-04-26T21:34:32.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:31 smithi001 ceph-mon[28054]: pgmap v129: 129 pgs: 129 active+clean; 58 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 982 KiB/s wr, 122 op/s 2024-04-26T21:34:32.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:31 smithi038 ceph-mon[32724]: pgmap v129: 129 pgs: 129 active+clean; 58 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 982 KiB/s wr, 122 op/s 2024-04-26T21:34:32.548 INFO:tasks.workunit.client.0.smithi001.stdout: 1 83982 3.15 MB/sec execute 30 sec latency 32.615 ms 2024-04-26T21:34:33.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.149+0000 7fe371a00640 1 -- 172.21.15.1:0/2215175506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c1044c0 msgr2=0x7fe36c10c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.149+0000 7fe371a00640 1 --2- 172.21.15.1:0/2215175506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c1044c0 0x7fe36c10c9a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe358009900 tx=0x7fe35802a490 comp rx=0 tx=0).stop 2024-04-26T21:34:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.150+0000 7fe371a00640 1 -- 172.21.15.1:0/2215175506 shutdown_connections 2024-04-26T21:34:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.150+0000 7fe371a00640 1 --2- 172.21.15.1:0/2215175506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36c10cf70 0x7fe36c10f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.150+0000 7fe371a00640 1 --2- 172.21.15.1:0/2215175506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c1044c0 0x7fe36c10c9a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.150+0000 7fe371a00640 1 --2- 172.21.15.1:0/2215175506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe36c103ba0 0x7fe36c103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.150+0000 7fe371a00640 1 -- 172.21.15.1:0/2215175506 >> 172.21.15.1:0/2215175506 conn(0x7fe36c078950 msgr2=0x7fe36c078d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.150+0000 7fe371a00640 1 -- 172.21.15.1:0/2215175506 shutdown_connections 2024-04-26T21:34:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.150+0000 7fe371a00640 1 -- 172.21.15.1:0/2215175506 wait complete. 2024-04-26T21:34:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe371a00640 1 Processor -- start 2024-04-26T21:34:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe371a00640 1 -- start start 2024-04-26T21:34:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe371a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe36c103ba0 0x7fe36c19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe371a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c1044c0 0x7fe36c19b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe371a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36c10cf70 0x7fe36c19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe371a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c112c00 con 0x7fe36c103ba0 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe371a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c112a80 con 0x7fe36c1044c0 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe371a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c112d80 con 0x7fe36c10cf70 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe36c103ba0 0x7fe36c19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe36a000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c1044c0 0x7fe36c19b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.151+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe36c103ba0 0x7fe36c19b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36166/0 (socket says 172.21.15.1:36166) 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.152+0000 7fe36aa00640 1 -- 172.21.15.1:0/1834561948 learned_addr learned my addr 172.21.15.1:0/1834561948 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.152+0000 7fe36b400640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36c10cf70 0x7fe36c19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.152+0000 7fe36a000640 1 -- 172.21.15.1:0/1834561948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36c10cf70 msgr2=0x7fe36c19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.152+0000 7fe36a000640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36c10cf70 0x7fe36c19fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.154 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.152+0000 7fe36a000640 1 -- 172.21.15.1:0/1834561948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe36c103ba0 msgr2=0x7fe36c19b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.152+0000 7fe36a000640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe36c103ba0 0x7fe36c19b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.152+0000 7fe36a000640 1 -- 172.21.15.1:0/1834561948 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe36c1a0340 con 0x7fe36c1044c0 2024-04-26T21:34:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.152+0000 7fe36b400640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36c10cf70 0x7fe36c19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:33.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.152+0000 7fe36a000640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c1044c0 0x7fe36c19b8e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe35802a9b0 tx=0x7fe358003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.153+0000 7fe363400640 1 -- 172.21.15.1:0/1834561948 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35803f070 con 0x7fe36c1044c0 2024-04-26T21:34:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.153+0000 7fe363400640 1 -- 172.21.15.1:0/1834561948 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe358031b60 con 0x7fe36c1044c0 2024-04-26T21:34:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.153+0000 7fe363400640 1 -- 172.21.15.1:0/1834561948 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358035030 con 0x7fe36c1044c0 2024-04-26T21:34:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.153+0000 7fe371a00640 1 -- 172.21.15.1:0/1834561948 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe36c1a05d0 con 0x7fe36c1044c0 2024-04-26T21:34:33.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.153+0000 7fe371a00640 1 -- 172.21.15.1:0/1834561948 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe36c1a7ec0 con 0x7fe36c1044c0 2024-04-26T21:34:33.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.157+0000 7fe374795740 1 Processor -- start 2024-04-26T21:34:33.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.157+0000 7fe374795740 1 -- start start 2024-04-26T21:34:33.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe363400640 1 -- 172.21.15.1:0/1834561948 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fe358042070 con 0x7fe36c1044c0 2024-04-26T21:34:33.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe374795740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x5626644991b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe374795740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644996f0 0x5626644a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe374795740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644a3fb0 0x5626644a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe374795740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5626643fe380 con 0x5626644996f0 2024-04-26T21:34:33.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe374795740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56266439ac30 con 0x562664498dd0 2024-04-26T21:34:33.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe374795740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5626643fa340 con 0x5626644a3fb0 2024-04-26T21:34:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644996f0 0x5626644a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe36a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644a3fb0 0x5626644a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe36b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x5626644991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644996f0 0x5626644a3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36172/0 (socket says 172.21.15.1:36172) 2024-04-26T21:34:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.158+0000 7fe36aa00640 1 -- 172.21.15.1:0/573108988 learned_addr learned my addr 172.21.15.1:0/573108988 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.159+0000 7fe36aa00640 1 -- 172.21.15.1:0/573108988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644a3fb0 msgr2=0x5626644a63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.159+0000 7fe36aa00640 1 --2- 172.21.15.1:0/573108988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644a3fb0 0x5626644a63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.159+0000 7fe36aa00640 1 -- 172.21.15.1:0/573108988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 msgr2=0x5626644991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.159+0000 7fe36aa00640 1 --2- 172.21.15.1:0/573108988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x5626644991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.159+0000 7fe36aa00640 1 -- 172.21.15.1:0/573108988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626643bc940 con 0x5626644996f0 2024-04-26T21:34:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.159+0000 7fe36b400640 1 --2- 172.21.15.1:0/573108988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x5626644991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.159+0000 7fe36aa00640 1 --2- 172.21.15.1:0/573108988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644996f0 0x5626644a3a70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe35c00b7e0 tx=0x7fe35c00bca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65adaed9c4af87e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.160+0000 7fe357e00640 1 -- 172.21.15.1:0/573108988 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c003dd0 con 0x5626644996f0 2024-04-26T21:34:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.160+0000 7fe357e00640 1 -- 172.21.15.1:0/573108988 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe35c015070 con 0x5626644996f0 2024-04-26T21:34:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.160+0000 7fe357e00640 1 -- 172.21.15.1:0/573108988 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c010730 con 0x5626644996f0 2024-04-26T21:34:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.160+0000 7fe363400640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe3500a3d90 0x7fe3500a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.160+0000 7fe363400640 1 -- 172.21.15.1:0/1834561948 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fe3580f1500 con 0x7fe36c1044c0 2024-04-26T21:34:33.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.160+0000 7fe36b400640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe3500a3d90 0x7fe3500a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.161+0000 7fe374795740 1 -- 172.21.15.1:0/573108988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644996f0 msgr2=0x5626644a3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.161+0000 7fe374795740 1 --2- 172.21.15.1:0/573108988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644996f0 0x5626644a3a70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe35c00b7e0 tx=0x7fe35c00bca0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.161+0000 7fe374795740 1 -- 172.21.15.1:0/573108988 shutdown_connections 2024-04-26T21:34:33.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.161+0000 7fe374795740 1 --2- 172.21.15.1:0/573108988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644a3fb0 0x5626644a63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.161+0000 7fe374795740 1 --2- 172.21.15.1:0/573108988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644996f0 0x5626644a3a70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.161+0000 7fe374795740 1 --2- 172.21.15.1:0/573108988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x5626644991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.161+0000 7fe374795740 1 -- 172.21.15.1:0/573108988 >> 172.21.15.1:0/573108988 conn(0x5626644952e0 msgr2=0x5626644956c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.161+0000 7fe374795740 1 -- 172.21.15.1:0/573108988 shutdown_connections 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.161+0000 7fe36b400640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe3500a3d90 0x7fe3500a6250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe36c06b230 tx=0x7fe364009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.161+0000 7fe374795740 1 -- 172.21.15.1:0/573108988 wait complete. 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.162+0000 7fe374795740 1 -- start start 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.162+0000 7fe374795740 1 Processor -- start 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.162+0000 7fe374795740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562664498dd0 0x5626644a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.162+0000 7fe374795740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.162+0000 7fe374795740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626644a3fb0 0x5626644a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.162+0000 7fe374795740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5626644a6be0 con 0x562664498dd0 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.162+0000 7fe374795740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562664385c60 con 0x5626644a3fb0 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.162+0000 7fe374795740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5626644a6d60 con 0x5626644996f0 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe374795740 10 client.0 _ll_register_callbacks cb 0x5626643b1e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:33.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562664498dd0 0x5626644a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52822/0 (socket says 172.21.15.1:52822) 2024-04-26T21:34:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36a000640 1 -- 172.21.15.1:0/41658580 learned_addr learned my addr 172.21.15.1:0/41658580 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:33.166 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36b400640 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626644a3fb0 0x5626644a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36a000640 1 -- 172.21.15.1:0/41658580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626644a3fb0 msgr2=0x5626644a19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36a000640 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626644a3fb0 0x5626644a19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36a000640 1 -- 172.21.15.1:0/41658580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562664498dd0 msgr2=0x5626644a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36a000640 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562664498dd0 0x5626644a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36a000640 1 -- 172.21.15.1:0/41658580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562664394510 con 0x5626644996f0 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36aa00640 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562664498dd0 0x5626644a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe36a000640 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a1470 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe358005ab0 tx=0x7fe358005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1195b9f9b372c24f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.163+0000 7fe354c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.164+0000 7fe354c00640 1 -- 172.21.15.1:0/41658580 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358049070 con 0x5626644996f0 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.164+0000 7fe354c00640 1 -- 172.21.15.1:0/41658580 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fe3580b96a0 con 0x5626644996f0 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.164+0000 7fe374795740 1 -- 172.21.15.1:0/41658580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe35c015070 con 0x5626644996f0 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.164+0000 7fe354c00640 1 -- 172.21.15.1:0/41658580 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe3580bcb70 con 0x5626644996f0 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.164+0000 7fe374795740 10 client.25837 fetch_fsmap learned FSMap version 32 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.164+0000 7fe374795740 1 -- 172.21.15.1:0/41658580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562664110210 con 0x5626644996f0 2024-04-26T21:34:33.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.165+0000 7fe354c00640 1 -- 172.21.15.1:0/41658580 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe35c015070 con 0x5626644996f0 2024-04-26T21:34:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.165+0000 7fe354c00640 1 -- 172.21.15.1:0/41658580 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe3580f1130 con 0x5626644996f0 2024-04-26T21:34:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.165+0000 7fe374795740 10 client.25837 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T21:34:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.165+0000 7fe374795740 10 client.25837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:34:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.165+0000 7fe374795740 20 client.25837 populate_metadata read hostname 'smithi001' 2024-04-26T21:34:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.165+0000 7fe374795740 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5626644b3c80 0x5626644b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.165+0000 7fe374795740 4 client.25837 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.165+0000 7fe374795740 1 -- 172.21.15.1:0/41658580 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56266440bf90 con 0x5626644b3c80 2024-04-26T21:34:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.165+0000 7fe36aa00640 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5626644b3c80 0x5626644b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.166+0000 7fe36aa00640 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5626644b3c80 0x5626644b60d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dcd333e3e8a3c33 server_cookie=d1f316d214f5768c in_seq=0 out_seq=0 2024-04-26T21:34:33.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.166+0000 7fe354c00640 10 client.25837 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.166+0000 7fe354c00640 1 -- 172.21.15.1:0/41658580 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56266440bf90 con 0x5626644b3c80 2024-04-26T21:34:33.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe354c00640 10 client.25837 handle_command_reply: tid=0 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 client.25837 shutdown 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 -- 172.21.15.1:0/41658580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 msgr2=0x5626644a1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a1470 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe358005ab0 tx=0x7fe358005d50 comp rx=0 tx=0).stop 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 -- 172.21.15.1:0/41658580 shutdown_connections 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5626644b3c80 0x5626644b60d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a1470 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5626644a3fb0 0x5626644a19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 --2- 172.21.15.1:0/41658580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562664498dd0 0x5626644a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 -- 172.21.15.1:0/41658580 >> 172.21.15.1:0/41658580 conn(0x5626644952e0 msgr2=0x56266449b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe354c00640 0 client.25837 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 -- 172.21.15.1:0/41658580 shutdown_connections 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 1 -- 172.21.15.1:0/41658580 wait complete. 2024-04-26T21:34:33.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.167+0000 7fe374795740 20 client.25837 trim_cache size 0 max 16384 2024-04-26T21:34:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.186+0000 7fe374795740 1 Processor -- start 2024-04-26T21:34:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe374795740 1 -- start start 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe374795740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x562664494bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe374795740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe374795740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644a3fb0 0x5626644a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe374795740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1b4a30 con 0x5626644a3fb0 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe374795740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1b2240 con 0x562664498dd0 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe374795740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1b0090 con 0x5626644996f0 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x562664494bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe36a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe36aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x562664494bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37164/0 (socket says 172.21.15.1:37164) 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe36aa00640 1 -- 172.21.15.1:0/540901574 learned_addr learned my addr 172.21.15.1:0/540901574 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.187+0000 7fe36b400640 1 --2- 172.21.15.1:0/540901574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644a3fb0 0x5626644a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.188+0000 7fe36a000640 1 -- 172.21.15.1:0/540901574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 msgr2=0x562664494bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.188+0000 7fe36a000640 1 --2- 172.21.15.1:0/540901574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x562664494bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.188+0000 7fe36a000640 1 -- 172.21.15.1:0/540901574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644a3fb0 msgr2=0x5626644a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.188+0000 7fe36a000640 1 --2- 172.21.15.1:0/540901574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644a3fb0 0x5626644a2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.188+0000 7fe36a000640 1 -- 172.21.15.1:0/540901574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562664490940 con 0x5626644996f0 2024-04-26T21:34:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.188+0000 7fe36aa00640 1 --2- 172.21.15.1:0/540901574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x562664494bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.188+0000 7fe36a000640 1 --2- 172.21.15.1:0/540901574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644afaa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe3580bcd30 tx=0x7fe3580b93b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=350de110702f39fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.188+0000 7fe357400640 1 -- 172.21.15.1:0/540901574 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe358049070 con 0x5626644996f0 2024-04-26T21:34:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.188+0000 7fe357400640 1 -- 172.21.15.1:0/540901574 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3580bd730 con 0x5626644996f0 2024-04-26T21:34:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.188+0000 7fe357400640 1 -- 172.21.15.1:0/540901574 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3580bc990 con 0x5626644996f0 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.189+0000 7fe374795740 1 -- 172.21.15.1:0/540901574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 msgr2=0x5626644afaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.189+0000 7fe374795740 1 --2- 172.21.15.1:0/540901574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644afaa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe3580bcd30 tx=0x7fe3580b93b0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.189+0000 7fe374795740 1 -- 172.21.15.1:0/540901574 shutdown_connections 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.189+0000 7fe374795740 1 --2- 172.21.15.1:0/540901574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644afaa0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.189+0000 7fe374795740 1 --2- 172.21.15.1:0/540901574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644a3fb0 0x5626644a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.189+0000 7fe374795740 1 --2- 172.21.15.1:0/540901574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x562664494bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.189+0000 7fe374795740 1 -- 172.21.15.1:0/540901574 >> 172.21.15.1:0/540901574 conn(0x5626644952e0 msgr2=0x5626644a2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.189+0000 7fe374795740 1 -- 172.21.15.1:0/540901574 shutdown_connections 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.189+0000 7fe374795740 1 -- 172.21.15.1:0/540901574 wait complete. 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.190+0000 7fe374795740 1 -- start start 2024-04-26T21:34:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.190+0000 7fe374795740 1 Processor -- start 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.190+0000 7fe374795740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x5626644ac350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.190+0000 7fe374795740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe374795740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644a3fb0 0x5626644a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe374795740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1b0700 con 0x5626644a3fb0 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe374795740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1b1b70 con 0x562664498dd0 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe374795740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe36c1b0dd0 con 0x5626644996f0 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe374795740 10 client.0 _ll_register_callbacks cb 0x7fe358009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36a000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52848/0 (socket says 172.21.15.1:52848) 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36a000640 1 -- 172.21.15.1:0/735438607 learned_addr learned my addr 172.21.15.1:0/735438607 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36aa00640 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x5626644ac350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36b400640 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644a3fb0 0x5626644a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36aa00640 1 -- 172.21.15.1:0/735438607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 msgr2=0x5626644a76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:33.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36aa00640 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36aa00640 1 -- 172.21.15.1:0/735438607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644a3fb0 msgr2=0x5626644a7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36aa00640 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644a3fb0 0x5626644a7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36aa00640 1 -- 172.21.15.1:0/735438607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626641dc620 con 0x562664498dd0 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.191+0000 7fe36a000640 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.192+0000 7fe36aa00640 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x5626644ac350 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe35c006010 tx=0x7fe35c004380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1706dd632b464289 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.192+0000 7fe354c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.192+0000 7fe354c00640 1 -- 172.21.15.1:0/735438607 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c003fe0 con 0x562664498dd0 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.192+0000 7fe354c00640 1 -- 172.21.15.1:0/735438607 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fe35c005050 con 0x562664498dd0 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.192+0000 7fe374795740 1 -- 172.21.15.1:0/735438607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3580bd730 con 0x562664498dd0 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.192+0000 7fe354c00640 1 -- 172.21.15.1:0/735438607 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe35c005250 con 0x562664498dd0 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.192+0000 7fe374795740 10 client.25859 fetch_fsmap learned FSMap version 32 2024-04-26T21:34:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.192+0000 7fe374795740 1 -- 172.21.15.1:0/735438607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5626643b8cb0 con 0x562664498dd0 2024-04-26T21:34:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.193+0000 7fe354c00640 1 -- 172.21.15.1:0/735438607 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3580bd730 con 0x562664498dd0 2024-04-26T21:34:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.193+0000 7fe354c00640 1 -- 172.21.15.1:0/735438607 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 32) ==== 8362+0+0 (secure 0 0 0) 0x7fe35c011740 con 0x562664498dd0 2024-04-26T21:34:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.193+0000 7fe374795740 10 client.25859 fetch_fsmap finished waiting for FSMap version 32 2024-04-26T21:34:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.193+0000 7fe374795740 10 client.25859 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:34:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.193+0000 7fe374795740 20 client.25859 populate_metadata read hostname 'smithi001' 2024-04-26T21:34:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.193+0000 7fe374795740 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56266451a410 0x56266451c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.193+0000 7fe374795740 4 client.25859 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:34:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.193+0000 7fe374795740 1 -- 172.21.15.1:0/735438607 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562664405e00 con 0x56266451a410 2024-04-26T21:34:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.193+0000 7fe36a000640 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56266451a410 0x56266451c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.194+0000 7fe36a000640 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56266451a410 0x56266451c800 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a5aaf0c45cabe45 server_cookie=9cbb6669b39644f4 in_seq=0 out_seq=0 2024-04-26T21:34:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.194+0000 7fe354c00640 10 client.25859 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:33.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe354c00640 1 -- 172.21.15.1:0/735438607 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562664405e00 con 0x56266451a410 2024-04-26T21:34:33.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe354c00640 10 client.25859 handle_command_reply: tid=0 2024-04-26T21:34:33.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe374795740 1 client.25859 shutdown 2024-04-26T21:34:33.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe374795740 1 -- 172.21.15.1:0/735438607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 msgr2=0x5626644ac350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe374795740 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x5626644ac350 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe35c006010 tx=0x7fe35c004380 comp rx=0 tx=0).stop 2024-04-26T21:34:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe374795740 1 -- 172.21.15.1:0/735438607 shutdown_connections 2024-04-26T21:34:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe374795740 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56266451a410 0x56266451c800 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe374795740 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626644996f0 0x5626644a76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe374795740 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5626644a3fb0 0x5626644a7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe374795740 1 --2- 172.21.15.1:0/735438607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562664498dd0 0x5626644ac350 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe374795740 1 -- 172.21.15.1:0/735438607 >> 172.21.15.1:0/735438607 conn(0x5626644952e0 msgr2=0x5626644a1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.237+0000 7fe354c00640 0 client.25859 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.238+0000 7fe374795740 1 -- 172.21.15.1:0/735438607 shutdown_connections 2024-04-26T21:34:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.238+0000 7fe374795740 1 -- 172.21.15.1:0/735438607 wait complete. 2024-04-26T21:34:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.238+0000 7fe374795740 20 client.25859 trim_cache size 0 max 16384 2024-04-26T21:34:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:34:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:34:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:34:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:34:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 -- 172.21.15.1:0/1834561948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe3500a3d90 msgr2=0x7fe3500a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe3500a3d90 0x7fe3500a6250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe36c06b230 tx=0x7fe364009500 comp rx=0 tx=0).stop 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 -- 172.21.15.1:0/1834561948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c1044c0 msgr2=0x7fe36c19b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c1044c0 0x7fe36c19b8e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe35802a9b0 tx=0x7fe358003c10 comp rx=0 tx=0).stop 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 -- 172.21.15.1:0/1834561948 shutdown_connections 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe3500a3d90 0x7fe3500a6250 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe36c10cf70 0x7fe36c19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe36c1044c0 0x7fe36c19b8e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 --2- 172.21.15.1:0/1834561948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe36c103ba0 0x7fe36c19b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 -- 172.21.15.1:0/1834561948 >> 172.21.15.1:0/1834561948 conn(0x7fe36c078950 msgr2=0x7fe36c108e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 -- 172.21.15.1:0/1834561948 shutdown_connections 2024-04-26T21:34:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:33.239+0000 7fe371a00640 1 -- 172.21.15.1:0/1834561948 wait complete. 2024-04-26T21:34:33.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:32 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:34:33.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:32 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:34:33.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:32 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:34:33.546 INFO:tasks.workunit.client.0.smithi001.stdout: 1 85176 3.39 MB/sec execute 31 sec latency 39.196 ms 2024-04-26T21:34:34.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:33 smithi001 ceph-mon[28054]: pgmap v130: 129 pgs: 129 active+clean; 58 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 696 KiB/s wr, 105 op/s 2024-04-26T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:33 smithi038 ceph-mon[32724]: pgmap v130: 129 pgs: 129 active+clean; 58 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 696 KiB/s wr, 105 op/s 2024-04-26T21:34:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:33 smithi186 ceph-mon[32991]: pgmap v130: 129 pgs: 129 active+clean; 58 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 696 KiB/s wr, 105 op/s 2024-04-26T21:34:34.547 INFO:tasks.workunit.client.0.smithi001.stdout: 1 85631 3.34 MB/sec execute 32 sec latency 27.026 ms 2024-04-26T21:34:35.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:34 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:34:35.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:34 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:34:35.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:34 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:34:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:34 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:34:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:34 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:34:35.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:34 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:34:35.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:34 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:34:35.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:34 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:34:35.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:34 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:34:35.547 INFO:tasks.workunit.client.0.smithi001.stdout: 1 85843 3.24 MB/sec execute 33 sec latency 594.991 ms 2024-04-26T21:34:36.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:35 smithi001 ceph-mon[28054]: pgmap v131: 129 pgs: 129 active+clean; 61 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1020 KiB/s wr, 166 op/s 2024-04-26T21:34:36.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:35 smithi038 ceph-mon[32724]: pgmap v131: 129 pgs: 129 active+clean; 61 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1020 KiB/s wr, 166 op/s 2024-04-26T21:34:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:35 smithi186 ceph-mon[32991]: pgmap v131: 129 pgs: 129 active+clean; 61 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1020 KiB/s wr, 166 op/s 2024-04-26T21:34:36.547 INFO:tasks.workunit.client.0.smithi001.stdout: 1 85916 3.15 MB/sec execute 34 sec latency 1348.265 ms 2024-04-26T21:34:37.547 INFO:tasks.workunit.client.0.smithi001.stdout: 1 86880 3.20 MB/sec execute 35 sec latency 283.391 ms 2024-04-26T21:34:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:38 smithi001 ceph-mon[28054]: pgmap v132: 129 pgs: 129 active+clean; 61 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 719 KiB/s wr, 101 op/s 2024-04-26T21:34:38.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:38 smithi038 ceph-mon[32724]: pgmap v132: 129 pgs: 129 active+clean; 61 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 719 KiB/s wr, 101 op/s 2024-04-26T21:34:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:38 smithi186 ceph-mon[32991]: pgmap v132: 129 pgs: 129 active+clean; 61 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 719 KiB/s wr, 101 op/s 2024-04-26T21:34:38.547 INFO:tasks.workunit.client.0.smithi001.stdout: 1 88164 3.32 MB/sec execute 36 sec latency 25.721 ms 2024-04-26T21:34:39.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:39 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:34:39.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:39 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:34:39.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:39 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:34:39.547 INFO:tasks.workunit.client.0.smithi001.stdout: 1 89022 3.40 MB/sec execute 37 sec latency 24.458 ms 2024-04-26T21:34:40.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:40 smithi001 ceph-mon[28054]: pgmap v133: 129 pgs: 129 active+clean; 61 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 719 KiB/s wr, 101 op/s 2024-04-26T21:34:40.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:40 smithi038 ceph-mon[32724]: pgmap v133: 129 pgs: 129 active+clean; 61 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 719 KiB/s wr, 101 op/s 2024-04-26T21:34:40.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:40 smithi186 ceph-mon[32991]: pgmap v133: 129 pgs: 129 active+clean; 61 MiB data, 707 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 719 KiB/s wr, 101 op/s 2024-04-26T21:34:40.548 INFO:tasks.workunit.client.0.smithi001.stdout: 1 89392 3.32 MB/sec execute 38 sec latency 26.853 ms 2024-04-26T21:34:41.548 INFO:tasks.workunit.client.0.smithi001.stdout: 1 90375 3.36 MB/sec execute 39 sec latency 21.709 ms 2024-04-26T21:34:42.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:42 smithi001 ceph-mon[28054]: pgmap v134: 129 pgs: 129 active+clean; 62 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 150 op/s 2024-04-26T21:34:42.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:42 smithi038 ceph-mon[32724]: pgmap v134: 129 pgs: 129 active+clean; 62 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 150 op/s 2024-04-26T21:34:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:42 smithi186 ceph-mon[32991]: pgmap v134: 129 pgs: 129 active+clean; 62 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 1.1 MiB/s wr, 150 op/s 2024-04-26T21:34:42.548 INFO:tasks.workunit.client.0.smithi001.stdout: 1 91698 3.46 MB/sec execute 40 sec latency 28.525 ms 2024-04-26T21:34:43.548 INFO:tasks.workunit.client.0.smithi001.stdout: 1 92608 3.54 MB/sec execute 41 sec latency 37.037 ms 2024-04-26T21:34:44.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:44 smithi001 ceph-mon[28054]: pgmap v135: 129 pgs: 129 active+clean; 62 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 783 KiB/s wr, 110 op/s 2024-04-26T21:34:44.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:44 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:34:44.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:44 smithi038 ceph-mon[32724]: pgmap v135: 129 pgs: 129 active+clean; 62 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 783 KiB/s wr, 110 op/s 2024-04-26T21:34:44.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:44 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:34:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:44 smithi186 ceph-mon[32991]: pgmap v135: 129 pgs: 129 active+clean; 62 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 783 KiB/s wr, 110 op/s 2024-04-26T21:34:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:44 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:34:44.548 INFO:tasks.workunit.client.0.smithi001.stdout: 1 92992 3.46 MB/sec execute 42 sec latency 61.802 ms 2024-04-26T21:34:45.548 INFO:tasks.workunit.client.0.smithi001.stdout: 1 92992 3.38 MB/sec execute 43 sec latency 1061.964 ms 2024-04-26T21:34:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:46 smithi001 ceph-mon[28054]: pgmap v136: 129 pgs: 129 active+clean; 64 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.3 MiB/s wr, 168 op/s 2024-04-26T21:34:46.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:46 smithi038 ceph-mon[32724]: pgmap v136: 129 pgs: 129 active+clean; 64 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.3 MiB/s wr, 168 op/s 2024-04-26T21:34:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.407+0000 7f7a63c00640 1 -- 172.21.15.1:0/3303931193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a5c108930 msgr2=0x7f7a5c108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.407+0000 7f7a63c00640 1 --2- 172.21.15.1:0/3303931193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a5c108930 0x7f7a5c108d10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c009cd0 tx=0x7f7a4c02a8e0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.407+0000 7f7a63c00640 1 -- 172.21.15.1:0/3303931193 shutdown_connections 2024-04-26T21:34:46.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.407+0000 7f7a63c00640 1 --2- 172.21.15.1:0/3303931193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a5c102920 0x7f7a5c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.407+0000 7f7a63c00640 1 --2- 172.21.15.1:0/3303931193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a5c1032e0 0x7f7a5c10f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.407+0000 7f7a63c00640 1 --2- 172.21.15.1:0/3303931193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a5c108930 0x7f7a5c108d10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.407+0000 7f7a63c00640 1 -- 172.21.15.1:0/3303931193 >> 172.21.15.1:0/3303931193 conn(0x7f7a5c0fe3f0 msgr2=0x7f7a5c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.408+0000 7f7a63c00640 1 -- 172.21.15.1:0/3303931193 shutdown_connections 2024-04-26T21:34:46.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.408+0000 7f7a63c00640 1 -- 172.21.15.1:0/3303931193 wait complete. 2024-04-26T21:34:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a63c00640 1 Processor -- start 2024-04-26T21:34:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a63c00640 1 -- start start 2024-04-26T21:34:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a63c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a5c102920 0x7f7a5c19b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a63c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a5c1032e0 0x7f7a5c19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a63c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a5c108930 0x7f7a5c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a63c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c112bf0 con 0x7f7a5c108930 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a63c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c112a70 con 0x7f7a5c1032e0 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a63c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c112d70 con 0x7f7a5c102920 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a61c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a5c108930 0x7f7a5c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a61200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a5c102920 0x7f7a5c19b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a61200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a5c102920 0x7f7a5c19b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:60748/0 (socket says 172.21.15.1:60748) 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a61200640 1 -- 172.21.15.1:0/612735001 learned_addr learned my addr 172.21.15.1:0/612735001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.409+0000 7f7a5be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a5c1032e0 0x7f7a5c19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.410+0000 7f7a5be00640 1 -- 172.21.15.1:0/612735001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a5c102920 msgr2=0x7f7a5c19b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.410+0000 7f7a5be00640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a5c102920 0x7f7a5c19b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.410+0000 7f7a5be00640 1 -- 172.21.15.1:0/612735001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a5c108930 msgr2=0x7f7a5c19fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.410+0000 7f7a5be00640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a5c108930 0x7f7a5c19fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.410+0000 7f7a5be00640 1 -- 172.21.15.1:0/612735001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5c1a0390 con 0x7f7a5c1032e0 2024-04-26T21:34:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.410+0000 7f7a5be00640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a5c1032e0 0x7f7a5c19b930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400e9f0 tx=0x7f7a4400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.411+0000 7f7a59600640 1 -- 172.21.15.1:0/612735001 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4400ccc0 con 0x7f7a5c1032e0 2024-04-26T21:34:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.411+0000 7f7a59600640 1 -- 172.21.15.1:0/612735001 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a44005020 con 0x7f7a5c1032e0 2024-04-26T21:34:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.411+0000 7f7a59600640 1 -- 172.21.15.1:0/612735001 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a44004a00 con 0x7f7a5c1032e0 2024-04-26T21:34:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.411+0000 7f7a63c00640 1 -- 172.21.15.1:0/612735001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a5c1a0680 con 0x7f7a5c1032e0 2024-04-26T21:34:46.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.411+0000 7f7a63c00640 1 -- 172.21.15.1:0/612735001 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a5c1a7f10 con 0x7f7a5c1032e0 2024-04-26T21:34:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.414+0000 7f7a66a73740 1 Processor -- start 2024-04-26T21:34:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.414+0000 7f7a59600640 1 -- 172.21.15.1:0/612735001 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f7a44004ba0 con 0x7f7a5c1032e0 2024-04-26T21:34:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.414+0000 7f7a66a73740 1 -- start start 2024-04-26T21:34:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb0ffb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1005a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb1005fb0 0x55ebb10083a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a66a73740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ebb0f60380 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a66a73740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ebb0efcc30 con 0x55ebb1005fb0 2024-04-26T21:34:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a66a73740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebb0f5c340 con 0x55ebb0ffadd0 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a61c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb0ffb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a61200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1005a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a5be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb1005fb0 0x55ebb10083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a61c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb0ffb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:60762/0 (socket says 172.21.15.1:60762) 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a61c00640 1 -- 172.21.15.1:0/1062005774 learned_addr learned my addr 172.21.15.1:0/1062005774 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a61c00640 1 -- 172.21.15.1:0/1062005774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb1005fb0 msgr2=0x55ebb10083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a61c00640 1 --2- 172.21.15.1:0/1062005774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb1005fb0 0x55ebb10083a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a61c00640 1 -- 172.21.15.1:0/1062005774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb0ffb6f0 msgr2=0x55ebb1005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a61c00640 1 --2- 172.21.15.1:0/1062005774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1005a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.415+0000 7f7a61c00640 1 -- 172.21.15.1:0/1062005774 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebb0f1e940 con 0x55ebb0ffadd0 2024-04-26T21:34:46.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.416+0000 7f7a61c00640 1 --2- 172.21.15.1:0/1062005774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb0ffb1b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009c20 tx=0x7f7a54011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff529c3913d66d28 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.416+0000 7f7a59600640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7a400a3d90 0x7f7a400a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.416+0000 7f7a52000640 1 -- 172.21.15.1:0/1062005774 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5401e070 con 0x55ebb0ffadd0 2024-04-26T21:34:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.416+0000 7f7a52000640 1 -- 172.21.15.1:0/1062005774 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a54019ae0 con 0x55ebb0ffadd0 2024-04-26T21:34:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.416+0000 7f7a52000640 1 -- 172.21.15.1:0/1062005774 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5400e8b0 con 0x55ebb0ffadd0 2024-04-26T21:34:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.416+0000 7f7a59600640 1 -- 172.21.15.1:0/612735001 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f7a440d0b40 con 0x7f7a5c1032e0 2024-04-26T21:34:46.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.416+0000 7f7a61200640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7a400a3d90 0x7f7a400a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.417+0000 7f7a66a73740 1 -- 172.21.15.1:0/1062005774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 msgr2=0x55ebb0ffb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.417+0000 7f7a66a73740 1 --2- 172.21.15.1:0/1062005774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb0ffb1b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009c20 tx=0x7f7a54011810 comp rx=0 tx=0).stop 2024-04-26T21:34:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.417+0000 7f7a61200640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7a400a3d90 0x7f7a400a6250 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55ebb0ffc370 tx=0x7f7a4c03d040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:46.420 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.417+0000 7f7a66a73740 1 -- 172.21.15.1:0/1062005774 shutdown_connections 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.417+0000 7f7a66a73740 1 --2- 172.21.15.1:0/1062005774 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb1005fb0 0x55ebb10083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.417+0000 7f7a66a73740 1 --2- 172.21.15.1:0/1062005774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1005a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.417+0000 7f7a66a73740 1 --2- 172.21.15.1:0/1062005774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb0ffb1b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.417+0000 7f7a66a73740 1 -- 172.21.15.1:0/1062005774 >> 172.21.15.1:0/1062005774 conn(0x55ebb0ff72e0 msgr2=0x55ebb0ff76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.417+0000 7f7a66a73740 1 -- 172.21.15.1:0/1062005774 shutdown_connections 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.417+0000 7f7a66a73740 1 -- 172.21.15.1:0/1062005774 wait complete. 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.418+0000 7f7a66a73740 1 -- start start 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.418+0000 7f7a66a73740 1 Processor -- start 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb1002fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1003500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb1005fb0 0x55ebb1003a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a66a73740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ebb1008be0 con 0x55ebb1005fb0 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a66a73740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ebb0ee7c60 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a66a73740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebb1008d60 con 0x55ebb0ffadd0 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a66a73740 10 client.0 _ll_register_callbacks cb 0x55ebb0f13e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a5be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1003500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a5be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1003500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:52538/0 (socket says 172.21.15.1:52538) 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a5be00640 1 -- 172.21.15.1:0/2750175624 learned_addr learned my addr 172.21.15.1:0/2750175624 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a61c00640 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb1005fb0 0x55ebb1003a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a61200640 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb1002fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a5be00640 1 -- 172.21.15.1:0/2750175624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 msgr2=0x55ebb1002fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a5be00640 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb1002fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a5be00640 1 -- 172.21.15.1:0/2750175624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb1005fb0 msgr2=0x55ebb1003a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a5be00640 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb1005fb0 0x55ebb1003a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a5be00640 1 -- 172.21.15.1:0/2750175624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebb0ef6510 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.419+0000 7f7a61200640 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb1002fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.420+0000 7f7a5be00640 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1003500 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a440cde80 tx=0x7f7a440096e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1df4abcd5ca43248 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.420+0000 7f7a4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:34:46.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.420+0000 7f7a4aa00640 1 -- 172.21.15.1:0/2750175624 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4401c070 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.420+0000 7f7a4aa00640 1 -- 172.21.15.1:0/2750175624 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f7a440054f0 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.423 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.420+0000 7f7a66a73740 1 -- 172.21.15.1:0/2750175624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a54019ae0 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.420+0000 7f7a4aa00640 1 -- 172.21.15.1:0/2750175624 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7a440b3070 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.420+0000 7f7a66a73740 10 client.25871 fetch_fsmap learned FSMap version 34 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.420+0000 7f7a66a73740 1 -- 172.21.15.1:0/2750175624 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ebb0c72210 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.421+0000 7f7a4aa00640 1 -- 172.21.15.1:0/2750175624 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a54019ae0 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.421+0000 7f7a4aa00640 1 -- 172.21.15.1:0/2750175624 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f7a440ce0a0 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.421+0000 7f7a66a73740 10 client.25871 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.421+0000 7f7a66a73740 10 client.25871 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.421+0000 7f7a66a73740 20 client.25871 populate_metadata read hostname 'smithi001' 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.421+0000 7f7a66a73740 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ebb1015de0 0x55ebb10181d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.421+0000 7f7a66a73740 4 client.25871 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.421+0000 7f7a66a73740 1 -- 172.21.15.1:0/2750175624 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebb0f6df90 con 0x55ebb1015de0 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.422+0000 7f7a61200640 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ebb1015de0 0x55ebb10181d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.422+0000 7f7a61200640 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ebb1015de0 0x55ebb10181d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d94a993dfbfe754c server_cookie=425f5933fa36afb6 in_seq=0 out_seq=0 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.422+0000 7f7a4aa00640 10 client.25871 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a4aa00640 1 -- 172.21.15.1:0/2750175624 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ebb0f6df90 con 0x55ebb1015de0 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a4aa00640 10 client.25871 handle_command_reply: tid=0 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a66a73740 1 client.25871 shutdown 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a66a73740 1 -- 172.21.15.1:0/2750175624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 msgr2=0x55ebb1003500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a66a73740 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1003500 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7a440cde80 tx=0x7f7a440096e0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a66a73740 1 -- 172.21.15.1:0/2750175624 shutdown_connections 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a66a73740 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ebb1015de0 0x55ebb10181d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a66a73740 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb1005fb0 0x55ebb1003a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a66a73740 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1003500 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a66a73740 1 --2- 172.21.15.1:0/2750175624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb1002fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a66a73740 1 -- 172.21.15.1:0/2750175624 >> 172.21.15.1:0/2750175624 conn(0x55ebb0ff72e0 msgr2=0x55ebb0ffd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a4aa00640 0 client.25871 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:46.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.423+0000 7f7a66a73740 1 -- 172.21.15.1:0/2750175624 shutdown_connections 2024-04-26T21:34:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.424+0000 7f7a66a73740 1 -- 172.21.15.1:0/2750175624 wait complete. 2024-04-26T21:34:46.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.424+0000 7f7a66a73740 20 client.25871 trim_cache size 0 max 16384 2024-04-26T21:34:46.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.441+0000 7f7a66a73740 1 Processor -- start 2024-04-26T21:34:46.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.441+0000 7f7a66a73740 1 -- start start 2024-04-26T21:34:46.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffadd0 0x55ebb10056f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb0ffb6f0 0x55ebb0ff6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb1005fb0 0x55ebb0ffd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a66a73740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c1b49a0 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a66a73740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c1b2210 con 0x55ebb0ffadd0 2024-04-26T21:34:46.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a66a73740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c1b0060 con 0x55ebb1005fb0 2024-04-26T21:34:46.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a61200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffadd0 0x55ebb10056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a5be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb0ffb6f0 0x55ebb0ff6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a61200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffadd0 0x55ebb10056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:52546/0 (socket says 172.21.15.1:52546) 2024-04-26T21:34:46.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a61200640 1 -- 172.21.15.1:0/3159479313 learned_addr learned my addr 172.21.15.1:0/3159479313 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:46.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.442+0000 7f7a61c00640 1 --2- 172.21.15.1:0/3159479313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb1005fb0 0x55ebb0ffd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.443+0000 7f7a61200640 1 -- 172.21.15.1:0/3159479313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb1005fb0 msgr2=0x55ebb0ffd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:46.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.443+0000 7f7a61200640 1 --2- 172.21.15.1:0/3159479313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb1005fb0 0x55ebb0ffd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.443+0000 7f7a61200640 1 -- 172.21.15.1:0/3159479313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb0ffb6f0 msgr2=0x55ebb0ff6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:46.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.443+0000 7f7a61200640 1 --2- 172.21.15.1:0/3159479313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb0ffb6f0 0x55ebb0ff6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.443+0000 7f7a61200640 1 -- 172.21.15.1:0/3159479313 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebb0d3e620 con 0x55ebb0ffadd0 2024-04-26T21:34:46.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.443+0000 7f7a61200640 1 --2- 172.21.15.1:0/3159479313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffadd0 0x55ebb10056f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c005e30 tx=0x7f7a4c041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23f0bec39535d72d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:46.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.443+0000 7f7a51600640 1 -- 172.21.15.1:0/3159479313 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c003e00 con 0x55ebb0ffadd0 2024-04-26T21:34:46.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.443+0000 7f7a51600640 1 -- 172.21.15.1:0/3159479313 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a4c058070 con 0x55ebb0ffadd0 2024-04-26T21:34:46.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.443+0000 7f7a51600640 1 -- 172.21.15.1:0/3159479313 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c030800 con 0x55ebb0ffadd0 2024-04-26T21:34:46.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.444+0000 7f7a66a73740 1 -- 172.21.15.1:0/3159479313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffadd0 msgr2=0x55ebb10056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.444+0000 7f7a66a73740 1 --2- 172.21.15.1:0/3159479313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffadd0 0x55ebb10056f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c005e30 tx=0x7f7a4c041000 comp rx=0 tx=0).stop 2024-04-26T21:34:46.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.444+0000 7f7a66a73740 1 -- 172.21.15.1:0/3159479313 shutdown_connections 2024-04-26T21:34:46.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.444+0000 7f7a66a73740 1 --2- 172.21.15.1:0/3159479313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb1005fb0 0x55ebb0ffd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.444+0000 7f7a66a73740 1 --2- 172.21.15.1:0/3159479313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb0ffb6f0 0x55ebb0ff6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.444+0000 7f7a66a73740 1 --2- 172.21.15.1:0/3159479313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffadd0 0x55ebb10056f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.444+0000 7f7a66a73740 1 -- 172.21.15.1:0/3159479313 >> 172.21.15.1:0/3159479313 conn(0x55ebb0ff72e0 msgr2=0x55ebb1003810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:46.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.445+0000 7f7a66a73740 1 -- 172.21.15.1:0/3159479313 shutdown_connections 2024-04-26T21:34:46.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.445+0000 7f7a66a73740 1 -- 172.21.15.1:0/3159479313 wait complete. 2024-04-26T21:34:46.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.445+0000 7f7a66a73740 1 -- start start 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.445+0000 7f7a66a73740 1 Processor -- start 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb100e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a66a73740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb1005fb0 0x55ebb1009f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a66a73740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c1b06d0 con 0x55ebb1005fb0 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a66a73740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c1b1b40 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a66a73740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5c1b0da0 con 0x55ebb0ffadd0 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a66a73740 10 client.0 _ll_register_callbacks cb 0x7f7a4400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a5be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a61200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb100e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a5be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1009840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:52554/0 (socket says 172.21.15.1:52554) 2024-04-26T21:34:46.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a5be00640 1 -- 172.21.15.1:0/3724327849 learned_addr learned my addr 172.21.15.1:0/3724327849 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.446+0000 7f7a61c00640 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb1005fb0 0x55ebb1009f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a5be00640 1 -- 172.21.15.1:0/3724327849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 msgr2=0x55ebb100e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a5be00640 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb100e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a5be00640 1 -- 172.21.15.1:0/3724327849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb1005fb0 msgr2=0x55ebb1009f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a5be00640 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb1005fb0 0x55ebb1009f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a5be00640 1 -- 172.21.15.1:0/3724327849 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebb0f1acb0 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a61200640 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb100e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a5be00640 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1009840 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7a4409c600 tx=0x7f7a4400c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f7715e508a126c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a4aa00640 1 -- 172.21.15.1:0/3724327849 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a440b9030 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a4aa00640 1 -- 172.21.15.1:0/3724327849 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f7a440054f0 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.447+0000 7f7a66a73740 1 -- 172.21.15.1:0/3724327849 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a4c058070 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a4aa00640 1 -- 172.21.15.1:0/3724327849 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f7a44083050 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a66a73740 10 client.25877 fetch_fsmap learned FSMap version 34 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a66a73740 1 -- 172.21.15.1:0/3724327849 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ebb1084b00 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a4aa00640 1 -- 172.21.15.1:0/3724327849 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a4c058070 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a4aa00640 1 -- 172.21.15.1:0/3724327849 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f7a440a4e50 con 0x55ebb0ffb6f0 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a66a73740 10 client.25877 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a66a73740 10 client.25877 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a66a73740 20 client.25877 populate_metadata read hostname 'smithi001' 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a66a73740 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ebb10852d0 0x55ebb10876f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a66a73740 4 client.25877 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a66a73740 1 -- 172.21.15.1:0/3724327849 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ebb0f67e00 con 0x55ebb10852d0 2024-04-26T21:34:46.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.448+0000 7f7a61200640 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ebb10852d0 0x55ebb10876f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:46.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.449+0000 7f7a61200640 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ebb10852d0 0x55ebb10876f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9501f4f192eac6b2 server_cookie=4a83a79e68d5e259 in_seq=0 out_seq=0 2024-04-26T21:34:46.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.449+0000 7f7a4aa00640 10 client.25877 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:46.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:46 smithi186 ceph-mon[32991]: pgmap v136: 129 pgs: 129 active+clean; 64 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.3 MiB/s wr, 168 op/s 2024-04-26T21:34:46.502 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.501+0000 7f7a4aa00640 1 -- 172.21.15.1:0/3724327849 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ebb0f67e00 con 0x55ebb10852d0 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.501+0000 7f7a4aa00640 10 client.25877 handle_command_reply: tid=0 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.501+0000 7f7a66a73740 1 client.25877 shutdown 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.501+0000 7f7a66a73740 1 -- 172.21.15.1:0/3724327849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 msgr2=0x55ebb1009840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a66a73740 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1009840 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7a4409c600 tx=0x7f7a4400c7d0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a66a73740 1 -- 172.21.15.1:0/3724327849 shutdown_connections 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a66a73740 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ebb10852d0 0x55ebb10876f0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a66a73740 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ebb1005fb0 0x55ebb1009f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a66a73740 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebb0ffb6f0 0x55ebb1009840 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a66a73740 1 --2- 172.21.15.1:0/3724327849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebb0ffadd0 0x55ebb100e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a4aa00640 0 client.25877 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a66a73740 1 -- 172.21.15.1:0/3724327849 >> 172.21.15.1:0/3724327849 conn(0x55ebb0ff72e0 msgr2=0x55ebb0ff5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a66a73740 1 -- 172.21.15.1:0/3724327849 shutdown_connections 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a66a73740 1 -- 172.21.15.1:0/3724327849 wait complete. 2024-04-26T21:34:46.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.502+0000 7f7a66a73740 20 client.25877 trim_cache size 0 max 16384 2024-04-26T21:34:46.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:34:46.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:34:46.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:34:46.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:34:46.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 -- 172.21.15.1:0/612735001 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7a400a3d90 msgr2=0x7f7a400a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7a400a3d90 0x7f7a400a6250 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55ebb0ffc370 tx=0x7f7a4c03d040 comp rx=0 tx=0).stop 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 -- 172.21.15.1:0/612735001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a5c1032e0 msgr2=0x7f7a5c19b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a5c1032e0 0x7f7a5c19b930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400e9f0 tx=0x7f7a4400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 -- 172.21.15.1:0/612735001 shutdown_connections 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7a400a3d90 0x7f7a400a6250 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a5c108930 0x7f7a5c19fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a5c1032e0 0x7f7a5c19b930 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 --2- 172.21.15.1:0/612735001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a5c102920 0x7f7a5c19b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 -- 172.21.15.1:0/612735001 >> 172.21.15.1:0/612735001 conn(0x7f7a5c0fe3f0 msgr2=0x7f7a5c10db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 -- 172.21.15.1:0/612735001 shutdown_connections 2024-04-26T21:34:46.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:46.503+0000 7f7a63c00640 1 -- 172.21.15.1:0/612735001 wait complete. 2024-04-26T21:34:46.548 INFO:tasks.workunit.client.0.smithi001.stdout: 1 93182 3.31 MB/sec execute 44 sec latency 1797.310 ms 2024-04-26T21:34:47.548 INFO:tasks.workunit.client.0.smithi001.stdout: 1 94673 3.38 MB/sec execute 45 sec latency 54.771 ms 2024-04-26T21:34:48.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:48 smithi001 ceph-mon[28054]: pgmap v137: 129 pgs: 129 active+clean; 64 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 981 KiB/s wr, 107 op/s 2024-04-26T21:34:48.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:48 smithi038 ceph-mon[32724]: pgmap v137: 129 pgs: 129 active+clean; 64 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 981 KiB/s wr, 107 op/s 2024-04-26T21:34:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:48 smithi186 ceph-mon[32991]: pgmap v137: 129 pgs: 129 active+clean; 64 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 981 KiB/s wr, 107 op/s 2024-04-26T21:34:48.549 INFO:tasks.workunit.client.0.smithi001.stdout: 1 95973 3.54 MB/sec execute 46 sec latency 20.467 ms 2024-04-26T21:34:49.549 INFO:tasks.workunit.client.0.smithi001.stdout: 1 96257 3.49 MB/sec execute 47 sec latency 23.987 ms 2024-04-26T21:34:50.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:50 smithi001 ceph-mon[28054]: pgmap v138: 129 pgs: 129 active+clean; 64 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 981 KiB/s wr, 107 op/s 2024-04-26T21:34:50.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:50 smithi038 ceph-mon[32724]: pgmap v138: 129 pgs: 129 active+clean; 64 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 981 KiB/s wr, 107 op/s 2024-04-26T21:34:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:50 smithi186 ceph-mon[32991]: pgmap v138: 129 pgs: 129 active+clean; 64 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 981 KiB/s wr, 107 op/s 2024-04-26T21:34:50.549 INFO:tasks.workunit.client.0.smithi001.stdout: 1 96483 3.42 MB/sec execute 48 sec latency 570.262 ms 2024-04-26T21:34:51.549 INFO:tasks.workunit.client.0.smithi001.stdout: 1 96539 3.36 MB/sec execute 49 sec latency 1325.674 ms 2024-04-26T21:34:52.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:52 smithi038 ceph-mon[32724]: pgmap v139: 129 pgs: 129 active+clean; 62 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.4 MiB/s wr, 176 op/s 2024-04-26T21:34:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:52 smithi186 ceph-mon[32991]: pgmap v139: 129 pgs: 129 active+clean; 62 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.4 MiB/s wr, 176 op/s 2024-04-26T21:34:52.549 INFO:tasks.workunit.client.0.smithi001.stdout: 1 96940 3.32 MB/sec execute 50 sec latency 707.647 ms 2024-04-26T21:34:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:52 smithi001 ceph-mon[28054]: pgmap v139: 129 pgs: 129 active+clean; 62 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 1.4 MiB/s wr, 176 op/s 2024-04-26T21:34:53.549 INFO:tasks.workunit.client.0.smithi001.stdout: 1 98393 3.38 MB/sec execute 51 sec latency 35.951 ms 2024-04-26T21:34:54.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:54 smithi038 ceph-mon[32724]: pgmap v140: 129 pgs: 129 active+clean; 62 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 959 KiB/s wr, 127 op/s 2024-04-26T21:34:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:54 smithi186 ceph-mon[32991]: pgmap v140: 129 pgs: 129 active+clean; 62 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 959 KiB/s wr, 127 op/s 2024-04-26T21:34:54.549 INFO:tasks.workunit.client.0.smithi001.stdout: 1 99613 3.52 MB/sec execute 52 sec latency 23.014 ms 2024-04-26T21:34:54.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:54 smithi001 ceph-mon[28054]: pgmap v140: 129 pgs: 129 active+clean; 62 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 959 KiB/s wr, 127 op/s 2024-04-26T21:34:55.550 INFO:tasks.workunit.client.0.smithi001.stdout: 1 99925 3.46 MB/sec execute 53 sec latency 23.302 ms 2024-04-26T21:34:56.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:56 smithi186 ceph-mon[32991]: pgmap v141: 129 pgs: 129 active+clean; 64 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.4 MiB/s wr, 174 op/s 2024-04-26T21:34:56.550 INFO:tasks.workunit.client.0.smithi001.stdout: 1 100421 3.43 MB/sec execute 54 sec latency 441.032 ms 2024-04-26T21:34:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:56 smithi001 ceph-mon[28054]: pgmap v141: 129 pgs: 129 active+clean; 64 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.4 MiB/s wr, 174 op/s 2024-04-26T21:34:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:56 smithi038 ceph-mon[32724]: pgmap v141: 129 pgs: 129 active+clean; 64 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.4 MiB/s wr, 174 op/s 2024-04-26T21:34:57.550 INFO:tasks.workunit.client.0.smithi001.stdout: 1 101829 3.47 MB/sec execute 55 sec latency 24.066 ms 2024-04-26T21:34:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:34:58 smithi186 ceph-mon[32991]: pgmap v142: 129 pgs: 129 active+clean; 64 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 868 KiB/s wr, 115 op/s 2024-04-26T21:34:58.550 INFO:tasks.workunit.client.0.smithi001.stdout: 1 103128 3.61 MB/sec execute 56 sec latency 93.354 ms 2024-04-26T21:34:58.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:34:58 smithi001 ceph-mon[28054]: pgmap v142: 129 pgs: 129 active+clean; 64 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 868 KiB/s wr, 115 op/s 2024-04-26T21:34:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:34:58 smithi038 ceph-mon[32724]: pgmap v142: 129 pgs: 129 active+clean; 64 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 868 KiB/s wr, 115 op/s 2024-04-26T21:34:59.551 INFO:tasks.workunit.client.0.smithi001.stdout: 1 103382 3.55 MB/sec execute 57 sec latency 25.793 ms 2024-04-26T21:34:59.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.659+0000 7f4c25600640 1 -- 172.21.15.1:0/4133732722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c20103910 msgr2=0x7f4c20103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.659+0000 7f4c25600640 1 --2- 172.21.15.1:0/4133732722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c20103910 0x7f4c20103cf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009ca0 tx=0x7f4c1002a940 comp rx=0 tx=0).stop 2024-04-26T21:34:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.660+0000 7f4c25600640 1 -- 172.21.15.1:0/4133732722 shutdown_connections 2024-04-26T21:34:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.660+0000 7f4c25600640 1 --2- 172.21.15.1:0/4133732722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c20104230 0x7f4c2010c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.660+0000 7f4c25600640 1 --2- 172.21.15.1:0/4133732722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c2010cd30 0x7f4c2010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.660+0000 7f4c25600640 1 --2- 172.21.15.1:0/4133732722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c20103910 0x7f4c20103cf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.660+0000 7f4c25600640 1 -- 172.21.15.1:0/4133732722 >> 172.21.15.1:0/4133732722 conn(0x7f4c200fd110 msgr2=0x7f4c200ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.660+0000 7f4c25600640 1 -- 172.21.15.1:0/4133732722 shutdown_connections 2024-04-26T21:34:59.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.660+0000 7f4c25600640 1 -- 172.21.15.1:0/4133732722 wait complete. 2024-04-26T21:34:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.661+0000 7f4c25600640 1 Processor -- start 2024-04-26T21:34:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.661+0000 7f4c25600640 1 -- start start 2024-04-26T21:34:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.661+0000 7f4c25600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c20103910 0x7f4c2019b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.661+0000 7f4c25600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c20104230 0x7f4c2019b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.661+0000 7f4c25600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c2010cd30 0x7f4c2019f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.661+0000 7f4c25600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4c20112a30 con 0x7f4c20103910 2024-04-26T21:34:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.661+0000 7f4c25600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c201128b0 con 0x7f4c20104230 2024-04-26T21:34:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.661+0000 7f4c25600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c20112bb0 con 0x7f4c2010cd30 2024-04-26T21:34:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.662+0000 7f4c1fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c20103910 0x7f4c2019b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.662+0000 7f4c1fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c20103910 0x7f4c2019b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39608/0 (socket says 172.21.15.1:39608) 2024-04-26T21:34:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.662+0000 7f4c1fe00640 1 -- 172.21.15.1:0/397878388 learned_addr learned my addr 172.21.15.1:0/397878388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.662+0000 7f4c24c00640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c2010cd30 0x7f4c2019f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.662+0000 7f4c1f400640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c20104230 0x7f4c2019b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.662+0000 7f4c1fe00640 1 -- 172.21.15.1:0/397878388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c2010cd30 msgr2=0x7f4c2019f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:59.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.662+0000 7f4c1fe00640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c2010cd30 0x7f4c2019f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.662+0000 7f4c1fe00640 1 -- 172.21.15.1:0/397878388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c20104230 msgr2=0x7f4c2019b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.662+0000 7f4c1fe00640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c20104230 0x7f4c2019b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.662+0000 7f4c1fe00640 1 -- 172.21.15.1:0/397878388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c201a0110 con 0x7f4c20103910 2024-04-26T21:34:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.663+0000 7f4c1fe00640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c20103910 0x7f4c2019b170 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4c10005e30 tx=0x7f4c100309b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.663+0000 7f4c1cc00640 1 -- 172.21.15.1:0/397878388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10034040 con 0x7f4c20103910 2024-04-26T21:34:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.663+0000 7f4c1cc00640 1 -- 172.21.15.1:0/397878388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c1002fd10 con 0x7f4c20103910 2024-04-26T21:34:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.663+0000 7f4c1cc00640 1 -- 172.21.15.1:0/397878388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10004670 con 0x7f4c20103910 2024-04-26T21:34:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.663+0000 7f4c25600640 1 -- 172.21.15.1:0/397878388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c201a03a0 con 0x7f4c20103910 2024-04-26T21:34:59.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.663+0000 7f4c25600640 1 -- 172.21.15.1:0/397878388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c201a7c90 con 0x7f4c20103910 2024-04-26T21:34:59.667 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.667+0000 7f4c2996c740 1 Processor -- start 2024-04-26T21:34:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.667+0000 7f4c1cc00640 1 -- 172.21.15.1:0/397878388 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f4c10037020 con 0x7f4c20103910 2024-04-26T21:34:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.667+0000 7f4c2996c740 1 -- start start 2024-04-26T21:34:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 0x55ad33bcb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcb900 0x55ad33bd5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 0x55ad33bd85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c2996c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ad33b30380 con 0x55ad33bcb900 2024-04-26T21:34:59.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c2996c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ad33accc30 con 0x55ad33bd61c0 2024-04-26T21:34:59.669 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c2996c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ad33b2c340 con 0x55ad33bcafe0 2024-04-26T21:34:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c24c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 0x55ad33bcb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c1fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcb900 0x55ad33bd5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c24c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 0x55ad33bcb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58890/0 (socket says 172.21.15.1:58890) 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c24c00640 1 -- 172.21.15.1:0/4087561166 learned_addr learned my addr 172.21.15.1:0/4087561166 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.668+0000 7f4c1f400640 1 --2- 172.21.15.1:0/4087561166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 0x55ad33bd85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.669+0000 7f4c24c00640 1 -- 172.21.15.1:0/4087561166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 msgr2=0x55ad33bd85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.669+0000 7f4c24c00640 1 --2- 172.21.15.1:0/4087561166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 0x55ad33bd85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.669+0000 7f4c24c00640 1 -- 172.21.15.1:0/4087561166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcb900 msgr2=0x55ad33bd5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.669+0000 7f4c24c00640 1 --2- 172.21.15.1:0/4087561166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcb900 0x55ad33bd5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.669+0000 7f4c24c00640 1 -- 172.21.15.1:0/4087561166 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad33aee940 con 0x55ad33bcafe0 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.669+0000 7f4c1f400640 1 --2- 172.21.15.1:0/4087561166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 0x55ad33bd85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.669+0000 7f4c1cc00640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4c040a3d00 0x7f4c040a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.669+0000 7f4c24c00640 1 --2- 172.21.15.1:0/4087561166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 0x55ad33bcb3c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009c20 tx=0x7f4c18011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30026c371e14218 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.669+0000 7f4c1cc00640 1 -- 172.21.15.1:0/397878388 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f4c100f1a20 con 0x7f4c20103910 2024-04-26T21:34:59.671 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.670+0000 7f4c1fe00640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4c040a3d00 0x7f4c040a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.670+0000 7f4c15200640 1 -- 172.21.15.1:0/4087561166 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1801e070 con 0x55ad33bcafe0 2024-04-26T21:34:59.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.670+0000 7f4c15200640 1 -- 172.21.15.1:0/4087561166 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c18019ae0 con 0x55ad33bcafe0 2024-04-26T21:34:59.672 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.670+0000 7f4c15200640 1 -- 172.21.15.1:0/4087561166 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1800e960 con 0x55ad33bcafe0 2024-04-26T21:34:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 -- 172.21.15.1:0/4087561166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 msgr2=0x55ad33bcb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 --2- 172.21.15.1:0/4087561166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 0x55ad33bcb3c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009c20 tx=0x7f4c18011810 comp rx=0 tx=0).stop 2024-04-26T21:34:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c1fe00640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4c040a3d00 0x7f4c040a61c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4c100eef20 tx=0x7f4c100f54c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 -- 172.21.15.1:0/4087561166 shutdown_connections 2024-04-26T21:34:59.675 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 --2- 172.21.15.1:0/4087561166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 0x55ad33bd85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 --2- 172.21.15.1:0/4087561166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcb900 0x55ad33bd5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 --2- 172.21.15.1:0/4087561166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 0x55ad33bcb3c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 -- 172.21.15.1:0/4087561166 >> 172.21.15.1:0/4087561166 conn(0x55ad33bc74f0 msgr2=0x55ad33bc78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 -- 172.21.15.1:0/4087561166 shutdown_connections 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 -- 172.21.15.1:0/4087561166 wait complete. 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 -- start start 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.671+0000 7f4c2996c740 1 Processor -- start 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcafe0 0x55ad33bd3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33bd36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c2996c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ad33bd8df0 con 0x55ad33bd61c0 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c2996c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ad33ab7c60 con 0x55ad33bcafe0 2024-04-26T21:34:59.676 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c2996c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ad33bd8f70 con 0x55ad33bcb900 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c2996c740 10 client.0 _ll_register_callbacks cb 0x55ad33ae3e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c24c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c1f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33bd36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c24c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39628/0 (socket says 172.21.15.1:39628) 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c24c00640 1 -- 172.21.15.1:0/3303041256 learned_addr learned my addr 172.21.15.1:0/3303041256 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.672+0000 7f4c1f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33bd36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58894/0 (socket says 172.21.15.1:58894) 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c24c00640 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcafe0 0x55ad33bd3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c1f400640 1 -- 172.21.15.1:0/3303041256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcafe0 msgr2=0x55ad33bd3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c1f400640 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcafe0 0x55ad33bd3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c1f400640 1 -- 172.21.15.1:0/3303041256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 msgr2=0x55ad33bd3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c1f400640 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c1f400640 1 -- 172.21.15.1:0/3303041256 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad33ac6510 con 0x55ad33bcb900 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c24c00640 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:59.677 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c1f400640 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33bd36d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x55ad33bc8680 tx=0x7f4c08012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99ac6531239ffdfc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c0d600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c0d600640 1 -- 172.21.15.1:0/3303041256 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08004220 con 0x55ad33bcb900 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.673+0000 7f4c0d600640 1 -- 172.21.15.1:0/3303041256 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f4c080043c0 con 0x55ad33bcb900 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c2996c740 1 -- 172.21.15.1:0/3303041256 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c18019ae0 con 0x55ad33bcb900 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c0d600640 1 -- 172.21.15.1:0/3303041256 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4c08025050 con 0x55ad33bcb900 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c2996c740 10 client.25873 fetch_fsmap learned FSMap version 34 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c2996c740 1 -- 172.21.15.1:0/3303041256 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ad33842210 con 0x55ad33bcb900 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c0d600640 1 -- 172.21.15.1:0/3303041256 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c18019ae0 con 0x55ad33bcb900 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c0d600640 1 -- 172.21.15.1:0/3303041256 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f4c08019030 con 0x55ad33bcb900 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c2996c740 10 client.25873 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c2996c740 10 client.25873 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c2996c740 20 client.25873 populate_metadata read hostname 'smithi001' 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c2996c740 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ad33be6000 0x55ad33be8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.678 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c2996c740 4 client.25873 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:34:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.674+0000 7f4c2996c740 1 -- 172.21.15.1:0/3303041256 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad33b3df90 con 0x55ad33be6000 2024-04-26T21:34:59.679 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.675+0000 7f4c24c00640 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ad33be6000 0x55ad33be8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.675+0000 7f4c24c00640 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ad33be6000 0x55ad33be8410 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3496c91e5c81778f server_cookie=da4217976930c8a9 in_seq=0 out_seq=0 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.675+0000 7f4c0d600640 10 client.25873 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.676+0000 7f4c0d600640 1 -- 172.21.15.1:0/3303041256 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ad33b3df90 con 0x55ad33be6000 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.676+0000 7f4c0d600640 10 client.25873 handle_command_reply: tid=0 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.676+0000 7f4c2996c740 1 client.25873 shutdown 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.676+0000 7f4c2996c740 1 -- 172.21.15.1:0/3303041256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 msgr2=0x55ad33bd36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.676+0000 7f4c2996c740 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33bd36d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x55ad33bc8680 tx=0x7f4c08012820 comp rx=0 tx=0).stop 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.676+0000 7f4c2996c740 1 -- 172.21.15.1:0/3303041256 shutdown_connections 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.676+0000 7f4c2996c740 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ad33be6000 0x55ad33be8410 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.676+0000 7f4c2996c740 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33bd36d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.676+0000 7f4c2996c740 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.676+0000 7f4c2996c740 1 --2- 172.21.15.1:0/3303041256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcafe0 0x55ad33bd3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.677+0000 7f4c0d600640 0 client.25873 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.677+0000 7f4c2996c740 1 -- 172.21.15.1:0/3303041256 >> 172.21.15.1:0/3303041256 conn(0x55ad33bc74f0 msgr2=0x55ad33bcd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.677+0000 7f4c2996c740 1 -- 172.21.15.1:0/3303041256 shutdown_connections 2024-04-26T21:34:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.677+0000 7f4c2996c740 1 -- 172.21.15.1:0/3303041256 wait complete. 2024-04-26T21:34:59.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.677+0000 7f4c2996c740 20 client.25873 trim_cache size 0 max 16384 2024-04-26T21:34:59.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.693+0000 7f4c2996c740 1 Processor -- start 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.693+0000 7f4c2996c740 1 -- start start 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcafe0 0x55ad33bd58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33be1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c2996c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4c201b47a0 con 0x55ad33bcafe0 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c2996c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c201b2010 con 0x55ad33bd61c0 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c2996c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c201afe60 con 0x55ad33bcb900 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c24c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcafe0 0x55ad33bd58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c24c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcafe0 0x55ad33bd58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39634/0 (socket says 172.21.15.1:39634) 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c24c00640 1 -- 172.21.15.1:0/946924208 learned_addr learned my addr 172.21.15.1:0/946924208 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c1f400640 1 --2- 172.21.15.1:0/946924208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33be1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c24c00640 1 --2- 172.21.15.1:0/946924208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.694+0000 7f4c1f400640 1 -- 172.21.15.1:0/946924208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 msgr2=0x55ad33bd3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.695+0000 7f4c1f400640 1 --2- 172.21.15.1:0/946924208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.695+0000 7f4c1f400640 1 -- 172.21.15.1:0/946924208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcafe0 msgr2=0x55ad33bd58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.695+0000 7f4c1f400640 1 --2- 172.21.15.1:0/946924208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcafe0 0x55ad33bd58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.695+0000 7f4c1f400640 1 -- 172.21.15.1:0/946924208 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad3390e620 con 0x55ad33bcb900 2024-04-26T21:34:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.695+0000 7f4c24c00640 1 --2- 172.21.15.1:0/946924208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:34:59.697 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.695+0000 7f4c1f400640 1 --2- 172.21.15.1:0/946924208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33be1cb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4c08003990 tx=0x7f4c08005c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75716022f6fabc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.695+0000 7f4c0fe00640 1 -- 172.21.15.1:0/946924208 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c08000e60 con 0x55ad33bcb900 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.695+0000 7f4c0fe00640 1 -- 172.21.15.1:0/946924208 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c08004480 con 0x55ad33bcb900 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.695+0000 7f4c0fe00640 1 -- 172.21.15.1:0/946924208 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0802d070 con 0x55ad33bcb900 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.696+0000 7f4c2996c740 1 -- 172.21.15.1:0/946924208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 msgr2=0x55ad33be1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.696+0000 7f4c2996c740 1 --2- 172.21.15.1:0/946924208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33be1cb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4c08003990 tx=0x7f4c08005c10 comp rx=0 tx=0).stop 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.696+0000 7f4c2996c740 1 -- 172.21.15.1:0/946924208 shutdown_connections 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.696+0000 7f4c2996c740 1 --2- 172.21.15.1:0/946924208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcb900 0x55ad33be1cb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.696+0000 7f4c2996c740 1 --2- 172.21.15.1:0/946924208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bd61c0 0x55ad33bd3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.696+0000 7f4c2996c740 1 --2- 172.21.15.1:0/946924208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bcafe0 0x55ad33bd58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.696+0000 7f4c2996c740 1 -- 172.21.15.1:0/946924208 >> 172.21.15.1:0/946924208 conn(0x55ad33bc74f0 msgr2=0x55ad33bd3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.696+0000 7f4c2996c740 1 -- 172.21.15.1:0/946924208 shutdown_connections 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.696+0000 7f4c2996c740 1 -- 172.21.15.1:0/946924208 wait complete. 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.697+0000 7f4c2996c740 1 -- start start 2024-04-26T21:34:59.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.697+0000 7f4c2996c740 1 Processor -- start 2024-04-26T21:34:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.697+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 0x55ad33bde730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.697+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcb900 0x55ad33bcfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.697+0000 7f4c2996c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 0x55ad33bd0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.697+0000 7f4c2996c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4c201b04d0 con 0x55ad33bd61c0 2024-04-26T21:34:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.697+0000 7f4c2996c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f4c201b1940 con 0x55ad33bcb900 2024-04-26T21:34:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.697+0000 7f4c2996c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c201b0ba0 con 0x55ad33bcafe0 2024-04-26T21:34:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.697+0000 7f4c2996c740 10 client.0 _ll_register_callbacks cb 0x7f4c0800ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:34:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.698+0000 7f4c1f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcb900 0x55ad33bcfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.698+0000 7f4c24c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 0x55ad33bde730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.698+0000 7f4c1f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcb900 0x55ad33bcfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:35660/0 (socket says 172.21.15.1:35660) 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.698+0000 7f4c1f400640 1 -- 172.21.15.1:0/2031598623 learned_addr learned my addr 172.21.15.1:0/2031598623 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.698+0000 7f4c24c00640 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 0x55ad33bd0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.698+0000 7f4c1f400640 1 -- 172.21.15.1:0/2031598623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 msgr2=0x55ad33bde730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.698+0000 7f4c1f400640 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 0x55ad33bde730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.698+0000 7f4c1f400640 1 -- 172.21.15.1:0/2031598623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 msgr2=0x55ad33bd0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.698+0000 7f4c1f400640 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 0x55ad33bd0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.698+0000 7f4c1f400640 1 -- 172.21.15.1:0/2031598623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad33aeacb0 con 0x55ad33bcb900 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.699+0000 7f4c1f400640 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcb900 0x55ad33bcfca0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4c08003ab0 tx=0x7f4c08003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b071c4603862a5ed server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.699+0000 7f4c0d600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.699+0000 7f4c0d600640 1 -- 172.21.15.1:0/2031598623 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c0802d070 con 0x55ad33bcb900 2024-04-26T21:34:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.699+0000 7f4c2996c740 1 -- 172.21.15.1:0/2031598623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ad33b37e00 con 0x55ad33bcb900 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.699+0000 7f4c0d600640 1 -- 172.21.15.1:0/2031598623 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f4c08005520 con 0x55ad33bcb900 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.699+0000 7f4c0d600640 1 -- 172.21.15.1:0/2031598623 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f4c08003d90 con 0x55ad33bcb900 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.699+0000 7f4c2996c740 10 client.25895 fetch_fsmap learned FSMap version 34 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.699+0000 7f4c2996c740 1 -- 172.21.15.1:0/2031598623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55ad33c4c4a0 con 0x55ad33bcb900 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.699+0000 7f4c0d600640 1 -- 172.21.15.1:0/2031598623 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55ad33b37e00 con 0x55ad33bcb900 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.700+0000 7f4c0d600640 1 -- 172.21.15.1:0/2031598623 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7f4c08039070 con 0x55ad33bcb900 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.700+0000 7f4c2996c740 10 client.25895 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.700+0000 7f4c2996c740 10 client.25895 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.700+0000 7f4c2996c740 20 client.25895 populate_metadata read hostname 'smithi001' 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.700+0000 7f4c2996c740 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ad33c4cae0 0x55ad33c4ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.700+0000 7f4c2996c740 4 client.25895 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.700+0000 7f4c2996c740 1 -- 172.21.15.1:0/2031598623 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ad33ab34b0 con 0x55ad33c4cae0 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.700+0000 7f4c24c00640 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ad33c4cae0 0x55ad33c4ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.700+0000 7f4c24c00640 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ad33c4cae0 0x55ad33c4ef00 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8446f024096629a2 server_cookie=5fad6c94e6b30acf in_seq=0 out_seq=0 2024-04-26T21:34:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.700+0000 7f4c0d600640 10 client.25895 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:59.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.852+0000 7f4c0d600640 1 -- 172.21.15.1:0/2031598623 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ad33ab34b0 con 0x55ad33c4cae0 2024-04-26T21:34:59.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.852+0000 7f4c0d600640 10 client.25895 handle_command_reply: tid=0 2024-04-26T21:34:59.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.852+0000 7f4c2996c740 1 client.25895 shutdown 2024-04-26T21:34:59.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 1 -- 172.21.15.1:0/2031598623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcb900 msgr2=0x55ad33bcfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:59.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcb900 0x55ad33bcfca0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4c08003ab0 tx=0x7f4c08003820 comp rx=0 tx=0).stop 2024-04-26T21:34:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 1 -- 172.21.15.1:0/2031598623 shutdown_connections 2024-04-26T21:34:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ad33c4cae0 0x55ad33c4ef00 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ad33bd61c0 0x55ad33bd0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ad33bcb900 0x55ad33bcfca0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 1 --2- 172.21.15.1:0/2031598623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ad33bcafe0 0x55ad33bde730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c0d600640 0 client.25895 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:34:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 1 -- 172.21.15.1:0/2031598623 >> 172.21.15.1:0/2031598623 conn(0x55ad33bc74f0 msgr2=0x55ad33bc6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 1 -- 172.21.15.1:0/2031598623 shutdown_connections 2024-04-26T21:34:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 1 -- 172.21.15.1:0/2031598623 wait complete. 2024-04-26T21:34:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.853+0000 7f4c2996c740 20 client.25895 trim_cache size 0 max 16384 2024-04-26T21:34:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:34:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:34:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:34:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 -- 172.21.15.1:0/397878388 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4c040a3d00 msgr2=0x7f4c040a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4c040a3d00 0x7f4c040a61c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4c100eef20 tx=0x7f4c100f54c0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 -- 172.21.15.1:0/397878388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c20103910 msgr2=0x7f4c2019b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c20103910 0x7f4c2019b170 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4c10005e30 tx=0x7f4c100309b0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 -- 172.21.15.1:0/397878388 shutdown_connections 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f4c040a3d00 0x7f4c040a61c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c2010cd30 0x7f4c2019f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c20104230 0x7f4c2019b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 --2- 172.21.15.1:0/397878388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c20103910 0x7f4c2019b170 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 -- 172.21.15.1:0/397878388 >> 172.21.15.1:0/397878388 conn(0x7f4c200fd110 msgr2=0x7f4c20101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 -- 172.21.15.1:0/397878388 shutdown_connections 2024-04-26T21:34:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:34:59.854+0000 7f4c25600640 1 -- 172.21.15.1:0/397878388 wait complete. 2024-04-26T21:35:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:00 smithi186 ceph-mon[32991]: pgmap v143: 129 pgs: 129 active+clean; 64 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 868 KiB/s wr, 115 op/s 2024-04-26T21:35:00.550 INFO:tasks.workunit.client.0.smithi001.stdout: 1 104310 3.58 MB/sec execute 58 sec latency 24.359 ms 2024-04-26T21:35:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:00 smithi001 ceph-mon[28054]: pgmap v143: 129 pgs: 129 active+clean; 64 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 868 KiB/s wr, 115 op/s 2024-04-26T21:35:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:00 smithi038 ceph-mon[32724]: pgmap v143: 129 pgs: 129 active+clean; 64 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 868 KiB/s wr, 115 op/s 2024-04-26T21:35:01.550 INFO:tasks.workunit.client.0.smithi001.stdout: 1 105676 3.62 MB/sec execute 59 sec latency 26.033 ms 2024-04-26T21:35:02.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:02 smithi186 ceph-mon[32991]: pgmap v144: 129 pgs: 129 active+clean; 65 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.4 MiB/s wr, 171 op/s 2024-04-26T21:35:02.551 INFO:tasks.workunit.client.0.smithi001.stdout: 1 106165 3.61 MB/sec execute 60 sec latency 106.970 ms 2024-04-26T21:35:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:02 smithi001 ceph-mon[28054]: pgmap v144: 129 pgs: 129 active+clean; 65 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.4 MiB/s wr, 171 op/s 2024-04-26T21:35:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:02 smithi038 ceph-mon[32724]: pgmap v144: 129 pgs: 129 active+clean; 65 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.4 MiB/s wr, 171 op/s 2024-04-26T21:35:03.551 INFO:tasks.workunit.client.0.smithi001.stdout: 1 106778 3.63 MB/sec execute 61 sec latency 90.426 ms 2024-04-26T21:35:04.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:04 smithi186 ceph-mon[32991]: pgmap v145: 129 pgs: 129 active+clean; 65 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 953 KiB/s wr, 102 op/s 2024-04-26T21:35:04.551 INFO:tasks.workunit.client.0.smithi001.stdout: 1 107363 3.59 MB/sec execute 62 sec latency 26.513 ms 2024-04-26T21:35:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:04 smithi001 ceph-mon[28054]: pgmap v145: 129 pgs: 129 active+clean; 65 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 953 KiB/s wr, 102 op/s 2024-04-26T21:35:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:04 smithi038 ceph-mon[32724]: pgmap v145: 129 pgs: 129 active+clean; 65 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 953 KiB/s wr, 102 op/s 2024-04-26T21:35:05.551 INFO:tasks.workunit.client.0.smithi001.stdout: 1 108841 3.63 MB/sec execute 63 sec latency 23.250 ms 2024-04-26T21:35:06.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:06 smithi186 ceph-mon[32991]: pgmap v146: 129 pgs: 129 active+clean; 69 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.4 MiB/s wr, 211 op/s 2024-04-26T21:35:06.551 INFO:tasks.workunit.client.0.smithi001.stdout: 1 110003 3.73 MB/sec execute 64 sec latency 22.806 ms 2024-04-26T21:35:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:06 smithi001 ceph-mon[28054]: pgmap v146: 129 pgs: 129 active+clean; 69 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.4 MiB/s wr, 211 op/s 2024-04-26T21:35:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:06 smithi038 ceph-mon[32724]: pgmap v146: 129 pgs: 129 active+clean; 69 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 1.4 MiB/s wr, 211 op/s 2024-04-26T21:35:07.551 INFO:tasks.workunit.client.0.smithi001.stdout: 1 110425 3.70 MB/sec execute 65 sec latency 32.138 ms 2024-04-26T21:35:08.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:08 smithi186 ceph-mon[32991]: pgmap v147: 129 pgs: 129 active+clean; 69 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1008 KiB/s wr, 164 op/s 2024-04-26T21:35:08.552 INFO:tasks.workunit.client.0.smithi001.stdout: 1 111073 3.68 MB/sec execute 66 sec latency 22.435 ms 2024-04-26T21:35:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:08 smithi001 ceph-mon[28054]: pgmap v147: 129 pgs: 129 active+clean; 69 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1008 KiB/s wr, 164 op/s 2024-04-26T21:35:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:08 smithi038 ceph-mon[32724]: pgmap v147: 129 pgs: 129 active+clean; 69 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1008 KiB/s wr, 164 op/s 2024-04-26T21:35:09.551 INFO:tasks.workunit.client.0.smithi001.stdout: 1 112471 3.71 MB/sec execute 67 sec latency 32.148 ms 2024-04-26T21:35:10.369 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:10 smithi001 ceph-mon[28054]: pgmap v148: 129 pgs: 129 active+clean; 69 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1008 KiB/s wr, 164 op/s 2024-04-26T21:35:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:10 smithi186 ceph-mon[32991]: pgmap v148: 129 pgs: 129 active+clean; 69 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1008 KiB/s wr, 164 op/s 2024-04-26T21:35:10.551 INFO:tasks.workunit.client.0.smithi001.stdout: 1 113759 3.82 MB/sec execute 68 sec latency 23.027 ms 2024-04-26T21:35:10.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:10 smithi038 ceph-mon[32724]: pgmap v148: 129 pgs: 129 active+clean; 69 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 1008 KiB/s wr, 164 op/s 2024-04-26T21:35:11.552 INFO:tasks.workunit.client.0.smithi001.stdout: 1 113986 3.76 MB/sec execute 69 sec latency 40.431 ms 2024-04-26T21:35:12.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:12 smithi186 ceph-mon[32991]: pgmap v149: 129 pgs: 129 active+clean; 71 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 1.5 MiB/s wr, 226 op/s 2024-04-26T21:35:12.552 INFO:tasks.workunit.client.0.smithi001.stdout: 1 114707 3.74 MB/sec execute 70 sec latency 20.572 ms 2024-04-26T21:35:12.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:12 smithi001 ceph-mon[28054]: pgmap v149: 129 pgs: 129 active+clean; 71 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 1.5 MiB/s wr, 226 op/s 2024-04-26T21:35:12.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:12 smithi038 ceph-mon[32724]: pgmap v149: 129 pgs: 129 active+clean; 71 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 1.5 MiB/s wr, 226 op/s 2024-04-26T21:35:13.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.006+0000 7fa6efe00640 1 -- 172.21.15.1:0/2306732258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f010ef90 msgr2=0x7fa6f01113e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.006+0000 7fa6efe00640 1 --2- 172.21.15.1:0/2306732258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f010ef90 0x7fa6f01113e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009ca0 tx=0x7fa6e802a950 comp rx=0 tx=0).stop 2024-04-26T21:35:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.007+0000 7fa6efe00640 1 -- 172.21.15.1:0/2306732258 shutdown_connections 2024-04-26T21:35:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.007+0000 7fa6efe00640 1 --2- 172.21.15.1:0/2306732258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f010ef90 0x7fa6f01113e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.007+0000 7fa6efe00640 1 --2- 172.21.15.1:0/2306732258 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6f0075680 0x7fa6f010ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.007+0000 7fa6efe00640 1 --2- 172.21.15.1:0/2306732258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6f0074d60 0x7fa6f0075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.007+0000 7fa6efe00640 1 -- 172.21.15.1:0/2306732258 >> 172.21.15.1:0/2306732258 conn(0x7fa6f00fe240 msgr2=0x7fa6f0100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.007+0000 7fa6efe00640 1 -- 172.21.15.1:0/2306732258 shutdown_connections 2024-04-26T21:35:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.007+0000 7fa6efe00640 1 -- 172.21.15.1:0/2306732258 wait complete. 2024-04-26T21:35:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.008+0000 7fa6efe00640 1 Processor -- start 2024-04-26T21:35:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.008+0000 7fa6efe00640 1 -- start start 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6efe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f0074d60 0x7fa6f019b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6efe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6f0075680 0x7fa6f019b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6efe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6f010ef90 0x7fa6f019f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6efe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f0114c30 con 0x7fa6f0075680 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6efe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f0114ab0 con 0x7fa6f0074d60 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6efe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f0114db0 con 0x7fa6f010ef90 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6eea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f0074d60 0x7fa6f019b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6f010ef90 0x7fa6f019f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6eea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f0074d60 0x7fa6f019b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:57526/0 (socket says 172.21.15.1:57526) 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6eea00640 1 -- 172.21.15.1:0/3151890302 learned_addr learned my addr 172.21.15.1:0/3151890302 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6ee000640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6f0075680 0x7fa6f019b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.009+0000 7fa6ef400640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6f010ef90 0x7fa6f019f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52082/0 (socket says 172.21.15.1:52082) 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.010+0000 7fa6eea00640 1 -- 172.21.15.1:0/3151890302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6f010ef90 msgr2=0x7fa6f019f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.010+0000 7fa6eea00640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6f010ef90 0x7fa6f019f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.010+0000 7fa6eea00640 1 -- 172.21.15.1:0/3151890302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6f0075680 msgr2=0x7fa6f019b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.010+0000 7fa6eea00640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6f0075680 0x7fa6f019b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.010+0000 7fa6eea00640 1 -- 172.21.15.1:0/3151890302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f01a00e0 con 0x7fa6f0074d60 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.010+0000 7fa6eea00640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f0074d60 0x7fa6f019b0f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000e9f0 tx=0x7fa6e000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.011+0000 7fa6e7400640 1 -- 172.21.15.1:0/3151890302 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e000cd80 con 0x7fa6f0074d60 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.011+0000 7fa6e7400640 1 -- 172.21.15.1:0/3151890302 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6e00054d0 con 0x7fa6f0074d60 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.011+0000 7fa6efe00640 1 -- 172.21.15.1:0/3151890302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f01a03d0 con 0x7fa6f0074d60 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.011+0000 7fa6e7400640 1 -- 172.21.15.1:0/3151890302 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0010690 con 0x7fa6f0074d60 2024-04-26T21:35:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.011+0000 7fa6efe00640 1 -- 172.21.15.1:0/3151890302 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f01a7c60 con 0x7fa6f0074d60 2024-04-26T21:35:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.016+0000 7fa6f83c5740 1 Processor -- start 2024-04-26T21:35:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.016+0000 7fa6f83c5740 1 -- start start 2024-04-26T21:35:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.016+0000 7fa6e7400640 1 -- 172.21.15.1:0/3151890302 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa6e0004e90 con 0x7fa6f0074d60 2024-04-26T21:35:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d56fe0 0x5589e1d573c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 0x5589e1d61c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 0x5589e1d645b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5589e1cbc380 con 0x5589e1d57900 2024-04-26T21:35:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5589e1c58c30 con 0x5589e1d621c0 2024-04-26T21:35:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5589e1cb8340 con 0x5589e1d56fe0 2024-04-26T21:35:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6eea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 0x5589e1d61c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d56fe0 0x5589e1d573c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6ee000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 0x5589e1d645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d56fe0 0x5589e1d573c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52096/0 (socket says 172.21.15.1:52096) 2024-04-26T21:35:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6ef400640 1 -- 172.21.15.1:0/2316527186 learned_addr learned my addr 172.21.15.1:0/2316527186 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6ee000640 1 -- 172.21.15.1:0/2316527186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d56fe0 msgr2=0x5589e1d573c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6ee000640 1 --2- 172.21.15.1:0/2316527186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d56fe0 0x5589e1d573c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6ee000640 1 -- 172.21.15.1:0/2316527186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 msgr2=0x5589e1d61c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6ee000640 1 --2- 172.21.15.1:0/2316527186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 0x5589e1d61c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.017+0000 7fa6ee000640 1 -- 172.21.15.1:0/2316527186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589e1c7a940 con 0x5589e1d621c0 2024-04-26T21:35:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.018+0000 7fa6ee000640 1 --2- 172.21.15.1:0/2316527186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 0x5589e1d645b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009bf0 tx=0x7fa6dc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35fac77eee946261 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.018+0000 7fa6e7400640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa6d40a3d00 0x7fa6d40a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.018+0000 7fa6db400640 1 -- 172.21.15.1:0/2316527186 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc004050 con 0x5589e1d621c0 2024-04-26T21:35:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.018+0000 7fa6db400640 1 -- 172.21.15.1:0/2316527186 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6dc0054d0 con 0x5589e1d621c0 2024-04-26T21:35:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.018+0000 7fa6db400640 1 -- 172.21.15.1:0/2316527186 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc004ca0 con 0x5589e1d621c0 2024-04-26T21:35:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.018+0000 7fa6e7400640 1 -- 172.21.15.1:0/3151890302 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fa6e00d1060 con 0x7fa6f0074d60 2024-04-26T21:35:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.018+0000 7fa6ef400640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa6d40a3d00 0x7fa6d40a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.019+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2316527186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 msgr2=0x5589e1d645b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.019+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/2316527186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 0x5589e1d645b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc009bf0 tx=0x7fa6dc011810 comp rx=0 tx=0).stop 2024-04-26T21:35:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.019+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2316527186 shutdown_connections 2024-04-26T21:35:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.019+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/2316527186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 0x5589e1d645b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.019+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/2316527186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 0x5589e1d61c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.019+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/2316527186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d56fe0 0x5589e1d573c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.019+0000 7fa6ef400640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa6d40a3d00 0x7fa6d40a61c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009a60 tx=0x7fa6e8005d80 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.019+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2316527186 >> 172.21.15.1:0/2316527186 conn(0x5589e1d534f0 msgr2=0x5589e1d538d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.019+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2316527186 shutdown_connections 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.019+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2316527186 wait complete. 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.020+0000 7fa6f83c5740 1 -- start start 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.020+0000 7fa6f83c5740 1 Processor -- start 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.020+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d56fe0 0x5589e1d5f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.020+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d57900 0x5589e1d5f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.020+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d621c0 0x5589e1d5fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.020+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5589e1d64df0 con 0x5589e1d56fe0 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5589e1c43c60 con 0x5589e1d57900 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5589e1d64f70 con 0x5589e1d621c0 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6f83c5740 10 client.0 _ll_register_callbacks cb 0x5589e1c6fe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6ee000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d57900 0x5589e1d5f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6eea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d56fe0 0x5589e1d5f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6ee000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d57900 0x5589e1d5f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:57546/0 (socket says 172.21.15.1:57546) 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6ee000640 1 -- 172.21.15.1:0/2614457688 learned_addr learned my addr 172.21.15.1:0/2614457688 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6ef400640 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d621c0 0x5589e1d5fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6ee000640 1 -- 172.21.15.1:0/2614457688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d621c0 msgr2=0x5589e1d5fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6ee000640 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d621c0 0x5589e1d5fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6ee000640 1 -- 172.21.15.1:0/2614457688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d56fe0 msgr2=0x5589e1d5f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6ee000640 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d56fe0 0x5589e1d5f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6ee000640 1 -- 172.21.15.1:0/2614457688 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589e1c52510 con 0x5589e1d57900 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.021+0000 7fa6eea00640 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d56fe0 0x5589e1d5f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.022+0000 7fa6ee000640 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d57900 0x5589e1d5f5f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc01b040 tx=0x7fa6dc009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79fa85e674f16d7e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.022+0000 7fa6cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.022+0000 7fa6cfe00640 1 -- 172.21.15.1:0/2614457688 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc029070 con 0x5589e1d57900 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.022+0000 7fa6cfe00640 1 -- 172.21.15.1:0/2614457688 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fa6dc003a00 con 0x5589e1d57900 2024-04-26T21:35:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.022+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2614457688 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6dc0054d0 con 0x5589e1d57900 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.022+0000 7fa6cfe00640 1 -- 172.21.15.1:0/2614457688 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa6dc035050 con 0x5589e1d57900 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.022+0000 7fa6f83c5740 10 client.25907 fetch_fsmap learned FSMap version 34 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.022+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2614457688 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5589e19ce210 con 0x5589e1d57900 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.023+0000 7fa6cfe00640 1 -- 172.21.15.1:0/2614457688 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6dc0054d0 con 0x5589e1d57900 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.023+0000 7fa6cfe00640 1 -- 172.21.15.1:0/2614457688 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fa6dc004f90 con 0x5589e1d57900 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.023+0000 7fa6f83c5740 10 client.25907 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.023+0000 7fa6f83c5740 10 client.25907 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.023+0000 7fa6f83c5740 20 client.25907 populate_metadata read hostname 'smithi001' 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.023+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5589e1d71d50 0x5589e1d74140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.023+0000 7fa6f83c5740 4 client.25907 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.023+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2614457688 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5589e1cc9f90 con 0x5589e1d71d50 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.023+0000 7fa6eea00640 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5589e1d71d50 0x5589e1d74140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.024+0000 7fa6eea00640 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5589e1d71d50 0x5589e1d74140 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=570d00775100a8ef server_cookie=a732dc905b942a51 in_seq=0 out_seq=0 2024-04-26T21:35:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.024+0000 7fa6cfe00640 10 client.25907 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.024+0000 7fa6cfe00640 1 -- 172.21.15.1:0/2614457688 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5589e1cc9f90 con 0x5589e1d71d50 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.024+0000 7fa6cfe00640 10 client.25907 handle_command_reply: tid=0 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 client.25907 shutdown 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2614457688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d57900 msgr2=0x5589e1d5f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d57900 0x5589e1d5f5f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc01b040 tx=0x7fa6dc009fd0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2614457688 shutdown_connections 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5589e1d71d50 0x5589e1d74140 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d621c0 0x5589e1d5fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d57900 0x5589e1d5f5f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/2614457688 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d56fe0 0x5589e1d5f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2614457688 >> 172.21.15.1:0/2614457688 conn(0x5589e1d534f0 msgr2=0x5589e1d59270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6cfe00640 0 client.25907 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2614457688 shutdown_connections 2024-04-26T21:35:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 1 -- 172.21.15.1:0/2614457688 wait complete. 2024-04-26T21:35:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.025+0000 7fa6f83c5740 20 client.25907 trim_cache size 0 max 16384 2024-04-26T21:35:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.053+0000 7fa6f83c5740 1 Processor -- start 2024-04-26T21:35:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.054+0000 7fa6f83c5740 1 -- start start 2024-04-26T21:35:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.060+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d56fe0 0x5589e1d6dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.060+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 0x5589e1d60ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.060+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 0x5589e1d79860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.060+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f01b4810 con 0x5589e1d57900 2024-04-26T21:35:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.060+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f01b2080 con 0x5589e1d621c0 2024-04-26T21:35:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.060+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f01afed0 con 0x5589e1d56fe0 2024-04-26T21:35:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.061+0000 7fa6ee000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 0x5589e1d60ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.062+0000 7fa6ee000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 0x5589e1d60ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45700/0 (socket says 172.21.15.1:45700) 2024-04-26T21:35:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.062+0000 7fa6ee000640 1 -- 172.21.15.1:0/361776550 learned_addr learned my addr 172.21.15.1:0/361776550 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.062+0000 7fa6ef400640 1 --2- 172.21.15.1:0/361776550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 0x5589e1d79860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.064+0000 7fa6ef400640 1 -- 172.21.15.1:0/361776550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d56fe0 msgr2=0x5589e1d6dcb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:35:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.064+0000 7fa6ef400640 1 --2- 172.21.15.1:0/361776550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d56fe0 0x5589e1d6dcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.064+0000 7fa6ef400640 1 -- 172.21.15.1:0/361776550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 msgr2=0x5589e1d60ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.064+0000 7fa6ef400640 1 --2- 172.21.15.1:0/361776550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 0x5589e1d60ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.064+0000 7fa6ef400640 1 -- 172.21.15.1:0/361776550 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589e1a9a620 con 0x5589e1d621c0 2024-04-26T21:35:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.066+0000 7fa6ef400640 1 --2- 172.21.15.1:0/361776550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 0x5589e1d79860 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8004af0 tx=0x7fa6e8045000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6561d8375fee5522 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.066+0000 7fa6daa00640 1 -- 172.21.15.1:0/361776550 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8056070 con 0x5589e1d621c0 2024-04-26T21:35:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.066+0000 7fa6daa00640 1 -- 172.21.15.1:0/361776550 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6e802fbf0 con 0x5589e1d621c0 2024-04-26T21:35:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.066+0000 7fa6daa00640 1 -- 172.21.15.1:0/361776550 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8034040 con 0x5589e1d621c0 2024-04-26T21:35:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.068+0000 7fa6f83c5740 1 -- 172.21.15.1:0/361776550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 msgr2=0x5589e1d79860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.068+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/361776550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 0x5589e1d79860 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8004af0 tx=0x7fa6e8045000 comp rx=0 tx=0).stop 2024-04-26T21:35:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.068+0000 7fa6f83c5740 1 -- 172.21.15.1:0/361776550 shutdown_connections 2024-04-26T21:35:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.068+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/361776550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d621c0 0x5589e1d79860 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.068+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/361776550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d57900 0x5589e1d60ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.068+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/361776550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d56fe0 0x5589e1d6dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.068+0000 7fa6f83c5740 1 -- 172.21.15.1:0/361776550 >> 172.21.15.1:0/361776550 conn(0x5589e1d534f0 msgr2=0x5589e1d607d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.068+0000 7fa6f83c5740 1 -- 172.21.15.1:0/361776550 shutdown_connections 2024-04-26T21:35:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.069+0000 7fa6f83c5740 1 -- 172.21.15.1:0/361776550 wait complete. 2024-04-26T21:35:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.069+0000 7fa6f83c5740 1 -- start start 2024-04-26T21:35:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.071+0000 7fa6f83c5740 1 Processor -- start 2024-04-26T21:35:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.073+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d56fe0 0x5589e1d6a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.073+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d57900 0x5589e1d65a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.073+0000 7fa6f83c5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d621c0 0x5589e1d66030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.073+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f01b0540 con 0x5589e1d621c0 2024-04-26T21:35:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.073+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f01b19b0 con 0x5589e1d56fe0 2024-04-26T21:35:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.073+0000 7fa6f83c5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f01b0c10 con 0x5589e1d57900 2024-04-26T21:35:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.073+0000 7fa6f83c5740 10 client.0 _ll_register_callbacks cb 0x7fa6e00d49e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:35:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.076+0000 7fa6eea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d56fe0 0x5589e1d6a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.076+0000 7fa6ee000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d57900 0x5589e1d65a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.078+0000 7fa6ef400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d621c0 0x5589e1d66030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.078+0000 7fa6ee000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d57900 0x5589e1d65a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52126/0 (socket says 172.21.15.1:52126) 2024-04-26T21:35:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.078+0000 7fa6ee000640 1 -- 172.21.15.1:0/4096114796 learned_addr learned my addr 172.21.15.1:0/4096114796 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:13.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.078+0000 7fa6eea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d56fe0 0x5589e1d6a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:57564/0 (socket says 172.21.15.1:57564) 2024-04-26T21:35:13.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.080+0000 7fa6ee000640 1 -- 172.21.15.1:0/4096114796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d56fe0 msgr2=0x5589e1d6a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:13.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.080+0000 7fa6ee000640 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d56fe0 0x5589e1d6a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.080+0000 7fa6ee000640 1 -- 172.21.15.1:0/4096114796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d621c0 msgr2=0x5589e1d66030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:13.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.080+0000 7fa6ee000640 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d621c0 0x5589e1d66030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.080+0000 7fa6ee000640 1 -- 172.21.15.1:0/4096114796 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589e1c76cb0 con 0x5589e1d57900 2024-04-26T21:35:13.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.081+0000 7fa6ef400640 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d621c0 0x5589e1d66030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:35:13.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.081+0000 7fa6eea00640 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d56fe0 0x5589e1d6a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:35:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.082+0000 7fa6ee000640 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d57900 0x5589e1d65a20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc000c00 tx=0x7fa6dc005540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b60f4a0f73541eeb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.083+0000 7fa6cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:35:13.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.083+0000 7fa6cfe00640 1 -- 172.21.15.1:0/4096114796 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc029040 con 0x5589e1d57900 2024-04-26T21:35:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.083+0000 7fa6cfe00640 1 -- 172.21.15.1:0/4096114796 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fa6dc023920 con 0x5589e1d57900 2024-04-26T21:35:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.084+0000 7fa6f83c5740 1 -- 172.21.15.1:0/4096114796 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6e802fbf0 con 0x5589e1d57900 2024-04-26T21:35:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.085+0000 7fa6cfe00640 1 -- 172.21.15.1:0/4096114796 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fa6dc023b20 con 0x5589e1d57900 2024-04-26T21:35:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.086+0000 7fa6f83c5740 10 client.25894 fetch_fsmap learned FSMap version 34 2024-04-26T21:35:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.086+0000 7fa6f83c5740 1 -- 172.21.15.1:0/4096114796 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x5589e1de9060 con 0x5589e1d57900 2024-04-26T21:35:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.087+0000 7fa6cfe00640 1 -- 172.21.15.1:0/4096114796 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6e802fbf0 con 0x5589e1d57900 2024-04-26T21:35:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.087+0000 7fa6cfe00640 1 -- 172.21.15.1:0/4096114796 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 34) ==== 8354+0+0 (secure 0 0 0) 0x7fa6dc043590 con 0x5589e1d57900 2024-04-26T21:35:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.088+0000 7fa6f83c5740 10 client.25894 fetch_fsmap finished waiting for FSMap version 34 2024-04-26T21:35:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.088+0000 7fa6f83c5740 10 client.25894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:35:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.088+0000 7fa6f83c5740 20 client.25894 populate_metadata read hostname 'smithi001' 2024-04-26T21:35:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.088+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5589e1de9720 0x5589e1debb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.088+0000 7fa6f83c5740 4 client.25894 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:35:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.088+0000 7fa6f83c5740 1 -- 172.21.15.1:0/4096114796 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5589e1cc3e00 con 0x5589e1de9720 2024-04-26T21:35:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.089+0000 7fa6eea00640 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5589e1de9720 0x5589e1debb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.090+0000 7fa6eea00640 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5589e1de9720 0x5589e1debb40 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eef833759617c7b9 server_cookie=d2727b5c6a3b6698 in_seq=0 out_seq=0 2024-04-26T21:35:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.090+0000 7fa6cfe00640 10 client.25894 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:13.552 INFO:tasks.workunit.client.0.smithi001.stdout: 1 116302 3.81 MB/sec execute 71 sec latency 24.971 ms 2024-04-26T21:35:13.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.627+0000 7fa6cfe00640 1 -- 172.21.15.1:0/4096114796 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5589e1cc3e00 con 0x5589e1de9720 2024-04-26T21:35:13.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.627+0000 7fa6cfe00640 10 client.25894 handle_command_reply: tid=0 2024-04-26T21:35:13.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.627+0000 7fa6f83c5740 1 client.25894 shutdown 2024-04-26T21:35:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.628+0000 7fa6f83c5740 1 -- 172.21.15.1:0/4096114796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d57900 msgr2=0x5589e1d65a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.628+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d57900 0x5589e1d65a20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc000c00 tx=0x7fa6dc005540 comp rx=0 tx=0).stop 2024-04-26T21:35:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.628+0000 7fa6f83c5740 1 -- 172.21.15.1:0/4096114796 shutdown_connections 2024-04-26T21:35:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.628+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5589e1de9720 0x5589e1debb40 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.628+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589e1d57900 0x5589e1d65a20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.628+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5589e1d621c0 0x5589e1d66030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.628+0000 7fa6f83c5740 1 --2- 172.21.15.1:0/4096114796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5589e1d56fe0 0x5589e1d6a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.628+0000 7fa6f83c5740 1 -- 172.21.15.1:0/4096114796 >> 172.21.15.1:0/4096114796 conn(0x5589e1d534f0 msgr2=0x5589e1d591f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:13.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.628+0000 7fa6cfe00640 0 client.25894 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.629+0000 7fa6f83c5740 1 -- 172.21.15.1:0/4096114796 shutdown_connections 2024-04-26T21:35:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.629+0000 7fa6f83c5740 1 -- 172.21.15.1:0/4096114796 wait complete. 2024-04-26T21:35:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.629+0000 7fa6f83c5740 20 client.25894 trim_cache size 0 max 16384 2024-04-26T21:35:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:35:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:35:13.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:35:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:35:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 -- 172.21.15.1:0/3151890302 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa6d40a3d00 msgr2=0x7fa6d40a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa6d40a3d00 0x7fa6d40a61c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009a60 tx=0x7fa6e8005d80 comp rx=0 tx=0).stop 2024-04-26T21:35:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 -- 172.21.15.1:0/3151890302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f0074d60 msgr2=0x7fa6f019b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:13.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f0074d60 0x7fa6f019b0f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa6e000e9f0 tx=0x7fa6e000edf0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 -- 172.21.15.1:0/3151890302 shutdown_connections 2024-04-26T21:35:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa6d40a3d00 0x7fa6d40a61c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6f010ef90 0x7fa6f019f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa6f0075680 0x7fa6f019b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 --2- 172.21.15.1:0/3151890302 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6f0074d60 0x7fa6f019b0f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 -- 172.21.15.1:0/3151890302 >> 172.21.15.1:0/3151890302 conn(0x7fa6f00fe240 msgr2=0x7fa6f00fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 -- 172.21.15.1:0/3151890302 shutdown_connections 2024-04-26T21:35:13.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:13.631+0000 7fa6efe00640 1 -- 172.21.15.1:0/3151890302 wait complete. 2024-04-26T21:35:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:14 smithi186 ceph-mon[32991]: pgmap v150: 129 pgs: 129 active+clean; 71 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 1.0 MiB/s wr, 171 op/s 2024-04-26T21:35:14.552 INFO:tasks.workunit.client.0.smithi001.stdout: 1 117358 3.87 MB/sec execute 72 sec latency 49.446 ms 2024-04-26T21:35:14.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:14 smithi001 ceph-mon[28054]: pgmap v150: 129 pgs: 129 active+clean; 71 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 1.0 MiB/s wr, 171 op/s 2024-04-26T21:35:14.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:14 smithi038 ceph-mon[32724]: pgmap v150: 129 pgs: 129 active+clean; 71 MiB data, 816 MiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 1.0 MiB/s wr, 171 op/s 2024-04-26T21:35:15.552 INFO:tasks.workunit.client.0.smithi001.stdout: 1 117771 3.82 MB/sec execute 73 sec latency 27.177 ms 2024-04-26T21:35:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:16 smithi186 ceph-mon[32991]: pgmap v151: 129 pgs: 129 active+clean; 74 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 1.8 MiB/s wr, 271 op/s 2024-04-26T21:35:16.552 INFO:tasks.workunit.client.0.smithi001.stdout: 1 118936 3.84 MB/sec execute 74 sec latency 18.721 ms 2024-04-26T21:35:16.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:16 smithi001 ceph-mon[28054]: pgmap v151: 129 pgs: 129 active+clean; 74 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 1.8 MiB/s wr, 271 op/s 2024-04-26T21:35:16.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:16 smithi038 ceph-mon[32724]: pgmap v151: 129 pgs: 129 active+clean; 74 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 1.8 MiB/s wr, 271 op/s 2024-04-26T21:35:17.552 INFO:tasks.workunit.client.0.smithi001.stdout: 1 120113 3.89 MB/sec execute 75 sec latency 26.309 ms 2024-04-26T21:35:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:18 smithi186 ceph-mon[32991]: pgmap v152: 129 pgs: 129 active+clean; 74 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-26T21:35:18.552 INFO:tasks.workunit.client.0.smithi001.stdout: 1 121025 3.92 MB/sec execute 76 sec latency 41.589 ms 2024-04-26T21:35:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:18 smithi001 ceph-mon[28054]: pgmap v152: 129 pgs: 129 active+clean; 74 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-26T21:35:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:18 smithi038 ceph-mon[32724]: pgmap v152: 129 pgs: 129 active+clean; 74 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-26T21:35:19.553 INFO:tasks.workunit.client.0.smithi001.stdout: 1 121706 3.90 MB/sec execute 77 sec latency 22.161 ms 2024-04-26T21:35:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:20 smithi186 ceph-mon[32991]: pgmap v153: 129 pgs: 129 active+clean; 74 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-26T21:35:20.553 INFO:tasks.workunit.client.0.smithi001.stdout: 1 123074 3.91 MB/sec execute 78 sec latency 27.476 ms 2024-04-26T21:35:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:20 smithi001 ceph-mon[28054]: pgmap v153: 129 pgs: 129 active+clean; 74 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-26T21:35:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:20 smithi038 ceph-mon[32724]: pgmap v153: 129 pgs: 129 active+clean; 74 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 1.3 MiB/s wr, 161 op/s 2024-04-26T21:35:21.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:21 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:35:21.553 INFO:tasks.workunit.client.0.smithi001.stdout: 1 124175 3.99 MB/sec execute 79 sec latency 28.066 ms 2024-04-26T21:35:21.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:21 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:35:21.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:21 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:35:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:22 smithi186 ceph-mon[32991]: pgmap v154: 129 pgs: 129 active+clean; 77 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 226 op/s 2024-04-26T21:35:22.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:22 smithi038 ceph-mon[32724]: pgmap v154: 129 pgs: 129 active+clean; 77 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 226 op/s 2024-04-26T21:35:22.553 INFO:tasks.workunit.client.0.smithi001.stdout: 1 124565 3.96 MB/sec execute 80 sec latency 32.766 ms 2024-04-26T21:35:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:22 smithi001 ceph-mon[28054]: pgmap v154: 129 pgs: 129 active+clean; 77 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 226 op/s 2024-04-26T21:35:23.553 INFO:tasks.workunit.client.0.smithi001.stdout: 1 124990 3.92 MB/sec execute 81 sec latency 25.091 ms 2024-04-26T21:35:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:24 smithi186 ceph-mon[32991]: pgmap v155: 129 pgs: 129 active+clean; 77 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-26T21:35:24.553 INFO:tasks.workunit.client.0.smithi001.stdout: 1 126365 3.94 MB/sec execute 82 sec latency 29.091 ms 2024-04-26T21:35:24.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:24 smithi001 ceph-mon[28054]: pgmap v155: 129 pgs: 129 active+clean; 77 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-26T21:35:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:24 smithi038 ceph-mon[32724]: pgmap v155: 129 pgs: 129 active+clean; 77 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 1.3 MiB/s wr, 164 op/s 2024-04-26T21:35:25.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:25 smithi186 ceph-mon[32991]: pgmap v156: 129 pgs: 129 active+clean; 79 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.7 MiB/s wr, 217 op/s 2024-04-26T21:35:25.553 INFO:tasks.workunit.client.0.smithi001.stdout: 1 127117 3.97 MB/sec execute 83 sec latency 27.033 ms 2024-04-26T21:35:25.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:25 smithi001 ceph-mon[28054]: pgmap v156: 129 pgs: 129 active+clean; 79 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.7 MiB/s wr, 217 op/s 2024-04-26T21:35:25.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:25 smithi038 ceph-mon[32724]: pgmap v156: 129 pgs: 129 active+clean; 79 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 1.7 MiB/s wr, 217 op/s 2024-04-26T21:35:26.553 INFO:tasks.workunit.client.0.smithi001.stdout: 1 127991 4.00 MB/sec execute 84 sec latency 32.404 ms 2024-04-26T21:35:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.796+0000 7ffa09c00640 1 -- 172.21.15.1:0/3561167885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa04105510 msgr2=0x7ffa041119d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.796+0000 7ffa09c00640 1 --2- 172.21.15.1:0/3561167885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa04105510 0x7ffa041119d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009cd0 tx=0x7ff9fc02a8e0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.796+0000 7ffa09c00640 1 -- 172.21.15.1:0/3561167885 shutdown_connections 2024-04-26T21:35:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.796+0000 7ffa09c00640 1 --2- 172.21.15.1:0/3561167885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa04105510 0x7ffa041119d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.796+0000 7ffa09c00640 1 --2- 172.21.15.1:0/3561167885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa04104b50 0x7ffa04104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.796+0000 7ffa09c00640 1 --2- 172.21.15.1:0/3561167885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa0410ac10 0x7ffa0410aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.796+0000 7ffa09c00640 1 -- 172.21.15.1:0/3561167885 >> 172.21.15.1:0/3561167885 conn(0x7ffa04100620 msgr2=0x7ffa04102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.796+0000 7ffa09c00640 1 -- 172.21.15.1:0/3561167885 shutdown_connections 2024-04-26T21:35:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.796+0000 7ffa09c00640 1 -- 172.21.15.1:0/3561167885 wait complete. 2024-04-26T21:35:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.798+0000 7ffa09c00640 1 Processor -- start 2024-04-26T21:35:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.798+0000 7ffa09c00640 1 -- start start 2024-04-26T21:35:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa09c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa04104b50 0x7ffa04111840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa09c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa04105510 0x7ffa0410e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa09c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa0410ac10 0x7ffa0410ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa09c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04114e50 con 0x7ffa04104b50 2024-04-26T21:35:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa09c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04114cd0 con 0x7ffa0410ac10 2024-04-26T21:35:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa09c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04114fd0 con 0x7ffa04105510 2024-04-26T21:35:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa02000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa04105510 0x7ffa0410e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa02000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa04105510 0x7ffa0410e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:35972/0 (socket says 172.21.15.1:35972) 2024-04-26T21:35:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa02000640 1 -- 172.21.15.1:0/3489273087 learned_addr learned my addr 172.21.15.1:0/3489273087 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa03400640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa0410ac10 0x7ffa0410ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.799+0000 7ffa02a00640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa04104b50 0x7ffa04111840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.800+0000 7ffa02000640 1 -- 172.21.15.1:0/3489273087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa0410ac10 msgr2=0x7ffa0410ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:26.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.800+0000 7ffa02000640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa0410ac10 0x7ffa0410ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.800+0000 7ffa02000640 1 -- 172.21.15.1:0/3489273087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa04104b50 msgr2=0x7ffa04111840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.800+0000 7ffa02000640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa04104b50 0x7ffa04111840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.800+0000 7ffa02000640 1 -- 172.21.15.1:0/3489273087 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0410f6b0 con 0x7ffa04105510 2024-04-26T21:35:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.800+0000 7ffa02000640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa04105510 0x7ffa0410e8e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00d950 tx=0x7ff9ec00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.800+0000 7ff9fb400640 1 -- 172.21.15.1:0/3489273087 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec017030 con 0x7ffa04105510 2024-04-26T21:35:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.800+0000 7ff9fb400640 1 -- 172.21.15.1:0/3489273087 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9ec004020 con 0x7ffa04105510 2024-04-26T21:35:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.800+0000 7ff9fb400640 1 -- 172.21.15.1:0/3489273087 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0047a0 con 0x7ffa04105510 2024-04-26T21:35:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.800+0000 7ffa09c00640 1 -- 172.21.15.1:0/3489273087 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa041abbf0 con 0x7ffa04105510 2024-04-26T21:35:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.801+0000 7ffa09c00640 1 -- 172.21.15.1:0/3489273087 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa041ac0c0 con 0x7ffa04105510 2024-04-26T21:35:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.804+0000 7ff9fb400640 1 -- 172.21.15.1:0/3489273087 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7ff9ec009a30 con 0x7ffa04105510 2024-04-26T21:35:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.804+0000 7ffa0ca14740 1 Processor -- start 2024-04-26T21:35:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa0ca14740 1 -- start start 2024-04-26T21:35:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 0x5591ad5bb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5c5fb0 0x5591ad5c83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5591ad520380 con 0x5591ad5c5fb0 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5591ad4bcc30 con 0x5591ad5bb6f0 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591ad51c340 con 0x5591ad5badd0 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa02a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa03400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 0x5591ad5bb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa02a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36130/0 (socket says 172.21.15.1:36130) 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.805+0000 7ffa02a00640 1 -- 172.21.15.1:0/760489366 learned_addr learned my addr 172.21.15.1:0/760489366 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.806+0000 7ffa02000640 1 --2- 172.21.15.1:0/760489366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5c5fb0 0x5591ad5c83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.806+0000 7ffa03400640 1 -- 172.21.15.1:0/760489366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5bb6f0 msgr2=0x5591ad5c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.806+0000 7ffa03400640 1 --2- 172.21.15.1:0/760489366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.806+0000 7ffa03400640 1 -- 172.21.15.1:0/760489366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5c5fb0 msgr2=0x5591ad5c83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.806+0000 7ffa03400640 1 --2- 172.21.15.1:0/760489366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5c5fb0 0x5591ad5c83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.806+0000 7ffa03400640 1 -- 172.21.15.1:0/760489366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591ad4de940 con 0x5591ad5badd0 2024-04-26T21:35:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.806+0000 7ffa02a00640 1 --2- 172.21.15.1:0/760489366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:35:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.806+0000 7ff9fb400640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff9e80a3d90 0x7ff9e80a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.806+0000 7ffa03400640 1 --2- 172.21.15.1:0/760489366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 0x5591ad5bb1b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009ca0 tx=0x7ff9fc005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75cfcdf7e47932c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.806+0000 7ff9fb400640 1 -- 172.21.15.1:0/3489273087 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7ff9ec0d0670 con 0x7ffa04105510 2024-04-26T21:35:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.807+0000 7ffa02a00640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff9e80a3d90 0x7ff9e80a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.807+0000 7ff9f3e00640 1 -- 172.21.15.1:0/760489366 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc048070 con 0x5591ad5badd0 2024-04-26T21:35:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.807+0000 7ff9f3e00640 1 -- 172.21.15.1:0/760489366 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9fc036070 con 0x5591ad5badd0 2024-04-26T21:35:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.807+0000 7ff9f3e00640 1 -- 172.21.15.1:0/760489366 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc0043a0 con 0x5591ad5badd0 2024-04-26T21:35:26.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa02a00640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff9e80a3d90 0x7ff9e80a6250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x5591ad5bc370 tx=0x7ff9f4005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa0ca14740 1 -- 172.21.15.1:0/760489366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 msgr2=0x5591ad5bb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/760489366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 0x5591ad5bb1b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009ca0 tx=0x7ff9fc005f80 comp rx=0 tx=0).stop 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa0ca14740 1 -- 172.21.15.1:0/760489366 shutdown_connections 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/760489366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5c5fb0 0x5591ad5c83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/760489366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/760489366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 0x5591ad5bb1b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa0ca14740 1 -- 172.21.15.1:0/760489366 >> 172.21.15.1:0/760489366 conn(0x5591ad5b72e0 msgr2=0x5591ad5b76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa0ca14740 1 -- 172.21.15.1:0/760489366 shutdown_connections 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa0ca14740 1 -- 172.21.15.1:0/760489366 wait complete. 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.808+0000 7ffa0ca14740 1 -- start start 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.809+0000 7ffa0ca14740 1 Processor -- start 2024-04-26T21:35:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.809+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5badd0 0x5591ad5c2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.809+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.809+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.809+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5591ad5c8b80 con 0x5591ad5badd0 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.809+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5591ad4a7c60 con 0x5591ad5c5fb0 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.809+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5591ad5c8d00 con 0x5591ad5bb6f0 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.809+0000 7ffa0ca14740 10 client.0 _ll_register_callbacks cb 0x5591ad4d3e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.809+0000 7ffa03400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa03400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5c39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36146/0 (socket says 172.21.15.1:36146) 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa03400640 1 -- 172.21.15.1:0/4104090156 learned_addr learned my addr 172.21.15.1:0/4104090156 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa02000640 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa02a00640 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5badd0 0x5591ad5c2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa03400640 1 -- 172.21.15.1:0/4104090156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 msgr2=0x5591ad5c34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa03400640 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa03400640 1 -- 172.21.15.1:0/4104090156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5badd0 msgr2=0x5591ad5c2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa03400640 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5badd0 0x5591ad5c2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa03400640 1 -- 172.21.15.1:0/4104090156 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591ad4b6510 con 0x5591ad5c5fb0 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa02000640 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ffa03400640 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5c39e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc003e40 tx=0x7ff9fc003e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=536b00dbde602fc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.810+0000 7ff9f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.811+0000 7ff9f0c00640 1 -- 172.21.15.1:0/4104090156 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc052020 con 0x5591ad5c5fb0 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.811+0000 7ff9f0c00640 1 -- 172.21.15.1:0/4104090156 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7ff9fc031850 con 0x5591ad5c5fb0 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.811+0000 7ffa0ca14740 1 -- 172.21.15.1:0/4104090156 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9fc036070 con 0x5591ad5c5fb0 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.811+0000 7ff9f0c00640 1 -- 172.21.15.1:0/4104090156 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff9fc048040 con 0x5591ad5c5fb0 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.811+0000 7ffa0ca14740 10 client.25925 fetch_fsmap learned FSMap version 35 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.811+0000 7ffa0ca14740 1 -- 172.21.15.1:0/4104090156 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5591ad232210 con 0x5591ad5c5fb0 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.811+0000 7ff9f0c00640 1 -- 172.21.15.1:0/4104090156 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9fc036070 con 0x5591ad5c5fb0 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.812+0000 7ff9f0c00640 1 -- 172.21.15.1:0/4104090156 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7ff9fc04b030 con 0x5591ad5c5fb0 2024-04-26T21:35:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.812+0000 7ffa0ca14740 10 client.25925 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T21:35:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.812+0000 7ffa0ca14740 10 client.25925 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:35:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.812+0000 7ffa0ca14740 20 client.25925 populate_metadata read hostname 'smithi001' 2024-04-26T21:35:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.812+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5591ad5d5bf0 0x5591ad5d7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.812+0000 7ffa0ca14740 4 client.25925 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:35:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.812+0000 7ffa0ca14740 1 -- 172.21.15.1:0/4104090156 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591ad52df90 con 0x5591ad5d5bf0 2024-04-26T21:35:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.812+0000 7ffa02a00640 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5591ad5d5bf0 0x5591ad5d7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.813+0000 7ffa02a00640 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5591ad5d5bf0 0x5591ad5d7fe0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c3a62a70961374 server_cookie=afa670057171f374 in_seq=0 out_seq=0 2024-04-26T21:35:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.813+0000 7ff9f0c00640 10 client.25925 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ff9f0c00640 1 -- 172.21.15.1:0/4104090156 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5591ad52df90 con 0x5591ad5d5bf0 2024-04-26T21:35:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ff9f0c00640 10 client.25925 handle_command_reply: tid=0 2024-04-26T21:35:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ffa0ca14740 1 client.25925 shutdown 2024-04-26T21:35:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ffa0ca14740 1 -- 172.21.15.1:0/4104090156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 msgr2=0x5591ad5c39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5c39e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc003e40 tx=0x7ff9fc003e70 comp rx=0 tx=0).stop 2024-04-26T21:35:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ffa0ca14740 1 -- 172.21.15.1:0/4104090156 shutdown_connections 2024-04-26T21:35:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5591ad5d5bf0 0x5591ad5d7fe0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 0x5591ad5c34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5c39e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/4104090156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5badd0 0x5591ad5c2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.814+0000 7ffa0ca14740 1 -- 172.21.15.1:0/4104090156 >> 172.21.15.1:0/4104090156 conn(0x5591ad5b72e0 msgr2=0x5591ad5bd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.815+0000 7ff9f0c00640 0 client.25925 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.815+0000 7ffa0ca14740 1 -- 172.21.15.1:0/4104090156 shutdown_connections 2024-04-26T21:35:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.815+0000 7ffa0ca14740 1 -- 172.21.15.1:0/4104090156 wait complete. 2024-04-26T21:35:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.815+0000 7ffa0ca14740 20 client.25925 trim_cache size 0 max 16384 2024-04-26T21:35:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.835+0000 7ffa0ca14740 1 Processor -- start 2024-04-26T21:35:26.835 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.835+0000 7ffa0ca14740 1 -- start start 2024-04-26T21:35:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.835+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 0x5591ad5d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.835+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5bb6f0 0x5591ad5b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.835+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5dd580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.835+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041b8c70 con 0x5591ad5bb6f0 2024-04-26T21:35:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.835+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041b6510 con 0x5591ad5c5fb0 2024-04-26T21:35:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.835+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041b4360 con 0x5591ad5badd0 2024-04-26T21:35:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa02000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5bb6f0 0x5591ad5b62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa03400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5dd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa02a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 0x5591ad5d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa03400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5dd580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36150/0 (socket says 172.21.15.1:36150) 2024-04-26T21:35:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa03400640 1 -- 172.21.15.1:0/757217617 learned_addr learned my addr 172.21.15.1:0/757217617 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa03400640 1 -- 172.21.15.1:0/757217617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 msgr2=0x5591ad5d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa03400640 1 --2- 172.21.15.1:0/757217617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 0x5591ad5d1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa03400640 1 -- 172.21.15.1:0/757217617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5bb6f0 msgr2=0x5591ad5b62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa03400640 1 --2- 172.21.15.1:0/757217617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5bb6f0 0x5591ad5b62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa03400640 1 -- 172.21.15.1:0/757217617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591ad5b2940 con 0x5591ad5c5fb0 2024-04-26T21:35:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.836+0000 7ffa02a00640 1 --2- 172.21.15.1:0/757217617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 0x5591ad5d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:35:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.837+0000 7ffa03400640 1 --2- 172.21.15.1:0/757217617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5dd580 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0306a0 tx=0x7ff9fc009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9cd99324ad6a069 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.837+0000 7ff9f3400640 1 -- 172.21.15.1:0/757217617 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc052070 con 0x5591ad5c5fb0 2024-04-26T21:35:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.837+0000 7ff9f3400640 1 -- 172.21.15.1:0/757217617 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9fc036040 con 0x5591ad5c5fb0 2024-04-26T21:35:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.837+0000 7ff9f3400640 1 -- 172.21.15.1:0/757217617 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9fc004080 con 0x5591ad5c5fb0 2024-04-26T21:35:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.838+0000 7ffa0ca14740 1 -- 172.21.15.1:0/757217617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 msgr2=0x5591ad5dd580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.838+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/757217617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5dd580 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0306a0 tx=0x7ff9fc009b20 comp rx=0 tx=0).stop 2024-04-26T21:35:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.838+0000 7ffa0ca14740 1 -- 172.21.15.1:0/757217617 shutdown_connections 2024-04-26T21:35:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.838+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/757217617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5c5fb0 0x5591ad5dd580 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.838+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/757217617 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5bb6f0 0x5591ad5b62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.838+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/757217617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5badd0 0x5591ad5d1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.838+0000 7ffa0ca14740 1 -- 172.21.15.1:0/757217617 >> 172.21.15.1:0/757217617 conn(0x5591ad5b72e0 msgr2=0x5591ad5b5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.838+0000 7ffa0ca14740 1 -- 172.21.15.1:0/757217617 shutdown_connections 2024-04-26T21:35:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.838+0000 7ffa0ca14740 1 -- 172.21.15.1:0/757217617 wait complete. 2024-04-26T21:35:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.839+0000 7ffa0ca14740 1 -- start start 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.839+0000 7ffa0ca14740 1 Processor -- start 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.839+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5badd0 0x5591ad5ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 0x5591ad5cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa0ca14740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5c5fb0 0x5591ad5cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041b49d0 con 0x5591ad5c5fb0 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041b5e40 con 0x5591ad5badd0 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa0ca14740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041b50a0 con 0x5591ad5bb6f0 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa0ca14740 10 client.0 _ll_register_callbacks cb 0x7ff9fc046070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa02a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5badd0 0x5591ad5ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa02a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5badd0 0x5591ad5ce2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36154/0 (socket says 172.21.15.1:36154) 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa02a00640 1 -- 172.21.15.1:0/3526535553 learned_addr learned my addr 172.21.15.1:0/3526535553 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa02000640 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 0x5591ad5cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa03400640 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5c5fb0 0x5591ad5cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa02a00640 1 -- 172.21.15.1:0/3526535553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 msgr2=0x5591ad5cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa02a00640 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 0x5591ad5cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa02a00640 1 -- 172.21.15.1:0/3526535553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5c5fb0 msgr2=0x5591ad5cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa02a00640 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5c5fb0 0x5591ad5cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.840+0000 7ffa02a00640 1 -- 172.21.15.1:0/3526535553 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591ad5c8710 con 0x5591ad5badd0 2024-04-26T21:35:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.841+0000 7ffa02a00640 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5badd0 0x5591ad5ce2d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40040c0 tx=0x7ff9f4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed2962e73b96912d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.841+0000 7ff9f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:35:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.841+0000 7ff9f0c00640 1 -- 172.21.15.1:0/3526535553 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f402b420 con 0x5591ad5badd0 2024-04-26T21:35:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.841+0000 7ff9f0c00640 1 -- 172.21.15.1:0/3526535553 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7ff9f402b5c0 con 0x5591ad5badd0 2024-04-26T21:35:26.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.841+0000 7ffa0ca14740 1 -- 172.21.15.1:0/3526535553 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9fc036040 con 0x5591ad5badd0 2024-04-26T21:35:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.841+0000 7ff9f0c00640 1 -- 172.21.15.1:0/3526535553 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff9f4039050 con 0x5591ad5badd0 2024-04-26T21:35:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.841+0000 7ffa0ca14740 10 client.25931 fetch_fsmap learned FSMap version 35 2024-04-26T21:35:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.841+0000 7ffa0ca14740 1 -- 172.21.15.1:0/3526535553 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5591ad2fe620 con 0x5591ad5badd0 2024-04-26T21:35:26.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.842+0000 7ff9f0c00640 1 -- 172.21.15.1:0/3526535553 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9fc036040 con 0x5591ad5badd0 2024-04-26T21:35:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.842+0000 7ff9f0c00640 1 -- 172.21.15.1:0/3526535553 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 35) ==== 8350+0+0 (secure 0 0 0) 0x7ff9f4004bc0 con 0x5591ad5badd0 2024-04-26T21:35:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.842+0000 7ffa0ca14740 10 client.25931 fetch_fsmap finished waiting for FSMap version 35 2024-04-26T21:35:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.842+0000 7ffa0ca14740 10 client.25931 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:35:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.842+0000 7ffa0ca14740 20 client.25931 populate_metadata read hostname 'smithi001' 2024-04-26T21:35:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.842+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5591ad63c220 0x5591ad63e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.842+0000 7ffa0ca14740 4 client.25931 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:35:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.842+0000 7ffa0ca14740 1 -- 172.21.15.1:0/3526535553 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5591ad527e00 con 0x5591ad63c220 2024-04-26T21:35:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.842+0000 7ffa02000640 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5591ad63c220 0x5591ad63e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.843+0000 7ffa02000640 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5591ad63c220 0x5591ad63e660 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5e53cc5625f695c server_cookie=e2debb340e3173e1 in_seq=0 out_seq=0 2024-04-26T21:35:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.843+0000 7ff9f0c00640 10 client.25931 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.909+0000 7ff9f0c00640 1 -- 172.21.15.1:0/3526535553 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5591ad527e00 con 0x5591ad63c220 2024-04-26T21:35:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.909+0000 7ff9f0c00640 10 client.25931 handle_command_reply: tid=0 2024-04-26T21:35:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.909+0000 7ffa0ca14740 1 client.25931 shutdown 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.909+0000 7ffa0ca14740 1 -- 172.21.15.1:0/3526535553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5badd0 msgr2=0x5591ad5ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.909+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5badd0 0x5591ad5ce2d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff9f40040c0 tx=0x7ff9f4004730 comp rx=0 tx=0).stop 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.910+0000 7ffa0ca14740 1 -- 172.21.15.1:0/3526535553 shutdown_connections 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.910+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5591ad63c220 0x5591ad63e660 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.910+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5591ad5bb6f0 0x5591ad5cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.910+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5591ad5c5fb0 0x5591ad5cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.910+0000 7ffa0ca14740 1 --2- 172.21.15.1:0/3526535553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5591ad5badd0 0x5591ad5ce2d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.910+0000 7ff9f0c00640 0 client.25931 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.910+0000 7ffa0ca14740 1 -- 172.21.15.1:0/3526535553 >> 172.21.15.1:0/3526535553 conn(0x5591ad5b72e0 msgr2=0x5591ad5bd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.910+0000 7ffa0ca14740 1 -- 172.21.15.1:0/3526535553 shutdown_connections 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.910+0000 7ffa0ca14740 1 -- 172.21.15.1:0/3526535553 wait complete. 2024-04-26T21:35:26.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.910+0000 7ffa0ca14740 20 client.25931 trim_cache size 0 max 16384 2024-04-26T21:35:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:35:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:35:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:35:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 -- 172.21.15.1:0/3489273087 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff9e80a3d90 msgr2=0x7ff9e80a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff9e80a3d90 0x7ff9e80a6250 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x5591ad5bc370 tx=0x7ff9f4005f90 comp rx=0 tx=0).stop 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 -- 172.21.15.1:0/3489273087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa04105510 msgr2=0x7ffa0410e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa04105510 0x7ffa0410e8e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00d950 tx=0x7ff9ec00de10 comp rx=0 tx=0).stop 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 -- 172.21.15.1:0/3489273087 shutdown_connections 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff9e80a3d90 0x7ff9e80a6250 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa04105510 0x7ffa0410e8e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa0410ac10 0x7ffa0410ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 --2- 172.21.15.1:0/3489273087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa04104b50 0x7ffa04111840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 -- 172.21.15.1:0/3489273087 >> 172.21.15.1:0/3489273087 conn(0x7ffa04100620 msgr2=0x7ffa04101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 -- 172.21.15.1:0/3489273087 shutdown_connections 2024-04-26T21:35:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:26.911+0000 7ffa09c00640 1 -- 172.21.15.1:0/3489273087 wait complete. 2024-04-26T21:35:27.553 INFO:tasks.workunit.client.0.smithi001.stdout: 1 128385 3.96 MB/sec execute 85 sec latency 53.061 ms 2024-04-26T21:35:28.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:27 smithi001 ceph-mon[28054]: pgmap v157: 129 pgs: 129 active+clean; 79 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 942 KiB/s wr, 117 op/s 2024-04-26T21:35:28.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:27 smithi038 ceph-mon[32724]: pgmap v157: 129 pgs: 129 active+clean; 79 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 942 KiB/s wr, 117 op/s 2024-04-26T21:35:28.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:27 smithi186 ceph-mon[32991]: pgmap v157: 129 pgs: 129 active+clean; 79 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 942 KiB/s wr, 117 op/s 2024-04-26T21:35:28.553 INFO:tasks.workunit.client.0.smithi001.stdout: 1 129137 3.97 MB/sec execute 86 sec latency 57.136 ms 2024-04-26T21:35:29.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 130344 3.96 MB/sec execute 87 sec latency 40.734 ms 2024-04-26T21:35:30.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:29 smithi001 ceph-mon[28054]: pgmap v158: 129 pgs: 129 active+clean; 79 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 942 KiB/s wr, 117 op/s 2024-04-26T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:29 smithi038 ceph-mon[32724]: pgmap v158: 129 pgs: 129 active+clean; 79 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 942 KiB/s wr, 117 op/s 2024-04-26T21:35:30.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:29 smithi186 ceph-mon[32991]: pgmap v158: 129 pgs: 129 active+clean; 79 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 942 KiB/s wr, 117 op/s 2024-04-26T21:35:30.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 131497 4.03 MB/sec execute 88 sec latency 39.967 ms 2024-04-26T21:35:31.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:30 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:35:31.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:30 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:35:31.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:30 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:35:31.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 131731 3.99 MB/sec execute 89 sec latency 28.003 ms 2024-04-26T21:35:32.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:31 smithi038 ceph-mon[32724]: pgmap v159: 129 pgs: 129 active+clean; 81 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.7 MiB/s wr, 206 op/s 2024-04-26T21:35:32.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:31 smithi186 ceph-mon[32991]: pgmap v159: 129 pgs: 129 active+clean; 81 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.7 MiB/s wr, 206 op/s 2024-04-26T21:35:32.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:31 smithi001 ceph-mon[28054]: pgmap v159: 129 pgs: 129 active+clean; 81 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 1.7 MiB/s wr, 206 op/s 2024-04-26T21:35:32.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 132384 3.97 MB/sec execute 90 sec latency 23.109 ms 2024-04-26T21:35:33.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 133685 3.98 MB/sec execute 91 sec latency 29.141 ms 2024-04-26T21:35:34.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:33 smithi186 ceph-mon[32991]: pgmap v160: 129 pgs: 129 active+clean; 81 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-26T21:35:34.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:33 smithi001 ceph-mon[28054]: pgmap v160: 129 pgs: 129 active+clean; 81 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-26T21:35:34.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:33 smithi038 ceph-mon[32724]: pgmap v160: 129 pgs: 129 active+clean; 81 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-26T21:35:34.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 134893 4.05 MB/sec execute 92 sec latency 54.699 ms 2024-04-26T21:35:35.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:34 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:35:35.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:34 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:35:35.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:34 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:35:35.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 135329 4.03 MB/sec execute 93 sec latency 19.006 ms 2024-04-26T21:35:36.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:35 smithi186 ceph-mon[32991]: pgmap v161: 129 pgs: 129 active+clean; 85 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-04-26T21:35:36.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:35 smithi001 ceph-mon[28054]: pgmap v161: 129 pgs: 129 active+clean; 85 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-04-26T21:35:36.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:35 smithi038 ceph-mon[32724]: pgmap v161: 129 pgs: 129 active+clean; 85 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-04-26T21:35:36.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 136281 4.04 MB/sec execute 94 sec latency 31.641 ms 2024-04-26T21:35:37.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:36 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:35:37.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:36 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:35:37.205 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:36 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:35:37.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:36 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:35:37.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:36 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:35:37.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:36 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:35:37.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:36 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:35:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:36 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:35:37.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:36 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:35:37.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 137719 4.07 MB/sec execute 95 sec latency 25.424 ms 2024-04-26T21:35:38.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:38 smithi001 ceph-mon[28054]: pgmap v162: 129 pgs: 129 active+clean; 85 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:35:38.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:38 smithi038 ceph-mon[32724]: pgmap v162: 129 pgs: 129 active+clean; 85 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:35:38.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:38 smithi186 ceph-mon[32991]: pgmap v162: 129 pgs: 129 active+clean; 85 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:35:38.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 138661 4.10 MB/sec execute 96 sec latency 23.780 ms 2024-04-26T21:35:39.554 INFO:tasks.workunit.client.0.smithi001.stdout: 1 139100 4.06 MB/sec execute 97 sec latency 60.417 ms 2024-04-26T21:35:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.061+0000 7f97b6e00640 1 -- 172.21.15.1:0/2751402071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97b0104230 msgr2=0x7f97b010c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.061+0000 7f97b6e00640 1 --2- 172.21.15.1:0/2751402071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97b0104230 0x7f97b010c710 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f979c009900 tx=0x7f979c02a480 comp rx=0 tx=0).stop 2024-04-26T21:35:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.061+0000 7f97b6e00640 1 -- 172.21.15.1:0/2751402071 shutdown_connections 2024-04-26T21:35:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.061+0000 7f97b6e00640 1 --2- 172.21.15.1:0/2751402071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b010cd30 0x7f97b010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.061+0000 7f97b6e00640 1 --2- 172.21.15.1:0/2751402071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97b0104230 0x7f97b010c710 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.061+0000 7f97b6e00640 1 --2- 172.21.15.1:0/2751402071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97b0103910 0x7f97b0103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.061+0000 7f97b6e00640 1 -- 172.21.15.1:0/2751402071 >> 172.21.15.1:0/2751402071 conn(0x7f97b00fd110 msgr2=0x7f97b00ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.061+0000 7f97b6e00640 1 -- 172.21.15.1:0/2751402071 shutdown_connections 2024-04-26T21:35:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.061+0000 7f97b6e00640 1 -- 172.21.15.1:0/2751402071 wait complete. 2024-04-26T21:35:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.062+0000 7f97b6e00640 1 Processor -- start 2024-04-26T21:35:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b6e00640 1 -- start start 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b6e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b0103910 0x7f97b019b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b6e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97b0104230 0x7f97b019b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b6e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97b010cd30 0x7f97b019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b6e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f97b01020e0 con 0x7f97b0103910 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b6e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f97b0101f60 con 0x7f97b010cd30 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b6e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97b0102260 con 0x7f97b0104230 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b6400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97b010cd30 0x7f97b019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b5a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b0103910 0x7f97b019b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b6400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97b010cd30 0x7f97b019fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:45578/0 (socket says 172.21.15.1:45578) 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.063+0000 7f97b6400640 1 -- 172.21.15.1:0/4118980480 learned_addr learned my addr 172.21.15.1:0/4118980480 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.064+0000 7f97b6400640 1 -- 172.21.15.1:0/4118980480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97b0104230 msgr2=0x7f97b019b6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.064+0000 7f97b6400640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97b0104230 0x7f97b019b6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.064+0000 7f97b6400640 1 -- 172.21.15.1:0/4118980480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b0103910 msgr2=0x7f97b019b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.064+0000 7f97b6400640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b0103910 0x7f97b019b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.064+0000 7f97b6400640 1 -- 172.21.15.1:0/4118980480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b01a0190 con 0x7f97b010cd30 2024-04-26T21:35:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.064+0000 7f97b6400640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97b010cd30 0x7f97b019fa60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f97a800eaa0 tx=0x7f97a800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.064+0000 7f97ae000640 1 -- 172.21.15.1:0/4118980480 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a800cdb0 con 0x7f97b010cd30 2024-04-26T21:35:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.065+0000 7f97ae000640 1 -- 172.21.15.1:0/4118980480 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97a8005020 con 0x7f97b010cd30 2024-04-26T21:35:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.065+0000 7f97ae000640 1 -- 172.21.15.1:0/4118980480 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a80116d0 con 0x7f97b010cd30 2024-04-26T21:35:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.065+0000 7f97b6e00640 1 -- 172.21.15.1:0/4118980480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b01a0480 con 0x7f97b010cd30 2024-04-26T21:35:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.065+0000 7f97b6e00640 1 -- 172.21.15.1:0/4118980480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b01a7d10 con 0x7f97b010cd30 2024-04-26T21:35:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.069+0000 7f97bb171740 1 Processor -- start 2024-04-26T21:35:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.069+0000 7f97ae000640 1 -- 172.21.15.1:0/4118980480 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f97a80051d0 con 0x7f97b010cd30 2024-04-26T21:35:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.070+0000 7f97bb171740 1 -- start start 2024-04-26T21:35:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.070+0000 7f97bb171740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca619fe0 0x55b4ca61a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.070+0000 7f97bb171740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca624c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.070+0000 7f97bb171740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 0x55b4ca6275b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.070+0000 7f97bb171740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b4ca57f380 con 0x55b4ca619fe0 2024-04-26T21:35:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.070+0000 7f97bb171740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b4ca51bc30 con 0x55b4ca6251c0 2024-04-26T21:35:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.070+0000 7f97bb171740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b4ca57b340 con 0x55b4ca61a900 2024-04-26T21:35:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97b5a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca624c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97b5000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca619fe0 0x55b4ca61a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97b5a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca624c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58704/0 (socket says 172.21.15.1:58704) 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97b5a00640 1 -- 172.21.15.1:0/3506794051 learned_addr learned my addr 172.21.15.1:0/3506794051 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97b5000640 1 --2- 172.21.15.1:0/3506794051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 0x55b4ca6275b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97b5a00640 1 -- 172.21.15.1:0/3506794051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 msgr2=0x55b4ca6275b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97b5a00640 1 --2- 172.21.15.1:0/3506794051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 0x55b4ca6275b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97b5a00640 1 -- 172.21.15.1:0/3506794051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca619fe0 msgr2=0x55b4ca61a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97b5a00640 1 --2- 172.21.15.1:0/3506794051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca619fe0 0x55b4ca61a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97b5a00640 1 -- 172.21.15.1:0/3506794051 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4ca53d940 con 0x55b4ca61a900 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.071+0000 7f97ae000640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f97980a3d80 0x7f97980a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.072+0000 7f97b5a00640 1 --2- 172.21.15.1:0/3506794051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca624c80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009d20 tx=0x7f97a4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=328a0767bdb11334 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.072+0000 7f97ae000640 1 -- 172.21.15.1:0/4118980480 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f97a80d0da0 con 0x7f97b010cd30 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.072+0000 7f97b5000640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f97980a3d80 0x7f97980a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.072+0000 7f97a2a00640 1 -- 172.21.15.1:0/3506794051 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a401e070 con 0x55b4ca61a900 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.072+0000 7f97a2a00640 1 -- 172.21.15.1:0/3506794051 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97a4019ae0 con 0x55b4ca61a900 2024-04-26T21:35:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.072+0000 7f97a2a00640 1 -- 172.21.15.1:0/3506794051 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a400eac0 con 0x55b4ca61a900 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.073+0000 7f97bb171740 1 -- 172.21.15.1:0/3506794051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 msgr2=0x55b4ca624c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.073+0000 7f97bb171740 1 --2- 172.21.15.1:0/3506794051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca624c80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009d20 tx=0x7f97a4011810 comp rx=0 tx=0).stop 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.073+0000 7f97bb171740 1 -- 172.21.15.1:0/3506794051 shutdown_connections 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.073+0000 7f97bb171740 1 --2- 172.21.15.1:0/3506794051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca624c80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.073+0000 7f97bb171740 1 --2- 172.21.15.1:0/3506794051 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 0x55b4ca6275b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.073+0000 7f97bb171740 1 --2- 172.21.15.1:0/3506794051 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca619fe0 0x55b4ca61a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.073+0000 7f97bb171740 1 -- 172.21.15.1:0/3506794051 >> 172.21.15.1:0/3506794051 conn(0x55b4ca6164f0 msgr2=0x55b4ca6168d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.073+0000 7f97bb171740 1 -- 172.21.15.1:0/3506794051 shutdown_connections 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.073+0000 7f97bb171740 1 -- 172.21.15.1:0/3506794051 wait complete. 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.073+0000 7f97b5000640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f97980a3d80 0x7f97980a6240 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55b4ca617910 tx=0x7f979c031400 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.074+0000 7f97bb171740 1 -- start start 2024-04-26T21:35:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.074+0000 7f97bb171740 1 Processor -- start 2024-04-26T21:35:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97bb171740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca619fe0 0x55b4ca622150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97bb171740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca61a900 0x55b4ca622690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97bb171740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca6251c0 0x55b4ca622bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97bb171740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b4ca627df0 con 0x55b4ca6251c0 2024-04-26T21:35:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97bb171740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b4ca506c60 con 0x55b4ca61a900 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97bb171740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b4ca627f70 con 0x55b4ca619fe0 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97bb171740 10 client.0 _ll_register_callbacks cb 0x55b4ca532e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97b5000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca61a900 0x55b4ca622690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97b6400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca6251c0 0x55b4ca622bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97b5a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca619fe0 0x55b4ca622150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97b5000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca61a900 0x55b4ca622690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:45604/0 (socket says 172.21.15.1:45604) 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.075+0000 7f97b5000640 1 -- 172.21.15.1:0/952196300 learned_addr learned my addr 172.21.15.1:0/952196300 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97b6400640 1 -- 172.21.15.1:0/952196300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca619fe0 msgr2=0x55b4ca622150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97b6400640 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca619fe0 0x55b4ca622150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97b6400640 1 -- 172.21.15.1:0/952196300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca61a900 msgr2=0x55b4ca622690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97b6400640 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca61a900 0x55b4ca622690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97b6400640 1 -- 172.21.15.1:0/952196300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4ca515510 con 0x55b4ca6251c0 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97b5a00640 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca619fe0 0x55b4ca622150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97b5000640 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca61a900 0x55b4ca622690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97b6400640 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca6251c0 0x55b4ca622bd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f97a80ce050 tx=0x7f97a809ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2da39945e8e6713 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f9797400640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f9797400640 1 -- 172.21.15.1:0/952196300 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a801f040 con 0x55b4ca6251c0 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f9797400640 1 -- 172.21.15.1:0/952196300 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f97a80a50b0 con 0x55b4ca6251c0 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97bb171740 1 -- 172.21.15.1:0/952196300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97a4019ae0 con 0x55b4ca6251c0 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f9797400640 1 -- 172.21.15.1:0/952196300 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f97a80a52b0 con 0x55b4ca6251c0 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97bb171740 10 client.15786 fetch_fsmap learned FSMap version 36 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.076+0000 7f97bb171740 1 -- 172.21.15.1:0/952196300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b4ca291210 con 0x55b4ca6251c0 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.077+0000 7f9797400640 1 -- 172.21.15.1:0/952196300 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97a4019ae0 con 0x55b4ca6251c0 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.078+0000 7f9797400640 1 -- 172.21.15.1:0/952196300 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f97a800f040 con 0x55b4ca6251c0 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.078+0000 7f97bb171740 10 client.15786 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.078+0000 7f97bb171740 10 client.15786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.078+0000 7f97bb171740 20 client.15786 populate_metadata read hostname 'smithi001' 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.078+0000 7f97bb171740 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b4ca634f70 0x55b4ca6373a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.078+0000 7f97bb171740 4 client.15786 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.078+0000 7f97bb171740 1 -- 172.21.15.1:0/952196300 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4ca58cf90 con 0x55b4ca634f70 2024-04-26T21:35:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.078+0000 7f97b5a00640 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b4ca634f70 0x55b4ca6373a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.078+0000 7f97b5a00640 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b4ca634f70 0x55b4ca6373a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48723709632f1628 server_cookie=61e6ef55793b9bf in_seq=0 out_seq=0 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.078+0000 7f9797400640 10 client.15786 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.079+0000 7f9797400640 1 -- 172.21.15.1:0/952196300 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b4ca58cf90 con 0x55b4ca634f70 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.079+0000 7f9797400640 10 client.15786 handle_command_reply: tid=0 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.079+0000 7f97bb171740 1 client.15786 shutdown 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 1 -- 172.21.15.1:0/952196300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca6251c0 msgr2=0x55b4ca622bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca6251c0 0x55b4ca622bd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f97a80ce050 tx=0x7f97a809ca10 comp rx=0 tx=0).stop 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 1 -- 172.21.15.1:0/952196300 shutdown_connections 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b4ca634f70 0x55b4ca6373a0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca6251c0 0x55b4ca622bd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca61a900 0x55b4ca622690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 1 --2- 172.21.15.1:0/952196300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca619fe0 0x55b4ca622150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 1 -- 172.21.15.1:0/952196300 >> 172.21.15.1:0/952196300 conn(0x55b4ca6164f0 msgr2=0x55b4ca61c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f9797400640 0 client.15786 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 1 -- 172.21.15.1:0/952196300 shutdown_connections 2024-04-26T21:35:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 1 -- 172.21.15.1:0/952196300 wait complete. 2024-04-26T21:35:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.080+0000 7f97bb171740 20 client.15786 trim_cache size 0 max 16384 2024-04-26T21:35:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.098+0000 7f97bb171740 1 Processor -- start 2024-04-26T21:35:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.098+0000 7f97bb171740 1 -- start start 2024-04-26T21:35:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97bb171740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca619fe0 0x55b4ca630cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97bb171740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca61a900 0x55b4ca63ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97bb171740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca6251c0 0x55b4ca63d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97bb171740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f97b01b4870 con 0x55b4ca61a900 2024-04-26T21:35:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97bb171740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f97b01b2140 con 0x55b4ca619fe0 2024-04-26T21:35:40.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97bb171740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97b01aff90 con 0x55b4ca6251c0 2024-04-26T21:35:40.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b5000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca61a900 0x55b4ca63ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b5a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca619fe0 0x55b4ca630cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b6400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca6251c0 0x55b4ca63d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b6400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca6251c0 0x55b4ca63d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58730/0 (socket says 172.21.15.1:58730) 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b6400640 1 -- 172.21.15.1:0/783078838 learned_addr learned my addr 172.21.15.1:0/783078838 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b5000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca61a900 0x55b4ca63ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59084/0 (socket says 172.21.15.1:59084) 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b5a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca619fe0 0x55b4ca630cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:45616/0 (socket says 172.21.15.1:45616) 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b5a00640 1 -- 172.21.15.1:0/783078838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca6251c0 msgr2=0x55b4ca63d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b5a00640 1 --2- 172.21.15.1:0/783078838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca6251c0 0x55b4ca63d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b5a00640 1 -- 172.21.15.1:0/783078838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca61a900 msgr2=0x55b4ca63ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b5a00640 1 --2- 172.21.15.1:0/783078838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca61a900 0x55b4ca63ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b5a00640 1 -- 172.21.15.1:0/783078838 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4ca61a730 con 0x55b4ca619fe0 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.099+0000 7f97b6400640 1 --2- 172.21.15.1:0/783078838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca6251c0 0x55b4ca63d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.100+0000 7f97b5a00640 1 --2- 172.21.15.1:0/783078838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca619fe0 0x55b4ca630cb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f97a401c040 tx=0x7f97a400e700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f83ca92d4c1f006b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.100+0000 7f97a2000640 1 -- 172.21.15.1:0/783078838 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a4029e60 con 0x55b4ca619fe0 2024-04-26T21:35:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.100+0000 7f97a2000640 1 -- 172.21.15.1:0/783078838 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97a401f040 con 0x55b4ca619fe0 2024-04-26T21:35:40.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.100+0000 7f97a2000640 1 -- 172.21.15.1:0/783078838 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a401e040 con 0x55b4ca619fe0 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.101+0000 7f97bb171740 1 -- 172.21.15.1:0/783078838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca619fe0 msgr2=0x55b4ca630cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.101+0000 7f97bb171740 1 --2- 172.21.15.1:0/783078838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca619fe0 0x55b4ca630cb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f97a401c040 tx=0x7f97a400e700 comp rx=0 tx=0).stop 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.101+0000 7f97bb171740 1 -- 172.21.15.1:0/783078838 shutdown_connections 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.101+0000 7f97bb171740 1 --2- 172.21.15.1:0/783078838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca6251c0 0x55b4ca63d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.101+0000 7f97bb171740 1 --2- 172.21.15.1:0/783078838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca61a900 0x55b4ca63ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.101+0000 7f97bb171740 1 --2- 172.21.15.1:0/783078838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca619fe0 0x55b4ca630cb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.101+0000 7f97bb171740 1 -- 172.21.15.1:0/783078838 >> 172.21.15.1:0/783078838 conn(0x55b4ca6164f0 msgr2=0x55b4ca61d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.101+0000 7f97bb171740 1 -- 172.21.15.1:0/783078838 shutdown_connections 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.101+0000 7f97bb171740 1 -- 172.21.15.1:0/783078838 wait complete. 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.102+0000 7f97bb171740 1 -- start start 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.102+0000 7f97bb171740 1 Processor -- start 2024-04-26T21:35:40.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97bb171740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca619fe0 0x55b4ca62d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97bb171740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca628970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97bb171740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 0x55b4ca628eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97bb171740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f97b01b0600 con 0x55b4ca619fe0 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97bb171740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f97b01b1a70 con 0x55b4ca6251c0 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97bb171740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97b01b0cd0 con 0x55b4ca61a900 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97bb171740 10 client.0 _ll_register_callbacks cb 0x7f97a8098c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97b5000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca628970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97b5000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca628970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58742/0 (socket says 172.21.15.1:58742) 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97b6400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 0x55b4ca628eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97b5000640 1 -- 172.21.15.1:0/2437499188 learned_addr learned my addr 172.21.15.1:0/2437499188 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97b5a00640 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca619fe0 0x55b4ca62d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97b6400640 1 -- 172.21.15.1:0/2437499188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 msgr2=0x55b4ca628970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97b6400640 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca628970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97b6400640 1 -- 172.21.15.1:0/2437499188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca619fe0 msgr2=0x55b4ca62d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97b6400640 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca619fe0 0x55b4ca62d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.103+0000 7f97b6400640 1 -- 172.21.15.1:0/2437499188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4ca35d620 con 0x55b4ca6251c0 2024-04-26T21:35:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.104+0000 7f97b6400640 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 0x55b4ca628eb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f97a809c860 tx=0x7f97a809b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abf6db4d079688d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.104+0000 7f9797400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:35:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.104+0000 7f9797400640 1 -- 172.21.15.1:0/2437499188 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a801f070 con 0x55b4ca6251c0 2024-04-26T21:35:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.104+0000 7f9797400640 1 -- 172.21.15.1:0/2437499188 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f97a809bd20 con 0x55b4ca6251c0 2024-04-26T21:35:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.104+0000 7f97bb171740 1 -- 172.21.15.1:0/2437499188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97a401f040 con 0x55b4ca6251c0 2024-04-26T21:35:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.104+0000 7f9797400640 1 -- 172.21.15.1:0/2437499188 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f97a8098830 con 0x55b4ca6251c0 2024-04-26T21:35:40.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.104+0000 7f97bb171740 10 client.25949 fetch_fsmap learned FSMap version 36 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.104+0000 7f97bb171740 1 -- 172.21.15.1:0/2437499188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b4ca539cb0 con 0x55b4ca6251c0 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.104+0000 7f9797400640 1 -- 172.21.15.1:0/2437499188 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97a401f040 con 0x55b4ca6251c0 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.104+0000 7f9797400640 1 -- 172.21.15.1:0/2437499188 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f97a80a1e10 con 0x55b4ca6251c0 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.105+0000 7f97bb171740 10 client.25949 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.105+0000 7f97bb171740 10 client.25949 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.105+0000 7f97bb171740 20 client.25949 populate_metadata read hostname 'smithi001' 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.105+0000 7f97bb171740 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b4ca69b9c0 0x55b4ca69dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.105+0000 7f97bb171740 4 client.25949 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.105+0000 7f97bb171740 1 -- 172.21.15.1:0/2437499188 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b4ca586e00 con 0x55b4ca69b9c0 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.105+0000 7f97b5a00640 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b4ca69b9c0 0x55b4ca69dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.105+0000 7f97b5a00640 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b4ca69b9c0 0x55b4ca69dde0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99062df9e552a0c6 server_cookie=8771b16e54ae3fc in_seq=0 out_seq=0 2024-04-26T21:35:40.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.105+0000 7f9797400640 10 client.25949 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f9797400640 1 -- 172.21.15.1:0/2437499188 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b4ca586e00 con 0x55b4ca69b9c0 2024-04-26T21:35:40.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f9797400640 10 client.25949 handle_command_reply: tid=0 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 client.25949 shutdown 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 -- 172.21.15.1:0/2437499188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 msgr2=0x55b4ca628eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 0x55b4ca628eb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f97a809c860 tx=0x7f97a809b7e0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 -- 172.21.15.1:0/2437499188 shutdown_connections 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b4ca69b9c0 0x55b4ca69dde0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b4ca61a900 0x55b4ca628970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b4ca6251c0 0x55b4ca628eb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 --2- 172.21.15.1:0/2437499188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b4ca619fe0 0x55b4ca62d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f9797400640 0 client.25949 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 -- 172.21.15.1:0/2437499188 >> 172.21.15.1:0/2437499188 conn(0x55b4ca6164f0 msgr2=0x55b4ca622110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 -- 172.21.15.1:0/2437499188 shutdown_connections 2024-04-26T21:35:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 1 -- 172.21.15.1:0/2437499188 wait complete. 2024-04-26T21:35:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.159+0000 7f97bb171740 20 client.25949 trim_cache size 0 max 16384 2024-04-26T21:35:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:35:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:35:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:35:40.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:35:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.160+0000 7f97b6e00640 1 -- 172.21.15.1:0/4118980480 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f97980a3d80 msgr2=0x7f97980a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.160+0000 7f97b6e00640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f97980a3d80 0x7f97980a6240 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55b4ca617910 tx=0x7f979c031400 comp rx=0 tx=0).stop 2024-04-26T21:35:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.160+0000 7f97b6e00640 1 -- 172.21.15.1:0/4118980480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97b010cd30 msgr2=0x7f97b019fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.160+0000 7f97b6e00640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97b010cd30 0x7f97b019fa60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f97a800eaa0 tx=0x7f97a800ef60 comp rx=0 tx=0).stop 2024-04-26T21:35:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.160+0000 7f97b6e00640 1 -- 172.21.15.1:0/4118980480 shutdown_connections 2024-04-26T21:35:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.160+0000 7f97b6e00640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f97980a3d80 0x7f97980a6240 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.160+0000 7f97b6e00640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97b0104230 0x7f97b019b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.160+0000 7f97b6e00640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f97b010cd30 0x7f97b019fa60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.160+0000 7f97b6e00640 1 --2- 172.21.15.1:0/4118980480 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97b0103910 0x7f97b019b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.160+0000 7f97b6e00640 1 -- 172.21.15.1:0/4118980480 >> 172.21.15.1:0/4118980480 conn(0x7f97b00fd110 msgr2=0x7f97b00ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.161+0000 7f97b6e00640 1 -- 172.21.15.1:0/4118980480 shutdown_connections 2024-04-26T21:35:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:40.161+0000 7f97b6e00640 1 -- 172.21.15.1:0/4118980480 wait complete. 2024-04-26T21:35:40.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:40 smithi001 ceph-mon[28054]: pgmap v163: 129 pgs: 129 active+clean; 85 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:35:40.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:40 smithi038 ceph-mon[32724]: pgmap v163: 129 pgs: 129 active+clean; 85 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:35:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:40 smithi186 ceph-mon[32991]: pgmap v163: 129 pgs: 129 active+clean; 85 MiB data, 926 MiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:35:40.555 INFO:tasks.workunit.client.0.smithi001.stdout: 1 140745 4.09 MB/sec execute 98 sec latency 21.254 ms 2024-04-26T21:35:41.555 INFO:tasks.workunit.client.0.smithi001.stdout: 1 141894 4.15 MB/sec execute 99 sec latency 21.042 ms 2024-04-26T21:35:42.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:42 smithi001 ceph-mon[28054]: pgmap v164: 129 pgs: 129 active+clean; 87 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-26T21:35:42.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:42 smithi038 ceph-mon[32724]: pgmap v164: 129 pgs: 129 active+clean; 87 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-26T21:35:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:42 smithi186 ceph-mon[32991]: pgmap v164: 129 pgs: 129 active+clean; 87 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-26T21:35:42.555 INFO:tasks.workunit.client.0.smithi001.stdout: 1 142341 4.13 MB/sec execute 100 sec latency 26.719 ms 2024-04-26T21:35:43.555 INFO:tasks.workunit.client.0.smithi001.stdout: 1 143019 4.11 MB/sec execute 101 sec latency 17.174 ms 2024-04-26T21:35:44.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:44 smithi001 ceph-mon[28054]: pgmap v165: 129 pgs: 129 active+clean; 87 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-26T21:35:44.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:44 smithi038 ceph-mon[32724]: pgmap v165: 129 pgs: 129 active+clean; 87 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-26T21:35:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:44 smithi186 ceph-mon[32991]: pgmap v165: 129 pgs: 129 active+clean; 87 MiB data, 946 MiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-26T21:35:44.555 INFO:tasks.workunit.client.0.smithi001.stdout: 1 144389 4.12 MB/sec execute 102 sec latency 35.899 ms 2024-04-26T21:35:45.555 INFO:tasks.workunit.client.0.smithi001.stdout: 1 145671 4.19 MB/sec execute 103 sec latency 74.070 ms 2024-04-26T21:35:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:46 smithi001 ceph-mon[28054]: pgmap v166: 129 pgs: 129 active+clean; 89 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 2.0 MiB/s wr, 211 op/s 2024-04-26T21:35:46.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:46 smithi038 ceph-mon[32724]: pgmap v166: 129 pgs: 129 active+clean; 89 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 2.0 MiB/s wr, 211 op/s 2024-04-26T21:35:46.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:46 smithi186 ceph-mon[32991]: pgmap v166: 129 pgs: 129 active+clean; 89 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 2.0 MiB/s wr, 211 op/s 2024-04-26T21:35:46.555 INFO:tasks.workunit.client.0.smithi001.stdout: 1 145995 4.15 MB/sec execute 104 sec latency 26.662 ms 2024-04-26T21:35:47.555 INFO:tasks.workunit.client.0.smithi001.stdout: 1 146884 4.16 MB/sec execute 105 sec latency 23.668 ms 2024-04-26T21:35:48.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:48 smithi001 ceph-mon[28054]: pgmap v167: 129 pgs: 129 active+clean; 89 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T21:35:48.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:48 smithi038 ceph-mon[32724]: pgmap v167: 129 pgs: 129 active+clean; 89 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T21:35:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:48 smithi186 ceph-mon[32991]: pgmap v167: 129 pgs: 129 active+clean; 89 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T21:35:48.555 INFO:tasks.workunit.client.0.smithi001.stdout: 1 148359 4.19 MB/sec execute 106 sec latency 23.354 ms 2024-04-26T21:35:49.555 INFO:tasks.workunit.client.0.smithi001.stdout: 1 149265 4.21 MB/sec execute 107 sec latency 40.187 ms 2024-04-26T21:35:50.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:50 smithi001 ceph-mon[28054]: pgmap v168: 129 pgs: 129 active+clean; 89 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T21:35:50.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:50 smithi038 ceph-mon[32724]: pgmap v168: 129 pgs: 129 active+clean; 89 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T21:35:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:50 smithi186 ceph-mon[32991]: pgmap v168: 129 pgs: 129 active+clean; 89 MiB data, 947 MiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T21:35:50.556 INFO:tasks.workunit.client.0.smithi001.stdout: 1 149686 4.18 MB/sec execute 108 sec latency 23.066 ms 2024-04-26T21:35:51.556 INFO:tasks.workunit.client.0.smithi001.stdout: 1 150692 4.19 MB/sec execute 109 sec latency 20.809 ms 2024-04-26T21:35:52.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:52 smithi001 ceph-mon[28054]: pgmap v169: 129 pgs: 129 active+clean; 90 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.0 MiB/s wr, 221 op/s 2024-04-26T21:35:52.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:52 smithi038 ceph-mon[32724]: pgmap v169: 129 pgs: 129 active+clean; 90 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.0 MiB/s wr, 221 op/s 2024-04-26T21:35:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:52 smithi186 ceph-mon[32991]: pgmap v169: 129 pgs: 129 active+clean; 90 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.0 MiB/s wr, 221 op/s 2024-04-26T21:35:52.556 INFO:tasks.workunit.client.0.smithi001.stdout: 1 151921 4.21 MB/sec execute 110 sec latency 26.764 ms 2024-04-26T21:35:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.317+0000 7f54b7e00640 1 -- 172.21.15.1:0/774329261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54b8108760 msgr2=0x7f54b8108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.317+0000 7f54b7e00640 1 --2- 172.21.15.1:0/774329261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54b8108760 0x7f54b8108b40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009980 tx=0x7f54a802e830 comp rx=0 tx=0).stop 2024-04-26T21:35:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.317+0000 7f54b7e00640 1 -- 172.21.15.1:0/774329261 shutdown_connections 2024-04-26T21:35:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.317+0000 7f54b7e00640 1 --2- 172.21.15.1:0/774329261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54b8103110 0x7f54b810f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.319 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.317+0000 7f54b7e00640 1 --2- 172.21.15.1:0/774329261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54b8102750 0x7f54b8102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.320 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.317+0000 7f54b7e00640 1 --2- 172.21.15.1:0/774329261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54b8108760 0x7f54b8108b40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.320 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.317+0000 7f54b7e00640 1 -- 172.21.15.1:0/774329261 >> 172.21.15.1:0/774329261 conn(0x7f54b80fe240 msgr2=0x7f54b8100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:53.320 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.318+0000 7f54b7e00640 1 -- 172.21.15.1:0/774329261 shutdown_connections 2024-04-26T21:35:53.320 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.318+0000 7f54b7e00640 1 -- 172.21.15.1:0/774329261 wait complete. 2024-04-26T21:35:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.319+0000 7f54b7e00640 1 Processor -- start 2024-04-26T21:35:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.319+0000 7f54b7e00640 1 -- start start 2024-04-26T21:35:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.319+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54b8102750 0x7f54b819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.319+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54b8103110 0x7f54b819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.319+0000 7f54b7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54b8108760 0x7f54b8196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.319+0000 7f54b7e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f54b8110860 con 0x7f54b8102750 2024-04-26T21:35:53.322 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.319+0000 7f54b7e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81106e0 con 0x7f54b8103110 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.319+0000 7f54b7e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81109e0 con 0x7f54b8108760 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.319+0000 7f54b6a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54b8102750 0x7f54b819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.320+0000 7f54b6000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54b8103110 0x7f54b819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.320+0000 7f54b6000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54b8103110 0x7f54b819cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34448/0 (socket says 172.21.15.1:34448) 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.320+0000 7f54b6000640 1 -- 172.21.15.1:0/2500685581 learned_addr learned my addr 172.21.15.1:0/2500685581 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.320+0000 7f54b7400640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54b8108760 0x7f54b8196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.320+0000 7f54b6000640 1 -- 172.21.15.1:0/2500685581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54b8108760 msgr2=0x7f54b8196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.320+0000 7f54b6000640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54b8108760 0x7f54b8196b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.320+0000 7f54b6000640 1 -- 172.21.15.1:0/2500685581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54b8102750 msgr2=0x7f54b819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.320+0000 7f54b6000640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54b8102750 0x7f54b819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.320+0000 7f54b6000640 1 -- 172.21.15.1:0/2500685581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b81973e0 con 0x7f54b8103110 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.320+0000 7f54b6000640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54b8103110 0x7f54b819cfa0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f54a400ea60 tx=0x7f54a400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.321+0000 7f54af400640 1 -- 172.21.15.1:0/2500685581 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a400cde0 con 0x7f54b8103110 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.321+0000 7f54af400640 1 -- 172.21.15.1:0/2500685581 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54a40035f0 con 0x7f54b8103110 2024-04-26T21:35:53.323 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.321+0000 7f54af400640 1 -- 172.21.15.1:0/2500685581 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4010690 con 0x7f54b8103110 2024-04-26T21:35:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.321+0000 7f54b7e00640 1 -- 172.21.15.1:0/2500685581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b81976d0 con 0x7f54b8103110 2024-04-26T21:35:53.324 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.321+0000 7f54b7e00640 1 -- 172.21.15.1:0/2500685581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b81a39e0 con 0x7f54b8103110 2024-04-26T21:35:53.325 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.324+0000 7f54af400640 1 -- 172.21.15.1:0/2500685581 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f54a40037a0 con 0x7f54b8103110 2024-04-26T21:35:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.324+0000 7f54c099e740 1 Processor -- start 2024-04-26T21:35:53.326 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.324+0000 7f54c099e740 1 -- start start 2024-04-26T21:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.325+0000 7f54c099e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c9675793c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.325+0000 7f54c099e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c967583c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.325+0000 7f54c099e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c9675841c0 0x55c9675865b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.325+0000 7f54c099e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c9674de380 con 0x55c967578fe0 2024-04-26T21:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.325+0000 7f54c099e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c96747ac30 con 0x55c967579900 2024-04-26T21:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.325+0000 7f54c099e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c9674da340 con 0x55c9675841c0 2024-04-26T21:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.325+0000 7f54b6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c967583c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.327 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.325+0000 7f54b6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c967583c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34462/0 (socket says 172.21.15.1:34462) 2024-04-26T21:35:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.325+0000 7f54b6a00640 1 -- 172.21.15.1:0/393384745 learned_addr learned my addr 172.21.15.1:0/393384745 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.325+0000 7f54b6000640 1 --2- 172.21.15.1:0/393384745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c9675841c0 0x55c9675865b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.326+0000 7f54b7400640 1 --2- 172.21.15.1:0/393384745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c9675793c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.326+0000 7f54af400640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f549c0a3d80 0x7f549c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.328 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.326+0000 7f54b6a00640 1 -- 172.21.15.1:0/393384745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c9675841c0 msgr2=0x55c9675865b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:53.329 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.326+0000 7f54b6a00640 1 --2- 172.21.15.1:0/393384745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c9675841c0 0x55c9675865b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.326+0000 7f54b6a00640 1 -- 172.21.15.1:0/393384745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 msgr2=0x55c9675793c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.326+0000 7f54b6a00640 1 --2- 172.21.15.1:0/393384745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c9675793c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.326+0000 7f54b6a00640 1 -- 172.21.15.1:0/393384745 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c96749c940 con 0x55c967579900 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.326+0000 7f54af400640 1 -- 172.21.15.1:0/2500685581 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f54a40d0f70 con 0x7f54b8103110 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.326+0000 7f54b7400640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f549c0a3d80 0x7f549c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.326+0000 7f54b6a00640 1 --2- 172.21.15.1:0/393384745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c967583c80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009230 tx=0x7f54a80095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8343fb403b3ab2b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.327+0000 7f54a3e00640 1 -- 172.21.15.1:0/393384745 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8045070 con 0x55c967579900 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.327+0000 7f54a3e00640 1 -- 172.21.15.1:0/393384745 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54a8004400 con 0x55c967579900 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.327+0000 7f54a3e00640 1 -- 172.21.15.1:0/393384745 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a803a040 con 0x55c967579900 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54c099e740 1 -- 172.21.15.1:0/393384745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 msgr2=0x55c967583c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54b7400640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f549c0a3d80 0x7f549c0a6240 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f54b0005fd0 tx=0x7f54b0004a40 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54c099e740 1 --2- 172.21.15.1:0/393384745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c967583c80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009230 tx=0x7f54a80095e0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54c099e740 1 -- 172.21.15.1:0/393384745 shutdown_connections 2024-04-26T21:35:53.330 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54c099e740 1 --2- 172.21.15.1:0/393384745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c9675841c0 0x55c9675865b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54c099e740 1 --2- 172.21.15.1:0/393384745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c967583c80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54c099e740 1 --2- 172.21.15.1:0/393384745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c9675793c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54c099e740 1 -- 172.21.15.1:0/393384745 >> 172.21.15.1:0/393384745 conn(0x55c9675754f0 msgr2=0x55c9675758d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54c099e740 1 -- 172.21.15.1:0/393384745 shutdown_connections 2024-04-26T21:35:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54c099e740 1 -- 172.21.15.1:0/393384745 wait complete. 2024-04-26T21:35:53.331 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.328+0000 7f54c099e740 1 -- start start 2024-04-26T21:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.329+0000 7f54c099e740 1 Processor -- start 2024-04-26T21:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.329+0000 7f54c099e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c967581100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.329+0000 7f54c099e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967579900 0x55c967581640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.329+0000 7f54c099e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c967581b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.329+0000 7f54c099e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c967586d90 con 0x55c967578fe0 2024-04-26T21:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.329+0000 7f54c099e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c967465c60 con 0x55c9675841c0 2024-04-26T21:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.329+0000 7f54c099e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c967586f10 con 0x55c967579900 2024-04-26T21:35:53.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.329+0000 7f54c099e740 10 client.0 _ll_register_callbacks cb 0x55c967491e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b6000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967579900 0x55c967581640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c967581b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b6000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967579900 0x55c967581640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52670/0 (socket says 172.21.15.1:52670) 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b6000640 1 -- 172.21.15.1:0/505601269 learned_addr learned my addr 172.21.15.1:0/505601269 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b6a00640 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c967581100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c967581b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34468/0 (socket says 172.21.15.1:34468) 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b6000640 1 -- 172.21.15.1:0/505601269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 msgr2=0x55c967581b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b6000640 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c967581b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b6000640 1 -- 172.21.15.1:0/505601269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 msgr2=0x55c967581100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b6000640 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c967581100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b6000640 1 -- 172.21.15.1:0/505601269 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c967474510 con 0x55c967579900 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b7400640 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c967581b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54b6000640 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967579900 0x55c967581640 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009920 tx=0x7f54a4098e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6fac14995673437 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:53.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.330+0000 7f54a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.331+0000 7f54a0c00640 1 -- 172.21.15.1:0/505601269 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a409d1a0 con 0x55c967579900 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.331+0000 7f54a0c00640 1 -- 172.21.15.1:0/505601269 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f54a4099110 con 0x55c967579900 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.331+0000 7f54c099e740 1 -- 172.21.15.1:0/505601269 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54a8004400 con 0x55c967579900 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.331+0000 7f54a0c00640 1 -- 172.21.15.1:0/505601269 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f54a409caa0 con 0x55c967579900 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.331+0000 7f54c099e740 10 client.25939 fetch_fsmap learned FSMap version 36 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.331+0000 7f54c099e740 1 -- 172.21.15.1:0/505601269 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c9671f0210 con 0x55c967579900 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.331+0000 7f54a0c00640 1 -- 172.21.15.1:0/505601269 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54a8004400 con 0x55c967579900 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.331+0000 7f54a0c00640 1 -- 172.21.15.1:0/505601269 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f54a4004000 con 0x55c967579900 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.332+0000 7f54c099e740 10 client.25939 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.332+0000 7f54c099e740 10 client.25939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.332+0000 7f54c099e740 20 client.25939 populate_metadata read hostname 'smithi001' 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.332+0000 7f54c099e740 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c967593e10 0x55c967596200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.332+0000 7f54c099e740 4 client.25939 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.332+0000 7f54c099e740 1 -- 172.21.15.1:0/505601269 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9674ebf90 con 0x55c967593e10 2024-04-26T21:35:53.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.332+0000 7f54b6a00640 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c967593e10 0x55c967596200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.333+0000 7f54b6a00640 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c967593e10 0x55c967596200 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5366ae3d26d8f0c7 server_cookie=675bbaf1512f0eed in_seq=0 out_seq=0 2024-04-26T21:35:53.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.333+0000 7f54a0c00640 10 client.25939 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.333+0000 7f54a0c00640 1 -- 172.21.15.1:0/505601269 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c9674ebf90 con 0x55c967593e10 2024-04-26T21:35:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.333+0000 7f54a0c00640 10 client.25939 handle_command_reply: tid=0 2024-04-26T21:35:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.333+0000 7f54c099e740 1 client.25939 shutdown 2024-04-26T21:35:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 1 -- 172.21.15.1:0/505601269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967579900 msgr2=0x55c967581640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967579900 0x55c967581640 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009920 tx=0x7f54a4098e00 comp rx=0 tx=0).stop 2024-04-26T21:35:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 1 -- 172.21.15.1:0/505601269 shutdown_connections 2024-04-26T21:35:53.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c967593e10 0x55c967596200 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967579900 0x55c967581640 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c967581b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 1 --2- 172.21.15.1:0/505601269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c967581100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 1 -- 172.21.15.1:0/505601269 >> 172.21.15.1:0/505601269 conn(0x55c9675754f0 msgr2=0x55c96757b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54a0c00640 0 client.25939 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 1 -- 172.21.15.1:0/505601269 shutdown_connections 2024-04-26T21:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 1 -- 172.21.15.1:0/505601269 wait complete. 2024-04-26T21:35:53.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.334+0000 7f54c099e740 20 client.25939 trim_cache size 0 max 16384 2024-04-26T21:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.351+0000 7f54c099e740 1 Processor -- start 2024-04-26T21:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.351+0000 7f54c099e740 1 -- start start 2024-04-26T21:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54c099e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967578fe0 0x55c96758fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54c099e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967579900 0x55c9675800e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54c099e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c96759bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54c099e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81b0330 con 0x55c967579900 2024-04-26T21:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54c099e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81adc00 con 0x55c9675841c0 2024-04-26T21:35:53.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54c099e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81aba50 con 0x55c967578fe0 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c96759bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967578fe0 0x55c96758fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967578fe0 0x55c96758fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52680/0 (socket says 172.21.15.1:52680) 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b6a00640 1 -- 172.21.15.1:0/1069761898 learned_addr learned my addr 172.21.15.1:0/1069761898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c96759bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34472/0 (socket says 172.21.15.1:34472) 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b6000640 1 --2- 172.21.15.1:0/1069761898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967579900 0x55c9675800e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b6a00640 1 -- 172.21.15.1:0/1069761898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 msgr2=0x55c96759bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b6a00640 1 --2- 172.21.15.1:0/1069761898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c96759bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b6a00640 1 -- 172.21.15.1:0/1069761898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967579900 msgr2=0x55c9675800e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b6a00640 1 --2- 172.21.15.1:0/1069761898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967579900 0x55c9675800e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b6a00640 1 -- 172.21.15.1:0/1069761898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c967596570 con 0x55c967578fe0 2024-04-26T21:35:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.352+0000 7f54b6000640 1 --2- 172.21.15.1:0/1069761898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967579900 0x55c9675800e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:35:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.353+0000 7f54b6a00640 1 --2- 172.21.15.1:0/1069761898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967578fe0 0x55c96758fc50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f54a8005dc0 tx=0x7f54a8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2456b4b25f87fa06 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.353+0000 7f54a3400640 1 -- 172.21.15.1:0/1069761898 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a8045040 con 0x55c967578fe0 2024-04-26T21:35:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.353+0000 7f54a3400640 1 -- 172.21.15.1:0/1069761898 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54a80043e0 con 0x55c967578fe0 2024-04-26T21:35:53.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.353+0000 7f54a3400640 1 -- 172.21.15.1:0/1069761898 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a803a040 con 0x55c967578fe0 2024-04-26T21:35:53.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.354+0000 7f54c099e740 1 -- 172.21.15.1:0/1069761898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967578fe0 msgr2=0x55c96758fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.354+0000 7f54c099e740 1 --2- 172.21.15.1:0/1069761898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967578fe0 0x55c96758fc50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f54a8005dc0 tx=0x7f54a8000bc0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.354+0000 7f54c099e740 1 -- 172.21.15.1:0/1069761898 shutdown_connections 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.354+0000 7f54c099e740 1 --2- 172.21.15.1:0/1069761898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c9675841c0 0x55c96759bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.354+0000 7f54c099e740 1 --2- 172.21.15.1:0/1069761898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967579900 0x55c9675800e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.354+0000 7f54c099e740 1 --2- 172.21.15.1:0/1069761898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c967578fe0 0x55c96758fc50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.354+0000 7f54c099e740 1 -- 172.21.15.1:0/1069761898 >> 172.21.15.1:0/1069761898 conn(0x55c9675754f0 msgr2=0x55c96757f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.354+0000 7f54c099e740 1 -- 172.21.15.1:0/1069761898 shutdown_connections 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.354+0000 7f54c099e740 1 -- 172.21.15.1:0/1069761898 wait complete. 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.354+0000 7f54c099e740 1 -- start start 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.355+0000 7f54c099e740 1 Processor -- start 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.355+0000 7f54c099e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c96758c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.355+0000 7f54c099e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c96758c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.355+0000 7f54c099e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c9675841c0 0x55c96757d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.355+0000 7f54c099e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81ac0c0 con 0x55c967578fe0 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.355+0000 7f54c099e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81ad530 con 0x55c967579900 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.355+0000 7f54c099e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f54b81ac790 con 0x55c9675841c0 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.355+0000 7f54c099e740 10 client.0 _ll_register_callbacks cb 0x7f54a4003d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b6000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c96758c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b7400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c9675841c0 0x55c96757d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b6000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c96758c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34484/0 (socket says 172.21.15.1:34484) 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b6a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c96758c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b6000640 1 -- 172.21.15.1:0/3886542484 learned_addr learned my addr 172.21.15.1:0/3886542484 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b6000640 1 -- 172.21.15.1:0/3886542484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c9675841c0 msgr2=0x55c96757d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b6000640 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c9675841c0 0x55c96757d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b6000640 1 -- 172.21.15.1:0/3886542484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 msgr2=0x55c96758c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b6000640 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c96758c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b6000640 1 -- 172.21.15.1:0/3886542484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c9672bc620 con 0x55c967579900 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54b6000640 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c96758c6a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f54a400c760 tx=0x7f54a4003cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a24647e7df1ef30f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54a0c00640 1 -- 172.21.15.1:0/3886542484 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a409d1a0 con 0x55c967579900 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.356+0000 7f54a0c00640 1 -- 172.21.15.1:0/3886542484 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f54a40988c0 con 0x55c967579900 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54c099e740 1 -- 172.21.15.1:0/3886542484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54a80043e0 con 0x55c967579900 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54a0c00640 1 -- 172.21.15.1:0/3886542484 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f54a40a0e50 con 0x55c967579900 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54c099e740 10 client.25964 fetch_fsmap learned FSMap version 36 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54c099e740 1 -- 172.21.15.1:0/3886542484 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c967498cb0 con 0x55c967579900 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54a0c00640 1 -- 172.21.15.1:0/3886542484 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f54a80043e0 con 0x55c967579900 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54a0c00640 1 -- 172.21.15.1:0/3886542484 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f54a40d3160 con 0x55c967579900 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54c099e740 10 client.25964 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54c099e740 10 client.25964 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54c099e740 20 client.25964 populate_metadata read hostname 'smithi001' 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54c099e740 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c9675f9f80 0x55c9675fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:35:53.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54c099e740 4 client.25964 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:35:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.357+0000 7f54c099e740 1 -- 172.21.15.1:0/3886542484 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c9674e5e00 con 0x55c9675f9f80 2024-04-26T21:35:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.358+0000 7f54b6a00640 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c9675f9f80 0x55c9675fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:35:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.358+0000 7f54b6a00640 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c9675f9f80 0x55c9675fc3a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91c48387b21ebcbb server_cookie=8086af5212a7f315 in_seq=0 out_seq=0 2024-04-26T21:35:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.358+0000 7f54a0c00640 10 client.25964 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.428+0000 7f54a0c00640 1 -- 172.21.15.1:0/3886542484 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c9674e5e00 con 0x55c9675f9f80 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.428+0000 7f54a0c00640 10 client.25964 handle_command_reply: tid=0 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.428+0000 7f54c099e740 1 client.25964 shutdown 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.428+0000 7f54c099e740 1 -- 172.21.15.1:0/3886542484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 msgr2=0x55c96758c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.428+0000 7f54c099e740 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c96758c6a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f54a400c760 tx=0x7f54a4003cf0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.428+0000 7f54c099e740 1 -- 172.21.15.1:0/3886542484 shutdown_connections 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.428+0000 7f54c099e740 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c9675f9f80 0x55c9675fc3a0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.428+0000 7f54c099e740 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c9675841c0 0x55c96757d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.429+0000 7f54a0c00640 0 client.25964 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.428+0000 7f54c099e740 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c967579900 0x55c96758c6a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.429+0000 7f54c099e740 1 --2- 172.21.15.1:0/3886542484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c967578fe0 0x55c96758c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.429+0000 7f54c099e740 1 -- 172.21.15.1:0/3886542484 >> 172.21.15.1:0/3886542484 conn(0x55c9675754f0 msgr2=0x55c96757a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.429+0000 7f54c099e740 1 -- 172.21.15.1:0/3886542484 shutdown_connections 2024-04-26T21:35:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.429+0000 7f54c099e740 1 -- 172.21.15.1:0/3886542484 wait complete. 2024-04-26T21:35:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.429+0000 7f54c099e740 20 client.25964 trim_cache size 0 max 16384 2024-04-26T21:35:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:35:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:35:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:35:53.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 -- 172.21.15.1:0/2500685581 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f549c0a3d80 msgr2=0x7f549c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f549c0a3d80 0x7f549c0a6240 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f54b0005fd0 tx=0x7f54b0004a40 comp rx=0 tx=0).stop 2024-04-26T21:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 -- 172.21.15.1:0/2500685581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54b8103110 msgr2=0x7f54b819cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54b8103110 0x7f54b819cfa0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f54a400ea60 tx=0x7f54a400ef20 comp rx=0 tx=0).stop 2024-04-26T21:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 -- 172.21.15.1:0/2500685581 shutdown_connections 2024-04-26T21:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f549c0a3d80 0x7f549c0a6240 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54b8108760 0x7f54b8196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f54b8103110 0x7f54b819cfa0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 --2- 172.21.15.1:0/2500685581 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54b8102750 0x7f54b819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:35:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 -- 172.21.15.1:0/2500685581 >> 172.21.15.1:0/2500685581 conn(0x7f54b80fe240 msgr2=0x7f54b80fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:35:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 -- 172.21.15.1:0/2500685581 shutdown_connections 2024-04-26T21:35:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:35:53.430+0000 7f54b7e00640 1 -- 172.21.15.1:0/2500685581 wait complete. 2024-04-26T21:35:53.556 INFO:tasks.workunit.client.0.smithi001.stdout: 1 152851 4.23 MB/sec execute 111 sec latency 62.578 ms 2024-04-26T21:35:54.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:54 smithi038 ceph-mon[32724]: pgmap v170: 129 pgs: 129 active+clean; 90 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-04-26T21:35:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:54 smithi186 ceph-mon[32991]: pgmap v170: 129 pgs: 129 active+clean; 90 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-04-26T21:35:54.556 INFO:tasks.workunit.client.0.smithi001.stdout: 1 153341 4.20 MB/sec execute 112 sec latency 50.757 ms 2024-04-26T21:35:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:54 smithi001 ceph-mon[28054]: pgmap v170: 129 pgs: 129 active+clean; 90 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-04-26T21:35:55.556 INFO:tasks.workunit.client.0.smithi001.stdout: 1 154896 4.22 MB/sec execute 113 sec latency 24.996 ms 2024-04-26T21:35:56.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:56 smithi186 ceph-mon[32991]: pgmap v171: 129 pgs: 129 active+clean; 92 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-04-26T21:35:56.556 INFO:tasks.workunit.client.0.smithi001.stdout: 1 155874 4.25 MB/sec execute 114 sec latency 21.342 ms 2024-04-26T21:35:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:56 smithi001 ceph-mon[28054]: pgmap v171: 129 pgs: 129 active+clean; 92 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-04-26T21:35:56.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:56 smithi038 ceph-mon[32724]: pgmap v171: 129 pgs: 129 active+clean; 92 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 2.0 MiB/s wr, 257 op/s 2024-04-26T21:35:57.556 INFO:tasks.workunit.client.0.smithi001.stdout: 1 156433 4.25 MB/sec execute 115 sec latency 23.822 ms 2024-04-26T21:35:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:35:58 smithi186 ceph-mon[32991]: pgmap v172: 129 pgs: 129 active+clean; 92 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.2 MiB/s wr, 168 op/s 2024-04-26T21:35:58.556 INFO:tasks.workunit.client.0.smithi001.stdout: 1 156873 4.22 MB/sec execute 116 sec latency 30.861 ms 2024-04-26T21:35:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:35:58 smithi001 ceph-mon[28054]: pgmap v172: 129 pgs: 129 active+clean; 92 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.2 MiB/s wr, 168 op/s 2024-04-26T21:35:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:35:58 smithi038 ceph-mon[32724]: pgmap v172: 129 pgs: 129 active+clean; 92 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.2 MiB/s wr, 168 op/s 2024-04-26T21:35:59.557 INFO:tasks.workunit.client.0.smithi001.stdout: 1 158185 4.23 MB/sec execute 117 sec latency 46.898 ms 2024-04-26T21:36:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:00 smithi186 ceph-mon[32991]: pgmap v173: 129 pgs: 129 active+clean; 92 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.2 MiB/s wr, 168 op/s 2024-04-26T21:36:00.557 INFO:tasks.workunit.client.0.smithi001.stdout: 1 159325 4.26 MB/sec execute 118 sec latency 33.887 ms 2024-04-26T21:36:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:00 smithi001 ceph-mon[28054]: pgmap v173: 129 pgs: 129 active+clean; 92 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.2 MiB/s wr, 168 op/s 2024-04-26T21:36:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:00 smithi038 ceph-mon[32724]: pgmap v173: 129 pgs: 129 active+clean; 92 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.2 MiB/s wr, 168 op/s 2024-04-26T21:36:01.557 INFO:tasks.workunit.client.0.smithi001.stdout: 1 159352 4.23 MB/sec execute 119 sec latency 107.315 ms 2024-04-26T21:36:02.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:02 smithi186 ceph-mon[32991]: pgmap v174: 129 pgs: 129 active+clean; 95 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-04-26T21:36:02.557 INFO:tasks.workunit.client.0.smithi001.stdout: 1 159965 4.23 MB/sec execute 120 sec latency 61.204 ms 2024-04-26T21:36:02.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:02 smithi001 ceph-mon[28054]: pgmap v174: 129 pgs: 129 active+clean; 95 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-04-26T21:36:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:02 smithi038 ceph-mon[32724]: pgmap v174: 129 pgs: 129 active+clean; 95 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 1.8 MiB/s wr, 227 op/s 2024-04-26T21:36:03.557 INFO:tasks.workunit.client.0.smithi001.stdout: 1 160350 4.20 MB/sec execute 121 sec latency 22.346 ms 2024-04-26T21:36:04.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:04 smithi186 ceph-mon[32991]: pgmap v175: 129 pgs: 129 active+clean; 95 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-04-26T21:36:04.558 INFO:tasks.workunit.client.0.smithi001.stdout: 1 161288 4.21 MB/sec execute 122 sec latency 51.684 ms 2024-04-26T21:36:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:04 smithi001 ceph-mon[28054]: pgmap v175: 129 pgs: 129 active+clean; 95 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-04-26T21:36:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:04 smithi038 ceph-mon[32724]: pgmap v175: 129 pgs: 129 active+clean; 95 MiB data, 1002 MiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 1.1 MiB/s wr, 159 op/s 2024-04-26T21:36:05.560 INFO:tasks.workunit.client.0.smithi001.stdout: 1 162561 4.23 MB/sec execute 123 sec latency 27.338 ms 2024-04-26T21:36:06.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:06 smithi186 ceph-mon[32991]: pgmap v176: 129 pgs: 129 active+clean; 100 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 2.0 MiB/s wr, 247 op/s 2024-04-26T21:36:06.558 INFO:tasks.workunit.client.0.smithi001.stdout: 1 163443 4.25 MB/sec execute 124 sec latency 25.112 ms 2024-04-26T21:36:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:06 smithi001 ceph-mon[28054]: pgmap v176: 129 pgs: 129 active+clean; 100 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 2.0 MiB/s wr, 247 op/s 2024-04-26T21:36:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:06 smithi038 ceph-mon[32724]: pgmap v176: 129 pgs: 129 active+clean; 100 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 2.0 MiB/s wr, 247 op/s 2024-04-26T21:36:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.590+0000 7f779dc00640 1 -- 172.21.15.1:0/241293100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798103110 msgr2=0x7f779810f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.590+0000 7f779dc00640 1 --2- 172.21.15.1:0/241293100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798103110 0x7f779810f620 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7790009cd0 tx=0x7f779002a8f0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.590+0000 7f779dc00640 1 -- 172.21.15.1:0/241293100 shutdown_connections 2024-04-26T21:36:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.590+0000 7f779dc00640 1 --2- 172.21.15.1:0/241293100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798103110 0x7f779810f620 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.590+0000 7f779dc00640 1 --2- 172.21.15.1:0/241293100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7798102750 0x7f7798102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.590+0000 7f779dc00640 1 --2- 172.21.15.1:0/241293100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7798108760 0x7f7798108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.590+0000 7f779dc00640 1 -- 172.21.15.1:0/241293100 >> 172.21.15.1:0/241293100 conn(0x7f77980fe240 msgr2=0x7f7798100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:06.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.590+0000 7f779dc00640 1 -- 172.21.15.1:0/241293100 shutdown_connections 2024-04-26T21:36:06.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.590+0000 7f779dc00640 1 -- 172.21.15.1:0/241293100 wait complete. 2024-04-26T21:36:06.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.592+0000 7f779dc00640 1 Processor -- start 2024-04-26T21:36:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.593+0000 7f779dc00640 1 -- start start 2024-04-26T21:36:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.593+0000 7f779dc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7798102750 0x7f779819b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.593+0000 7f779dc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798103110 0x7f779819b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f779dc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7798108760 0x7f779819f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f779dc00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7798112a20 con 0x7f7798103110 2024-04-26T21:36:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f779dc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f77981128a0 con 0x7f7798108760 2024-04-26T21:36:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f779dc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7798112ba0 con 0x7f7798102750 2024-04-26T21:36:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f7797e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7798102750 0x7f779819b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f7797e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7798102750 0x7f779819b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44650/0 (socket says 172.21.15.1:44650) 2024-04-26T21:36:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f7797e00640 1 -- 172.21.15.1:0/2175154303 learned_addr learned my addr 172.21.15.1:0/2175154303 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f7797400640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798103110 0x7f779819b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f779d200640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7798108760 0x7f779819f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f7797400640 1 -- 172.21.15.1:0/2175154303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7798102750 msgr2=0x7f779819b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f7797400640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7798102750 0x7f779819b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f7797400640 1 -- 172.21.15.1:0/2175154303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7798108760 msgr2=0x7f779819f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f7797400640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7798108760 0x7f779819f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.594+0000 7f7797400640 1 -- 172.21.15.1:0/2175154303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77981a00f0 con 0x7f7798103110 2024-04-26T21:36:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.595+0000 7f7797e00640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7798102750 0x7f779819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:36:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.595+0000 7f779d200640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7798108760 0x7f779819f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:36:06.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.595+0000 7f7797400640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798103110 0x7f779819b640 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f778400e9f0 tx=0x7f778400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:06.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.595+0000 7f7794c00640 1 -- 172.21.15.1:0/2175154303 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778400cd80 con 0x7f7798103110 2024-04-26T21:36:06.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.595+0000 7f7794c00640 1 -- 172.21.15.1:0/2175154303 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77840035f0 con 0x7f7798103110 2024-04-26T21:36:06.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.595+0000 7f7794c00640 1 -- 172.21.15.1:0/2175154303 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784010670 con 0x7f7798103110 2024-04-26T21:36:06.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.595+0000 7f779dc00640 1 -- 172.21.15.1:0/2175154303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77981a03e0 con 0x7f7798103110 2024-04-26T21:36:06.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.595+0000 7f779dc00640 1 -- 172.21.15.1:0/2175154303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77981a7c70 con 0x7f7798103110 2024-04-26T21:36:06.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.600+0000 7f77a1fce740 1 Processor -- start 2024-04-26T21:36:06.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.600+0000 7f77a1fce740 1 -- start start 2024-04-26T21:36:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.601+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696bfe0 0x55f00696c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.601+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696c900 0x55f006976c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.601+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f0069795b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.601+0000 7f77a1fce740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55f0068d1380 con 0x55f0069771c0 2024-04-26T21:36:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.601+0000 7f77a1fce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f00686dc30 con 0x55f00696bfe0 2024-04-26T21:36:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.601+0000 7f77a1fce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f0068cd340 con 0x55f00696c900 2024-04-26T21:36:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.601+0000 7f779d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696bfe0 0x55f00696c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.601+0000 7f779d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696bfe0 0x55f00696c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:44078/0 (socket says 172.21.15.1:44078) 2024-04-26T21:36:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.601+0000 7f779d200640 1 -- 172.21.15.1:0/2680767291 learned_addr learned my addr 172.21.15.1:0/2680767291 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:06.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.601+0000 7f7797e00640 1 --2- 172.21.15.1:0/2680767291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696c900 0x55f006976c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.602+0000 7f7794c00640 1 -- 172.21.15.1:0/2175154303 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f77840037a0 con 0x7f7798103110 2024-04-26T21:36:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.602+0000 7f779d200640 1 -- 172.21.15.1:0/2680767291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696c900 msgr2=0x55f006976c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.602+0000 7f779d200640 1 --2- 172.21.15.1:0/2680767291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696c900 0x55f006976c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.602+0000 7f779d200640 1 -- 172.21.15.1:0/2680767291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 msgr2=0x55f0069795b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:36:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.602+0000 7f779d200640 1 --2- 172.21.15.1:0/2680767291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f0069795b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.602+0000 7f779d200640 1 -- 172.21.15.1:0/2680767291 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f00688f940 con 0x55f00696bfe0 2024-04-26T21:36:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.602+0000 7f779d200640 1 --2- 172.21.15.1:0/2680767291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696bfe0 0x55f00696c3c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7790035000 tx=0x7f7790030f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=538628334b5b55a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.603+0000 7f778d600640 1 -- 172.21.15.1:0/2680767291 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790030a00 con 0x55f00696bfe0 2024-04-26T21:36:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.603+0000 7f778d600640 1 -- 172.21.15.1:0/2680767291 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7790036030 con 0x55f00696bfe0 2024-04-26T21:36:06.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.603+0000 7f778d600640 1 -- 172.21.15.1:0/2680767291 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77900046e0 con 0x55f00696bfe0 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.603+0000 7f7794c00640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f777c0a3d00 0x7f777c0a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.603+0000 7f7794c00640 1 -- 172.21.15.1:0/2175154303 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f77840d0ee0 con 0x7f7798103110 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.604+0000 7f7797e00640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f777c0a3d00 0x7f777c0a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.604+0000 7f77a1fce740 1 -- 172.21.15.1:0/2680767291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696bfe0 msgr2=0x55f00696c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.604+0000 7f77a1fce740 1 --2- 172.21.15.1:0/2680767291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696bfe0 0x55f00696c3c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7790035000 tx=0x7f7790030f50 comp rx=0 tx=0).stop 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.604+0000 7f77a1fce740 1 -- 172.21.15.1:0/2680767291 shutdown_connections 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.604+0000 7f77a1fce740 1 --2- 172.21.15.1:0/2680767291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696c900 0x55f006976c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.604+0000 7f77a1fce740 1 --2- 172.21.15.1:0/2680767291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f0069795b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.604+0000 7f77a1fce740 1 --2- 172.21.15.1:0/2680767291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696bfe0 0x55f00696c3c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.604+0000 7f77a1fce740 1 -- 172.21.15.1:0/2680767291 >> 172.21.15.1:0/2680767291 conn(0x55f0069684f0 msgr2=0x55f0069688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.605+0000 7f77a1fce740 1 -- 172.21.15.1:0/2680767291 shutdown_connections 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.605+0000 7f7797e00640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f777c0a3d00 0x7f777c0a61c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55f00696d580 tx=0x7f778800b4a0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:06.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.605+0000 7f77a1fce740 1 -- 172.21.15.1:0/2680767291 wait complete. 2024-04-26T21:36:06.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.605+0000 7f77a1fce740 1 -- start start 2024-04-26T21:36:06.609 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.605+0000 7f77a1fce740 1 Processor -- start 2024-04-26T21:36:06.609 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f0069740f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.609 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f00696c900 0x55f006974630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0069771c0 0x55f006974b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f77a1fce740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55f006979d90 con 0x55f00696c900 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f77a1fce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f006858c60 con 0x55f0069771c0 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f77a1fce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f006979f10 con 0x55f00696bfe0 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f77a1fce740 10 client.0 _ll_register_callbacks cb 0x55f006884e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f7797e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f0069740f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f7797e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f0069740f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44666/0 (socket says 172.21.15.1:44666) 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f7797e00640 1 -- 172.21.15.1:0/241225698 learned_addr learned my addr 172.21.15.1:0/241225698 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.606+0000 7f7797400640 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f00696c900 0x55f006974630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.607+0000 7f779d200640 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0069771c0 0x55f006974b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.607+0000 7f7797e00640 1 -- 172.21.15.1:0/241225698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0069771c0 msgr2=0x55f006974b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.607+0000 7f7797e00640 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0069771c0 0x55f006974b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.607+0000 7f7797e00640 1 -- 172.21.15.1:0/241225698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f00696c900 msgr2=0x55f006974630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.607+0000 7f7797e00640 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f00696c900 0x55f006974630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.607+0000 7f7797e00640 1 -- 172.21.15.1:0/241225698 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f006867510 con 0x55f00696bfe0 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.607+0000 7f7797400640 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f00696c900 0x55f006974630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.607+0000 7f7797e00640 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f0069740f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f77880042f0 tx=0x7f7788004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c68842291d4176fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.607+0000 7f7782000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.608+0000 7f7782000640 1 -- 172.21.15.1:0/241225698 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77880044a0 con 0x55f00696bfe0 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.608+0000 7f77a1fce740 1 -- 172.21.15.1:0/241225698 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7790036030 con 0x55f00696bfe0 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.608+0000 7f7782000640 1 -- 172.21.15.1:0/241225698 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f7788003d80 con 0x55f00696bfe0 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.608+0000 7f7782000640 1 -- 172.21.15.1:0/241225698 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7788031050 con 0x55f00696bfe0 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.608+0000 7f77a1fce740 10 client.25957 fetch_fsmap learned FSMap version 36 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.608+0000 7f77a1fce740 1 -- 172.21.15.1:0/241225698 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55f0065e3210 con 0x55f00696bfe0 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.608+0000 7f7782000640 1 -- 172.21.15.1:0/241225698 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7790036030 con 0x55f00696bfe0 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.608+0000 7f7782000640 1 -- 172.21.15.1:0/241225698 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f7788029780 con 0x55f00696bfe0 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.609+0000 7f77a1fce740 10 client.25957 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:36:06.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.609+0000 7f77a1fce740 10 client.25957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:36:06.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.609+0000 7f77a1fce740 20 client.25957 populate_metadata read hostname 'smithi001' 2024-04-26T21:36:06.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.609+0000 7f77a1fce740 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f006986f00 0x55f006989310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.609+0000 7f77a1fce740 4 client.25957 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:36:06.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.609+0000 7f77a1fce740 1 -- 172.21.15.1:0/241225698 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0068def90 con 0x55f006986f00 2024-04-26T21:36:06.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.609+0000 7f7797400640 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f006986f00 0x55f006989310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.610+0000 7f7797400640 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f006986f00 0x55f006989310 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=268e8703144a3ab7 server_cookie=da9cc206689cda1c in_seq=0 out_seq=0 2024-04-26T21:36:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.610+0000 7f7782000640 10 client.25957 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.610+0000 7f7782000640 1 -- 172.21.15.1:0/241225698 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f0068def90 con 0x55f006986f00 2024-04-26T21:36:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.610+0000 7f7782000640 10 client.25957 handle_command_reply: tid=0 2024-04-26T21:36:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.610+0000 7f77a1fce740 1 client.25957 shutdown 2024-04-26T21:36:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 1 -- 172.21.15.1:0/241225698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 msgr2=0x55f0069740f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f0069740f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f77880042f0 tx=0x7f7788004730 comp rx=0 tx=0).stop 2024-04-26T21:36:06.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 1 -- 172.21.15.1:0/241225698 shutdown_connections 2024-04-26T21:36:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f006986f00 0x55f006989310 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0069771c0 0x55f006974b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f00696c900 0x55f006974630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f7782000640 0 client.25957 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 1 --2- 172.21.15.1:0/241225698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f0069740f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 1 -- 172.21.15.1:0/241225698 >> 172.21.15.1:0/241225698 conn(0x55f0069684f0 msgr2=0x55f00696e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 1 -- 172.21.15.1:0/241225698 shutdown_connections 2024-04-26T21:36:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 1 -- 172.21.15.1:0/241225698 wait complete. 2024-04-26T21:36:06.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.611+0000 7f77a1fce740 20 client.25957 trim_cache size 0 max 16384 2024-04-26T21:36:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.630+0000 7f77a1fce740 1 Processor -- start 2024-04-26T21:36:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f77a1fce740 1 -- start start 2024-04-26T21:36:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f006982c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 0x55f0069739d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f00696e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f77a1fce740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f77981b4710 con 0x55f0069771c0 2024-04-26T21:36:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f77a1fce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f77981b1fe0 con 0x55f00696c900 2024-04-26T21:36:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f77a1fce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f77981afe30 con 0x55f00696bfe0 2024-04-26T21:36:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f779d200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f00696e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f779d200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f00696e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51094/0 (socket says 172.21.15.1:51094) 2024-04-26T21:36:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f779d200640 1 -- 172.21.15.1:0/984758454 learned_addr learned my addr 172.21.15.1:0/984758454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f7797400640 1 --2- 172.21.15.1:0/984758454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 0x55f0069739d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.631+0000 7f7797e00640 1 --2- 172.21.15.1:0/984758454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f006982c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.632+0000 7f7797400640 1 -- 172.21.15.1:0/984758454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 msgr2=0x55f006982c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.632+0000 7f7797400640 1 --2- 172.21.15.1:0/984758454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f006982c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.632+0000 7f7797400640 1 -- 172.21.15.1:0/984758454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 msgr2=0x55f00696e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.632+0000 7f7797400640 1 --2- 172.21.15.1:0/984758454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f00696e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.632+0000 7f7797400640 1 -- 172.21.15.1:0/984758454 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0066af620 con 0x55f00696c900 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.632+0000 7f7797400640 1 --2- 172.21.15.1:0/984758454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 0x55f0069739d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f77840ce1c0 tx=0x7f7784098590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ce9daab4680c073 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.632+0000 7f778cc00640 1 -- 172.21.15.1:0/984758454 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77840b5020 con 0x55f00696c900 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.632+0000 7f778cc00640 1 -- 172.21.15.1:0/984758454 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f778409ca10 con 0x55f00696c900 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.632+0000 7f778cc00640 1 -- 172.21.15.1:0/984758454 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784099080 con 0x55f00696c900 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.633+0000 7f77a1fce740 1 -- 172.21.15.1:0/984758454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 msgr2=0x55f0069739d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.633+0000 7f77a1fce740 1 --2- 172.21.15.1:0/984758454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 0x55f0069739d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f77840ce1c0 tx=0x7f7784098590 comp rx=0 tx=0).stop 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.633+0000 7f77a1fce740 1 -- 172.21.15.1:0/984758454 shutdown_connections 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.633+0000 7f77a1fce740 1 --2- 172.21.15.1:0/984758454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f00696e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.633+0000 7f77a1fce740 1 --2- 172.21.15.1:0/984758454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 0x55f0069739d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.633+0000 7f77a1fce740 1 --2- 172.21.15.1:0/984758454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f006982c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.633+0000 7f77a1fce740 1 -- 172.21.15.1:0/984758454 >> 172.21.15.1:0/984758454 conn(0x55f0069684f0 msgr2=0x55f006966dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.633+0000 7f77a1fce740 1 -- 172.21.15.1:0/984758454 shutdown_connections 2024-04-26T21:36:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.633+0000 7f77a1fce740 1 -- 172.21.15.1:0/984758454 wait complete. 2024-04-26T21:36:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.634+0000 7f77a1fce740 1 -- start start 2024-04-26T21:36:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.634+0000 7f77a1fce740 1 Processor -- start 2024-04-26T21:36:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f00697a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 0x55f00697a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f77a1fce740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f0069e54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f77a1fce740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f77981b04a0 con 0x55f0069771c0 2024-04-26T21:36:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f77a1fce740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f77981b1910 con 0x55f00696c900 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f77a1fce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f77981b0b70 con 0x55f00696bfe0 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f77a1fce740 10 client.0 _ll_register_callbacks cb 0x7f7788005a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f7797e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f00697a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f7797e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f00697a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:44674/0 (socket says 172.21.15.1:44674) 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f7797e00640 1 -- 172.21.15.1:0/1256005154 learned_addr learned my addr 172.21.15.1:0/1256005154 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f779d200640 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f0069e54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f7797400640 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 0x55f00697a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f7797e00640 1 -- 172.21.15.1:0/1256005154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 msgr2=0x55f00697a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f7797e00640 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 0x55f00697a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f7797e00640 1 -- 172.21.15.1:0/1256005154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 msgr2=0x55f0069e54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f7797e00640 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f0069e54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.635+0000 7f7797e00640 1 -- 172.21.15.1:0/1256005154 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f00688bcb0 con 0x55f00696bfe0 2024-04-26T21:36:06.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.636+0000 7f7797e00640 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f00697a430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7788029c20 tx=0x7f7788003ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5249d0bc8f2826fe server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.636+0000 7f7782000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.636+0000 7f7782000640 1 -- 172.21.15.1:0/1256005154 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77880044a0 con 0x55f00696bfe0 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.636+0000 7f77a1fce740 1 -- 172.21.15.1:0/1256005154 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f778409ca10 con 0x55f00696bfe0 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.636+0000 7f7782000640 1 -- 172.21.15.1:0/1256005154 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f7788029420 con 0x55f00696bfe0 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f7782000640 1 -- 172.21.15.1:0/1256005154 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7788047050 con 0x55f00696bfe0 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f77a1fce740 10 client.25963 fetch_fsmap learned FSMap version 36 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f77a1fce740 1 -- 172.21.15.1:0/1256005154 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55f0069eccc0 con 0x55f00696bfe0 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f7782000640 1 -- 172.21.15.1:0/1256005154 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f778409ca10 con 0x55f00696bfe0 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f7782000640 1 -- 172.21.15.1:0/1256005154 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f77880109f0 con 0x55f00696bfe0 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f77a1fce740 10 client.25963 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f77a1fce740 10 client.25963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f77a1fce740 20 client.25963 populate_metadata read hostname 'smithi001' 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f77a1fce740 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0069ed3b0 0x55f0069ef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f77a1fce740 4 client.25963 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:36:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.637+0000 7f77a1fce740 1 -- 172.21.15.1:0/1256005154 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f0068d8e00 con 0x55f0069ed3b0 2024-04-26T21:36:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.638+0000 7f7797400640 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0069ed3b0 0x55f0069ef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.638+0000 7f7797400640 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0069ed3b0 0x55f0069ef7d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71ebe882a62dba52 server_cookie=8ca7fa979dcc521d in_seq=0 out_seq=0 2024-04-26T21:36:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.638+0000 7f7782000640 10 client.25963 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f7782000640 1 -- 172.21.15.1:0/1256005154 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f0068d8e00 con 0x55f0069ed3b0 2024-04-26T21:36:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f7782000640 10 client.25963 handle_command_reply: tid=0 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f77a1fce740 1 client.25963 shutdown 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f77a1fce740 1 -- 172.21.15.1:0/1256005154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 msgr2=0x55f00697a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f77a1fce740 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f00697a430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7788029c20 tx=0x7f7788003ad0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f77a1fce740 1 -- 172.21.15.1:0/1256005154 shutdown_connections 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f77a1fce740 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0069ed3b0 0x55f0069ef7d0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f77a1fce740 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0069771c0 0x55f0069e54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f7782000640 0 client.25963 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f77a1fce740 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f00696c900 0x55f00697a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f77a1fce740 1 --2- 172.21.15.1:0/1256005154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f00696bfe0 0x55f00697a430 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.699+0000 7f77a1fce740 1 -- 172.21.15.1:0/1256005154 >> 172.21.15.1:0/1256005154 conn(0x55f0069684f0 msgr2=0x55f006974090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.700+0000 7f77a1fce740 1 -- 172.21.15.1:0/1256005154 shutdown_connections 2024-04-26T21:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.700+0000 7f77a1fce740 1 -- 172.21.15.1:0/1256005154 wait complete. 2024-04-26T21:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.700+0000 7f77a1fce740 20 client.25963 trim_cache size 0 max 16384 2024-04-26T21:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:36:06.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:36:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 -- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f777c0a3d00 msgr2=0x7f777c0a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f777c0a3d00 0x7f777c0a61c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55f00696d580 tx=0x7f778800b4a0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 -- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798103110 msgr2=0x7f779819b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798103110 0x7f779819b640 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f778400e9f0 tx=0x7f778400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 -- 172.21.15.1:0/2175154303 shutdown_connections 2024-04-26T21:36:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f777c0a3d00 0x7f777c0a61c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.703 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7798108760 0x7f779819f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7798103110 0x7f779819b640 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 --2- 172.21.15.1:0/2175154303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7798102750 0x7f779819b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 -- 172.21.15.1:0/2175154303 >> 172.21.15.1:0/2175154303 conn(0x7f77980fe240 msgr2=0x7f779810dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 -- 172.21.15.1:0/2175154303 shutdown_connections 2024-04-26T21:36:06.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:06.701+0000 7f779dc00640 1 -- 172.21.15.1:0/2175154303 wait complete. 2024-04-26T21:36:07.558 INFO:tasks.workunit.client.0.smithi001.stdout: 1 163867 4.22 MB/sec execute 125 sec latency 51.042 ms 2024-04-26T21:36:08.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:08 smithi186 ceph-mon[32991]: pgmap v177: 129 pgs: 129 active+clean; 100 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T21:36:08.558 INFO:tasks.workunit.client.0.smithi001.stdout: 1 164823 4.23 MB/sec execute 126 sec latency 20.603 ms 2024-04-26T21:36:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:08 smithi001 ceph-mon[28054]: pgmap v177: 129 pgs: 129 active+clean; 100 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T21:36:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:08 smithi038 ceph-mon[32724]: pgmap v177: 129 pgs: 129 active+clean; 100 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T21:36:09.558 INFO:tasks.workunit.client.0.smithi001.stdout: 1 166077 4.25 MB/sec execute 127 sec latency 53.610 ms 2024-04-26T21:36:10.370 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:10 smithi001 ceph-mon[28054]: pgmap v178: 129 pgs: 129 active+clean; 100 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T21:36:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:10 smithi186 ceph-mon[32991]: pgmap v178: 129 pgs: 129 active+clean; 100 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T21:36:10.558 INFO:tasks.workunit.client.0.smithi001.stdout: 1 166979 4.27 MB/sec execute 128 sec latency 36.948 ms 2024-04-26T21:36:10.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:10 smithi038 ceph-mon[32724]: pgmap v178: 129 pgs: 129 active+clean; 100 MiB data, 1021 MiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.4 MiB/s wr, 147 op/s 2024-04-26T21:36:11.558 INFO:tasks.workunit.client.0.smithi001.stdout: 1 167235 4.24 MB/sec execute 129 sec latency 25.389 ms 2024-04-26T21:36:12.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:12 smithi186 ceph-mon[32991]: pgmap v179: 129 pgs: 129 active+clean; 102 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 2.0 MiB/s wr, 232 op/s 2024-04-26T21:36:12.558 INFO:tasks.workunit.client.0.smithi001.stdout: 1 167915 4.22 MB/sec execute 130 sec latency 19.568 ms 2024-04-26T21:36:12.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:12 smithi001 ceph-mon[28054]: pgmap v179: 129 pgs: 129 active+clean; 102 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 2.0 MiB/s wr, 232 op/s 2024-04-26T21:36:12.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:12 smithi038 ceph-mon[32724]: pgmap v179: 129 pgs: 129 active+clean; 102 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 2.0 MiB/s wr, 232 op/s 2024-04-26T21:36:13.559 INFO:tasks.workunit.client.0.smithi001.stdout: 1 169218 4.23 MB/sec execute 131 sec latency 24.675 ms 2024-04-26T21:36:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:14 smithi186 ceph-mon[32991]: pgmap v180: 129 pgs: 129 active+clean; 102 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-04-26T21:36:14.559 INFO:tasks.workunit.client.0.smithi001.stdout: 1 170282 4.27 MB/sec execute 132 sec latency 55.556 ms 2024-04-26T21:36:14.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:14 smithi001 ceph-mon[28054]: pgmap v180: 129 pgs: 129 active+clean; 102 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-04-26T21:36:14.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:14 smithi038 ceph-mon[32724]: pgmap v180: 129 pgs: 129 active+clean; 102 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-04-26T21:36:15.559 INFO:tasks.workunit.client.0.smithi001.stdout: 1 170708 4.25 MB/sec execute 133 sec latency 65.823 ms 2024-04-26T21:36:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:16 smithi186 ceph-mon[32991]: pgmap v181: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-04-26T21:36:16.559 INFO:tasks.workunit.client.0.smithi001.stdout: 1 171415 4.24 MB/sec execute 134 sec latency 30.692 ms 2024-04-26T21:36:16.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:16 smithi001 ceph-mon[28054]: pgmap v181: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-04-26T21:36:16.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:16 smithi038 ceph-mon[32724]: pgmap v181: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-04-26T21:36:17.559 INFO:tasks.workunit.client.0.smithi001.stdout: 1 172764 4.25 MB/sec execute 135 sec latency 24.833 ms 2024-04-26T21:36:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:18 smithi186 ceph-mon[32991]: pgmap v182: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:36:18.559 INFO:tasks.workunit.client.0.smithi001.stdout: 1 173965 4.29 MB/sec execute 136 sec latency 22.640 ms 2024-04-26T21:36:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:18 smithi001 ceph-mon[28054]: pgmap v182: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:36:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:18 smithi038 ceph-mon[32724]: pgmap v182: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:36:19.563 INFO:tasks.workunit.client.0.smithi001.stdout: 1 174239 4.27 MB/sec execute 137 sec latency 26.102 ms 2024-04-26T21:36:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.853+0000 7f98be800640 1 -- 172.21.15.1:0/1332793312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98b81137d0 msgr2=0x7f98b8115bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.853+0000 7f98be800640 1 --2- 172.21.15.1:0/1332793312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98b81137d0 0x7f98b8115bc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f98b0009cd0 tx=0x7f98b002a8e0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.853+0000 7f98be800640 1 -- 172.21.15.1:0/1332793312 shutdown_connections 2024-04-26T21:36:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.853+0000 7f98be800640 1 --2- 172.21.15.1:0/1332793312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b8077780 0x7f98b8113290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.853+0000 7f98be800640 1 --2- 172.21.15.1:0/1332793312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98b81137d0 0x7f98b8115bc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.853+0000 7f98be800640 1 --2- 172.21.15.1:0/1332793312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98b8076e60 0x7f98b8077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.853+0000 7f98be800640 1 -- 172.21.15.1:0/1332793312 >> 172.21.15.1:0/1332793312 conn(0x7f98b8100600 msgr2=0x7f98b8102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.854+0000 7f98be800640 1 -- 172.21.15.1:0/1332793312 shutdown_connections 2024-04-26T21:36:19.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.854+0000 7f98be800640 1 -- 172.21.15.1:0/1332793312 wait complete. 2024-04-26T21:36:19.856 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.855+0000 7f98be800640 1 Processor -- start 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.855+0000 7f98be800640 1 -- start start 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.855+0000 7f98be800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98b8076e60 0x7f98b819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.855+0000 7f98be800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98b8077780 0x7f98b819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.855+0000 7f98be800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b81137d0 0x7f98b81a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.855+0000 7f98be800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f98b81191f0 con 0x7f98b8076e60 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.855+0000 7f98be800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8119070 con 0x7f98b8077780 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.855+0000 7f98be800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8119370 con 0x7f98b81137d0 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.855+0000 7f98b7400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98b8076e60 0x7f98b819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.855+0000 7f98b6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98b8077780 0x7f98b819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.856+0000 7f98b6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98b8077780 0x7f98b819fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43328/0 (socket says 172.21.15.1:43328) 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.856+0000 7f98b6a00640 1 -- 172.21.15.1:0/1334632959 learned_addr learned my addr 172.21.15.1:0/1334632959 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.856+0000 7f98b7e00640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b81137d0 0x7f98b81a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.856+0000 7f98b6a00640 1 -- 172.21.15.1:0/1334632959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b81137d0 msgr2=0x7f98b81a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.856+0000 7f98b6a00640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b81137d0 0x7f98b81a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.857 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.856+0000 7f98b6a00640 1 -- 172.21.15.1:0/1334632959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98b8076e60 msgr2=0x7f98b819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:19.858 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.856+0000 7f98b6a00640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98b8076e60 0x7f98b819f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.858 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.856+0000 7f98b6a00640 1 -- 172.21.15.1:0/1334632959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b81a4750 con 0x7f98b8077780 2024-04-26T21:36:19.858 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.856+0000 7f98b6a00640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98b8077780 0x7f98b819fcf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f98a400e9f0 tx=0x7f98a400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:19.858 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.857+0000 7f98afe00640 1 -- 172.21.15.1:0/1334632959 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a400cd80 con 0x7f98b8077780 2024-04-26T21:36:19.858 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.857+0000 7f98be800640 1 -- 172.21.15.1:0/1334632959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b81a4a40 con 0x7f98b8077780 2024-04-26T21:36:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.857+0000 7f98afe00640 1 -- 172.21.15.1:0/1334632959 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a40108e0 con 0x7f98b8077780 2024-04-26T21:36:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.857+0000 7f98afe00640 1 -- 172.21.15.1:0/1334632959 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a4018860 con 0x7f98b8077780 2024-04-26T21:36:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.857+0000 7f98be800640 1 -- 172.21.15.1:0/1334632959 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b81ac2d0 con 0x7f98b8077780 2024-04-26T21:36:19.863 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.862+0000 7f98c17b7740 1 Processor -- start 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.862+0000 7f98c17b7740 1 -- start start 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.862+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f68122dd0 0x564f681231b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.862+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.862+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f681303a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.862+0000 7f98c17b7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564f68088380 con 0x564f6812dfb0 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.862+0000 7f98c17b7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564f68024c30 con 0x564f681236f0 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.862+0000 7f98c17b7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564f68084340 con 0x564f68122dd0 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f68122dd0 0x564f681231b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43336/0 (socket says 172.21.15.1:43336) 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7400640 1 -- 172.21.15.1:0/4259931882 learned_addr learned my addr 172.21.15.1:0/4259931882 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98afe00640 1 -- 172.21.15.1:0/1334632959 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f98a4010430 con 0x7f98b8077780 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7400640 1 -- 172.21.15.1:0/4259931882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f68122dd0 msgr2=0x564f681231b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:19.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7400640 1 --2- 172.21.15.1:0/4259931882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f68122dd0 0x564f681231b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7400640 1 -- 172.21.15.1:0/4259931882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 msgr2=0x564f681303a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b6a00640 1 --2- 172.21.15.1:0/4259931882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f681303a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7400640 1 --2- 172.21.15.1:0/4259931882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f681303a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7400640 1 -- 172.21.15.1:0/4259931882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f68046940 con 0x564f681236f0 2024-04-26T21:36:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7e00640 1 --2- 172.21.15.1:0/4259931882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f68122dd0 0x564f681231b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:36:19.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b6a00640 1 --2- 172.21.15.1:0/4259931882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f681303a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:36:19.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.863+0000 7f98b7400640 1 --2- 172.21.15.1:0/4259931882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812da70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009cf0 tx=0x7f98a8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e51bce97f4c53d15 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:19.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.864+0000 7f98acc00640 1 -- 172.21.15.1:0/4259931882 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a8004230 con 0x564f681236f0 2024-04-26T21:36:19.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.864+0000 7f98acc00640 1 -- 172.21.15.1:0/4259931882 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a80043d0 con 0x564f681236f0 2024-04-26T21:36:19.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.864+0000 7f98acc00640 1 -- 172.21.15.1:0/4259931882 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a80197e0 con 0x564f681236f0 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98afe00640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f989c0a3d90 0x7f989c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98c17b7740 1 -- 172.21.15.1:0/4259931882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 msgr2=0x564f6812da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98c17b7740 1 --2- 172.21.15.1:0/4259931882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812da70 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009cf0 tx=0x7f98a8011810 comp rx=0 tx=0).stop 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98afe00640 1 -- 172.21.15.1:0/1334632959 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f98a40d0de0 con 0x7f98b8077780 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98c17b7740 1 -- 172.21.15.1:0/4259931882 shutdown_connections 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98c17b7740 1 --2- 172.21.15.1:0/4259931882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f681303a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98c17b7740 1 --2- 172.21.15.1:0/4259931882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812da70 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98c17b7740 1 --2- 172.21.15.1:0/4259931882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f68122dd0 0x564f681231b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98c17b7740 1 -- 172.21.15.1:0/4259931882 >> 172.21.15.1:0/4259931882 conn(0x564f6811f2e0 msgr2=0x564f6811f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98b7e00640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f989c0a3d90 0x7f989c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98c17b7740 1 -- 172.21.15.1:0/4259931882 shutdown_connections 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.865+0000 7f98c17b7740 1 -- 172.21.15.1:0/4259931882 wait complete. 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.866+0000 7f98c17b7740 1 -- start start 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.866+0000 7f98b7e00640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f989c0a3d90 0x7f989c0a6250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f98b0005e30 tx=0x7f98b0005d80 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:19.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.866+0000 7f98c17b7740 1 Processor -- start 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f6812ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 0x564f6812b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f6812b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98c17b7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564f68130be0 con 0x564f6812dfb0 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98c17b7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564f6800fc60 con 0x564f68122dd0 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98c17b7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564f68130d60 con 0x564f681236f0 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98c17b7740 10 client.0 _ll_register_callbacks cb 0x564f6803be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f6812ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 0x564f6812b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f6812ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43340/0 (socket says 172.21.15.1:43340) 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b7400640 1 -- 172.21.15.1:0/2063062429 learned_addr learned my addr 172.21.15.1:0/2063062429 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:19.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b7e00640 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f6812b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b7400640 1 -- 172.21.15.1:0/2063062429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 msgr2=0x564f6812b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b7400640 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 0x564f6812b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b7400640 1 -- 172.21.15.1:0/2063062429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 msgr2=0x564f6812b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b7400640 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f6812b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b7400640 1 -- 172.21.15.1:0/2063062429 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f6801e510 con 0x564f68122dd0 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.867+0000 7f98b6a00640 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 0x564f6812b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.868+0000 7f98b7400640 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f6812ae90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009340 tx=0x7f98a8003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f86e42243bd541e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.868+0000 7f98a1600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.868+0000 7f98a1600640 1 -- 172.21.15.1:0/2063062429 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a8027070 con 0x564f68122dd0 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.868+0000 7f98a1600640 1 -- 172.21.15.1:0/2063062429 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f98a8005240 con 0x564f68122dd0 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.868+0000 7f98c17b7740 1 -- 172.21.15.1:0/2063062429 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98a80043d0 con 0x564f68122dd0 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.868+0000 7f98a1600640 1 -- 172.21.15.1:0/2063062429 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f98a8033050 con 0x564f68122dd0 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.868+0000 7f98c17b7740 10 client.25994 fetch_fsmap learned FSMap version 36 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.868+0000 7f98c17b7740 1 -- 172.21.15.1:0/2063062429 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564f67d9a210 con 0x564f68122dd0 2024-04-26T21:36:19.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.869+0000 7f98a1600640 1 -- 172.21.15.1:0/2063062429 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a80043d0 con 0x564f68122dd0 2024-04-26T21:36:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.869+0000 7f98a1600640 1 -- 172.21.15.1:0/2063062429 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f98a8004b70 con 0x564f68122dd0 2024-04-26T21:36:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.869+0000 7f98c17b7740 10 client.25994 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:36:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.869+0000 7f98c17b7740 10 client.25994 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:36:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.869+0000 7f98c17b7740 20 client.25994 populate_metadata read hostname 'smithi001' 2024-04-26T21:36:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.869+0000 7f98c17b7740 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564f6813db10 0x564f6813ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.869+0000 7f98c17b7740 4 client.25994 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:36:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.869+0000 7f98c17b7740 1 -- 172.21.15.1:0/2063062429 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f68095f90 con 0x564f6813db10 2024-04-26T21:36:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.869+0000 7f98b6a00640 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564f6813db10 0x564f6813ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.870+0000 7f98b6a00640 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564f6813db10 0x564f6813ff00 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abe5e4fe54a52a95 server_cookie=300ceecc81f06133 in_seq=0 out_seq=0 2024-04-26T21:36:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.870+0000 7f98a1600640 10 client.25994 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:19.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.872+0000 7f98a1600640 1 -- 172.21.15.1:0/2063062429 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f68095f90 con 0x564f6813db10 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.872+0000 7f98a1600640 10 client.25994 handle_command_reply: tid=0 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.872+0000 7f98c17b7740 1 client.25994 shutdown 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.872+0000 7f98c17b7740 1 -- 172.21.15.1:0/2063062429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 msgr2=0x564f6812ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.872+0000 7f98c17b7740 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f6812ae90 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009340 tx=0x7f98a8003820 comp rx=0 tx=0).stop 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.872+0000 7f98c17b7740 1 -- 172.21.15.1:0/2063062429 shutdown_connections 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.872+0000 7f98c17b7740 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564f6813db10 0x564f6813ff00 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.872+0000 7f98c17b7740 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 0x564f6812b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.873+0000 7f98c17b7740 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f6812b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.873+0000 7f98c17b7740 1 --2- 172.21.15.1:0/2063062429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f6812ae90 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.873+0000 7f98c17b7740 1 -- 172.21.15.1:0/2063062429 >> 172.21.15.1:0/2063062429 conn(0x564f6811f2e0 msgr2=0x564f68125010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.873+0000 7f98a1600640 0 client.25994 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.873+0000 7f98c17b7740 1 -- 172.21.15.1:0/2063062429 shutdown_connections 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.873+0000 7f98c17b7740 1 -- 172.21.15.1:0/2063062429 wait complete. 2024-04-26T21:36:19.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.873+0000 7f98c17b7740 20 client.25994 trim_cache size 0 max 16384 2024-04-26T21:36:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.891+0000 7f98c17b7740 1 Processor -- start 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.891+0000 7f98c17b7740 1 -- start start 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.891+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f68122dd0 0x564f68139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.891+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.891+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f6812dfb0 0x564f6811e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.891+0000 7f98c17b7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f98b81b8ea0 con 0x564f68122dd0 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.891+0000 7f98c17b7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98b81b6770 con 0x564f681236f0 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.891+0000 7f98c17b7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98b81b45c0 con 0x564f6812dfb0 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.892+0000 7f98b6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.892+0000 7f98b6a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43344/0 (socket says 172.21.15.1:43344) 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.892+0000 7f98b6a00640 1 -- 172.21.15.1:0/2231091700 learned_addr learned my addr 172.21.15.1:0/2231091700 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.892+0000 7f98b7400640 1 --2- 172.21.15.1:0/2231091700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f68122dd0 0x564f68139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.892+0000 7f98b7e00640 1 --2- 172.21.15.1:0/2231091700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f6812dfb0 0x564f6811e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.892+0000 7f98b6a00640 1 -- 172.21.15.1:0/2231091700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f6812dfb0 msgr2=0x564f6811e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.892+0000 7f98b6a00640 1 --2- 172.21.15.1:0/2231091700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f6812dfb0 0x564f6811e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.892+0000 7f98b6a00640 1 -- 172.21.15.1:0/2231091700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f68122dd0 msgr2=0x564f68139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.892+0000 7f98b6a00640 1 --2- 172.21.15.1:0/2231091700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f68122dd0 0x564f68139aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.892+0000 7f98b6a00640 1 -- 172.21.15.1:0/2231091700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f67e66620 con 0x564f681236f0 2024-04-26T21:36:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.893+0000 7f98b6a00640 1 --2- 172.21.15.1:0/2231091700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812a790 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f98a400cfd0 tx=0x7f98a40d32f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a14db8c166273a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:19.897 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.893+0000 7f98a3e00640 1 -- 172.21.15.1:0/2231091700 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a40a55e0 con 0x564f681236f0 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.893+0000 7f98a3e00640 1 -- 172.21.15.1:0/2231091700 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a409ad20 con 0x564f681236f0 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.893+0000 7f98a3e00640 1 -- 172.21.15.1:0/2231091700 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a409c0d0 con 0x564f681236f0 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 -- 172.21.15.1:0/2231091700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 msgr2=0x564f6812a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 --2- 172.21.15.1:0/2231091700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812a790 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f98a400cfd0 tx=0x7f98a40d32f0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 -- 172.21.15.1:0/2231091700 shutdown_connections 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 --2- 172.21.15.1:0/2231091700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f6812dfb0 0x564f6811e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 --2- 172.21.15.1:0/2231091700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f681236f0 0x564f6812a790 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 --2- 172.21.15.1:0/2231091700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f68122dd0 0x564f68139aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 -- 172.21.15.1:0/2231091700 >> 172.21.15.1:0/2231091700 conn(0x564f6811f2e0 msgr2=0x564f6811dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 -- 172.21.15.1:0/2231091700 shutdown_connections 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 -- 172.21.15.1:0/2231091700 wait complete. 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 -- start start 2024-04-26T21:36:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.894+0000 7f98c17b7740 1 Processor -- start 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f68136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 0x564f68127720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98c17b7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f68127c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98c17b7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f98b81b4c30 con 0x564f6812dfb0 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98c17b7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f98b81b60a0 con 0x564f68122dd0 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98c17b7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f98b81b5300 con 0x564f681236f0 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98c17b7740 10 client.0 _ll_register_callbacks cb 0x7f98b0031f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f68136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98b7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f68136210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43360/0 (socket says 172.21.15.1:43360) 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98b7400640 1 -- 172.21.15.1:0/1102745473 learned_addr learned my addr 172.21.15.1:0/1102745473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98b6a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 0x564f68127720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.895+0000 7f98b7e00640 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f68127c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98b7400640 1 -- 172.21.15.1:0/1102745473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 msgr2=0x564f68127720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98b7400640 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 0x564f68127720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98b7400640 1 -- 172.21.15.1:0/1102745473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 msgr2=0x564f68127c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98b7400640 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f68127c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98b7400640 1 -- 172.21.15.1:0/1102745473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f68042cb0 con 0x564f68122dd0 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98b7e00640 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f68127c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98b7400640 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f68136210 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009310 tx=0x7f98a8005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea80c16b2c356c9b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98a1600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98a1600640 1 -- 172.21.15.1:0/1102745473 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a803c060 con 0x564f68122dd0 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98c17b7740 1 -- 172.21.15.1:0/1102745473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98a409ad20 con 0x564f68122dd0 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98a1600640 1 -- 172.21.15.1:0/1102745473 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f98a8005410 con 0x564f68122dd0 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98a1600640 1 -- 172.21.15.1:0/1102745473 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f98a8043050 con 0x564f68122dd0 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98c17b7740 10 client.26000 fetch_fsmap learned FSMap version 36 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.896+0000 7f98c17b7740 1 -- 172.21.15.1:0/1102745473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564f681a3bc0 con 0x564f68122dd0 2024-04-26T21:36:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.897+0000 7f98a1600640 1 -- 172.21.15.1:0/1102745473 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98a409ad20 con 0x564f68122dd0 2024-04-26T21:36:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.897+0000 7f98a1600640 1 -- 172.21.15.1:0/1102745473 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f98a80043d0 con 0x564f68122dd0 2024-04-26T21:36:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.897+0000 7f98c17b7740 10 client.26000 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:36:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.897+0000 7f98c17b7740 10 client.26000 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:36:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.897+0000 7f98c17b7740 20 client.26000 populate_metadata read hostname 'smithi001' 2024-04-26T21:36:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.897+0000 7f98c17b7740 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564f681a4280 0x564f681a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.897+0000 7f98c17b7740 4 client.26000 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:36:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.897+0000 7f98c17b7740 1 -- 172.21.15.1:0/1102745473 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564f6808fe00 con 0x564f681a4280 2024-04-26T21:36:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.897+0000 7f98b6a00640 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564f681a4280 0x564f681a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.898+0000 7f98b6a00640 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564f681a4280 0x564f681a66a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36847b7d0d7d645a server_cookie=d49da4a5a0d5b2eb in_seq=0 out_seq=0 2024-04-26T21:36:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:19.898+0000 7f98a1600640 10 client.26000 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98a1600640 1 -- 172.21.15.1:0/1102745473 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564f6808fe00 con 0x564f681a4280 2024-04-26T21:36:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98a1600640 10 client.26000 handle_command_reply: tid=0 2024-04-26T21:36:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98c17b7740 1 client.26000 shutdown 2024-04-26T21:36:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98c17b7740 1 -- 172.21.15.1:0/1102745473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 msgr2=0x564f68136210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98c17b7740 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f68136210 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009310 tx=0x7f98a8005b20 comp rx=0 tx=0).stop 2024-04-26T21:36:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98c17b7740 1 -- 172.21.15.1:0/1102745473 shutdown_connections 2024-04-26T21:36:20.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98c17b7740 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564f681a4280 0x564f681a66a0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98c17b7740 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f681236f0 0x564f68127720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98c17b7740 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564f6812dfb0 0x564f68127c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98c17b7740 1 --2- 172.21.15.1:0/1102745473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f68122dd0 0x564f68136210 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98a1600640 0 client.26000 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98c17b7740 1 -- 172.21.15.1:0/1102745473 >> 172.21.15.1:0/1102745473 conn(0x564f6811f2e0 msgr2=0x564f6811e440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.039+0000 7f98c17b7740 1 -- 172.21.15.1:0/1102745473 shutdown_connections 2024-04-26T21:36:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.040+0000 7f98c17b7740 1 -- 172.21.15.1:0/1102745473 wait complete. 2024-04-26T21:36:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.040+0000 7f98c17b7740 20 client.26000 trim_cache size 0 max 16384 2024-04-26T21:36:20.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:36:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:36:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:36:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:36:20.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 -- 172.21.15.1:0/1334632959 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f989c0a3d90 msgr2=0x7f989c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f989c0a3d90 0x7f989c0a6250 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f98b0005e30 tx=0x7f98b0005d80 comp rx=0 tx=0).stop 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 -- 172.21.15.1:0/1334632959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98b8077780 msgr2=0x7f98b819fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98b8077780 0x7f98b819fcf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f98a400e9f0 tx=0x7f98a400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 -- 172.21.15.1:0/1334632959 shutdown_connections 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f989c0a3d90 0x7f989c0a6250 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b81137d0 0x7f98b81a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98b8077780 0x7f98b819fcf0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 --2- 172.21.15.1:0/1334632959 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98b8076e60 0x7f98b819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 -- 172.21.15.1:0/1334632959 >> 172.21.15.1:0/1334632959 conn(0x7f98b8100600 msgr2=0x7f98b8114f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.041+0000 7f98be800640 1 -- 172.21.15.1:0/1334632959 shutdown_connections 2024-04-26T21:36:20.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:20.042+0000 7f98be800640 1 -- 172.21.15.1:0/1334632959 wait complete. 2024-04-26T21:36:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:20 smithi186 ceph-mon[32991]: pgmap v183: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:36:20.560 INFO:tasks.workunit.client.0.smithi001.stdout: 1 174946 4.25 MB/sec execute 138 sec latency 37.727 ms 2024-04-26T21:36:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:20 smithi001 ceph-mon[28054]: pgmap v183: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:36:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:20 smithi038 ceph-mon[32724]: pgmap v183: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 1.3 MiB/s wr, 147 op/s 2024-04-26T21:36:21.560 INFO:tasks.workunit.client.0.smithi001.stdout: 1 176308 4.26 MB/sec execute 139 sec latency 34.731 ms 2024-04-26T21:36:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:22 smithi186 ceph-mon[32991]: pgmap v184: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-04-26T21:36:22.560 INFO:tasks.workunit.client.0.smithi001.stdout: 1 177486 4.30 MB/sec execute 140 sec latency 20.966 ms 2024-04-26T21:36:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:22 smithi001 ceph-mon[28054]: pgmap v184: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-04-26T21:36:22.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:22 smithi038 ceph-mon[32724]: pgmap v184: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.6 MiB/s wr, 207 op/s 2024-04-26T21:36:23.560 INFO:tasks.workunit.client.0.smithi001.stdout: 1 177805 4.28 MB/sec execute 141 sec latency 26.407 ms 2024-04-26T21:36:24.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:24 smithi186 ceph-mon[32991]: pgmap v185: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-04-26T21:36:24.560 INFO:tasks.workunit.client.0.smithi001.stdout: 1 178434 4.27 MB/sec execute 142 sec latency 23.289 ms 2024-04-26T21:36:24.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:24 smithi001 ceph-mon[28054]: pgmap v185: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-04-26T21:36:24.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:24 smithi038 ceph-mon[32724]: pgmap v185: 129 pgs: 129 active+clean; 106 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-04-26T21:36:25.560 INFO:tasks.workunit.client.0.smithi001.stdout: 1 179814 4.27 MB/sec execute 143 sec latency 74.566 ms 2024-04-26T21:36:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:26 smithi186 ceph-mon[32991]: pgmap v186: 129 pgs: 129 active+clean; 104 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-04-26T21:36:26.560 INFO:tasks.workunit.client.0.smithi001.stdout: 1 180999 4.32 MB/sec execute 144 sec latency 33.674 ms 2024-04-26T21:36:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:26 smithi001 ceph-mon[28054]: pgmap v186: 129 pgs: 129 active+clean; 104 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-04-26T21:36:26.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:26 smithi038 ceph-mon[32724]: pgmap v186: 129 pgs: 129 active+clean; 104 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 1.7 MiB/s wr, 203 op/s 2024-04-26T21:36:27.560 INFO:tasks.workunit.client.0.smithi001.stdout: 1 181359 4.30 MB/sec execute 145 sec latency 21.289 ms 2024-04-26T21:36:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:28 smithi186 ceph-mon[32991]: pgmap v187: 129 pgs: 129 active+clean; 104 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-04-26T21:36:28.560 INFO:tasks.workunit.client.0.smithi001.stdout: 1 182005 4.28 MB/sec execute 146 sec latency 22.775 ms 2024-04-26T21:36:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:28 smithi001 ceph-mon[28054]: pgmap v187: 129 pgs: 129 active+clean; 104 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-04-26T21:36:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:28 smithi038 ceph-mon[32724]: pgmap v187: 129 pgs: 129 active+clean; 104 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-04-26T21:36:29.561 INFO:tasks.workunit.client.0.smithi001.stdout: 1 183353 4.29 MB/sec execute 147 sec latency 21.581 ms 2024-04-26T21:36:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:30 smithi186 ceph-mon[32991]: pgmap v188: 129 pgs: 129 active+clean; 104 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-04-26T21:36:30.561 INFO:tasks.workunit.client.0.smithi001.stdout: 1 184464 4.33 MB/sec execute 148 sec latency 94.584 ms 2024-04-26T21:36:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:30 smithi001 ceph-mon[28054]: pgmap v188: 129 pgs: 129 active+clean; 104 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-04-26T21:36:30.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:30 smithi038 ceph-mon[32724]: pgmap v188: 129 pgs: 129 active+clean; 104 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-04-26T21:36:31.561 INFO:tasks.workunit.client.0.smithi001.stdout: 1 184882 4.31 MB/sec execute 149 sec latency 27.398 ms 2024-04-26T21:36:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:32 smithi186 ceph-mon[32991]: pgmap v189: 129 pgs: 129 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-04-26T21:36:32.561 INFO:tasks.workunit.client.0.smithi001.stdout: 1 185435 4.29 MB/sec execute 150 sec latency 23.592 ms 2024-04-26T21:36:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:32 smithi001 ceph-mon[28054]: pgmap v189: 129 pgs: 129 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-04-26T21:36:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:32 smithi038 ceph-mon[32724]: pgmap v189: 129 pgs: 129 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 1.7 MiB/s wr, 200 op/s 2024-04-26T21:36:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.204+0000 7f6162e00640 1 -- 172.21.15.1:0/196136156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f615c077780 msgr2=0x7f615c113290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.204+0000 7f6162e00640 1 --2- 172.21.15.1:0/196136156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f615c077780 0x7f615c113290 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6148009930 tx=0x7f614802a420 comp rx=0 tx=0).stop 2024-04-26T21:36:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.205+0000 7f6162e00640 1 -- 172.21.15.1:0/196136156 shutdown_connections 2024-04-26T21:36:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.205+0000 7f6162e00640 1 --2- 172.21.15.1:0/196136156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f615c1137d0 0x7f615c115bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.205+0000 7f6162e00640 1 --2- 172.21.15.1:0/196136156 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f615c077780 0x7f615c113290 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.205+0000 7f6162e00640 1 --2- 172.21.15.1:0/196136156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f615c076e60 0x7f615c077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.205+0000 7f6162e00640 1 -- 172.21.15.1:0/196136156 >> 172.21.15.1:0/196136156 conn(0x7f615c100620 msgr2=0x7f615c102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:33.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.205+0000 7f6162e00640 1 -- 172.21.15.1:0/196136156 shutdown_connections 2024-04-26T21:36:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.205+0000 7f6162e00640 1 -- 172.21.15.1:0/196136156 wait complete. 2024-04-26T21:36:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.206+0000 7f6162e00640 1 Processor -- start 2024-04-26T21:36:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.206+0000 7f6162e00640 1 -- start start 2024-04-26T21:36:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f6162e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f615c076e60 0x7f615c1a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f6162e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f615c077780 0x7f615c1a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f6162e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f615c1137d0 0x7f615c19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f6162e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f615c119220 con 0x7f615c076e60 2024-04-26T21:36:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f6162e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1190a0 con 0x7f615c1137d0 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f6162e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1193a0 con 0x7f615c077780 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f615b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f615c077780 0x7f615c1a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f615be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f615c076e60 0x7f615c1a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f6160e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f615c1137d0 0x7f615c19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f615b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f615c077780 0x7f615c1a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41452/0 (socket says 172.21.15.1:41452) 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.207+0000 7f615b400640 1 -- 172.21.15.1:0/3801047757 learned_addr learned my addr 172.21.15.1:0/3801047757 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.208+0000 7f6160e00640 1 -- 172.21.15.1:0/3801047757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f615c077780 msgr2=0x7f615c1a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.208+0000 7f6160e00640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f615c077780 0x7f615c1a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.208+0000 7f6160e00640 1 -- 172.21.15.1:0/3801047757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f615c076e60 msgr2=0x7f615c1a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.208+0000 7f6160e00640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f615c076e60 0x7f615c1a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.208+0000 7f6160e00640 1 -- 172.21.15.1:0/3801047757 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615c19ff40 con 0x7f615c1137d0 2024-04-26T21:36:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.208+0000 7f6160e00640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f615c1137d0 0x7f615c19f680 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6154005fd0 tx=0x7f615400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.209+0000 7f6158c00640 1 -- 172.21.15.1:0/3801047757 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615400cdf0 con 0x7f615c1137d0 2024-04-26T21:36:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.209+0000 7f6158c00640 1 -- 172.21.15.1:0/3801047757 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6154005020 con 0x7f615c1137d0 2024-04-26T21:36:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.209+0000 7f6162e00640 1 -- 172.21.15.1:0/3801047757 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615c1a0230 con 0x7f615c1137d0 2024-04-26T21:36:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.209+0000 7f6158c00640 1 -- 172.21.15.1:0/3801047757 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6154010690 con 0x7f615c1137d0 2024-04-26T21:36:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.209+0000 7f6162e00640 1 -- 172.21.15.1:0/3801047757 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f615c1ac4a0 con 0x7f615c1137d0 2024-04-26T21:36:33.214 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.214+0000 7f6165bf9740 1 Processor -- start 2024-04-26T21:36:33.214 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.214+0000 7f6158c00640 1 -- 172.21.15.1:0/3801047757 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6154005460 con 0x7f615c1137d0 2024-04-26T21:36:33.214 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.214+0000 7f6165bf9740 1 -- start start 2024-04-26T21:36:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.214+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 0x556972b401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b4aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b4d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f6165bf9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x556972aa5380 con 0x556972b3fdd0 2024-04-26T21:36:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f6165bf9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556972a41c30 con 0x556972b4afb0 2024-04-26T21:36:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f6165bf9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556972aa1340 con 0x556972b406f0 2024-04-26T21:36:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f615b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 0x556972b401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f615be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b4aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f615b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 0x556972b401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46726/0 (socket says 172.21.15.1:46726) 2024-04-26T21:36:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f615b400640 1 -- 172.21.15.1:0/1976822705 learned_addr learned my addr 172.21.15.1:0/1976822705 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f615b400640 1 --2- 172.21.15.1:0/1976822705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b4d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f615be00640 1 -- 172.21.15.1:0/1976822705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 msgr2=0x556972b4d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f615be00640 1 --2- 172.21.15.1:0/1976822705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b4d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f615be00640 1 -- 172.21.15.1:0/1976822705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 msgr2=0x556972b401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f615be00640 1 --2- 172.21.15.1:0/1976822705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 0x556972b401b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.215+0000 7f615be00640 1 -- 172.21.15.1:0/1976822705 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556972a63940 con 0x556972b406f0 2024-04-26T21:36:33.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.216+0000 7f615be00640 1 --2- 172.21.15.1:0/1976822705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b4aa70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f614c00bbf0 tx=0x7f614c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e24089d26d47d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:33.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.216+0000 7f6158c00640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f61400a3d90 0x7f61400a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.216+0000 7f6158c00640 1 -- 172.21.15.1:0/3801047757 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f61540d1270 con 0x7f615c1137d0 2024-04-26T21:36:33.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.216+0000 7f6151600640 1 -- 172.21.15.1:0/1976822705 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c004030 con 0x556972b406f0 2024-04-26T21:36:33.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.216+0000 7f6151600640 1 -- 172.21.15.1:0/1976822705 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f614c0035f0 con 0x556972b406f0 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.216+0000 7f6151600640 1 -- 172.21.15.1:0/1976822705 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c011640 con 0x556972b406f0 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.216+0000 7f615b400640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f61400a3d90 0x7f61400a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.217+0000 7f6165bf9740 1 -- 172.21.15.1:0/1976822705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 msgr2=0x556972b4aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.217+0000 7f615b400640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f61400a3d90 0x7f61400a6250 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x556972b3d700 tx=0x7f6148005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.217+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1976822705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b4aa70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f614c00bbf0 tx=0x7f614c0107a0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.217+0000 7f6165bf9740 1 -- 172.21.15.1:0/1976822705 shutdown_connections 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.217+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1976822705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b4aa70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.217+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1976822705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b4d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.217+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1976822705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 0x556972b401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.217+0000 7f6165bf9740 1 -- 172.21.15.1:0/1976822705 >> 172.21.15.1:0/1976822705 conn(0x556972b3c2e0 msgr2=0x556972b3c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.217+0000 7f6165bf9740 1 -- 172.21.15.1:0/1976822705 shutdown_connections 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.217+0000 7f6165bf9740 1 -- 172.21.15.1:0/1976822705 wait complete. 2024-04-26T21:36:33.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.218+0000 7f6165bf9740 1 -- start start 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.218+0000 7f6165bf9740 1 Processor -- start 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.218+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b3fdd0 0x556972b47fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.218+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b406f0 0x556972b48500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.218+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b4afb0 0x556972b48a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.218+0000 7f6165bf9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x556972b4dbe0 con 0x556972b406f0 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.218+0000 7f6165bf9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x556972a2cc60 con 0x556972b3fdd0 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.218+0000 7f6165bf9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556972b4dd60 con 0x556972b4afb0 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.218+0000 7f6165bf9740 10 client.0 _ll_register_callbacks cb 0x556972a58e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f615be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b3fdd0 0x556972b47fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f615be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b3fdd0 0x556972b47fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53416/0 (socket says 172.21.15.1:53416) 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f615be00640 1 -- 172.21.15.1:0/1384579071 learned_addr learned my addr 172.21.15.1:0/1384579071 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f615b400640 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b406f0 0x556972b48500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f6160e00640 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b4afb0 0x556972b48a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f615be00640 1 -- 172.21.15.1:0/1384579071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b4afb0 msgr2=0x556972b48a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f615be00640 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b4afb0 0x556972b48a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f615be00640 1 -- 172.21.15.1:0/1384579071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b406f0 msgr2=0x556972b48500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f615be00640 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b406f0 0x556972b48500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f615be00640 1 -- 172.21.15.1:0/1384579071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556972a3b510 con 0x556972b3fdd0 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f615be00640 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b3fdd0 0x556972b47fc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f614c005930 tx=0x7f614c00b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b7a4532a2c28e96 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.219+0000 7f6146000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.220+0000 7f6146000640 1 -- 172.21.15.1:0/1384579071 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c0039b0 con 0x556972b3fdd0 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.220+0000 7f6146000640 1 -- 172.21.15.1:0/1384579071 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f614c003fe0 con 0x556972b3fdd0 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.220+0000 7f6165bf9740 1 -- 172.21.15.1:0/1384579071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f614c0035f0 con 0x556972b3fdd0 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.220+0000 7f6146000640 1 -- 172.21.15.1:0/1384579071 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f614c0041e0 con 0x556972b3fdd0 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.220+0000 7f6165bf9740 10 client.26012 fetch_fsmap learned FSMap version 36 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.220+0000 7f6165bf9740 1 -- 172.21.15.1:0/1384579071 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5569727b7210 con 0x556972b3fdd0 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.220+0000 7f6146000640 1 -- 172.21.15.1:0/1384579071 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f614c0035f0 con 0x556972b3fdd0 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.220+0000 7f6146000640 1 -- 172.21.15.1:0/1384579071 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f614c004730 con 0x556972b3fdd0 2024-04-26T21:36:33.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.220+0000 7f6165bf9740 10 client.26012 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:36:33.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.221+0000 7f6165bf9740 10 client.26012 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:36:33.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.221+0000 7f6165bf9740 20 client.26012 populate_metadata read hostname 'smithi001' 2024-04-26T21:36:33.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.221+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x556972b5acc0 0x556972b5d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.221+0000 7f6165bf9740 4 client.26012 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:36:33.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.221+0000 7f6165bf9740 1 -- 172.21.15.1:0/1384579071 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556972ab2f90 con 0x556972b5acc0 2024-04-26T21:36:33.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.221+0000 7f615b400640 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x556972b5acc0 0x556972b5d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.221+0000 7f615b400640 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x556972b5acc0 0x556972b5d0b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf2b1b3251644a6 server_cookie=2acbf6843e1b7d5a in_seq=0 out_seq=0 2024-04-26T21:36:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.221+0000 7f6146000640 10 client.26012 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.222+0000 7f6146000640 1 -- 172.21.15.1:0/1384579071 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556972ab2f90 con 0x556972b5acc0 2024-04-26T21:36:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.222+0000 7f6146000640 10 client.26012 handle_command_reply: tid=0 2024-04-26T21:36:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.222+0000 7f6165bf9740 1 client.26012 shutdown 2024-04-26T21:36:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 1 -- 172.21.15.1:0/1384579071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b3fdd0 msgr2=0x556972b47fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b3fdd0 0x556972b47fc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f614c005930 tx=0x7f614c00b820 comp rx=0 tx=0).stop 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 1 -- 172.21.15.1:0/1384579071 shutdown_connections 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x556972b5acc0 0x556972b5d0b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b4afb0 0x556972b48a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b406f0 0x556972b48500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1384579071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b3fdd0 0x556972b47fc0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6146000640 0 client.26012 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 1 -- 172.21.15.1:0/1384579071 >> 172.21.15.1:0/1384579071 conn(0x556972b3c2e0 msgr2=0x556972b42060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 1 -- 172.21.15.1:0/1384579071 shutdown_connections 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 1 -- 172.21.15.1:0/1384579071 wait complete. 2024-04-26T21:36:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.223+0000 7f6165bf9740 20 client.26012 trim_cache size 0 max 16384 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.245+0000 7f6165bf9740 1 Processor -- start 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.245+0000 7f6165bf9740 1 -- start start 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.245+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b3fdd0 0x556972b56aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b406f0 0x556972b627c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b62d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f6165bf9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1b8e30 con 0x556972b406f0 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f6165bf9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1b6700 con 0x556972b4afb0 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f6165bf9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1b4550 con 0x556972b3fdd0 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f615be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b3fdd0 0x556972b56aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f615be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b3fdd0 0x556972b56aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41486/0 (socket says 172.21.15.1:41486) 2024-04-26T21:36:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f615be00640 1 -- 172.21.15.1:0/1007127483 learned_addr learned my addr 172.21.15.1:0/1007127483 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:33.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f6160e00640 1 --2- 172.21.15.1:0/1007127483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b62d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f615b400640 1 --2- 172.21.15.1:0/1007127483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b406f0 0x556972b627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f615be00640 1 -- 172.21.15.1:0/1007127483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 msgr2=0x556972b62d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:33.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f615be00640 1 --2- 172.21.15.1:0/1007127483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b62d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f615be00640 1 -- 172.21.15.1:0/1007127483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b406f0 msgr2=0x556972b627c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:33.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f615be00640 1 --2- 172.21.15.1:0/1007127483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b406f0 0x556972b627c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.246+0000 7f615be00640 1 -- 172.21.15.1:0/1007127483 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556972883620 con 0x556972b3fdd0 2024-04-26T21:36:33.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.247+0000 7f615be00640 1 --2- 172.21.15.1:0/1007127483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b3fdd0 0x556972b56aa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f614c001eb0 tx=0x7f614c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=193bb3fa585b9172 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.247+0000 7f6150c00640 1 -- 172.21.15.1:0/1007127483 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c0035f0 con 0x556972b3fdd0 2024-04-26T21:36:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.247+0000 7f6150c00640 1 -- 172.21.15.1:0/1007127483 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f614c010ca0 con 0x556972b3fdd0 2024-04-26T21:36:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.247+0000 7f6150c00640 1 -- 172.21.15.1:0/1007127483 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f614c011670 con 0x556972b3fdd0 2024-04-26T21:36:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 -- 172.21.15.1:0/1007127483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b3fdd0 msgr2=0x556972b56aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1007127483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b3fdd0 0x556972b56aa0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f614c001eb0 tx=0x7f614c0107a0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 -- 172.21.15.1:0/1007127483 shutdown_connections 2024-04-26T21:36:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1007127483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b62d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1007127483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b406f0 0x556972b627c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1007127483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b3fdd0 0x556972b56aa0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 -- 172.21.15.1:0/1007127483 >> 172.21.15.1:0/1007127483 conn(0x556972b3c2e0 msgr2=0x556972b43330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 -- 172.21.15.1:0/1007127483 shutdown_connections 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 -- 172.21.15.1:0/1007127483 wait complete. 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 -- start start 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.248+0000 7f6165bf9740 1 Processor -- start 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 0x556972b44bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b45130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f6165bf9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b45670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f6165bf9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1b4bc0 con 0x556972b3fdd0 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f6165bf9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1b6030 con 0x556972b4afb0 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f6165bf9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f615c1b5290 con 0x556972b406f0 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f6165bf9740 10 client.0 _ll_register_callbacks cb 0x7f6148009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:36:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f615b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b45130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f615b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b45130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41492/0 (socket says 172.21.15.1:41492) 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f615b400640 1 -- 172.21.15.1:0/1562583470 learned_addr learned my addr 172.21.15.1:0/1562583470 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f6160e00640 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b45670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.249+0000 7f615be00640 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 0x556972b44bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.250+0000 7f615b400640 1 -- 172.21.15.1:0/1562583470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 msgr2=0x556972b45670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.250+0000 7f615b400640 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b45670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.250+0000 7f615b400640 1 -- 172.21.15.1:0/1562583470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 msgr2=0x556972b44bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.250+0000 7f615b400640 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 0x556972b44bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.250+0000 7f615b400640 1 -- 172.21.15.1:0/1562583470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556972a5fcb0 con 0x556972b406f0 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.250+0000 7f615be00640 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 0x556972b44bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.250+0000 7f615b400640 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b45130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6148004240 tx=0x7f614802a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc069e703d02395 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.250+0000 7f6146000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.250+0000 7f6146000640 1 -- 172.21.15.1:0/1562583470 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6148035040 con 0x556972b406f0 2024-04-26T21:36:33.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6165bf9740 1 -- 172.21.15.1:0/1562583470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f614c010ca0 con 0x556972b406f0 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6146000640 1 -- 172.21.15.1:0/1562583470 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f61480553a0 con 0x556972b406f0 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6146000640 1 -- 172.21.15.1:0/1562583470 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6148065050 con 0x556972b406f0 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6165bf9740 10 client.25999 fetch_fsmap learned FSMap version 36 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6165bf9740 1 -- 172.21.15.1:0/1562583470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556972bc10d0 con 0x556972b406f0 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6146000640 1 -- 172.21.15.1:0/1562583470 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f614c010ca0 con 0x556972b406f0 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6146000640 1 -- 172.21.15.1:0/1562583470 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f6148049040 con 0x556972b406f0 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6165bf9740 10 client.25999 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6165bf9740 10 client.25999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6165bf9740 20 client.25999 populate_metadata read hostname 'smithi001' 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x556972bc1820 0x556972bc3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6165bf9740 4 client.25999 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:36:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.251+0000 7f6165bf9740 1 -- 172.21.15.1:0/1562583470 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556972aace00 con 0x556972bc1820 2024-04-26T21:36:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.252+0000 7f615be00640 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x556972bc1820 0x556972bc3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.252+0000 7f615be00640 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x556972bc1820 0x556972bc3c40 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc063026c4715274 server_cookie=c9eeb4ff85a26ffa in_seq=0 out_seq=0 2024-04-26T21:36:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.252+0000 7f6146000640 10 client.25999 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6146000640 1 -- 172.21.15.1:0/1562583470 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556972aace00 con 0x556972bc1820 2024-04-26T21:36:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6146000640 10 client.25999 handle_command_reply: tid=0 2024-04-26T21:36:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6165bf9740 1 client.25999 shutdown 2024-04-26T21:36:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6165bf9740 1 -- 172.21.15.1:0/1562583470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 msgr2=0x556972b45130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b45130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6148004240 tx=0x7f614802a420 comp rx=0 tx=0).stop 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6165bf9740 1 -- 172.21.15.1:0/1562583470 shutdown_connections 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x556972bc1820 0x556972bc3c40 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556972b406f0 0x556972b45130 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556972b4afb0 0x556972b45670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6165bf9740 1 --2- 172.21.15.1:0/1562583470 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556972b3fdd0 0x556972b44bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6146000640 0 client.25999 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.313+0000 7f6165bf9740 1 -- 172.21.15.1:0/1562583470 >> 172.21.15.1:0/1562583470 conn(0x556972b3c2e0 msgr2=0x556972b41780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.314+0000 7f6165bf9740 1 -- 172.21.15.1:0/1562583470 shutdown_connections 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.314+0000 7f6165bf9740 1 -- 172.21.15.1:0/1562583470 wait complete. 2024-04-26T21:36:33.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.314+0000 7f6165bf9740 20 client.25999 trim_cache size 0 max 16384 2024-04-26T21:36:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:36:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:36:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:36:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:36:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.314+0000 7f6162e00640 1 -- 172.21.15.1:0/3801047757 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f61400a3d90 msgr2=0x7f61400a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.314+0000 7f6162e00640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f61400a3d90 0x7f61400a6250 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x556972b3d700 tx=0x7f6148005a20 comp rx=0 tx=0).stop 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.315+0000 7f6162e00640 1 -- 172.21.15.1:0/3801047757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f615c1137d0 msgr2=0x7f615c19f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.315+0000 7f6162e00640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f615c1137d0 0x7f615c19f680 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6154005fd0 tx=0x7f615400ef10 comp rx=0 tx=0).stop 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.315+0000 7f6162e00640 1 -- 172.21.15.1:0/3801047757 shutdown_connections 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.315+0000 7f6162e00640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f61400a3d90 0x7f61400a6250 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.315+0000 7f6162e00640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f615c077780 0x7f615c1a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.315+0000 7f6162e00640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f615c1137d0 0x7f615c19f680 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.315+0000 7f6162e00640 1 --2- 172.21.15.1:0/3801047757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f615c076e60 0x7f615c1a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.315+0000 7f6162e00640 1 -- 172.21.15.1:0/3801047757 >> 172.21.15.1:0/3801047757 conn(0x7f615c100620 msgr2=0x7f615c114f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.315+0000 7f6162e00640 1 -- 172.21.15.1:0/3801047757 shutdown_connections 2024-04-26T21:36:33.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:33.315+0000 7f6162e00640 1 -- 172.21.15.1:0/3801047757 wait complete. 2024-04-26T21:36:33.561 INFO:tasks.workunit.client.0.smithi001.stdout: 1 186923 4.30 MB/sec execute 151 sec latency 27.681 ms 2024-04-26T21:36:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:34 smithi186 ceph-mon[32991]: pgmap v190: 129 pgs: 129 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-26T21:36:34.561 INFO:tasks.workunit.client.0.smithi001.stdout: 1 188112 4.34 MB/sec execute 152 sec latency 25.534 ms 2024-04-26T21:36:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:34 smithi001 ceph-mon[28054]: pgmap v190: 129 pgs: 129 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-26T21:36:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:34 smithi038 ceph-mon[32724]: pgmap v190: 129 pgs: 129 active+clean; 107 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-26T21:36:35.561 INFO:tasks.workunit.client.0.smithi001.stdout: 1 188435 4.32 MB/sec execute 153 sec latency 44.149 ms 2024-04-26T21:36:36.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:36 smithi186 ceph-mon[32991]: pgmap v191: 129 pgs: 129 active+clean; 110 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 1.9 MiB/s wr, 228 op/s 2024-04-26T21:36:36.562 INFO:tasks.workunit.client.0.smithi001.stdout: 1 189193 4.31 MB/sec execute 154 sec latency 27.446 ms 2024-04-26T21:36:36.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:36 smithi001 ceph-mon[28054]: pgmap v191: 129 pgs: 129 active+clean; 110 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 1.9 MiB/s wr, 228 op/s 2024-04-26T21:36:36.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:36 smithi038 ceph-mon[32724]: pgmap v191: 129 pgs: 129 active+clean; 110 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 1.9 MiB/s wr, 228 op/s 2024-04-26T21:36:37.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:37 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:36:37.562 INFO:tasks.workunit.client.0.smithi001.stdout: 1 190765 4.34 MB/sec execute 155 sec latency 20.524 ms 2024-04-26T21:36:37.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:37 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:36:37.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:37 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:36:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:38 smithi186 ceph-mon[32991]: pgmap v192: 129 pgs: 129 active+clean; 110 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-26T21:36:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:38 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:36:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:38 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:36:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:38 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:36:38.562 INFO:tasks.workunit.client.0.smithi001.stdout: 1 191792 4.36 MB/sec execute 156 sec latency 28.591 ms 2024-04-26T21:36:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:38 smithi001 ceph-mon[28054]: pgmap v192: 129 pgs: 129 active+clean; 110 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-26T21:36:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:38 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:36:38.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:38 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:36:38.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:38 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:36:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:38 smithi038 ceph-mon[32724]: pgmap v192: 129 pgs: 129 active+clean; 110 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-26T21:36:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:38 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:36:38.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:38 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:36:38.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:38 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:36:39.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:39 smithi186 ceph-mon[32991]: pgmap v193: 129 pgs: 129 active+clean; 110 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-26T21:36:39.562 INFO:tasks.workunit.client.0.smithi001.stdout: 1 192017 4.33 MB/sec execute 157 sec latency 25.509 ms 2024-04-26T21:36:39.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:39 smithi001 ceph-mon[28054]: pgmap v193: 129 pgs: 129 active+clean; 110 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-26T21:36:39.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:39 smithi038 ceph-mon[32724]: pgmap v193: 129 pgs: 129 active+clean; 110 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-04-26T21:36:40.562 INFO:tasks.workunit.client.0.smithi001.stdout: 1 192689 4.32 MB/sec execute 158 sec latency 39.478 ms 2024-04-26T21:36:41.562 INFO:tasks.workunit.client.0.smithi001.stdout: 1 194038 4.33 MB/sec execute 159 sec latency 30.156 ms 2024-04-26T21:36:42.157 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:41 smithi186 ceph-mon[32991]: pgmap v194: 129 pgs: 129 active+clean; 112 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 208 op/s 2024-04-26T21:36:42.271 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:41 smithi001 ceph-mon[28054]: pgmap v194: 129 pgs: 129 active+clean; 112 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 208 op/s 2024-04-26T21:36:42.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:41 smithi038 ceph-mon[32724]: pgmap v194: 129 pgs: 129 active+clean; 112 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 208 op/s 2024-04-26T21:36:42.562 INFO:tasks.workunit.client.0.smithi001.stdout: 1 195211 4.36 MB/sec execute 160 sec latency 21.141 ms 2024-04-26T21:36:43.562 INFO:tasks.workunit.client.0.smithi001.stdout: 1 195528 4.34 MB/sec execute 161 sec latency 23.480 ms 2024-04-26T21:36:44.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:43 smithi186 ceph-mon[32991]: pgmap v195: 129 pgs: 129 active+clean; 112 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-26T21:36:44.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:43 smithi001 ceph-mon[28054]: pgmap v195: 129 pgs: 129 active+clean; 112 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-26T21:36:44.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:43 smithi038 ceph-mon[32724]: pgmap v195: 129 pgs: 129 active+clean; 112 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-26T21:36:44.562 INFO:tasks.workunit.client.0.smithi001.stdout: 1 196165 4.33 MB/sec execute 162 sec latency 22.623 ms 2024-04-26T21:36:45.563 INFO:tasks.workunit.client.0.smithi001.stdout: 1 197550 4.34 MB/sec execute 163 sec latency 24.701 ms 2024-04-26T21:36:46.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:45 smithi186 ceph-mon[32991]: pgmap v196: 129 pgs: 129 active+clean; 115 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-26T21:36:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:45 smithi001 ceph-mon[28054]: pgmap v196: 129 pgs: 129 active+clean; 115 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-26T21:36:46.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:45 smithi038 ceph-mon[32724]: pgmap v196: 129 pgs: 129 active+clean; 115 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-26T21:36:46.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.472+0000 7f3b88a00640 1 -- 172.21.15.1:0/2428280757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b84103910 msgr2=0x7f3b84103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.472+0000 7f3b88a00640 1 --2- 172.21.15.1:0/2428280757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b84103910 0x7f3b84103cf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009900 tx=0x7f3b7402a480 comp rx=0 tx=0).stop 2024-04-26T21:36:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.473+0000 7f3b88a00640 1 -- 172.21.15.1:0/2428280757 shutdown_connections 2024-04-26T21:36:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.473+0000 7f3b88a00640 1 --2- 172.21.15.1:0/2428280757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8410cd30 0x7f3b8410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.473+0000 7f3b88a00640 1 --2- 172.21.15.1:0/2428280757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b84104230 0x7f3b8410c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.473+0000 7f3b88a00640 1 --2- 172.21.15.1:0/2428280757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b84103910 0x7f3b84103cf0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.473+0000 7f3b88a00640 1 -- 172.21.15.1:0/2428280757 >> 172.21.15.1:0/2428280757 conn(0x7f3b840fd110 msgr2=0x7f3b840ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.473+0000 7f3b88a00640 1 -- 172.21.15.1:0/2428280757 shutdown_connections 2024-04-26T21:36:46.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.473+0000 7f3b88a00640 1 -- 172.21.15.1:0/2428280757 wait complete. 2024-04-26T21:36:46.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.474+0000 7f3b88a00640 1 Processor -- start 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.474+0000 7f3b88a00640 1 -- start start 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.474+0000 7f3b88a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b84103910 0x7f3b84198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.474+0000 7f3b88a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b84104230 0x7f3b841994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.474+0000 7f3b88a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8410cd30 0x7f3b8419d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.474+0000 7f3b88a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3b84112a30 con 0x7f3b8410cd30 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.474+0000 7f3b88a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841128b0 con 0x7f3b84104230 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.474+0000 7f3b88a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b84112bb0 con 0x7f3b84103910 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.474+0000 7f3b7be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b84104230 0x7f3b841994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.475+0000 7f3b83400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b84103910 0x7f3b84198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.475+0000 7f3b7be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b84104230 0x7f3b841994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36856/0 (socket says 172.21.15.1:36856) 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.475+0000 7f3b7be00640 1 -- 172.21.15.1:0/1515522861 learned_addr learned my addr 172.21.15.1:0/1515522861 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.475+0000 7f3b83e00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8410cd30 0x7f3b8419d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.475+0000 7f3b7be00640 1 -- 172.21.15.1:0/1515522861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b84103910 msgr2=0x7f3b84198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.475+0000 7f3b7be00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b84103910 0x7f3b84198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.475+0000 7f3b7be00640 1 -- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8410cd30 msgr2=0x7f3b8419d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.475+0000 7f3b7be00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8410cd30 0x7f3b8419d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.475+0000 7f3b7be00640 1 -- 172.21.15.1:0/1515522861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b8419df80 con 0x7f3b84104230 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.475+0000 7f3b83e00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8410cd30 0x7f3b8419d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.476+0000 7f3b7be00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b84104230 0x7f3b841994d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00e9f0 tx=0x7f3b6c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.476+0000 7f3b80c00640 1 -- 172.21.15.1:0/1515522861 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c00ccc0 con 0x7f3b84104230 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.476+0000 7f3b80c00640 1 -- 172.21.15.1:0/1515522861 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b6c0108e0 con 0x7f3b84104230 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.476+0000 7f3b80c00640 1 -- 172.21.15.1:0/1515522861 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c018880 con 0x7f3b84104230 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.476+0000 7f3b88a00640 1 -- 172.21.15.1:0/1515522861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b8419e270 con 0x7f3b84104230 2024-04-26T21:36:46.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.476+0000 7f3b88a00640 1 -- 172.21.15.1:0/1515522861 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b841a5b00 con 0x7f3b84104230 2024-04-26T21:36:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.479+0000 7f3b8ce00740 1 Processor -- start 2024-04-26T21:36:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b80c00640 1 -- 172.21.15.1:0/1515522861 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f3b6c010430 con 0x7f3b84104230 2024-04-26T21:36:46.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b8ce00740 1 -- start start 2024-04-26T21:36:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d97ff00 0x564b3d9802e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d980820 0x564b3d98f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 0x564b3d9919c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564b3d8e5380 con 0x564b3d97ff00 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564b3d881c30 con 0x564b3d980820 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564b3d8e1340 con 0x564b3d98f5d0 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b83400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d980820 0x564b3d98f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d97ff00 0x564b3d9802e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b83400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d980820 0x564b3d98f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36872/0 (socket says 172.21.15.1:36872) 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.480+0000 7f3b83400640 1 -- 172.21.15.1:0/2012867037 learned_addr learned my addr 172.21.15.1:0/2012867037 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.481+0000 7f3b7be00640 1 --2- 172.21.15.1:0/2012867037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 0x564b3d9919c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.481+0000 7f3b83400640 1 -- 172.21.15.1:0/2012867037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 msgr2=0x564b3d9919c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.481+0000 7f3b83400640 1 --2- 172.21.15.1:0/2012867037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 0x564b3d9919c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.481+0000 7f3b83400640 1 -- 172.21.15.1:0/2012867037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d97ff00 msgr2=0x564b3d9802e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.481+0000 7f3b83400640 1 --2- 172.21.15.1:0/2012867037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d97ff00 0x564b3d9802e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.481+0000 7f3b83400640 1 -- 172.21.15.1:0/2012867037 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b3d8a3940 con 0x564b3d980820 2024-04-26T21:36:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.481+0000 7f3b83400640 1 --2- 172.21.15.1:0/2012867037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d980820 0x564b3d98f090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3b74000c00 tx=0x7f3b74004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8f4b1dc6dc2ae45 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.482+0000 7f3b80c00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b680a3d80 0x7f3b680a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.482+0000 7f3b73e00640 1 -- 172.21.15.1:0/2012867037 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74035040 con 0x564b3d980820 2024-04-26T21:36:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.482+0000 7f3b73e00640 1 -- 172.21.15.1:0/2012867037 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b740054f0 con 0x564b3d980820 2024-04-26T21:36:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.482+0000 7f3b73e00640 1 -- 172.21.15.1:0/2012867037 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b740035f0 con 0x564b3d980820 2024-04-26T21:36:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.482+0000 7f3b80c00640 1 -- 172.21.15.1:0/1515522861 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f3b6c0d0f40 con 0x7f3b84104230 2024-04-26T21:36:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.482+0000 7f3b83e00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b680a3d80 0x7f3b680a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.483+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2012867037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d980820 msgr2=0x564b3d98f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:46.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.483+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/2012867037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d980820 0x564b3d98f090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3b74000c00 tx=0x7f3b74004040 comp rx=0 tx=0).stop 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.483+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2012867037 shutdown_connections 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.483+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/2012867037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 0x564b3d9919c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.483+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/2012867037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d980820 0x564b3d98f090 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.483+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/2012867037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d97ff00 0x564b3d9802e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.483+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2012867037 >> 172.21.15.1:0/2012867037 conn(0x564b3d97c4f0 msgr2=0x564b3d97c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.483+0000 7f3b83e00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b680a3d80 0x7f3b680a6240 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c005fd0 tx=0x7f3b7c009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.483+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2012867037 shutdown_connections 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.483+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2012867037 wait complete. 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.484+0000 7f3b8ce00740 1 -- start start 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.484+0000 7f3b8ce00740 1 Processor -- start 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.484+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d97ff00 0x564b3d98c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.484+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d980820 0x564b3d98ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.484+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.484+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564b3d9921a0 con 0x564b3d97ff00 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.484+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564b3d86cc60 con 0x564b3d98f5d0 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.484+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564b3d992320 con 0x564b3d980820 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b8ce00740 10 client.0 _ll_register_callbacks cb 0x564b3d898e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36876/0 (socket says 172.21.15.1:36876) 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b83e00640 1 -- 172.21.15.1:0/3113147691 learned_addr learned my addr 172.21.15.1:0/3113147691 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b83400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d97ff00 0x564b3d98c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b7be00640 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d980820 0x564b3d98ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b83e00640 1 -- 172.21.15.1:0/3113147691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d980820 msgr2=0x564b3d98ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b83e00640 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d980820 0x564b3d98ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b83e00640 1 -- 172.21.15.1:0/3113147691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d97ff00 msgr2=0x564b3d98c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b83e00640 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d97ff00 0x564b3d98c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b83e00640 1 -- 172.21.15.1:0/3113147691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b3d87b510 con 0x564b3d98f5d0 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.485+0000 7f3b7be00640 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d980820 0x564b3d98ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:36:46.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.486+0000 7f3b83e00640 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cfb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c010ab0 tx=0x7f3b7c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5d7e11668991041 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.486+0000 7f3b70c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:36:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.486+0000 7f3b70c00640 1 -- 172.21.15.1:0/3113147691 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c01fc80 con 0x564b3d98f5d0 2024-04-26T21:36:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.486+0000 7f3b70c00640 1 -- 172.21.15.1:0/3113147691 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f3b7c01c3f0 con 0x564b3d98f5d0 2024-04-26T21:36:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.486+0000 7f3b8ce00740 1 -- 172.21.15.1:0/3113147691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b740054f0 con 0x564b3d98f5d0 2024-04-26T21:36:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.486+0000 7f3b70c00640 1 -- 172.21.15.1:0/3113147691 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3b7c01fe20 con 0x564b3d98f5d0 2024-04-26T21:36:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.486+0000 7f3b8ce00740 10 client.26030 fetch_fsmap learned FSMap version 36 2024-04-26T21:36:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.486+0000 7f3b8ce00740 1 -- 172.21.15.1:0/3113147691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564b3d5f7210 con 0x564b3d98f5d0 2024-04-26T21:36:46.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.487+0000 7f3b70c00640 1 -- 172.21.15.1:0/3113147691 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b740054f0 con 0x564b3d98f5d0 2024-04-26T21:36:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.487+0000 7f3b70c00640 1 -- 172.21.15.1:0/3113147691 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f3b7c026070 con 0x564b3d98f5d0 2024-04-26T21:36:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.487+0000 7f3b8ce00740 10 client.26030 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:36:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.487+0000 7f3b8ce00740 10 client.26030 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:36:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.487+0000 7f3b8ce00740 20 client.26030 populate_metadata read hostname 'smithi001' 2024-04-26T21:36:46.492 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.487+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564b3d99f380 0x564b3d9a1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.487+0000 7f3b8ce00740 4 client.26030 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.487+0000 7f3b8ce00740 1 -- 172.21.15.1:0/3113147691 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b3d8f2f90 con 0x564b3d99f380 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.487+0000 7f3b83400640 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564b3d99f380 0x564b3d9a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.488+0000 7f3b83400640 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564b3d99f380 0x564b3d9a1790 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=500989750419f384 server_cookie=bbc2c2856fd958e0 in_seq=0 out_seq=0 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.488+0000 7f3b70c00640 10 client.26030 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b70c00640 1 -- 172.21.15.1:0/3113147691 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564b3d8f2f90 con 0x564b3d99f380 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b70c00640 10 client.26030 handle_command_reply: tid=0 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b8ce00740 1 client.26030 shutdown 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b8ce00740 1 -- 172.21.15.1:0/3113147691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 msgr2=0x564b3d98cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cfb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c010ab0 tx=0x7f3b7c004a40 comp rx=0 tx=0).stop 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b8ce00740 1 -- 172.21.15.1:0/3113147691 shutdown_connections 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564b3d99f380 0x564b3d9a1790 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d980820 0x564b3d98ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cfb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/3113147691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d97ff00 0x564b3d98c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b70c00640 0 client.26030 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.489+0000 7f3b8ce00740 1 -- 172.21.15.1:0/3113147691 >> 172.21.15.1:0/3113147691 conn(0x564b3d97c4f0 msgr2=0x564b3d982360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.490+0000 7f3b8ce00740 1 -- 172.21.15.1:0/3113147691 shutdown_connections 2024-04-26T21:36:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.490+0000 7f3b8ce00740 1 -- 172.21.15.1:0/3113147691 wait complete. 2024-04-26T21:36:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.490+0000 7f3b8ce00740 20 client.26030 trim_cache size 0 max 16384 2024-04-26T21:36:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.509+0000 7f3b8ce00740 1 Processor -- start 2024-04-26T21:36:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b8ce00740 1 -- start start 2024-04-26T21:36:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d97ff00 0x564b3d98ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 0x564b3d99b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841b03d0 con 0x564b3d980820 2024-04-26T21:36:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841adca0 con 0x564b3d98f5d0 2024-04-26T21:36:46.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841abaf0 con 0x564b3d97ff00 2024-04-26T21:36:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b83400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d97ff00 0x564b3d98ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b83400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d97ff00 0x564b3d98ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41226/0 (socket says 172.21.15.1:41226) 2024-04-26T21:36:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b83400640 1 -- 172.21.15.1:0/1165494195 learned_addr learned my addr 172.21.15.1:0/1165494195 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b7be00640 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 0x564b3d99b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.510+0000 7f3b83e00640 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b83400640 1 -- 172.21.15.1:0/1165494195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 msgr2=0x564b3d98cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b83400640 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b83400640 1 -- 172.21.15.1:0/1165494195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 msgr2=0x564b3d99b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b83400640 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 0x564b3d99b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b83400640 1 -- 172.21.15.1:0/1165494195 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b3d98b500 con 0x564b3d97ff00 2024-04-26T21:36:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b7be00640 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 0x564b3d99b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:36:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b83e00640 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:36:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b83400640 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d97ff00 0x564b3d98ec60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009230 tx=0x7f3b74004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34d31a484b796bf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b73400640 1 -- 172.21.15.1:0/1165494195 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74035040 con 0x564b3d97ff00 2024-04-26T21:36:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b73400640 1 -- 172.21.15.1:0/1165494195 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b74054680 con 0x564b3d97ff00 2024-04-26T21:36:46.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.511+0000 7f3b73400640 1 -- 172.21.15.1:0/1165494195 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74003a90 con 0x564b3d97ff00 2024-04-26T21:36:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.512+0000 7f3b8ce00740 1 -- 172.21.15.1:0/1165494195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d97ff00 msgr2=0x564b3d98ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:46.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.512+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d97ff00 0x564b3d98ec60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009230 tx=0x7f3b74004ab0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.512+0000 7f3b8ce00740 1 -- 172.21.15.1:0/1165494195 shutdown_connections 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.512+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d98f5d0 0x564b3d98cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.512+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 0x564b3d99b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.512+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/1165494195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d97ff00 0x564b3d98ec60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.512+0000 7f3b8ce00740 1 -- 172.21.15.1:0/1165494195 >> 172.21.15.1:0/1165494195 conn(0x564b3d97c4f0 msgr2=0x564b3d97c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.512+0000 7f3b8ce00740 1 -- 172.21.15.1:0/1165494195 shutdown_connections 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.512+0000 7f3b8ce00740 1 -- 172.21.15.1:0/1165494195 wait complete. 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.513+0000 7f3b8ce00740 1 -- start start 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.513+0000 7f3b8ce00740 1 Processor -- start 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.513+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 0x564b3d997570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.513+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 0x564b3d997ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b8ce00740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d994540 0x564b3d9949c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841ac160 con 0x564b3d980820 2024-04-26T21:36:46.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841ad5d0 con 0x564b3d994540 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b8ce00740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841ac830 con 0x564b3d98f5d0 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b8ce00740 10 client.0 _ll_register_callbacks cb 0x7f3b7c009df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b83400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 0x564b3d997570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d994540 0x564b3d9949c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b83400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 0x564b3d997570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35132/0 (socket says 172.21.15.1:35132) 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b83400640 1 -- 172.21.15.1:0/2471627781 learned_addr learned my addr 172.21.15.1:0/2471627781 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b7be00640 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 0x564b3d997ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b83e00640 1 -- 172.21.15.1:0/2471627781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 msgr2=0x564b3d997ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b83e00640 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 0x564b3d997ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b83e00640 1 -- 172.21.15.1:0/2471627781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 msgr2=0x564b3d997570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b83e00640 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 0x564b3d997570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b83e00640 1 -- 172.21.15.1:0/2471627781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b3d6c3620 con 0x564b3d994540 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.514+0000 7f3b7be00640 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 0x564b3d997ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:36:46.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.515+0000 7f3b83e00640 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d994540 0x564b3d9949c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009f80 tx=0x7f3b7c028670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e151d0ac7f72509f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.515+0000 7f3b70c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.515+0000 7f3b70c00640 1 -- 172.21.15.1:0/2471627781 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c010880 con 0x564b3d994540 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.515+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2471627781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b74054680 con 0x564b3d994540 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.515+0000 7f3b70c00640 1 -- 172.21.15.1:0/2471627781 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f3b7c036480 con 0x564b3d994540 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.515+0000 7f3b70c00640 1 -- 172.21.15.1:0/2471627781 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3b7c043050 con 0x564b3d994540 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.515+0000 7f3b8ce00740 10 client.26036 fetch_fsmap learned FSMap version 36 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.515+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2471627781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564b3d89fcb0 con 0x564b3d994540 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.515+0000 7f3b70c00640 1 -- 172.21.15.1:0/2471627781 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b74054680 con 0x564b3d994540 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.516+0000 7f3b70c00640 1 -- 172.21.15.1:0/2471627781 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f3b7c026050 con 0x564b3d994540 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.516+0000 7f3b8ce00740 10 client.26036 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.516+0000 7f3b8ce00740 10 client.26036 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.516+0000 7f3b8ce00740 20 client.26036 populate_metadata read hostname 'smithi001' 2024-04-26T21:36:46.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.516+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564b3da06ad0 0x564b3da08ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.516+0000 7f3b8ce00740 4 client.26036 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:36:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.516+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2471627781 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564b3d8ece00 con 0x564b3da06ad0 2024-04-26T21:36:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.516+0000 7f3b83400640 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564b3da06ad0 0x564b3da08ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.517+0000 7f3b83400640 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564b3da06ad0 0x564b3da08ef0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc992255e0d8bd81 server_cookie=f0f5486caafa3e29 in_seq=0 out_seq=0 2024-04-26T21:36:46.520 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.517+0000 7f3b70c00640 10 client.26036 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:46.563 INFO:tasks.workunit.client.0.smithi001.stdout: 1 198749 4.37 MB/sec execute 164 sec latency 23.146 ms 2024-04-26T21:36:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b70c00640 1 -- 172.21.15.1:0/2471627781 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564b3d8ece00 con 0x564b3da06ad0 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b70c00640 10 client.26036 handle_command_reply: tid=0 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 client.26036 shutdown 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2471627781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d994540 msgr2=0x564b3d9949c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d994540 0x564b3d9949c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009f80 tx=0x7f3b7c028670 comp rx=0 tx=0).stop 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2471627781 shutdown_connections 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564b3da06ad0 0x564b3da08ef0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564b3d98f5d0 0x564b3d997ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564b3d994540 0x564b3d9949c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 --2- 172.21.15.1:0/2471627781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564b3d980820 0x564b3d997570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b70c00640 0 client.26036 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2471627781 >> 172.21.15.1:0/2471627781 conn(0x564b3d97c4f0 msgr2=0x564b3d982250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2471627781 shutdown_connections 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.576+0000 7f3b8ce00740 1 -- 172.21.15.1:0/2471627781 wait complete. 2024-04-26T21:36:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.577+0000 7f3b8ce00740 20 client.26036 trim_cache size 0 max 16384 2024-04-26T21:36:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:36:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:36:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:36:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:36:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.577+0000 7f3b88a00640 1 -- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b680a3d80 msgr2=0x7f3b680a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.577+0000 7f3b88a00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b680a3d80 0x7f3b680a6240 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c005fd0 tx=0x7f3b7c009500 comp rx=0 tx=0).stop 2024-04-26T21:36:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.577+0000 7f3b88a00640 1 -- 172.21.15.1:0/1515522861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b84104230 msgr2=0x7f3b841994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.577+0000 7f3b88a00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b84104230 0x7f3b841994d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00e9f0 tx=0x7f3b6c00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.578+0000 7f3b88a00640 1 -- 172.21.15.1:0/1515522861 shutdown_connections 2024-04-26T21:36:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.578+0000 7f3b88a00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b680a3d80 0x7f3b680a6240 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.578+0000 7f3b88a00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b8410cd30 0x7f3b8419d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.578+0000 7f3b88a00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b84104230 0x7f3b841994d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.578+0000 7f3b88a00640 1 --2- 172.21.15.1:0/1515522861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b84103910 0x7f3b84198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:46.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.578+0000 7f3b88a00640 1 -- 172.21.15.1:0/1515522861 >> 172.21.15.1:0/1515522861 conn(0x7f3b840fd110 msgr2=0x7f3b840fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:46.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.578+0000 7f3b88a00640 1 -- 172.21.15.1:0/1515522861 shutdown_connections 2024-04-26T21:36:46.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:46.578+0000 7f3b88a00640 1 -- 172.21.15.1:0/1515522861 wait complete. 2024-04-26T21:36:47.563 INFO:tasks.workunit.client.0.smithi001.stdout: 1 199083 4.36 MB/sec execute 165 sec latency 27.258 ms 2024-04-26T21:36:48.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:47 smithi186 ceph-mon[32991]: pgmap v197: 129 pgs: 129 active+clean; 115 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-26T21:36:48.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:47 smithi001 ceph-mon[28054]: pgmap v197: 129 pgs: 129 active+clean; 115 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-26T21:36:48.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:47 smithi038 ceph-mon[32724]: pgmap v197: 129 pgs: 129 active+clean; 115 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-26T21:36:48.563 INFO:tasks.workunit.client.0.smithi001.stdout: 1 199711 4.34 MB/sec execute 166 sec latency 22.728 ms 2024-04-26T21:36:49.563 INFO:tasks.workunit.client.0.smithi001.stdout: 1 201094 4.35 MB/sec execute 167 sec latency 29.358 ms 2024-04-26T21:36:50.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:49 smithi186 ceph-mon[32991]: pgmap v198: 129 pgs: 129 active+clean; 115 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-26T21:36:50.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:49 smithi001 ceph-mon[28054]: pgmap v198: 129 pgs: 129 active+clean; 115 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-26T21:36:50.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:49 smithi038 ceph-mon[32724]: pgmap v198: 129 pgs: 129 active+clean; 115 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 1.3 MiB/s wr, 124 op/s 2024-04-26T21:36:50.563 INFO:tasks.workunit.client.0.smithi001.stdout: 1 202246 4.38 MB/sec execute 168 sec latency 46.005 ms 2024-04-26T21:36:51.563 INFO:tasks.workunit.client.0.smithi001.stdout: 1 202609 4.37 MB/sec execute 169 sec latency 31.832 ms 2024-04-26T21:36:52.128 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:51 smithi038 ceph-mon[32724]: pgmap v199: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 212 op/s 2024-04-26T21:36:52.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:51 smithi186 ceph-mon[32991]: pgmap v199: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 212 op/s 2024-04-26T21:36:52.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:51 smithi001 ceph-mon[28054]: pgmap v199: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 212 op/s 2024-04-26T21:36:52.563 INFO:tasks.workunit.client.0.smithi001.stdout: 1 203235 4.35 MB/sec execute 170 sec latency 27.719 ms 2024-04-26T21:36:53.563 INFO:tasks.workunit.client.0.smithi001.stdout: 1 204605 4.36 MB/sec execute 171 sec latency 23.209 ms 2024-04-26T21:36:54.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:53 smithi186 ceph-mon[32991]: pgmap v200: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-04-26T21:36:54.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:53 smithi001 ceph-mon[28054]: pgmap v200: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-04-26T21:36:54.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:53 smithi038 ceph-mon[32724]: pgmap v200: 129 pgs: 129 active+clean; 118 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.4 MiB/s wr, 151 op/s 2024-04-26T21:36:54.564 INFO:tasks.workunit.client.0.smithi001.stdout: 1 205713 4.39 MB/sec execute 172 sec latency 23.066 ms 2024-04-26T21:36:55.564 INFO:tasks.workunit.client.0.smithi001.stdout: 1 206132 4.37 MB/sec execute 173 sec latency 71.648 ms 2024-04-26T21:36:56.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:55 smithi186 ceph-mon[32991]: pgmap v201: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-26T21:36:56.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:55 smithi001 ceph-mon[28054]: pgmap v201: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-26T21:36:56.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:55 smithi038 ceph-mon[32724]: pgmap v201: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 1.9 MiB/s wr, 212 op/s 2024-04-26T21:36:56.564 INFO:tasks.workunit.client.0.smithi001.stdout: 1 206855 4.36 MB/sec execute 174 sec latency 21.277 ms 2024-04-26T21:36:57.564 INFO:tasks.workunit.client.0.smithi001.stdout: 1 208179 4.37 MB/sec execute 175 sec latency 25.185 ms 2024-04-26T21:36:58.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:57 smithi186 ceph-mon[32991]: pgmap v202: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-26T21:36:58.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:57 smithi001 ceph-mon[28054]: pgmap v202: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-26T21:36:58.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:57 smithi038 ceph-mon[32724]: pgmap v202: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-26T21:36:58.564 INFO:tasks.workunit.client.0.smithi001.stdout: 1 209337 4.40 MB/sec execute 176 sec latency 25.649 ms 2024-04-26T21:36:59.564 INFO:tasks.workunit.client.0.smithi001.stdout: 1 209685 4.38 MB/sec execute 177 sec latency 25.869 ms 2024-04-26T21:36:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.748+0000 7f8dc2600640 1 -- 172.21.15.1:0/1013219150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dbc10ac10 msgr2=0x7f8dbc10aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.748+0000 7f8dc2600640 1 --2- 172.21.15.1:0/1013219150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dbc10ac10 0x7f8dbc10aff0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009930 tx=0x7f8dac02a420 comp rx=0 tx=0).stop 2024-04-26T21:36:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.748+0000 7f8dc2600640 1 -- 172.21.15.1:0/1013219150 shutdown_connections 2024-04-26T21:36:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.748+0000 7f8dc2600640 1 --2- 172.21.15.1:0/1013219150 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dbc104b50 0x7f8dbc104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.748+0000 7f8dc2600640 1 --2- 172.21.15.1:0/1013219150 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dbc105510 0x7f8dbc1119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.748+0000 7f8dc2600640 1 --2- 172.21.15.1:0/1013219150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dbc10ac10 0x7f8dbc10aff0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.748+0000 7f8dc2600640 1 -- 172.21.15.1:0/1013219150 >> 172.21.15.1:0/1013219150 conn(0x7f8dbc100620 msgr2=0x7f8dbc102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.748+0000 7f8dc2600640 1 -- 172.21.15.1:0/1013219150 shutdown_connections 2024-04-26T21:36:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.748+0000 7f8dc2600640 1 -- 172.21.15.1:0/1013219150 wait complete. 2024-04-26T21:36:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.750+0000 7f8dc2600640 1 Processor -- start 2024-04-26T21:36:59.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.750+0000 7f8dc2600640 1 -- start start 2024-04-26T21:36:59.752 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.750+0000 7f8dc2600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dbc104b50 0x7f8dbc1118b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.751+0000 7f8dc2600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dbc105510 0x7f8dbc10e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.751+0000 7f8dc2600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dbc10ac10 0x7f8dbc10eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.751+0000 7f8dc2600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc114e10 con 0x7f8dbc104b50 2024-04-26T21:36:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.751+0000 7f8dc2600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc114c90 con 0x7f8dbc105510 2024-04-26T21:36:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.751+0000 7f8dc2600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc114f90 con 0x7f8dbc10ac10 2024-04-26T21:36:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.751+0000 7f8dbbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dbc10ac10 0x7f8dbc10eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.751+0000 7f8dbbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dbc10ac10 0x7f8dbc10eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42264/0 (socket says 172.21.15.1:42264) 2024-04-26T21:36:59.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.751+0000 7f8dbbe00640 1 -- 172.21.15.1:0/2100817022 learned_addr learned my addr 172.21.15.1:0/2100817022 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.752+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dbc105510 0x7f8dbc10e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.752+0000 7f8dbbe00640 1 -- 172.21.15.1:0/2100817022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dbc105510 msgr2=0x7f8dbc10e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.752+0000 7f8dbbe00640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dbc105510 0x7f8dbc10e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.752+0000 7f8dbbe00640 1 -- 172.21.15.1:0/2100817022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dbc104b50 msgr2=0x7f8dbc1118b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.752+0000 7f8dbb400640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dbc104b50 0x7f8dbc1118b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.753+0000 7f8dbbe00640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dbc104b50 0x7f8dbc1118b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.753+0000 7f8dbbe00640 1 -- 172.21.15.1:0/2100817022 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dbc10f790 con 0x7f8dbc10ac10 2024-04-26T21:36:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.753+0000 7f8dbbe00640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dbc10ac10 0x7f8dbc10eed0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8db400eb10 tx=0x7f8db400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.754+0000 7f8db3e00640 1 -- 172.21.15.1:0/2100817022 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4004b20 con 0x7f8dbc10ac10 2024-04-26T21:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.754+0000 7f8db3e00640 1 -- 172.21.15.1:0/2100817022 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8db4004cc0 con 0x7f8dbc10ac10 2024-04-26T21:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.754+0000 7f8db3e00640 1 -- 172.21.15.1:0/2100817022 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4004b20 con 0x7f8dbc10ac10 2024-04-26T21:36:59.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.755+0000 7f8dc2600640 1 -- 172.21.15.1:0/2100817022 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dbc1abc30 con 0x7f8dbc10ac10 2024-04-26T21:36:59.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.755+0000 7f8dc2600640 1 -- 172.21.15.1:0/2100817022 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dbc1ac130 con 0x7f8dbc10ac10 2024-04-26T21:36:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.761+0000 7f8dc547b740 1 Processor -- start 2024-04-26T21:36:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.762+0000 7f8db3e00640 1 -- 172.21.15.1:0/2100817022 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f8db4005000 con 0x7f8dbc10ac10 2024-04-26T21:36:59.763 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.762+0000 7f8dc547b740 1 -- start start 2024-04-26T21:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.763+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 0x55bb0d5861b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.763+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d5866f0 0x55bb0d590a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.763+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 0x55bb0d5933a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.763+0000 7f8dc547b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55bb0d4eb380 con 0x55bb0d590fb0 2024-04-26T21:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.763+0000 7f8dc547b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bb0d487c30 con 0x55bb0d5866f0 2024-04-26T21:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.763+0000 7f8dc547b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bb0d4e7340 con 0x55bb0d585dd0 2024-04-26T21:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.763+0000 7f8dbaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 0x55bb0d5861b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.763+0000 7f8dbaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 0x55bb0d5861b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42266/0 (socket says 172.21.15.1:42266) 2024-04-26T21:36:59.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.763+0000 7f8dbaa00640 1 -- 172.21.15.1:0/769904334 learned_addr learned my addr 172.21.15.1:0/769904334 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.763+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/769904334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 0x55bb0d5933a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.764+0000 7f8dbb400640 1 --2- 172.21.15.1:0/769904334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d5866f0 0x55bb0d590a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.764+0000 7f8dbaa00640 1 -- 172.21.15.1:0/769904334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d5866f0 msgr2=0x55bb0d590a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.764+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/769904334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d5866f0 0x55bb0d590a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.764+0000 7f8dbaa00640 1 -- 172.21.15.1:0/769904334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 msgr2=0x55bb0d5933a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.764+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/769904334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 0x55bb0d5933a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.764+0000 7f8dbaa00640 1 -- 172.21.15.1:0/769904334 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb0d4a9940 con 0x55bb0d585dd0 2024-04-26T21:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.765+0000 7f8db3e00640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8da00a3fd0 0x7f8da00a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.765+0000 7f8db3e00640 1 -- 172.21.15.1:0/2100817022 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f8db40d0ec0 con 0x7f8dbc10ac10 2024-04-26T21:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.765+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/769904334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 0x55bb0d5861b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009fd0 tx=0x7f8da8005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba589f3137172525 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:59.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.765+0000 7f8dbb400640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8da00a3fd0 0x7f8da00a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.766+0000 7f8db0c00640 1 -- 172.21.15.1:0/769904334 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8004030 con 0x55bb0d585dd0 2024-04-26T21:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.766+0000 7f8db0c00640 1 -- 172.21.15.1:0/769904334 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da8005020 con 0x55bb0d585dd0 2024-04-26T21:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.766+0000 7f8db0c00640 1 -- 172.21.15.1:0/769904334 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8004a90 con 0x55bb0d585dd0 2024-04-26T21:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.767+0000 7f8dc547b740 1 -- 172.21.15.1:0/769904334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 msgr2=0x55bb0d5861b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.767+0000 7f8dbb400640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8da00a3fd0 0x7f8da00a6490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009230 tx=0x7f8dac031400 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:59.768 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.767+0000 7f8dc547b740 1 --2- 172.21.15.1:0/769904334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 0x55bb0d5861b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009fd0 tx=0x7f8da8005850 comp rx=0 tx=0).stop 2024-04-26T21:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.767+0000 7f8dc547b740 1 -- 172.21.15.1:0/769904334 shutdown_connections 2024-04-26T21:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.767+0000 7f8dc547b740 1 --2- 172.21.15.1:0/769904334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 0x55bb0d5933a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.767+0000 7f8dc547b740 1 --2- 172.21.15.1:0/769904334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d5866f0 0x55bb0d590a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.767+0000 7f8dc547b740 1 --2- 172.21.15.1:0/769904334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 0x55bb0d5861b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.767+0000 7f8dc547b740 1 -- 172.21.15.1:0/769904334 >> 172.21.15.1:0/769904334 conn(0x55bb0d5822e0 msgr2=0x55bb0d5826c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.767+0000 7f8dc547b740 1 -- 172.21.15.1:0/769904334 shutdown_connections 2024-04-26T21:36:59.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.768+0000 7f8dc547b740 1 -- 172.21.15.1:0/769904334 wait complete. 2024-04-26T21:36:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.768+0000 7f8dc547b740 1 -- start start 2024-04-26T21:36:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.769+0000 7f8dc547b740 1 Processor -- start 2024-04-26T21:36:59.770 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.769+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d585dd0 0x55bb0d58df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.769+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 0x55bb0d58e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.769+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 0x55bb0d58e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.769+0000 7f8dc547b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55bb0d593be0 con 0x55bb0d590fb0 2024-04-26T21:36:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.769+0000 7f8dc547b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55bb0d472c60 con 0x55bb0d585dd0 2024-04-26T21:36:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.769+0000 7f8dc547b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bb0d593d60 con 0x55bb0d5866f0 2024-04-26T21:36:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.769+0000 7f8dc547b740 10 client.0 _ll_register_callbacks cb 0x55bb0d49ee50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:36:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.769+0000 7f8dbb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d585dd0 0x55bb0d58df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d585dd0 0x55bb0d58df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40852/0 (socket says 172.21.15.1:40852) 2024-04-26T21:36:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbb400640 1 -- 172.21.15.1:0/3903675473 learned_addr learned my addr 172.21.15.1:0/3903675473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbbe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 0x55bb0d58e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 0x55bb0d58e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbb400640 1 -- 172.21.15.1:0/3903675473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 msgr2=0x55bb0d58e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbb400640 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 0x55bb0d58e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbb400640 1 -- 172.21.15.1:0/3903675473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 msgr2=0x55bb0d58e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbb400640 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 0x55bb0d58e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbb400640 1 -- 172.21.15.1:0/3903675473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb0d481510 con 0x55bb0d585dd0 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbbe00640 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 0x55bb0d58e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8dbb400640 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d585dd0 0x55bb0d58df20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009930 tx=0x7f8dac02a420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72ceafe6f3c3cbc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.770+0000 7f8da5600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.771+0000 7f8da5600640 1 -- 172.21.15.1:0/3903675473 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac04d020 con 0x55bb0d585dd0 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.771+0000 7f8dc547b740 1 -- 172.21.15.1:0/3903675473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8da8005020 con 0x55bb0d585dd0 2024-04-26T21:36:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.771+0000 7f8da5600640 1 -- 172.21.15.1:0/3903675473 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f8dac0048c0 con 0x55bb0d585dd0 2024-04-26T21:36:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.771+0000 7f8da5600640 1 -- 172.21.15.1:0/3903675473 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8dac058050 con 0x55bb0d585dd0 2024-04-26T21:36:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.771+0000 7f8dc547b740 10 client.26048 fetch_fsmap learned FSMap version 36 2024-04-26T21:36:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.771+0000 7f8dc547b740 1 -- 172.21.15.1:0/3903675473 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55bb0d1fd210 con 0x55bb0d585dd0 2024-04-26T21:36:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.772+0000 7f8da5600640 1 -- 172.21.15.1:0/3903675473 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da8005020 con 0x55bb0d585dd0 2024-04-26T21:36:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.772+0000 7f8da5600640 1 -- 172.21.15.1:0/3903675473 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f8dac003ca0 con 0x55bb0d585dd0 2024-04-26T21:36:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.772+0000 7f8dc547b740 10 client.26048 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:36:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.772+0000 7f8dc547b740 10 client.26048 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:36:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.772+0000 7f8dc547b740 20 client.26048 populate_metadata read hostname 'smithi001' 2024-04-26T21:36:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.772+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bb0d5a0dd0 0x55bb0d5a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.772+0000 7f8dc547b740 4 client.26048 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:36:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.773+0000 7f8dc547b740 1 -- 172.21.15.1:0/3903675473 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb0d4f8f90 con 0x55bb0d5a0dd0 2024-04-26T21:36:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.773+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bb0d5a0dd0 0x55bb0d5a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.773+0000 7f8da5600640 10 client.26048 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:59.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.773+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bb0d5a0dd0 0x55bb0d5a31c0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c2344184e0aa782 server_cookie=198a2dba1757c19d in_seq=0 out_seq=0 2024-04-26T21:36:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.774+0000 7f8da5600640 1 -- 172.21.15.1:0/3903675473 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb0d4f8f90 con 0x55bb0d5a0dd0 2024-04-26T21:36:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.774+0000 7f8da5600640 10 client.26048 handle_command_reply: tid=0 2024-04-26T21:36:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 client.26048 shutdown 2024-04-26T21:36:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 -- 172.21.15.1:0/3903675473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d585dd0 msgr2=0x55bb0d58df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d585dd0 0x55bb0d58df20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009930 tx=0x7f8dac02a420 comp rx=0 tx=0).stop 2024-04-26T21:36:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 -- 172.21.15.1:0/3903675473 shutdown_connections 2024-04-26T21:36:59.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bb0d5a0dd0 0x55bb0d5a31c0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 0x55bb0d58e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d590fb0 0x55bb0d58e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8da5600640 0 client.26048 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3903675473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d585dd0 0x55bb0d58df20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 -- 172.21.15.1:0/3903675473 >> 172.21.15.1:0/3903675473 conn(0x55bb0d5822e0 msgr2=0x55bb0d588010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 -- 172.21.15.1:0/3903675473 shutdown_connections 2024-04-26T21:36:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.775+0000 7f8dc547b740 1 -- 172.21.15.1:0/3903675473 wait complete. 2024-04-26T21:36:59.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.776+0000 7f8dc547b740 20 client.26048 trim_cache size 0 max 16384 2024-04-26T21:36:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dc547b740 1 Processor -- start 2024-04-26T21:36:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dc547b740 1 -- start start 2024-04-26T21:36:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d585dd0 0x55bb0d59caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 0x55bb0d5812c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 0x55bb0d5a8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dc547b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc1b8d00 con 0x55bb0d585dd0 2024-04-26T21:36:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dc547b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc1b65d0 con 0x55bb0d590fb0 2024-04-26T21:36:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dc547b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc1b4420 con 0x55bb0d5866f0 2024-04-26T21:36:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dbaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 0x55bb0d5812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dbb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d585dd0 0x55bb0d59caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dbaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 0x55bb0d5812c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42282/0 (socket says 172.21.15.1:42282) 2024-04-26T21:36:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.796+0000 7f8dbaa00640 1 -- 172.21.15.1:0/2597070490 learned_addr learned my addr 172.21.15.1:0/2597070490 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:59.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8dbbe00640 1 --2- 172.21.15.1:0/2597070490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 0x55bb0d5a8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8dbaa00640 1 -- 172.21.15.1:0/2597070490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 msgr2=0x55bb0d5a8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/2597070490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 0x55bb0d5a8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8dbaa00640 1 -- 172.21.15.1:0/2597070490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d585dd0 msgr2=0x55bb0d59caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/2597070490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d585dd0 0x55bb0d59caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8dbaa00640 1 -- 172.21.15.1:0/2597070490 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb0d2c9620 con 0x55bb0d5866f0 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8dbb400640 1 --2- 172.21.15.1:0/2597070490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d585dd0 0x55bb0d59caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/2597070490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 0x55bb0d5812c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8da80117f0 tx=0x7f8da8009c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1ca9180ed0583ca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8da7e00640 1 -- 172.21.15.1:0/2597070490 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8011020 con 0x55bb0d5866f0 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8da7e00640 1 -- 172.21.15.1:0/2597070490 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da8005a70 con 0x55bb0d5866f0 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.797+0000 7f8da7e00640 1 -- 172.21.15.1:0/2597070490 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da80035f0 con 0x55bb0d5866f0 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.798+0000 7f8dc547b740 1 -- 172.21.15.1:0/2597070490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 msgr2=0x55bb0d5812c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.798+0000 7f8dc547b740 1 --2- 172.21.15.1:0/2597070490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 0x55bb0d5812c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8da80117f0 tx=0x7f8da8009c70 comp rx=0 tx=0).stop 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.798+0000 7f8dc547b740 1 -- 172.21.15.1:0/2597070490 shutdown_connections 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.798+0000 7f8dc547b740 1 --2- 172.21.15.1:0/2597070490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d5866f0 0x55bb0d5812c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.798+0000 7f8dc547b740 1 --2- 172.21.15.1:0/2597070490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 0x55bb0d5a8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.798+0000 7f8dc547b740 1 --2- 172.21.15.1:0/2597070490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d585dd0 0x55bb0d59caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.798+0000 7f8dc547b740 1 -- 172.21.15.1:0/2597070490 >> 172.21.15.1:0/2597070490 conn(0x55bb0d5822e0 msgr2=0x55bb0d580bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.799+0000 7f8dc547b740 1 -- 172.21.15.1:0/2597070490 shutdown_connections 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.799+0000 7f8dc547b740 1 -- 172.21.15.1:0/2597070490 wait complete. 2024-04-26T21:36:59.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.799+0000 7f8dc547b740 1 -- start start 2024-04-26T21:36:59.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.799+0000 7f8dc547b740 1 Processor -- start 2024-04-26T21:36:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 0x55bb0d58aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d5866f0 0x55bb0d58b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dc547b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 0x55bb0d58b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dc547b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc1b4a90 con 0x55bb0d5866f0 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dc547b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc1b5f00 con 0x55bb0d590fb0 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dc547b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8dbc1b5160 con 0x55bb0d585dd0 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dc547b740 10 client.0 _ll_register_callbacks cb 0x7f8dac040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dbbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 0x55bb0d58b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dbbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 0x55bb0d58b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40866/0 (socket says 172.21.15.1:40866) 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dbbe00640 1 -- 172.21.15.1:0/3969520833 learned_addr learned my addr 172.21.15.1:0/3969520833 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dbb400640 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 0x55bb0d58aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dbaa00640 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d5866f0 0x55bb0d58b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dbbe00640 1 -- 172.21.15.1:0/3969520833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 msgr2=0x55bb0d58aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dbbe00640 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 0x55bb0d58aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dbbe00640 1 -- 172.21.15.1:0/3969520833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d5866f0 msgr2=0x55bb0d58b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dbbe00640 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d5866f0 0x55bb0d58b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.800+0000 7f8dbbe00640 1 -- 172.21.15.1:0/3969520833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb0d4a5cb0 con 0x55bb0d590fb0 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.801+0000 7f8dbbe00640 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 0x55bb0d58b980 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8db40ce200 tx=0x7f8db409c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3afdd1c1efb5a7e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.801+0000 7f8da5600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.801+0000 7f8da5600640 1 -- 172.21.15.1:0/3969520833 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db40b7070 con 0x55bb0d590fb0 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.801+0000 7f8da5600640 1 -- 172.21.15.1:0/3969520833 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f8db40bbdb0 con 0x55bb0d590fb0 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.801+0000 7f8dc547b740 1 -- 172.21.15.1:0/3969520833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8da8005a70 con 0x55bb0d590fb0 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.801+0000 7f8da5600640 1 -- 172.21.15.1:0/3969520833 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8db40a0da0 con 0x55bb0d590fb0 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.802+0000 7f8dc547b740 10 client.26054 fetch_fsmap learned FSMap version 36 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.802+0000 7f8dc547b740 1 -- 172.21.15.1:0/3969520833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55bb0d607760 con 0x55bb0d590fb0 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.802+0000 7f8da5600640 1 -- 172.21.15.1:0/3969520833 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8da8005a70 con 0x55bb0d590fb0 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.802+0000 7f8da5600640 1 -- 172.21.15.1:0/3969520833 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f8db4098740 con 0x55bb0d590fb0 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.802+0000 7f8dc547b740 10 client.26054 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.802+0000 7f8dc547b740 10 client.26054 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.802+0000 7f8dc547b740 20 client.26054 populate_metadata read hostname 'smithi001' 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.802+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bb0d607ed0 0x55bb0d60a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:36:59.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.802+0000 7f8dc547b740 4 client.26054 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:36:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.802+0000 7f8dc547b740 1 -- 172.21.15.1:0/3969520833 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bb0d4f2e00 con 0x55bb0d607ed0 2024-04-26T21:36:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.803+0000 7f8dbb400640 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bb0d607ed0 0x55bb0d60a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:36:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.803+0000 7f8dbb400640 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bb0d607ed0 0x55bb0d60a2c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dec37a58ec205db7 server_cookie=17295e025d10cda2 in_seq=0 out_seq=0 2024-04-26T21:36:59.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.803+0000 7f8da5600640 10 client.26054 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8da5600640 1 -- 172.21.15.1:0/3969520833 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bb0d4f2e00 con 0x55bb0d607ed0 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8da5600640 10 client.26054 handle_command_reply: tid=0 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8dc547b740 1 client.26054 shutdown 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8dc547b740 1 -- 172.21.15.1:0/3969520833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 msgr2=0x55bb0d58b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 0x55bb0d58b980 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8db40ce200 tx=0x7f8db409c440 comp rx=0 tx=0).stop 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8dc547b740 1 -- 172.21.15.1:0/3969520833 shutdown_connections 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55bb0d607ed0 0x55bb0d60a2c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb0d590fb0 0x55bb0d58b980 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8da5600640 0 client.26054 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55bb0d5866f0 0x55bb0d58b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8dc547b740 1 --2- 172.21.15.1:0/3969520833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bb0d585dd0 0x55bb0d58aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.863+0000 7f8dc547b740 1 -- 172.21.15.1:0/3969520833 >> 172.21.15.1:0/3969520833 conn(0x55bb0d5822e0 msgr2=0x55bb0d588010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.864+0000 7f8dc547b740 1 -- 172.21.15.1:0/3969520833 shutdown_connections 2024-04-26T21:36:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.864+0000 7f8dc547b740 1 -- 172.21.15.1:0/3969520833 wait complete. 2024-04-26T21:36:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.864+0000 7f8dc547b740 20 client.26054 trim_cache size 0 max 16384 2024-04-26T21:36:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:36:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:36:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:36:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:36:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 -- 172.21.15.1:0/2100817022 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8da00a3fd0 msgr2=0x7f8da00a6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8da00a3fd0 0x7f8da00a6490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009230 tx=0x7f8dac031400 comp rx=0 tx=0).stop 2024-04-26T21:36:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 -- 172.21.15.1:0/2100817022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dbc10ac10 msgr2=0x7f8dbc10eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:36:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dbc10ac10 0x7f8dbc10eed0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8db400eb10 tx=0x7f8db400c2d0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 -- 172.21.15.1:0/2100817022 shutdown_connections 2024-04-26T21:36:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8da00a3fd0 0x7f8da00a6490 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8dbc10ac10 0x7f8dbc10eed0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dbc105510 0x7f8dbc10e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 --2- 172.21.15.1:0/2100817022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dbc104b50 0x7f8dbc1118b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:36:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 -- 172.21.15.1:0/2100817022 >> 172.21.15.1:0/2100817022 conn(0x7f8dbc100620 msgr2=0x7f8dbc101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:36:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 -- 172.21.15.1:0/2100817022 shutdown_connections 2024-04-26T21:36:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:36:59.865+0000 7f8dc2600640 1 -- 172.21.15.1:0/2100817022 wait complete. 2024-04-26T21:37:00.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:36:59 smithi186 ceph-mon[32991]: pgmap v203: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-26T21:37:00.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:36:59 smithi001 ceph-mon[28054]: pgmap v203: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-26T21:37:00.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:36:59 smithi038 ceph-mon[32724]: pgmap v203: 129 pgs: 129 active+clean; 120 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 1.2 MiB/s wr, 148 op/s 2024-04-26T21:37:00.565 INFO:tasks.workunit.client.0.smithi001.stdout: 1 210286 4.36 MB/sec execute 178 sec latency 37.061 ms 2024-04-26T21:37:01.565 INFO:tasks.workunit.client.0.smithi001.stdout: 1 210312 4.35 MB/sec execute 179 sec latency 175.331 ms 2024-04-26T21:37:02.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:01 smithi186 ceph-mon[32991]: pgmap v204: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-04-26T21:37:02.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:01 smithi001 ceph-mon[28054]: pgmap v204: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-04-26T21:37:02.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:01 smithi038 ceph-mon[32724]: pgmap v204: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-04-26T21:37:02.565 INFO:tasks.workunit.client.0.smithi001.stdout: 1 211633 4.35 MB/sec execute 180 sec latency 102.969 ms 2024-04-26T21:37:03.565 INFO:tasks.workunit.client.0.smithi001.stdout: 1 212485 4.36 MB/sec execute 181 sec latency 25.445 ms 2024-04-26T21:37:04.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:03 smithi186 ceph-mon[32991]: pgmap v205: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-26T21:37:04.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:03 smithi001 ceph-mon[28054]: pgmap v205: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-26T21:37:04.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:03 smithi038 ceph-mon[32724]: pgmap v205: 129 pgs: 129 active+clean; 123 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 1.1 MiB/s wr, 124 op/s 2024-04-26T21:37:04.565 INFO:tasks.workunit.client.0.smithi001.stdout: 1 213160 4.37 MB/sec execute 182 sec latency 24.200 ms 2024-04-26T21:37:05.566 INFO:tasks.workunit.client.0.smithi001.stdout: 1 213599 4.35 MB/sec execute 183 sec latency 34.589 ms 2024-04-26T21:37:06.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:05 smithi186 ceph-mon[32991]: pgmap v206: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 238 op/s 2024-04-26T21:37:06.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:05 smithi001 ceph-mon[28054]: pgmap v206: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 238 op/s 2024-04-26T21:37:06.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:05 smithi038 ceph-mon[32724]: pgmap v206: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 238 op/s 2024-04-26T21:37:06.565 INFO:tasks.workunit.client.0.smithi001.stdout: 1 214479 4.35 MB/sec execute 184 sec latency 78.498 ms 2024-04-26T21:37:07.566 INFO:tasks.workunit.client.0.smithi001.stdout: 1 215789 4.37 MB/sec execute 185 sec latency 60.888 ms 2024-04-26T21:37:08.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:07 smithi186 ceph-mon[32991]: pgmap v207: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-04-26T21:37:08.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:07 smithi186 ceph-mon[32991]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T21:37:08.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:07 smithi001 ceph-mon[28054]: pgmap v207: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-04-26T21:37:08.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:07 smithi001 ceph-mon[28054]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T21:37:08.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:07 smithi038 ceph-mon[32724]: pgmap v207: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 1.4 MiB/s wr, 177 op/s 2024-04-26T21:37:08.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:07 smithi038 ceph-mon[32724]: osdmap e97: 12 total, 12 up, 12 in 2024-04-26T21:37:08.566 INFO:tasks.workunit.client.0.smithi001.stdout: 1 216655 4.38 MB/sec execute 186 sec latency 25.784 ms 2024-04-26T21:37:09.566 INFO:tasks.workunit.client.0.smithi001.stdout: 1 216885 4.35 MB/sec execute 187 sec latency 25.676 ms 2024-04-26T21:37:10.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:09 smithi186 ceph-mon[32991]: pgmap v209: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-04-26T21:37:10.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:09 smithi001 ceph-mon[28054]: pgmap v209: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-04-26T21:37:10.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:09 smithi038 ceph-mon[32724]: pgmap v209: 129 pgs: 129 active+clean; 121 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 1.7 MiB/s wr, 212 op/s 2024-04-26T21:37:10.566 INFO:tasks.workunit.client.0.smithi001.stdout: 1 217482 4.34 MB/sec execute 188 sec latency 33.744 ms 2024-04-26T21:37:11.566 INFO:tasks.workunit.client.0.smithi001.stdout: 1 218830 4.35 MB/sec execute 189 sec latency 32.873 ms 2024-04-26T21:37:12.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:11 smithi186 ceph-mon[32991]: pgmap v210: 129 pgs: 129 active+clean; 124 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.7 MiB/s wr, 202 op/s 2024-04-26T21:37:12.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:11 smithi001 ceph-mon[28054]: pgmap v210: 129 pgs: 129 active+clean; 124 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.7 MiB/s wr, 202 op/s 2024-04-26T21:37:12.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:11 smithi038 ceph-mon[32724]: pgmap v210: 129 pgs: 129 active+clean; 124 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.7 MiB/s wr, 202 op/s 2024-04-26T21:37:12.566 INFO:tasks.workunit.client.0.smithi001.stdout: 1 219772 4.36 MB/sec execute 190 sec latency 30.737 ms 2024-04-26T21:37:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.026+0000 7f39a1c00640 1 -- 172.21.15.1:0/3169827805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f399c108760 msgr2=0x7f399c108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.026+0000 7f39a1c00640 1 --2- 172.21.15.1:0/3169827805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f399c108760 0x7f399c108b40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f398c009900 tx=0x7f398c02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.027+0000 7f39a1c00640 1 -- 172.21.15.1:0/3169827805 shutdown_connections 2024-04-26T21:37:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.027+0000 7f39a1c00640 1 --2- 172.21.15.1:0/3169827805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f399c103110 0x7f399c10f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.027+0000 7f39a1c00640 1 --2- 172.21.15.1:0/3169827805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f399c102750 0x7f399c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.027+0000 7f39a1c00640 1 --2- 172.21.15.1:0/3169827805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f399c108760 0x7f399c108b40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.027+0000 7f39a1c00640 1 -- 172.21.15.1:0/3169827805 >> 172.21.15.1:0/3169827805 conn(0x7f399c0fe240 msgr2=0x7f399c100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.027+0000 7f39a1c00640 1 -- 172.21.15.1:0/3169827805 shutdown_connections 2024-04-26T21:37:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.027+0000 7f39a1c00640 1 -- 172.21.15.1:0/3169827805 wait complete. 2024-04-26T21:37:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.028+0000 7f39a1c00640 1 Processor -- start 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.028+0000 7f39a1c00640 1 -- start start 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.028+0000 7f39a1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f399c102750 0x7f399c074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.028+0000 7f39a1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f399c103110 0x7f399c075170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.028+0000 7f39a1c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f399c108760 0x7f399c0756b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.028+0000 7f39a1c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f399c112a70 con 0x7f399c108760 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.028+0000 7f39a1c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f399c1128f0 con 0x7f399c103110 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.028+0000 7f39a1c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f399c112bf0 con 0x7f399c102750 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f399b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f399c103110 0x7f399c075170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f399b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f399c103110 0x7f399c075170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43752/0 (socket says 172.21.15.1:43752) 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f399b400640 1 -- 172.21.15.1:0/4031639995 learned_addr learned my addr 172.21.15.1:0/4031639995 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f39a1200640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f399c108760 0x7f399c0756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f399be00640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f399c102750 0x7f399c074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f399b400640 1 -- 172.21.15.1:0/4031639995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f399c102750 msgr2=0x7f399c074c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f399b400640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f399c102750 0x7f399c074c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f399b400640 1 -- 172.21.15.1:0/4031639995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f399c108760 msgr2=0x7f399c0756b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f399b400640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f399c108760 0x7f399c0756b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f399b400640 1 -- 172.21.15.1:0/4031639995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f399c1a7430 con 0x7f399c103110 2024-04-26T21:37:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.029+0000 7f399b400640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f399c103110 0x7f399c075170 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f398800d950 tx=0x7f398800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.030+0000 7f3998c00640 1 -- 172.21.15.1:0/4031639995 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3988017030 con 0x7f399c103110 2024-04-26T21:37:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.030+0000 7f3998c00640 1 -- 172.21.15.1:0/4031639995 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3988004020 con 0x7f399c103110 2024-04-26T21:37:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.030+0000 7f3998c00640 1 -- 172.21.15.1:0/4031639995 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39880047a0 con 0x7f399c103110 2024-04-26T21:37:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.030+0000 7f39a1c00640 1 -- 172.21.15.1:0/4031639995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f399c1a7720 con 0x7f399c103110 2024-04-26T21:37:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.030+0000 7f39a1c00640 1 -- 172.21.15.1:0/4031639995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f399c1a7cc0 con 0x7f399c103110 2024-04-26T21:37:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.035+0000 7f39a5fcb740 1 Processor -- start 2024-04-26T21:37:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.035+0000 7f39a5fcb740 1 -- start start 2024-04-26T21:37:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.035+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56323fe0 0x55ac563243c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.035+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 0x55ac5632ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac563315b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ac56289380 con 0x55ac5632f1c0 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ac56225c30 con 0x55ac56323fe0 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ac56285340 con 0x55ac56324900 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f39a1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56323fe0 0x55ac563243c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f39a1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56323fe0 0x55ac563243c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43764/0 (socket says 172.21.15.1:43764) 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f39a1200640 1 -- 172.21.15.1:0/2666085073 learned_addr learned my addr 172.21.15.1:0/2666085073 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f3998c00640 1 -- 172.21.15.1:0/4031639995 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f3988009a30 con 0x7f399c103110 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f399be00640 1 --2- 172.21.15.1:0/2666085073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 0x55ac5632ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f399be00640 1 -- 172.21.15.1:0/2666085073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56323fe0 msgr2=0x55ac563243c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f399be00640 1 --2- 172.21.15.1:0/2666085073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56323fe0 0x55ac563243c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f399be00640 1 -- 172.21.15.1:0/2666085073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 msgr2=0x55ac563315b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:13.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f399be00640 1 --2- 172.21.15.1:0/2666085073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac563315b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f399be00640 1 -- 172.21.15.1:0/2666085073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac56247940 con 0x55ac56324900 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.036+0000 7f39a1200640 1 --2- 172.21.15.1:0/2666085073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56323fe0 0x55ac563243c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.037+0000 7f399be00640 1 --2- 172.21.15.1:0/2666085073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 0x55ac5632ec80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f398c02a9e0 tx=0x7f398c004180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efbee7cd7dd0f1f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.037+0000 7f3991600640 1 -- 172.21.15.1:0/2666085073 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c035040 con 0x55ac56324900 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.038+0000 7f3991600640 1 -- 172.21.15.1:0/2666085073 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f398c0054f0 con 0x55ac56324900 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.038+0000 7f3991600640 1 -- 172.21.15.1:0/2666085073 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c003850 con 0x55ac56324900 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.038+0000 7f3998c00640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f39800a3d80 0x7f39800a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.038+0000 7f3998c00640 1 -- 172.21.15.1:0/4031639995 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9730+0+0 (secure 0 0 0) 0x7f39880d0600 con 0x7f399c103110 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.038+0000 7f39a1200640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f39800a3d80 0x7f39800a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.038+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2666085073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 msgr2=0x55ac5632ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.038+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2666085073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 0x55ac5632ec80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f398c02a9e0 tx=0x7f398c004180 comp rx=0 tx=0).stop 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.039+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2666085073 shutdown_connections 2024-04-26T21:37:13.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.039+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2666085073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 0x55ac5632ec80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.039+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2666085073 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac563315b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.039+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2666085073 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56323fe0 0x55ac563243c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.039+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2666085073 >> 172.21.15.1:0/2666085073 conn(0x55ac563204f0 msgr2=0x55ac563208d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:13.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.039+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2666085073 shutdown_connections 2024-04-26T21:37:13.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.039+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2666085073 wait complete. 2024-04-26T21:37:13.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.039+0000 7f39a1200640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f39800a3d80 0x7f39800a6240 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3994009d60 tx=0x7f3994009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.040+0000 7f39a5fcb740 1 -- start start 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.040+0000 7f39a5fcb740 1 Processor -- start 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.040+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac56323fe0 0x55ac5632c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.040+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 0x55ac5632c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.040+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac5632f1c0 0x55ac5632cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ac56331d90 con 0x55ac56323fe0 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ac56210c60 con 0x55ac5632f1c0 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ac56331f10 con 0x55ac56324900 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a5fcb740 10 client.0 _ll_register_callbacks cb 0x55ac5623ce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f399be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac56323fe0 0x55ac5632c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac5632f1c0 0x55ac5632cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f399b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 0x55ac5632c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f399be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac56323fe0 0x55ac5632c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45364/0 (socket says 172.21.15.1:45364) 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac5632f1c0 0x55ac5632cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43766/0 (socket says 172.21.15.1:43766) 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f399be00640 1 -- 172.21.15.1:0/2779480421 learned_addr learned my addr 172.21.15.1:0/2779480421 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a1200640 1 -- 172.21.15.1:0/2779480421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 msgr2=0x55ac5632c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a1200640 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 0x55ac5632c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a1200640 1 -- 172.21.15.1:0/2779480421 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac56323fe0 msgr2=0x55ac5632c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a1200640 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac56323fe0 0x55ac5632c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.041+0000 7f39a1200640 1 -- 172.21.15.1:0/2779480421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac5621f510 con 0x55ac5632f1c0 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.042+0000 7f39a1200640 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac5632f1c0 0x55ac5632cbe0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3994009fd0 tx=0x7f3994004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed797555cc43928 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.042+0000 7f3986000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.042+0000 7f3986000640 1 -- 172.21.15.1:0/2779480421 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994004b20 con 0x55ac5632f1c0 2024-04-26T21:37:13.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.042+0000 7f3986000640 1 -- 172.21.15.1:0/2779480421 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9730+0+0 (secure 0 0 0) 0x7f3994017030 con 0x55ac5632f1c0 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.042+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2779480421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f398c0054f0 con 0x55ac5632f1c0 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.042+0000 7f3986000640 1 -- 172.21.15.1:0/2779480421 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3994031050 con 0x55ac5632f1c0 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.042+0000 7f39a5fcb740 10 client.26066 fetch_fsmap learned FSMap version 36 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.042+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2779480421 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ac55f9b210 con 0x55ac5632f1c0 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.043+0000 7f3986000640 1 -- 172.21.15.1:0/2779480421 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f398c0054f0 con 0x55ac5632f1c0 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.043+0000 7f3986000640 1 -- 172.21.15.1:0/2779480421 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f399401ca10 con 0x55ac5632f1c0 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.043+0000 7f39a5fcb740 10 client.26066 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.043+0000 7f39a5fcb740 10 client.26066 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.043+0000 7f39a5fcb740 20 client.26066 populate_metadata read hostname 'smithi001' 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.043+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ac5633ef60 0x55ac56341350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.043+0000 7f39a5fcb740 4 client.26066 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.043+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2779480421 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac56296f90 con 0x55ac5633ef60 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.044+0000 7f399be00640 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ac5633ef60 0x55ac56341350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.044+0000 7f399be00640 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ac5633ef60 0x55ac56341350 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95ff64448c246eb4 server_cookie=ec1fdb7b0e8926e5 in_seq=0 out_seq=0 2024-04-26T21:37:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.044+0000 7f3986000640 10 client.26066 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f3986000640 1 -- 172.21.15.1:0/2779480421 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ac56296f90 con 0x55ac5633ef60 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f3986000640 10 client.26066 handle_command_reply: tid=0 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 client.26066 shutdown 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2779480421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac5632f1c0 msgr2=0x55ac5632cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac5632f1c0 0x55ac5632cbe0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3994009fd0 tx=0x7f3994004ab0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2779480421 shutdown_connections 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ac5633ef60 0x55ac56341350 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56324900 0x55ac5632c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac5632f1c0 0x55ac5632cbe0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2779480421 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac56323fe0 0x55ac5632c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2779480421 >> 172.21.15.1:0/2779480421 conn(0x55ac563204f0 msgr2=0x55ac56326220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:13.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f3986000640 0 client.26066 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:13.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2779480421 shutdown_connections 2024-04-26T21:37:13.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.045+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2779480421 wait complete. 2024-04-26T21:37:13.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.046+0000 7f39a5fcb740 20 client.26066 trim_cache size 0 max 16384 2024-04-26T21:37:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.066+0000 7f39a5fcb740 1 Processor -- start 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.066+0000 7f39a5fcb740 1 -- start start 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.066+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56323fe0 0x55ac5633ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.066+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac56346c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.066+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac56347160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.066+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f399c1b49f0 con 0x55ac5632f1c0 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.066+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f399c1b22c0 con 0x55ac56324900 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.066+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f399c1b0130 con 0x55ac56323fe0 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.066+0000 7f399b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac56346c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.066+0000 7f399b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac56346c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43772/0 (socket says 172.21.15.1:43772) 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.067+0000 7f399b400640 1 -- 172.21.15.1:0/1573747010 learned_addr learned my addr 172.21.15.1:0/1573747010 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.067+0000 7f399be00640 1 --2- 172.21.15.1:0/1573747010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56323fe0 0x55ac5633ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.067+0000 7f39a1200640 1 --2- 172.21.15.1:0/1573747010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac56347160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.067+0000 7f399b400640 1 -- 172.21.15.1:0/1573747010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56323fe0 msgr2=0x55ac5633ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.067+0000 7f399b400640 1 --2- 172.21.15.1:0/1573747010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56323fe0 0x55ac5633ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.067+0000 7f399b400640 1 -- 172.21.15.1:0/1573747010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 msgr2=0x55ac56347160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.067+0000 7f399b400640 1 --2- 172.21.15.1:0/1573747010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac56347160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.067+0000 7f399b400640 1 -- 172.21.15.1:0/1573747010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac56067620 con 0x55ac56324900 2024-04-26T21:37:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.067+0000 7f399b400640 1 --2- 172.21.15.1:0/1573747010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac56346c20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f39880cd8f0 tx=0x7f398809bd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=454a963ab270aa15 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.068+0000 7f3990c00640 1 -- 172.21.15.1:0/1573747010 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398809c0e0 con 0x55ac56324900 2024-04-26T21:37:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.068+0000 7f3990c00640 1 -- 172.21.15.1:0/1573747010 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39880a0490 con 0x55ac56324900 2024-04-26T21:37:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.068+0000 7f3990c00640 1 -- 172.21.15.1:0/1573747010 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39880cdb10 con 0x55ac56324900 2024-04-26T21:37:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.069+0000 7f39a5fcb740 1 -- 172.21.15.1:0/1573747010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 msgr2=0x55ac56346c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.069+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/1573747010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac56346c20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f39880cd8f0 tx=0x7f398809bd40 comp rx=0 tx=0).stop 2024-04-26T21:37:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.069+0000 7f39a5fcb740 1 -- 172.21.15.1:0/1573747010 shutdown_connections 2024-04-26T21:37:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.069+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/1573747010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac56347160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.069+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/1573747010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac56346c20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.069+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/1573747010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56323fe0 0x55ac5633ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.069+0000 7f39a5fcb740 1 -- 172.21.15.1:0/1573747010 >> 172.21.15.1:0/1573747010 conn(0x55ac563204f0 msgr2=0x55ac56327440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.069+0000 7f39a5fcb740 1 -- 172.21.15.1:0/1573747010 shutdown_connections 2024-04-26T21:37:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.069+0000 7f39a5fcb740 1 -- 172.21.15.1:0/1573747010 wait complete. 2024-04-26T21:37:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.069+0000 7f39a5fcb740 1 -- start start 2024-04-26T21:37:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.070+0000 7f39a5fcb740 1 Processor -- start 2024-04-26T21:37:13.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.070+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56323fe0 0x55ac56332ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.070+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac563333e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.070+0000 7f39a5fcb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac563a4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.070+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f399c1b0780 con 0x55ac5632f1c0 2024-04-26T21:37:13.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.070+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f399c1b1bf0 con 0x55ac56324900 2024-04-26T21:37:13.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.070+0000 7f39a5fcb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f399c1b0e50 con 0x55ac56323fe0 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f39a5fcb740 10 client.0 _ll_register_callbacks cb 0x7f3994011f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f399b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac563333e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f399b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac563333e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43784/0 (socket says 172.21.15.1:43784) 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f399b400640 1 -- 172.21.15.1:0/2122716627 learned_addr learned my addr 172.21.15.1:0/2122716627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f39a1200640 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac563a4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f399be00640 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56323fe0 0x55ac56332ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f399b400640 1 -- 172.21.15.1:0/2122716627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56323fe0 msgr2=0x55ac56332ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f399b400640 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56323fe0 0x55ac56332ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f399b400640 1 -- 172.21.15.1:0/2122716627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 msgr2=0x55ac563a4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f399b400640 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac563a4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f399b400640 1 -- 172.21.15.1:0/2122716627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac56243cb0 con 0x55ac56324900 2024-04-26T21:37:13.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f399b400640 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac563333e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f39880a4820 tx=0x7f398809a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0ce43bc884cc2b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.071+0000 7f3986000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f3986000640 1 -- 172.21.15.1:0/2122716627 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39880a4d50 con 0x55ac56324900 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f3986000640 1 -- 172.21.15.1:0/2122716627 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9730+0+0 (secure 0 0 0) 0x7f39880041d0 con 0x55ac56324900 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2122716627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39880a0490 con 0x55ac56324900 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f3986000640 1 -- 172.21.15.1:0/2122716627 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f39880a0120 con 0x55ac56324900 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f39a5fcb740 10 client.26072 fetch_fsmap learned FSMap version 36 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2122716627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ac563a58d0 con 0x55ac56324900 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f3986000640 1 -- 172.21.15.1:0/2122716627 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39880a0490 con 0x55ac56324900 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f3986000640 1 -- 172.21.15.1:0/2122716627 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f39880b0b80 con 0x55ac56324900 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f39a5fcb740 10 client.26072 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f39a5fcb740 10 client.26072 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.072+0000 7f39a5fcb740 20 client.26072 populate_metadata read hostname 'smithi001' 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.073+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ac563a5f10 0x55ac563a8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.073+0000 7f39a5fcb740 4 client.26072 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:37:13.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.073+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2122716627 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ac56290e00 con 0x55ac563a5f10 2024-04-26T21:37:13.077 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.073+0000 7f399be00640 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ac563a5f10 0x55ac563a8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:13.077 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.073+0000 7f399be00640 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ac563a5f10 0x55ac563a8330 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92e489d5e89abc14 server_cookie=99c19ef0864f6c47 in_seq=0 out_seq=0 2024-04-26T21:37:13.077 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.073+0000 7f3986000640 10 client.26072 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.145+0000 7f3986000640 1 -- 172.21.15.1:0/2122716627 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ac56290e00 con 0x55ac563a5f10 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.145+0000 7f3986000640 10 client.26072 handle_command_reply: tid=0 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.145+0000 7f39a5fcb740 1 client.26072 shutdown 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2122716627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 msgr2=0x55ac563333e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac563333e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f39880a4820 tx=0x7f398809a5f0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2122716627 shutdown_connections 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ac563a5f10 0x55ac563a8330 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ac5632f1c0 0x55ac563a4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ac56324900 0x55ac563333e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 1 --2- 172.21.15.1:0/2122716627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ac56323fe0 0x55ac56332ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f3986000640 0 client.26072 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:13.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2122716627 >> 172.21.15.1:0/2122716627 conn(0x55ac563204f0 msgr2=0x55ac56325990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2122716627 shutdown_connections 2024-04-26T21:37:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 1 -- 172.21.15.1:0/2122716627 wait complete. 2024-04-26T21:37:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.146+0000 7f39a5fcb740 20 client.26072 trim_cache size 0 max 16384 2024-04-26T21:37:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:37:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:37:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:37:13.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:37:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 -- 172.21.15.1:0/4031639995 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f39800a3d80 msgr2=0x7f39800a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f39800a3d80 0x7f39800a6240 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3994009d60 tx=0x7f3994009500 comp rx=0 tx=0).stop 2024-04-26T21:37:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 -- 172.21.15.1:0/4031639995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f399c103110 msgr2=0x7f399c075170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f399c103110 0x7f399c075170 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f398800d950 tx=0x7f398800de10 comp rx=0 tx=0).stop 2024-04-26T21:37:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 -- 172.21.15.1:0/4031639995 shutdown_connections 2024-04-26T21:37:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f39800a3d80 0x7f39800a6240 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f399c108760 0x7f399c0756b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f399c103110 0x7f399c075170 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 --2- 172.21.15.1:0/4031639995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f399c102750 0x7f399c074c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:13.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 -- 172.21.15.1:0/4031639995 >> 172.21.15.1:0/4031639995 conn(0x7f399c0fe240 msgr2=0x7f399c0fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.147+0000 7f39a1c00640 1 -- 172.21.15.1:0/4031639995 shutdown_connections 2024-04-26T21:37:13.150 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:13.148+0000 7f39a1c00640 1 -- 172.21.15.1:0/4031639995 wait complete. 2024-04-26T21:37:13.566 INFO:tasks.workunit.client.0.smithi001.stdout: 1 220347 4.36 MB/sec execute 191 sec latency 45.113 ms 2024-04-26T21:37:14.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:13 smithi186 ceph-mon[32991]: pgmap v211: 129 pgs: 129 active+clean; 124 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.7 MiB/s wr, 202 op/s 2024-04-26T21:37:14.207 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:13 smithi186 ceph-mon[32991]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T21:37:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:13 smithi001 ceph-mon[28054]: pgmap v211: 129 pgs: 129 active+clean; 124 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.7 MiB/s wr, 202 op/s 2024-04-26T21:37:14.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:13 smithi001 ceph-mon[28054]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T21:37:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:13 smithi038 ceph-mon[32724]: pgmap v211: 129 pgs: 129 active+clean; 124 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 1.7 MiB/s wr, 202 op/s 2024-04-26T21:37:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:13 smithi038 ceph-mon[32724]: osdmap e98: 12 total, 12 up, 12 in 2024-04-26T21:37:14.566 INFO:tasks.workunit.client.0.smithi001.stdout: 1 220755 4.34 MB/sec execute 192 sec latency 27.736 ms 2024-04-26T21:37:15.567 INFO:tasks.workunit.client.0.smithi001.stdout: 1 221657 4.35 MB/sec execute 193 sec latency 20.325 ms 2024-04-26T21:37:16.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:15 smithi186 ceph-mon[32991]: pgmap v213: 129 pgs: 129 active+clean; 128 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 208 op/s 2024-04-26T21:37:16.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:15 smithi001 ceph-mon[28054]: pgmap v213: 129 pgs: 129 active+clean; 128 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 208 op/s 2024-04-26T21:37:16.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:15 smithi038 ceph-mon[32724]: pgmap v213: 129 pgs: 129 active+clean; 128 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 208 op/s 2024-04-26T21:37:16.567 INFO:tasks.workunit.client.0.smithi001.stdout: 1 222879 4.36 MB/sec execute 194 sec latency 70.813 ms 2024-04-26T21:37:17.567 INFO:tasks.workunit.client.0.smithi001.stdout: 1 223759 4.37 MB/sec execute 195 sec latency 34.616 ms 2024-04-26T21:37:18.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:17 smithi186 ceph-mon[32991]: pgmap v214: 129 pgs: 129 active+clean; 128 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-04-26T21:37:18.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:17 smithi001 ceph-mon[28054]: pgmap v214: 129 pgs: 129 active+clean; 128 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-04-26T21:37:18.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:17 smithi038 ceph-mon[32724]: pgmap v214: 129 pgs: 129 active+clean; 128 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.5 MiB/s wr, 168 op/s 2024-04-26T21:37:18.567 INFO:tasks.workunit.client.0.smithi001.stdout: 1 224021 4.35 MB/sec execute 196 sec latency 24.233 ms 2024-04-26T21:37:19.567 INFO:tasks.workunit.client.0.smithi001.stdout: 1 224735 4.34 MB/sec execute 197 sec latency 36.802 ms 2024-04-26T21:37:20.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:19 smithi186 ceph-mon[32991]: pgmap v215: 129 pgs: 129 active+clean; 128 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.5 MiB/s wr, 166 op/s 2024-04-26T21:37:20.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:19 smithi001 ceph-mon[28054]: pgmap v215: 129 pgs: 129 active+clean; 128 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.5 MiB/s wr, 166 op/s 2024-04-26T21:37:20.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:19 smithi038 ceph-mon[32724]: pgmap v215: 129 pgs: 129 active+clean; 128 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 1.5 MiB/s wr, 166 op/s 2024-04-26T21:37:20.567 INFO:tasks.workunit.client.0.smithi001.stdout: 1 226117 4.35 MB/sec execute 198 sec latency 30.047 ms 2024-04-26T21:37:21.567 INFO:tasks.workunit.client.0.smithi001.stdout: 1 227280 4.38 MB/sec execute 199 sec latency 30.782 ms 2024-04-26T21:37:22.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:21 smithi186 ceph-mon[32991]: pgmap v216: 129 pgs: 129 active+clean; 130 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-04-26T21:37:22.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:21 smithi001 ceph-mon[28054]: pgmap v216: 129 pgs: 129 active+clean; 130 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-04-26T21:37:22.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:21 smithi038 ceph-mon[32724]: pgmap v216: 129 pgs: 129 active+clean; 130 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-04-26T21:37:22.567 INFO:tasks.workunit.client.0.smithi001.stdout: 1 227565 4.36 MB/sec execute 200 sec latency 26.093 ms 2024-04-26T21:37:23.568 INFO:tasks.workunit.client.0.smithi001.stdout: 1 228293 4.35 MB/sec execute 201 sec latency 19.145 ms 2024-04-26T21:37:24.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:23 smithi186 ceph-mon[32991]: pgmap v217: 129 pgs: 129 active+clean; 130 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-04-26T21:37:24.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:23 smithi001 ceph-mon[28054]: pgmap v217: 129 pgs: 129 active+clean; 130 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-04-26T21:37:24.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:23 smithi038 ceph-mon[32724]: pgmap v217: 129 pgs: 129 active+clean; 130 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 1.7 MiB/s wr, 177 op/s 2024-04-26T21:37:24.568 INFO:tasks.workunit.client.0.smithi001.stdout: 1 229651 4.35 MB/sec execute 202 sec latency 23.875 ms 2024-04-26T21:37:25.568 INFO:tasks.workunit.client.0.smithi001.stdout: 1 230815 4.39 MB/sec execute 203 sec latency 25.877 ms 2024-04-26T21:37:26.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:25 smithi001 ceph-mon[28054]: pgmap v218: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 230 op/s 2024-04-26T21:37:26.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:25 smithi038 ceph-mon[32724]: pgmap v218: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 230 op/s 2024-04-26T21:37:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.331+0000 7f0e5de00640 1 -- 172.21.15.1:0/2215016836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e58074d60 msgr2=0x7f0e58075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.331+0000 7f0e5de00640 1 --2- 172.21.15.1:0/2215016836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e58074d60 0x7f0e58075140 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0e48009900 tx=0x7f0e4802a480 comp rx=0 tx=0).stop 2024-04-26T21:37:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.331+0000 7f0e5de00640 1 -- 172.21.15.1:0/2215016836 shutdown_connections 2024-04-26T21:37:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.331+0000 7f0e5de00640 1 --2- 172.21.15.1:0/2215016836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e58075680 0x7f0e5810ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.331+0000 7f0e5de00640 1 --2- 172.21.15.1:0/2215016836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e5810f180 0x7f0e581115d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.331+0000 7f0e5de00640 1 --2- 172.21.15.1:0/2215016836 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e58074d60 0x7f0e58075140 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.331+0000 7f0e5de00640 1 -- 172.21.15.1:0/2215016836 >> 172.21.15.1:0/2215016836 conn(0x7f0e580fe3f0 msgr2=0x7f0e58100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.331+0000 7f0e5de00640 1 -- 172.21.15.1:0/2215016836 shutdown_connections 2024-04-26T21:37:26.332 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.331+0000 7f0e5de00640 1 -- 172.21.15.1:0/2215016836 wait complete. 2024-04-26T21:37:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.332+0000 7f0e5de00640 1 Processor -- start 2024-04-26T21:37:26.333 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.332+0000 7f0e5de00640 1 -- start start 2024-04-26T21:37:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e5de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e58074d60 0x7f0e5819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e5de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e58075680 0x7f0e5819b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e5de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e5810f180 0x7f0e5819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e5de00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58114dd0 con 0x7f0e58074d60 2024-04-26T21:37:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e5de00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58114c50 con 0x7f0e58075680 2024-04-26T21:37:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e5de00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e58114f50 con 0x7f0e5810f180 2024-04-26T21:37:26.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e56000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e58075680 0x7f0e5819b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e56000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e58075680 0x7f0e5819b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37114/0 (socket says 172.21.15.1:37114) 2024-04-26T21:37:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e56000640 1 -- 172.21.15.1:0/3220304506 learned_addr learned my addr 172.21.15.1:0/3220304506 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e56a00640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e58074d60 0x7f0e5819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.333+0000 7f0e57400640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e5810f180 0x7f0e5819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e56000640 1 -- 172.21.15.1:0/3220304506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e5810f180 msgr2=0x7f0e5819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e56000640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e5810f180 0x7f0e5819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e56000640 1 -- 172.21.15.1:0/3220304506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e58074d60 msgr2=0x7f0e5819b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:26.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e56000640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e58074d60 0x7f0e5819b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e56000640 1 -- 172.21.15.1:0/3220304506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e581a0320 con 0x7f0e58075680 2024-04-26T21:37:26.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e56000640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e58075680 0x7f0e5819b870 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400e9f0 tx=0x7f0e4400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e5ca00640 1 -- 172.21.15.1:0/3220304506 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4400cd80 con 0x7f0e58075680 2024-04-26T21:37:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e5ca00640 1 -- 172.21.15.1:0/3220304506 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e440054d0 con 0x7f0e58075680 2024-04-26T21:37:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e5de00640 1 -- 172.21.15.1:0/3220304506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e581a0610 con 0x7f0e58075680 2024-04-26T21:37:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e5ca00640 1 -- 172.21.15.1:0/3220304506 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44010690 con 0x7f0e58075680 2024-04-26T21:37:26.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.334+0000 7f0e5de00640 1 -- 172.21.15.1:0/3220304506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e581a7ea0 con 0x7f0e58075680 2024-04-26T21:37:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.339+0000 7f0e60c78740 1 Processor -- start 2024-04-26T21:37:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.339+0000 7f0e60c78740 1 -- start start 2024-04-26T21:37:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.339+0000 7f0e5ca00640 1 -- 172.21.15.1:0/3220304506 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0e44004e90 con 0x7f0e58075680 2024-04-26T21:37:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef06931b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef06936f0 0x555ef069da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef069dfb0 0x555ef06a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e60c78740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555ef05f8380 con 0x555ef0692dd0 2024-04-26T21:37:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e60c78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555ef0594c30 con 0x555ef069dfb0 2024-04-26T21:37:26.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e60c78740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555ef05f4340 con 0x555ef06936f0 2024-04-26T21:37:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e56a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef06936f0 0x555ef069da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e56a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef06936f0 0x555ef069da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:55776/0 (socket says 172.21.15.1:55776) 2024-04-26T21:37:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e56a00640 1 -- 172.21.15.1:0/1992923121 learned_addr learned my addr 172.21.15.1:0/1992923121 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e56000640 1 --2- 172.21.15.1:0/1992923121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef069dfb0 0x555ef06a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e57400640 1 --2- 172.21.15.1:0/1992923121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef06931b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.342 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e56a00640 1 -- 172.21.15.1:0/1992923121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef069dfb0 msgr2=0x555ef06a03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e56a00640 1 --2- 172.21.15.1:0/1992923121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef069dfb0 0x555ef06a03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e56a00640 1 -- 172.21.15.1:0/1992923121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 msgr2=0x555ef06931b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e56a00640 1 --2- 172.21.15.1:0/1992923121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef06931b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.340+0000 7f0e56a00640 1 -- 172.21.15.1:0/1992923121 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ef05b6940 con 0x555ef06936f0 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.341+0000 7f0e56a00640 1 --2- 172.21.15.1:0/1992923121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef06936f0 0x555ef069da70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0e48009900 tx=0x7f0e48032b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf2a675d952a1e84 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.341+0000 7f0e43e00640 1 -- 172.21.15.1:0/1992923121 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48040070 con 0x555ef06936f0 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.341+0000 7f0e43e00640 1 -- 172.21.15.1:0/1992923121 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e48004030 con 0x555ef06936f0 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.341+0000 7f0e43e00640 1 -- 172.21.15.1:0/1992923121 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48035040 con 0x555ef06936f0 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.341+0000 7f0e5ca00640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e3c0a3d90 0x7f0e3c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.341+0000 7f0e5ca00640 1 -- 172.21.15.1:0/3220304506 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f0e440d0fd0 con 0x7f0e58075680 2024-04-26T21:37:26.343 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.341+0000 7f0e57400640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e3c0a3d90 0x7f0e3c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.342+0000 7f0e60c78740 1 -- 172.21.15.1:0/1992923121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef06936f0 msgr2=0x555ef069da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.342+0000 7f0e60c78740 1 --2- 172.21.15.1:0/1992923121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef06936f0 0x555ef069da70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0e48009900 tx=0x7f0e48032b20 comp rx=0 tx=0).stop 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.342+0000 7f0e60c78740 1 -- 172.21.15.1:0/1992923121 shutdown_connections 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.342+0000 7f0e60c78740 1 --2- 172.21.15.1:0/1992923121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef06936f0 0x555ef069da70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.342+0000 7f0e60c78740 1 --2- 172.21.15.1:0/1992923121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef069dfb0 0x555ef06a03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.342+0000 7f0e60c78740 1 --2- 172.21.15.1:0/1992923121 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef06931b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.342+0000 7f0e60c78740 1 -- 172.21.15.1:0/1992923121 >> 172.21.15.1:0/1992923121 conn(0x555ef068f2e0 msgr2=0x555ef068f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.342+0000 7f0e60c78740 1 -- 172.21.15.1:0/1992923121 shutdown_connections 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.343+0000 7f0e57400640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e3c0a3d90 0x7f0e3c0a6250 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0e50005fd0 tx=0x7f0e50009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.343+0000 7f0e60c78740 1 -- 172.21.15.1:0/1992923121 wait complete. 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.343+0000 7f0e60c78740 1 -- start start 2024-04-26T21:37:26.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e60c78740 1 Processor -- start 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef069af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef069b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef069b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e60c78740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555ef06a0b80 con 0x555ef0692dd0 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e60c78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555ef057fc60 con 0x555ef06936f0 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e60c78740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555ef06a0d00 con 0x555ef069dfb0 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e60c78740 10 client.0 _ll_register_callbacks cb 0x555ef05abe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e56a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef069af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e56000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef069b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e56a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef069af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35128/0 (socket says 172.21.15.1:35128) 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e56000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef069b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37128/0 (socket says 172.21.15.1:37128) 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.344+0000 7f0e56a00640 1 -- 172.21.15.1:0/3621345113 learned_addr learned my addr 172.21.15.1:0/3621345113 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.345+0000 7f0e57400640 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef069b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.345+0000 7f0e56a00640 1 -- 172.21.15.1:0/3621345113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 msgr2=0x555ef069b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.345+0000 7f0e56a00640 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef069b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.345+0000 7f0e56a00640 1 -- 172.21.15.1:0/3621345113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 msgr2=0x555ef069b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.345+0000 7f0e56a00640 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef069b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.345+0000 7f0e56a00640 1 -- 172.21.15.1:0/3621345113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ef058e510 con 0x555ef0692dd0 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.345+0000 7f0e56000640 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef069b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.345+0000 7f0e56a00640 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef069af00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e48032f40 tx=0x7f0e48031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d604a20a3ea10a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.345+0000 7f0e40c00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.346+0000 7f0e40c00640 1 -- 172.21.15.1:0/3621345113 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48040070 con 0x555ef0692dd0 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.346+0000 7f0e40c00640 1 -- 172.21.15.1:0/3621345113 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f0e48005430 con 0x555ef0692dd0 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.346+0000 7f0e60c78740 1 -- 172.21.15.1:0/3621345113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e48004030 con 0x555ef0692dd0 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.346+0000 7f0e40c00640 1 -- 172.21.15.1:0/3621345113 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0e48056050 con 0x555ef0692dd0 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.346+0000 7f0e60c78740 10 client.15915 fetch_fsmap learned FSMap version 36 2024-04-26T21:37:26.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.346+0000 7f0e60c78740 1 -- 172.21.15.1:0/3621345113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555ef030a210 con 0x555ef0692dd0 2024-04-26T21:37:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.347+0000 7f0e40c00640 1 -- 172.21.15.1:0/3621345113 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e48004030 con 0x555ef0692dd0 2024-04-26T21:37:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.348+0000 7f0e40c00640 1 -- 172.21.15.1:0/3621345113 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f0e48004cd0 con 0x555ef0692dd0 2024-04-26T21:37:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.348+0000 7f0e60c78740 10 client.15915 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:37:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.348+0000 7f0e60c78740 10 client.15915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:37:26.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.348+0000 7f0e60c78740 20 client.15915 populate_metadata read hostname 'smithi001' 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.348+0000 7f0e60c78740 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555ef06adaf0 0x555ef06afee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.348+0000 7f0e60c78740 4 client.15915 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.348+0000 7f0e60c78740 1 -- 172.21.15.1:0/3621345113 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ef0605f90 con 0x555ef06adaf0 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.348+0000 7f0e56000640 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555ef06adaf0 0x555ef06afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.349+0000 7f0e56000640 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555ef06adaf0 0x555ef06afee0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8cf33b8b361140 server_cookie=4085867bfa6b5afa in_seq=0 out_seq=0 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.349+0000 7f0e40c00640 10 client.15915 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e40c00640 1 -- 172.21.15.1:0/3621345113 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555ef0605f90 con 0x555ef06adaf0 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e40c00640 10 client.15915 handle_command_reply: tid=0 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e60c78740 1 client.15915 shutdown 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e60c78740 1 -- 172.21.15.1:0/3621345113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 msgr2=0x555ef069af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e60c78740 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef069af00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e48032f40 tx=0x7f0e48031400 comp rx=0 tx=0).stop 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e60c78740 1 -- 172.21.15.1:0/3621345113 shutdown_connections 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e60c78740 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555ef06adaf0 0x555ef06afee0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e60c78740 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef069b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e60c78740 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef069b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e60c78740 1 --2- 172.21.15.1:0/3621345113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef069af00 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e60c78740 1 -- 172.21.15.1:0/3621345113 >> 172.21.15.1:0/3621345113 conn(0x555ef068f2e0 msgr2=0x555ef0695060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.350+0000 7f0e40c00640 0 client.15915 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.351+0000 7f0e60c78740 1 -- 172.21.15.1:0/3621345113 shutdown_connections 2024-04-26T21:37:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.351+0000 7f0e60c78740 1 -- 172.21.15.1:0/3621345113 wait complete. 2024-04-26T21:37:26.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.351+0000 7f0e60c78740 20 client.15915 trim_cache size 0 max 16384 2024-04-26T21:37:26.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e60c78740 1 Processor -- start 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e60c78740 1 -- start start 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef0692dd0 0x555ef069a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef06936f0 0x555ef0698160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef06986a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e60c78740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b4a30 con 0x555ef06936f0 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e60c78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b2300 con 0x555ef0692dd0 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e60c78740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b0170 con 0x555ef069dfb0 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e56a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef0692dd0 0x555ef069a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e56a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef0692dd0 0x555ef069a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37130/0 (socket says 172.21.15.1:37130) 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.371+0000 7f0e56a00640 1 -- 172.21.15.1:0/712364977 learned_addr learned my addr 172.21.15.1:0/712364977 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.372+0000 7f0e56000640 1 --2- 172.21.15.1:0/712364977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef06936f0 0x555ef0698160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.372+0000 7f0e57400640 1 --2- 172.21.15.1:0/712364977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef06986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.372+0000 7f0e56a00640 1 -- 172.21.15.1:0/712364977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 msgr2=0x555ef06986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:26.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.372+0000 7f0e56a00640 1 --2- 172.21.15.1:0/712364977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef06986a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.372+0000 7f0e56a00640 1 -- 172.21.15.1:0/712364977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef06936f0 msgr2=0x555ef0698160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.372+0000 7f0e56a00640 1 --2- 172.21.15.1:0/712364977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef06936f0 0x555ef0698160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.372+0000 7f0e56a00640 1 -- 172.21.15.1:0/712364977 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ef03d6620 con 0x555ef0692dd0 2024-04-26T21:37:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.372+0000 7f0e57400640 1 --2- 172.21.15.1:0/712364977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef06986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:37:26.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.372+0000 7f0e56a00640 1 --2- 172.21.15.1:0/712364977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef0692dd0 0x555ef069a7e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0e48005120 tx=0x7f0e4802a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18980b56e9c12707 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.372+0000 7f0e43400640 1 -- 172.21.15.1:0/712364977 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4805f070 con 0x555ef0692dd0 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.373+0000 7f0e43400640 1 -- 172.21.15.1:0/712364977 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e4804d070 con 0x555ef0692dd0 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.373+0000 7f0e43400640 1 -- 172.21.15.1:0/712364977 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48040070 con 0x555ef0692dd0 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.373+0000 7f0e60c78740 1 -- 172.21.15.1:0/712364977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef0692dd0 msgr2=0x555ef069a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.373+0000 7f0e60c78740 1 --2- 172.21.15.1:0/712364977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef0692dd0 0x555ef069a7e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0e48005120 tx=0x7f0e4802a480 comp rx=0 tx=0).stop 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.374+0000 7f0e60c78740 1 -- 172.21.15.1:0/712364977 shutdown_connections 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.374+0000 7f0e60c78740 1 --2- 172.21.15.1:0/712364977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef06986a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.374+0000 7f0e60c78740 1 --2- 172.21.15.1:0/712364977 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef06936f0 0x555ef0698160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.374+0000 7f0e60c78740 1 --2- 172.21.15.1:0/712364977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef0692dd0 0x555ef069a7e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.374+0000 7f0e60c78740 1 -- 172.21.15.1:0/712364977 >> 172.21.15.1:0/712364977 conn(0x555ef068f2e0 msgr2=0x555ef06a97e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.374+0000 7f0e60c78740 1 -- 172.21.15.1:0/712364977 shutdown_connections 2024-04-26T21:37:26.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.374+0000 7f0e60c78740 1 -- 172.21.15.1:0/712364977 wait complete. 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.374+0000 7f0e60c78740 1 -- start start 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.374+0000 7f0e60c78740 1 Processor -- start 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef070b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef070b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e60c78740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef06b0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e60c78740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b07c0 con 0x555ef0692dd0 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e60c78740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b1c30 con 0x555ef06936f0 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e60c78740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581b0e90 con 0x555ef069dfb0 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e60c78740 10 client.0 _ll_register_callbacks cb 0x7f0e440d49f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e56a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef070b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e56000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef070b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e56000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef070b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:37132/0 (socket says 172.21.15.1:37132) 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e56000640 1 -- 172.21.15.1:0/356685299 learned_addr learned my addr 172.21.15.1:0/356685299 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:26.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.375+0000 7f0e57400640 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef06b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.376+0000 7f0e56a00640 1 -- 172.21.15.1:0/356685299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 msgr2=0x555ef06b0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:26.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.376+0000 7f0e56a00640 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef06b0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.376+0000 7f0e56a00640 1 -- 172.21.15.1:0/356685299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 msgr2=0x555ef070b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.376+0000 7f0e56a00640 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef070b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.376+0000 7f0e56a00640 1 -- 172.21.15.1:0/356685299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ef05b2cb0 con 0x555ef0692dd0 2024-04-26T21:37:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.376+0000 7f0e56000640 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef070b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:37:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.376+0000 7f0e56a00640 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef070b050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0e4805d040 tx=0x7f0e48003a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be0f2fc8b5fcd83 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.376+0000 7f0e40c00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:37:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.376+0000 7f0e40c00640 1 -- 172.21.15.1:0/356685299 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4805f020 con 0x555ef0692dd0 2024-04-26T21:37:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.376+0000 7f0e60c78740 1 -- 172.21.15.1:0/356685299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e4804d070 con 0x555ef0692dd0 2024-04-26T21:37:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e40c00640 1 -- 172.21.15.1:0/356685299 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f0e480051c0 con 0x555ef0692dd0 2024-04-26T21:37:26.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e40c00640 1 -- 172.21.15.1:0/356685299 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0e48040040 con 0x555ef0692dd0 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e60c78740 10 client.15921 fetch_fsmap learned FSMap version 36 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e60c78740 1 -- 172.21.15.1:0/356685299 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555ef0713610 con 0x555ef0692dd0 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e40c00640 1 -- 172.21.15.1:0/356685299 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e4804d070 con 0x555ef0692dd0 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e40c00640 1 -- 172.21.15.1:0/356685299 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f0e480314a0 con 0x555ef0692dd0 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e60c78740 10 client.15921 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e60c78740 10 client.15921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e60c78740 20 client.15921 populate_metadata read hostname 'smithi001' 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e60c78740 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555ef0713b40 0x555ef0715f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.377+0000 7f0e60c78740 4 client.15921 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.378+0000 7f0e60c78740 1 -- 172.21.15.1:0/356685299 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555ef05ffe00 con 0x555ef0713b40 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.378+0000 7f0e56000640 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555ef0713b40 0x555ef0715f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.378+0000 7f0e56000640 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555ef0713b40 0x555ef0715f30 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffb41eaf15c885a8 server_cookie=c028d46c70837d9 in_seq=0 out_seq=0 2024-04-26T21:37:26.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.378+0000 7f0e40c00640 10 client.15921 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:26.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e40c00640 1 -- 172.21.15.1:0/356685299 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555ef05ffe00 con 0x555ef0713b40 2024-04-26T21:37:26.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e40c00640 10 client.15921 handle_command_reply: tid=0 2024-04-26T21:37:26.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 client.15921 shutdown 2024-04-26T21:37:26.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 -- 172.21.15.1:0/356685299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 msgr2=0x555ef070b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:26.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef070b050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0e4805d040 tx=0x7f0e48003a30 comp rx=0 tx=0).stop 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 -- 172.21.15.1:0/356685299 shutdown_connections 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555ef0713b40 0x555ef0715f30 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555ef069dfb0 0x555ef06b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555ef06936f0 0x555ef070b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 --2- 172.21.15.1:0/356685299 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555ef0692dd0 0x555ef070b050 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 -- 172.21.15.1:0/356685299 >> 172.21.15.1:0/356685299 conn(0x555ef068f2e0 msgr2=0x555ef068dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e40c00640 0 client.15921 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 -- 172.21.15.1:0/356685299 shutdown_connections 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.453+0000 7f0e60c78740 1 -- 172.21.15.1:0/356685299 wait complete. 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e60c78740 20 client.15921 trim_cache size 0 max 16384 2024-04-26T21:37:26.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:37:26.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:37:26.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:37:26.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:37:26.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e5de00640 1 -- 172.21.15.1:0/3220304506 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e3c0a3d90 msgr2=0x7f0e3c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:26.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e5de00640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e3c0a3d90 0x7f0e3c0a6250 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0e50005fd0 tx=0x7f0e50009500 comp rx=0 tx=0).stop 2024-04-26T21:37:26.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e5de00640 1 -- 172.21.15.1:0/3220304506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e58075680 msgr2=0x7f0e5819b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:26.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e5de00640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e58075680 0x7f0e5819b870 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400e9f0 tx=0x7f0e4400edf0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e5de00640 1 -- 172.21.15.1:0/3220304506 shutdown_connections 2024-04-26T21:37:26.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e5de00640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e3c0a3d90 0x7f0e3c0a6250 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e5de00640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e5810f180 0x7f0e5819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e5de00640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e58075680 0x7f0e5819b870 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e5de00640 1 --2- 172.21.15.1:0/3220304506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e58074d60 0x7f0e5819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:26.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.454+0000 7f0e5de00640 1 -- 172.21.15.1:0/3220304506 >> 172.21.15.1:0/3220304506 conn(0x7f0e580fe3f0 msgr2=0x7f0e580ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:26.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.455+0000 7f0e5de00640 1 -- 172.21.15.1:0/3220304506 shutdown_connections 2024-04-26T21:37:26.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:26.455+0000 7f0e5de00640 1 -- 172.21.15.1:0/3220304506 wait complete. 2024-04-26T21:37:26.457 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:25 smithi186 ceph-mon[32991]: pgmap v218: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 230 op/s 2024-04-26T21:37:26.568 INFO:tasks.workunit.client.0.smithi001.stdout: 1 231165 4.37 MB/sec execute 204 sec latency 60.558 ms 2024-04-26T21:37:27.568 INFO:tasks.workunit.client.0.smithi001.stdout: 1 231987 4.37 MB/sec execute 205 sec latency 23.144 ms 2024-04-26T21:37:28.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:27 smithi001 ceph-mon[28054]: pgmap v219: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.5 MiB/s wr, 125 op/s 2024-04-26T21:37:28.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:27 smithi038 ceph-mon[32724]: pgmap v219: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.5 MiB/s wr, 125 op/s 2024-04-26T21:37:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:27 smithi186 ceph-mon[32991]: pgmap v219: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.5 MiB/s wr, 125 op/s 2024-04-26T21:37:28.568 INFO:tasks.workunit.client.0.smithi001.stdout: 1 233187 4.36 MB/sec execute 206 sec latency 26.013 ms 2024-04-26T21:37:29.568 INFO:tasks.workunit.client.0.smithi001.stdout: 1 234281 4.39 MB/sec execute 207 sec latency 23.078 ms 2024-04-26T21:37:30.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:29 smithi001 ceph-mon[28054]: pgmap v220: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.5 MiB/s wr, 125 op/s 2024-04-26T21:37:30.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:29 smithi038 ceph-mon[32724]: pgmap v220: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.5 MiB/s wr, 125 op/s 2024-04-26T21:37:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:29 smithi186 ceph-mon[32991]: pgmap v220: 129 pgs: 129 active+clean; 134 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.5 MiB/s wr, 125 op/s 2024-04-26T21:37:30.568 INFO:tasks.workunit.client.0.smithi001.stdout: 1 234570 4.37 MB/sec execute 208 sec latency 26.829 ms 2024-04-26T21:37:31.568 INFO:tasks.workunit.client.0.smithi001.stdout: 1 235137 4.36 MB/sec execute 209 sec latency 50.387 ms 2024-04-26T21:37:32.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:31 smithi001 ceph-mon[28054]: pgmap v221: 129 pgs: 129 active+clean; 137 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 2.1 MiB/s wr, 219 op/s 2024-04-26T21:37:32.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:31 smithi038 ceph-mon[32724]: pgmap v221: 129 pgs: 129 active+clean; 137 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 2.1 MiB/s wr, 219 op/s 2024-04-26T21:37:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:31 smithi186 ceph-mon[32991]: pgmap v221: 129 pgs: 129 active+clean; 137 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 2.1 MiB/s wr, 219 op/s 2024-04-26T21:37:32.569 INFO:tasks.workunit.client.0.smithi001.stdout: 1 236555 4.37 MB/sec execute 210 sec latency 26.798 ms 2024-04-26T21:37:33.569 INFO:tasks.workunit.client.0.smithi001.stdout: 1 237408 4.38 MB/sec execute 211 sec latency 40.218 ms 2024-04-26T21:37:34.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:33 smithi001 ceph-mon[28054]: pgmap v222: 129 pgs: 129 active+clean; 137 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:37:34.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:33 smithi038 ceph-mon[32724]: pgmap v222: 129 pgs: 129 active+clean; 137 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:37:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:33 smithi186 ceph-mon[32991]: pgmap v222: 129 pgs: 129 active+clean; 137 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:37:34.569 INFO:tasks.workunit.client.0.smithi001.stdout: 1 238021 4.38 MB/sec execute 212 sec latency 25.524 ms 2024-04-26T21:37:35.569 INFO:tasks.workunit.client.0.smithi001.stdout: 1 238373 4.36 MB/sec execute 213 sec latency 26.021 ms 2024-04-26T21:37:36.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:35 smithi001 ceph-mon[28054]: pgmap v223: 129 pgs: 129 active+clean; 139 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-26T21:37:36.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:35 smithi038 ceph-mon[32724]: pgmap v223: 129 pgs: 129 active+clean; 139 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-26T21:37:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:35 smithi186 ceph-mon[32991]: pgmap v223: 129 pgs: 129 active+clean; 139 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-26T21:37:36.569 INFO:tasks.workunit.client.0.smithi001.stdout: 1 239122 4.36 MB/sec execute 214 sec latency 39.427 ms 2024-04-26T21:37:37.569 INFO:tasks.workunit.client.0.smithi001.stdout: 1 240589 4.38 MB/sec execute 215 sec latency 29.217 ms 2024-04-26T21:37:38.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:37 smithi001 ceph-mon[28054]: pgmap v224: 129 pgs: 129 active+clean; 139 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-04-26T21:37:38.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:37 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:37:38.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:37 smithi038 ceph-mon[32724]: pgmap v224: 129 pgs: 129 active+clean; 139 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-04-26T21:37:38.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:37 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:37:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:37 smithi186 ceph-mon[32991]: pgmap v224: 129 pgs: 129 active+clean; 139 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-04-26T21:37:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:37 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:37:38.569 INFO:tasks.workunit.client.0.smithi001.stdout: 1 241500 4.39 MB/sec execute 216 sec latency 22.754 ms 2024-04-26T21:37:39.569 INFO:tasks.workunit.client.0.smithi001.stdout: 1 242028 4.37 MB/sec execute 217 sec latency 21.414 ms 2024-04-26T21:37:39.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.616+0000 7f1b4f600640 1 -- 172.21.15.1:0/4140999488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b48108a20 msgr2=0x7f1b48108e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:39.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.616+0000 7f1b4f600640 1 --2- 172.21.15.1:0/4140999488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b48108a20 0x7f1b48108e00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009890 tx=0x7f1b3c02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.616+0000 7f1b4f600640 1 -- 172.21.15.1:0/4140999488 shutdown_connections 2024-04-26T21:37:39.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.616+0000 7f1b4f600640 1 --2- 172.21.15.1:0/4140999488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b4806b4e0 0x7f1b4810f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.616+0000 7f1b4f600640 1 --2- 172.21.15.1:0/4140999488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4806ab40 0x7f1b4806afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.616+0000 7f1b4f600640 1 --2- 172.21.15.1:0/4140999488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b48108a20 0x7f1b48108e00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.618 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.616+0000 7f1b4f600640 1 -- 172.21.15.1:0/4140999488 >> 172.21.15.1:0/4140999488 conn(0x7f1b48078710 msgr2=0x7f1b48078b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:39.618 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.617+0000 7f1b4f600640 1 -- 172.21.15.1:0/4140999488 shutdown_connections 2024-04-26T21:37:39.618 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.617+0000 7f1b4f600640 1 -- 172.21.15.1:0/4140999488 wait complete. 2024-04-26T21:37:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.619+0000 7f1b4f600640 1 Processor -- start 2024-04-26T21:37:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.619+0000 7f1b4f600640 1 -- start start 2024-04-26T21:37:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.619+0000 7f1b4f600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4806ab40 0x7f1b4810da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b4f600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b4806b4e0 0x7f1b4810df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b4f600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b48108a20 0x7f1b481a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b4f600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1b48112c10 con 0x7f1b4806ab40 2024-04-26T21:37:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b4f600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1b48112a90 con 0x7f1b48108a20 2024-04-26T21:37:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b4f600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1b48112d90 con 0x7f1b4806b4e0 2024-04-26T21:37:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b47e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b4806b4e0 0x7f1b4810df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b47e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b4806b4e0 0x7f1b4810df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:53480/0 (socket says 172.21.15.1:53480) 2024-04-26T21:37:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b47e00640 1 -- 172.21.15.1:0/2159569264 learned_addr learned my addr 172.21.15.1:0/2159569264 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:39.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b4cc00640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4806ab40 0x7f1b4810da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b4d600640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b48108a20 0x7f1b481a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b47e00640 1 -- 172.21.15.1:0/2159569264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b48108a20 msgr2=0x7f1b481a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b47e00640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b48108a20 0x7f1b481a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b47e00640 1 -- 172.21.15.1:0/2159569264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4806ab40 msgr2=0x7f1b4810da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b47e00640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4806ab40 0x7f1b4810da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.620+0000 7f1b47e00640 1 -- 172.21.15.1:0/2159569264 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b481a46e0 con 0x7f1b4806b4e0 2024-04-26T21:37:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.621+0000 7f1b47e00640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b4806b4e0 0x7f1b4810df50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400e9c0 tx=0x7f1b3400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:39.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.621+0000 7f1b45600640 1 -- 172.21.15.1:0/2159569264 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3400ccc0 con 0x7f1b4806b4e0 2024-04-26T21:37:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.621+0000 7f1b45600640 1 -- 172.21.15.1:0/2159569264 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b340054d0 con 0x7f1b4806b4e0 2024-04-26T21:37:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.621+0000 7f1b45600640 1 -- 172.21.15.1:0/2159569264 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b34010430 con 0x7f1b4806b4e0 2024-04-26T21:37:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.621+0000 7f1b4f600640 1 -- 172.21.15.1:0/2159569264 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b481a49d0 con 0x7f1b4806b4e0 2024-04-26T21:37:39.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.621+0000 7f1b4f600640 1 -- 172.21.15.1:0/2159569264 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b481ac260 con 0x7f1b4806b4e0 2024-04-26T21:37:39.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.626+0000 7f1b45600640 1 -- 172.21.15.1:0/2159569264 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f1b34004f80 con 0x7f1b4806b4e0 2024-04-26T21:37:39.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.626+0000 7f1b5247b740 1 Processor -- start 2024-04-26T21:37:39.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.626+0000 7f1b5247b740 1 -- start start 2024-04-26T21:37:39.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.626+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae73a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.626+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 0x55e0ae744a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.626+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.626+0000 7f1b5247b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e0ae69f380 con 0x55e0ae739dd0 2024-04-26T21:37:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.626+0000 7f1b5247b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e0ae63bc30 con 0x55e0ae744fb0 2024-04-26T21:37:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.626+0000 7f1b5247b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e0ae69b340 con 0x55e0ae73a6f0 2024-04-26T21:37:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b47e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b4cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 0x55e0ae744a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b4cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 0x55e0ae744a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:53494/0 (socket says 172.21.15.1:53494) 2024-04-26T21:37:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b4cc00640 1 -- 172.21.15.1:0/185481283 learned_addr learned my addr 172.21.15.1:0/185481283 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:39.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b4d600640 1 --2- 172.21.15.1:0/185481283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae73a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b47e00640 1 -- 172.21.15.1:0/185481283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 msgr2=0x55e0ae744a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b47e00640 1 --2- 172.21.15.1:0/185481283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 0x55e0ae744a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b47e00640 1 -- 172.21.15.1:0/185481283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 msgr2=0x55e0ae73a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b47e00640 1 --2- 172.21.15.1:0/185481283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae73a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b47e00640 1 -- 172.21.15.1:0/185481283 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0ae65d940 con 0x55e0ae744fb0 2024-04-26T21:37:39.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b4d600640 1 --2- 172.21.15.1:0/185481283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae73a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:37:39.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.627+0000 7f1b47e00640 1 --2- 172.21.15.1:0/185481283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7473a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1b340ce340 tx=0x7f1b340d9570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e8048d0f65966f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:39.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.628+0000 7f1b39c00640 1 -- 172.21.15.1:0/185481283 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b340e4070 con 0x55e0ae744fb0 2024-04-26T21:37:39.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.628+0000 7f1b39c00640 1 -- 172.21.15.1:0/185481283 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b340d32d0 con 0x55e0ae744fb0 2024-04-26T21:37:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.628+0000 7f1b39c00640 1 -- 172.21.15.1:0/185481283 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b340df6a0 con 0x55e0ae744fb0 2024-04-26T21:37:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.628+0000 7f1b45600640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1b300a3e10 0x7f1b300a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.628+0000 7f1b45600640 1 -- 172.21.15.1:0/2159569264 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f1b340d10b0 con 0x7f1b4806b4e0 2024-04-26T21:37:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.629+0000 7f1b4d600640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1b300a3e10 0x7f1b300a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.629+0000 7f1b5247b740 1 -- 172.21.15.1:0/185481283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 msgr2=0x55e0ae7473a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.629+0000 7f1b5247b740 1 --2- 172.21.15.1:0/185481283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7473a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1b340ce340 tx=0x7f1b340d9570 comp rx=0 tx=0).stop 2024-04-26T21:37:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.629+0000 7f1b5247b740 1 -- 172.21.15.1:0/185481283 shutdown_connections 2024-04-26T21:37:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.629+0000 7f1b5247b740 1 --2- 172.21.15.1:0/185481283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 0x55e0ae744a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.629+0000 7f1b5247b740 1 --2- 172.21.15.1:0/185481283 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7473a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.629+0000 7f1b5247b740 1 --2- 172.21.15.1:0/185481283 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae73a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.629+0000 7f1b5247b740 1 -- 172.21.15.1:0/185481283 >> 172.21.15.1:0/185481283 conn(0x55e0ae7362e0 msgr2=0x55e0ae7366c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.629+0000 7f1b4d600640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1b300a3e10 0x7f1b300a62d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1b40005fd0 tx=0x7f1b40009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.630+0000 7f1b5247b740 1 -- 172.21.15.1:0/185481283 shutdown_connections 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.630+0000 7f1b5247b740 1 -- 172.21.15.1:0/185481283 wait complete. 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.630+0000 7f1b5247b740 1 -- start start 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.631+0000 7f1b5247b740 1 Processor -- start 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.631+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae739dd0 0x55e0ae741f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.631+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae742460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.631+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae744fb0 0x55e0ae7429a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.631+0000 7f1b5247b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e0ae747be0 con 0x55e0ae744fb0 2024-04-26T21:37:39.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.631+0000 7f1b5247b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e0ae626c60 con 0x55e0ae73a6f0 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.631+0000 7f1b5247b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e0ae747d60 con 0x55e0ae739dd0 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.631+0000 7f1b5247b740 10 client.0 _ll_register_callbacks cb 0x55e0ae652e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b47e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae742460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b47e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae742460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36308/0 (socket says 172.21.15.1:36308) 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b47e00640 1 -- 172.21.15.1:0/1770249225 learned_addr learned my addr 172.21.15.1:0/1770249225 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b4cc00640 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae739dd0 0x55e0ae741f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b47e00640 1 -- 172.21.15.1:0/1770249225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae739dd0 msgr2=0x55e0ae741f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b47e00640 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae739dd0 0x55e0ae741f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b47e00640 1 -- 172.21.15.1:0/1770249225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae744fb0 msgr2=0x55e0ae7429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b47e00640 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae744fb0 0x55e0ae7429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b47e00640 1 -- 172.21.15.1:0/1770249225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0ae635510 con 0x55e0ae73a6f0 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b4cc00640 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae739dd0 0x55e0ae741f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b47e00640 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae742460 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1b3409cb20 tx=0x7f1b34009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afdfb24a4ee86aa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b2e000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b2e000640 1 -- 172.21.15.1:0/1770249225 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b34098e30 con 0x55e0ae73a6f0 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.632+0000 7f1b2e000640 1 -- 172.21.15.1:0/1770249225 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f1b340d1b80 con 0x55e0ae73a6f0 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.633+0000 7f1b5247b740 1 -- 172.21.15.1:0/1770249225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b340d32d0 con 0x55e0ae73a6f0 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.633+0000 7f1b2e000640 1 -- 172.21.15.1:0/1770249225 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1b3400ce60 con 0x55e0ae73a6f0 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.633+0000 7f1b5247b740 10 client.26102 fetch_fsmap learned FSMap version 36 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.633+0000 7f1b5247b740 1 -- 172.21.15.1:0/1770249225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e0ae3b1210 con 0x55e0ae73a6f0 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.633+0000 7f1b2e000640 1 -- 172.21.15.1:0/1770249225 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b340d32d0 con 0x55e0ae73a6f0 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.633+0000 7f1b2e000640 1 -- 172.21.15.1:0/1770249225 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f1b340ce5c0 con 0x55e0ae73a6f0 2024-04-26T21:37:39.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.634+0000 7f1b5247b740 10 client.26102 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:37:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.634+0000 7f1b5247b740 10 client.26102 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:37:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.634+0000 7f1b5247b740 20 client.26102 populate_metadata read hostname 'smithi001' 2024-04-26T21:37:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.634+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e0ae754ad0 0x55e0ae756ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.634+0000 7f1b5247b740 4 client.26102 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:37:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.634+0000 7f1b5247b740 1 -- 172.21.15.1:0/1770249225 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0ae6acf90 con 0x55e0ae754ad0 2024-04-26T21:37:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.635+0000 7f1b4cc00640 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e0ae754ad0 0x55e0ae756ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.635+0000 7f1b4cc00640 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e0ae754ad0 0x55e0ae756ec0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8920e23b77482fe7 server_cookie=cbe99afb7bd8e0da in_seq=0 out_seq=0 2024-04-26T21:37:39.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.635+0000 7f1b2e000640 10 client.26102 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.636+0000 7f1b2e000640 1 -- 172.21.15.1:0/1770249225 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e0ae6acf90 con 0x55e0ae754ad0 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.636+0000 7f1b2e000640 10 client.26102 handle_command_reply: tid=0 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.636+0000 7f1b5247b740 1 client.26102 shutdown 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 1 -- 172.21.15.1:0/1770249225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 msgr2=0x55e0ae742460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae742460 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1b3409cb20 tx=0x7f1b34009f90 comp rx=0 tx=0).stop 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 1 -- 172.21.15.1:0/1770249225 shutdown_connections 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e0ae754ad0 0x55e0ae756ec0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae744fb0 0x55e0ae7429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae742460 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b2e000640 0 client.26102 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:39.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1770249225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae739dd0 0x55e0ae741f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 1 -- 172.21.15.1:0/1770249225 >> 172.21.15.1:0/1770249225 conn(0x55e0ae7362e0 msgr2=0x55e0ae73c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 1 -- 172.21.15.1:0/1770249225 shutdown_connections 2024-04-26T21:37:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 1 -- 172.21.15.1:0/1770249225 wait complete. 2024-04-26T21:37:39.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.637+0000 7f1b5247b740 20 client.26102 trim_cache size 0 max 16384 2024-04-26T21:37:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.656+0000 7f1b5247b740 1 Processor -- start 2024-04-26T21:37:39.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.656+0000 7f1b5247b740 1 -- start start 2024-04-26T21:37:39.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae744650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 0x55e0ae73c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7506c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b5247b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1b481b8f10 con 0x55e0ae739dd0 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b5247b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1b481b67e0 con 0x55e0ae744fb0 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b5247b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1b481b4650 con 0x55e0ae73a6f0 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b4d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b4cc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae744650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b4d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36312/0 (socket says 172.21.15.1:36312) 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b4d600640 1 -- 172.21.15.1:0/1524658255 learned_addr learned my addr 172.21.15.1:0/1524658255 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b47e00640 1 --2- 172.21.15.1:0/1524658255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 0x55e0ae73c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b4d600640 1 -- 172.21.15.1:0/1524658255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 msgr2=0x55e0ae73c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.657+0000 7f1b4d600640 1 --2- 172.21.15.1:0/1524658255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 0x55e0ae73c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.658+0000 7f1b4d600640 1 -- 172.21.15.1:0/1524658255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 msgr2=0x55e0ae744650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.658+0000 7f1b4d600640 1 --2- 172.21.15.1:0/1524658255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae744650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.658+0000 7f1b4d600640 1 -- 172.21.15.1:0/1524658255 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0ae47d620 con 0x55e0ae744fb0 2024-04-26T21:37:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.658+0000 7f1b4cc00640 1 --2- 172.21.15.1:0/1524658255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae744650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:37:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.658+0000 7f1b4d600640 1 --2- 172.21.15.1:0/1524658255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7506c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1b40009950 tx=0x7f1b40014070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72324b6f3f259a1d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.658+0000 7f1b39200640 1 -- 172.21.15.1:0/1524658255 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4000f8d0 con 0x55e0ae744fb0 2024-04-26T21:37:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.658+0000 7f1b39200640 1 -- 172.21.15.1:0/1524658255 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b4001f420 con 0x55e0ae744fb0 2024-04-26T21:37:39.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.658+0000 7f1b39200640 1 -- 172.21.15.1:0/1524658255 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40027660 con 0x55e0ae744fb0 2024-04-26T21:37:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.659+0000 7f1b5247b740 1 -- 172.21.15.1:0/1524658255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 msgr2=0x55e0ae7506c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.659+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1524658255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7506c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1b40009950 tx=0x7f1b40014070 comp rx=0 tx=0).stop 2024-04-26T21:37:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.659+0000 7f1b5247b740 1 -- 172.21.15.1:0/1524658255 shutdown_connections 2024-04-26T21:37:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.659+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1524658255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae73a6f0 0x55e0ae73c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.659+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1524658255 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae744fb0 0x55e0ae7506c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.659+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1524658255 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae744650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.659+0000 7f1b5247b740 1 -- 172.21.15.1:0/1524658255 >> 172.21.15.1:0/1524658255 conn(0x55e0ae7362e0 msgr2=0x55e0ae74ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.660+0000 7f1b5247b740 1 -- 172.21.15.1:0/1524658255 shutdown_connections 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.660+0000 7f1b5247b740 1 -- 172.21.15.1:0/1524658255 wait complete. 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.660+0000 7f1b5247b740 1 -- start start 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.660+0000 7f1b5247b740 1 Processor -- start 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae75a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae7574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b5247b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae744fb0 0x55e0ae7579e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b5247b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1b481b4ca0 con 0x55e0ae739dd0 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b5247b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1b481b6110 con 0x55e0ae73a6f0 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b5247b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1b481b5370 con 0x55e0ae744fb0 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b47e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae7574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b47e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae7574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36316/0 (socket says 172.21.15.1:36316) 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b47e00640 1 -- 172.21.15.1:0/1063823653 learned_addr learned my addr 172.21.15.1:0/1063823653 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b5247b740 10 client.0 _ll_register_callbacks cb 0x7f1b340e2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b4cc00640 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae75a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.661+0000 7f1b4d600640 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae744fb0 0x55e0ae7579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.662+0000 7f1b47e00640 1 -- 172.21.15.1:0/1063823653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae744fb0 msgr2=0x55e0ae7579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.662+0000 7f1b47e00640 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae744fb0 0x55e0ae7579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.662+0000 7f1b47e00640 1 -- 172.21.15.1:0/1063823653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 msgr2=0x55e0ae75a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.662+0000 7f1b47e00640 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae75a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.662+0000 7f1b47e00640 1 -- 172.21.15.1:0/1063823653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0ae659cb0 con 0x55e0ae73a6f0 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.662+0000 7f1b47e00640 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae7574a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1b34099560 tx=0x7f1b340d9a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b6228faa558a74c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.662+0000 7f1b2e000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:37:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.662+0000 7f1b2e000640 1 -- 172.21.15.1:0/1063823653 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3409c4f0 con 0x55e0ae73a6f0 2024-04-26T21:37:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.662+0000 7f1b2e000640 1 -- 172.21.15.1:0/1063823653 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7f1b340bd0b0 con 0x55e0ae73a6f0 2024-04-26T21:37:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.662+0000 7f1b5247b740 1 -- 172.21.15.1:0/1063823653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b4001f420 con 0x55e0ae73a6f0 2024-04-26T21:37:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b2e000640 1 -- 172.21.15.1:0/1063823653 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1b3409c6d0 con 0x55e0ae73a6f0 2024-04-26T21:37:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b5247b740 10 client.26108 fetch_fsmap learned FSMap version 36 2024-04-26T21:37:39.666 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b5247b740 1 -- 172.21.15.1:0/1063823653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e0ae7ba960 con 0x55e0ae73a6f0 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b2e000640 1 -- 172.21.15.1:0/1063823653 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b4001f420 con 0x55e0ae73a6f0 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b2e000640 1 -- 172.21.15.1:0/1063823653 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f1b340a5370 con 0x55e0ae73a6f0 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b5247b740 10 client.26108 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b5247b740 10 client.26108 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b5247b740 20 client.26108 populate_metadata read hostname 'smithi001' 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e0ae7baea0 0x55e0ae7bd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b5247b740 4 client.26108 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.663+0000 7f1b5247b740 1 -- 172.21.15.1:0/1063823653 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e0ae6a6e00 con 0x55e0ae7baea0 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.664+0000 7f1b4cc00640 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e0ae7baea0 0x55e0ae7bd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.664+0000 7f1b4cc00640 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e0ae7baea0 0x55e0ae7bd2c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46ba1991ace227b4 server_cookie=6498090260905001 in_seq=0 out_seq=0 2024-04-26T21:37:39.668 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.664+0000 7f1b2e000640 10 client.26108 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:39.726 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b2e000640 1 -- 172.21.15.1:0/1063823653 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e0ae6a6e00 con 0x55e0ae7baea0 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b2e000640 10 client.26108 handle_command_reply: tid=0 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 client.26108 shutdown 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 -- 172.21.15.1:0/1063823653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 msgr2=0x55e0ae7574a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae7574a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1b34099560 tx=0x7f1b340d9a70 comp rx=0 tx=0).stop 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 -- 172.21.15.1:0/1063823653 shutdown_connections 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e0ae7baea0 0x55e0ae7bd2c0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e0ae744fb0 0x55e0ae7579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e0ae73a6f0 0x55e0ae7574a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 --2- 172.21.15.1:0/1063823653 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e0ae739dd0 0x55e0ae75a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 -- 172.21.15.1:0/1063823653 >> 172.21.15.1:0/1063823653 conn(0x55e0ae7362e0 msgr2=0x55e0ae7403b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b2e000640 0 client.26108 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 -- 172.21.15.1:0/1063823653 shutdown_connections 2024-04-26T21:37:39.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 1 -- 172.21.15.1:0/1063823653 wait complete. 2024-04-26T21:37:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.726+0000 7f1b5247b740 20 client.26108 trim_cache size 0 max 16384 2024-04-26T21:37:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:37:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:37:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:37:39.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:37:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 -- 172.21.15.1:0/2159569264 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1b300a3e10 msgr2=0x7f1b300a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1b300a3e10 0x7f1b300a62d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1b40005fd0 tx=0x7f1b40009500 comp rx=0 tx=0).stop 2024-04-26T21:37:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 -- 172.21.15.1:0/2159569264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b4806b4e0 msgr2=0x7f1b4810df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b4806b4e0 0x7f1b4810df50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400e9c0 tx=0x7f1b3400ee80 comp rx=0 tx=0).stop 2024-04-26T21:37:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 -- 172.21.15.1:0/2159569264 shutdown_connections 2024-04-26T21:37:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1b300a3e10 0x7f1b300a62d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.729 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b4806b4e0 0x7f1b4810df50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b48108a20 0x7f1b481a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 --2- 172.21.15.1:0/2159569264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4806ab40 0x7f1b4810da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 -- 172.21.15.1:0/2159569264 >> 172.21.15.1:0/2159569264 conn(0x7f1b48078710 msgr2=0x7f1b481066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 -- 172.21.15.1:0/2159569264 shutdown_connections 2024-04-26T21:37:39.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:39.727+0000 7f1b4f600640 1 -- 172.21.15.1:0/2159569264 wait complete. 2024-04-26T21:37:40.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:39 smithi001 ceph-mon[28054]: pgmap v225: 129 pgs: 129 active+clean; 139 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-04-26T21:37:40.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:39 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:37:40.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:39 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:37:40.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:39 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:37:40.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:39 smithi038 ceph-mon[32724]: pgmap v225: 129 pgs: 129 active+clean; 139 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-04-26T21:37:40.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:39 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:37:40.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:39 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:37:40.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:39 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:37:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:39 smithi186 ceph-mon[32991]: pgmap v225: 129 pgs: 129 active+clean; 139 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.2 MiB/s wr, 150 op/s 2024-04-26T21:37:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:39 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:37:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:39 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:37:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:39 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:37:40.570 INFO:tasks.workunit.client.0.smithi001.stdout: 1 243586 4.38 MB/sec execute 218 sec latency 31.978 ms 2024-04-26T21:37:41.570 INFO:tasks.workunit.client.0.smithi001.stdout: 1 244535 4.39 MB/sec execute 219 sec latency 45.702 ms 2024-04-26T21:37:42.257 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:41 smithi186 ceph-mon[32991]: pgmap v226: 129 pgs: 129 active+clean; 143 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-26T21:37:42.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:41 smithi001 ceph-mon[28054]: pgmap v226: 129 pgs: 129 active+clean; 143 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-26T21:37:42.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:41 smithi038 ceph-mon[32724]: pgmap v226: 129 pgs: 129 active+clean; 143 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 216 op/s 2024-04-26T21:37:42.570 INFO:tasks.workunit.client.0.smithi001.stdout: 1 245166 4.40 MB/sec execute 220 sec latency 25.603 ms 2024-04-26T21:37:43.570 INFO:tasks.workunit.client.0.smithi001.stdout: 1 245699 4.38 MB/sec execute 221 sec latency 27.415 ms 2024-04-26T21:37:44.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:44 smithi001 ceph-mon[28054]: pgmap v227: 129 pgs: 129 active+clean; 143 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-26T21:37:44.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:44 smithi038 ceph-mon[32724]: pgmap v227: 129 pgs: 129 active+clean; 143 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-26T21:37:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:44 smithi186 ceph-mon[32991]: pgmap v227: 129 pgs: 129 active+clean; 143 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-26T21:37:44.571 INFO:tasks.workunit.client.0.smithi001.stdout: 1 247195 4.39 MB/sec execute 222 sec latency 20.630 ms 2024-04-26T21:37:45.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 248441 4.42 MB/sec execute 223 sec latency 21.454 ms 2024-04-26T21:37:46.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:46 smithi001 ceph-mon[28054]: pgmap v228: 129 pgs: 129 active+clean; 146 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-26T21:37:46.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:46 smithi038 ceph-mon[32724]: pgmap v228: 129 pgs: 129 active+clean; 146 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-26T21:37:46.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:46 smithi186 ceph-mon[32991]: pgmap v228: 129 pgs: 129 active+clean; 146 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 211 op/s 2024-04-26T21:37:46.571 INFO:tasks.workunit.client.0.smithi001.stdout: 1 248778 4.40 MB/sec execute 224 sec latency 24.501 ms 2024-04-26T21:37:47.571 INFO:tasks.workunit.client.0.smithi001.stdout: 1 249514 4.39 MB/sec execute 225 sec latency 67.767 ms 2024-04-26T21:37:48.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:48 smithi001 ceph-mon[28054]: pgmap v229: 129 pgs: 129 active+clean; 146 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:37:48.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:48 smithi038 ceph-mon[32724]: pgmap v229: 129 pgs: 129 active+clean; 146 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:37:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:48 smithi186 ceph-mon[32991]: pgmap v229: 129 pgs: 129 active+clean; 146 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:37:48.571 INFO:tasks.workunit.client.0.smithi001.stdout: 1 250924 4.40 MB/sec execute 226 sec latency 23.381 ms 2024-04-26T21:37:49.571 INFO:tasks.workunit.client.0.smithi001.stdout: 1 252123 4.43 MB/sec execute 227 sec latency 23.187 ms 2024-04-26T21:37:50.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:50 smithi001 ceph-mon[28054]: pgmap v230: 129 pgs: 129 active+clean; 146 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:37:50.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:50 smithi038 ceph-mon[32724]: pgmap v230: 129 pgs: 129 active+clean; 146 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:37:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:50 smithi186 ceph-mon[32991]: pgmap v230: 129 pgs: 129 active+clean; 146 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-04-26T21:37:50.571 INFO:tasks.workunit.client.0.smithi001.stdout: 1 252413 4.41 MB/sec execute 228 sec latency 24.427 ms 2024-04-26T21:37:51.571 INFO:tasks.workunit.client.0.smithi001.stdout: 1 253289 4.41 MB/sec execute 229 sec latency 27.664 ms 2024-04-26T21:37:52.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:52 smithi001 ceph-mon[28054]: pgmap v231: 129 pgs: 129 active+clean; 150 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 222 op/s 2024-04-26T21:37:52.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:52 smithi038 ceph-mon[32724]: pgmap v231: 129 pgs: 129 active+clean; 150 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 222 op/s 2024-04-26T21:37:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:52 smithi186 ceph-mon[32991]: pgmap v231: 129 pgs: 129 active+clean; 150 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 222 op/s 2024-04-26T21:37:52.572 INFO:tasks.workunit.client.0.smithi001.stdout: 1 254772 4.42 MB/sec execute 230 sec latency 52.388 ms 2024-04-26T21:37:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.903+0000 7fd133a00640 1 -- 172.21.15.1:0/1363804108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd12c115a80 msgr2=0x7fd12c117e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.903+0000 7fd133a00640 1 --2- 172.21.15.1:0/1363804108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd12c115a80 0x7fd12c117e70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd124009ca0 tx=0x7fd12402a940 comp rx=0 tx=0).stop 2024-04-26T21:37:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.903+0000 7fd133a00640 1 -- 172.21.15.1:0/1363804108 shutdown_connections 2024-04-26T21:37:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.903+0000 7fd133a00640 1 --2- 172.21.15.1:0/1363804108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd12c10a460 0x7fd12c115540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.903+0000 7fd133a00640 1 --2- 172.21.15.1:0/1363804108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd12c115a80 0x7fd12c117e70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.903+0000 7fd133a00640 1 --2- 172.21.15.1:0/1363804108 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd12c109b40 0x7fd12c109f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.904+0000 7fd133a00640 1 -- 172.21.15.1:0/1363804108 >> 172.21.15.1:0/1363804108 conn(0x7fd12c06da10 msgr2=0x7fd12c06de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.904+0000 7fd133a00640 1 -- 172.21.15.1:0/1363804108 shutdown_connections 2024-04-26T21:37:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.904+0000 7fd133a00640 1 -- 172.21.15.1:0/1363804108 wait complete. 2024-04-26T21:37:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.905+0000 7fd133a00640 1 Processor -- start 2024-04-26T21:37:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.906+0000 7fd133a00640 1 -- start start 2024-04-26T21:37:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.906+0000 7fd133a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd12c109b40 0x7fd12c111f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.906+0000 7fd133a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd12c10a460 0x7fd12c112480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.906+0000 7fd133a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd12c115a80 0x7fd12c1b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.906+0000 7fd133a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c10c630 con 0x7fd12c10a460 2024-04-26T21:37:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.906+0000 7fd133a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c10c4b0 con 0x7fd12c109b40 2024-04-26T21:37:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.906+0000 7fd133a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c10c7b0 con 0x7fd12c115a80 2024-04-26T21:37:52.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd131000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd12c109b40 0x7fd12c111f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd131a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd12c115a80 0x7fd12c1b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd131a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd12c115a80 0x7fd12c1b3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43124/0 (socket says 172.21.15.1:43124) 2024-04-26T21:37:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd131a00640 1 -- 172.21.15.1:0/267557890 learned_addr learned my addr 172.21.15.1:0/267557890 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd131000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd12c109b40 0x7fd12c111f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:49408/0 (socket says 172.21.15.1:49408) 2024-04-26T21:37:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd12be00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd12c10a460 0x7fd12c112480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd131a00640 1 -- 172.21.15.1:0/267557890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd12c109b40 msgr2=0x7fd12c111f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd131a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd12c109b40 0x7fd12c111f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd131a00640 1 -- 172.21.15.1:0/267557890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd12c10a460 msgr2=0x7fd12c112480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd131a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd12c10a460 0x7fd12c112480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.907+0000 7fd131a00640 1 -- 172.21.15.1:0/267557890 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12c1b4100 con 0x7fd12c115a80 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.908+0000 7fd131a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd12c115a80 0x7fd12c1b3a60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd124003ba0 tx=0x7fd124003bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.908+0000 7fd129600640 1 -- 172.21.15.1:0/267557890 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd124034040 con 0x7fd12c115a80 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.908+0000 7fd129600640 1 -- 172.21.15.1:0/267557890 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd12403e070 con 0x7fd12c115a80 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.908+0000 7fd133a00640 1 -- 172.21.15.1:0/267557890 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd12c1b4390 con 0x7fd12c115a80 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.908+0000 7fd129600640 1 -- 172.21.15.1:0/267557890 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd124043470 con 0x7fd12c115a80 2024-04-26T21:37:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.908+0000 7fd133a00640 1 -- 172.21.15.1:0/267557890 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd12c1b48d0 con 0x7fd12c115a80 2024-04-26T21:37:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.912+0000 7fd129600640 1 -- 172.21.15.1:0/267557890 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd124037020 con 0x7fd12c115a80 2024-04-26T21:37:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.912+0000 7fd13685e740 1 Processor -- start 2024-04-26T21:37:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.912+0000 7fd13685e740 1 -- start start 2024-04-26T21:37:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.912+0000 7fd13685e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d00591b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.912+0000 7fd13685e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d00596f0 0x5590d0063a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.912+0000 7fd13685e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0063fb0 0x5590d00663a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.912+0000 7fd13685e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5590cffbe380 con 0x5590d0063fb0 2024-04-26T21:37:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.912+0000 7fd13685e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5590cff5ac30 con 0x5590d0058dd0 2024-04-26T21:37:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.912+0000 7fd13685e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5590cffba340 con 0x5590d00596f0 2024-04-26T21:37:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd131000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d00596f0 0x5590d0063a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd12be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d00591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd12be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d00591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:49422/0 (socket says 172.21.15.1:49422) 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd12be00640 1 -- 172.21.15.1:0/1329453865 learned_addr learned my addr 172.21.15.1:0/1329453865 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd131000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d00596f0 0x5590d0063a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43134/0 (socket says 172.21.15.1:43134) 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd12be00640 1 --2- 172.21.15.1:0/1329453865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0063fb0 0x5590d00663a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd129600640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1140a3dc0 0x7fd1140a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd12be00640 1 -- 172.21.15.1:0/1329453865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d00596f0 msgr2=0x5590d0063a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd12be00640 1 --2- 172.21.15.1:0/1329453865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d00596f0 0x5590d0063a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd12be00640 1 -- 172.21.15.1:0/1329453865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0063fb0 msgr2=0x5590d00663a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd12be00640 1 --2- 172.21.15.1:0/1329453865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0063fb0 0x5590d00663a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd12be00640 1 -- 172.21.15.1:0/1329453865 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590cff7c940 con 0x5590d0058dd0 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.913+0000 7fd129600640 1 -- 172.21.15.1:0/267557890 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7fd1240f1f40 con 0x7fd12c115a80 2024-04-26T21:37:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.914+0000 7fd12be00640 1 --2- 172.21.15.1:0/1329453865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d00591b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd11c005e10 tx=0x7fd11c00e430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a2d0b2efdbf9b45 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:52.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.914+0000 7fd11a000640 1 -- 172.21.15.1:0/1329453865 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c018070 con 0x5590d0058dd0 2024-04-26T21:37:52.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.914+0000 7fd11a000640 1 -- 172.21.15.1:0/1329453865 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd11c004220 con 0x5590d0058dd0 2024-04-26T21:37:52.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.914+0000 7fd131a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1140a3dc0 0x7fd1140a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.914+0000 7fd11a000640 1 -- 172.21.15.1:0/1329453865 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c003970 con 0x5590d0058dd0 2024-04-26T21:37:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd13685e740 1 -- 172.21.15.1:0/1329453865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 msgr2=0x5590d00591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd13685e740 1 --2- 172.21.15.1:0/1329453865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d00591b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd11c005e10 tx=0x7fd11c00e430 comp rx=0 tx=0).stop 2024-04-26T21:37:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd131a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1140a3dc0 0x7fd1140a6280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd1240ba500 tx=0x7fd124005ef0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:52.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd13685e740 1 -- 172.21.15.1:0/1329453865 shutdown_connections 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd13685e740 1 --2- 172.21.15.1:0/1329453865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d00596f0 0x5590d0063a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd13685e740 1 --2- 172.21.15.1:0/1329453865 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0063fb0 0x5590d00663a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd13685e740 1 --2- 172.21.15.1:0/1329453865 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d00591b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd13685e740 1 -- 172.21.15.1:0/1329453865 >> 172.21.15.1:0/1329453865 conn(0x5590d00552e0 msgr2=0x5590d00556c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd13685e740 1 -- 172.21.15.1:0/1329453865 shutdown_connections 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd13685e740 1 -- 172.21.15.1:0/1329453865 wait complete. 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.915+0000 7fd13685e740 1 -- start start 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd13685e740 1 Processor -- start 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd13685e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d0060ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd13685e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d00596f0 0x5590d0061430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd13685e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0063fb0 0x5590d0061970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd13685e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5590d0066be0 con 0x5590d00596f0 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd13685e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5590cff45c60 con 0x5590d0058dd0 2024-04-26T21:37:52.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd13685e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5590d0066d60 con 0x5590d0063fb0 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd13685e740 10 client.0 _ll_register_callbacks cb 0x5590cff71e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd12be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d0060ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd12be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d0060ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:49436/0 (socket says 172.21.15.1:49436) 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.916+0000 7fd12be00640 1 -- 172.21.15.1:0/3144430704 learned_addr learned my addr 172.21.15.1:0/3144430704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.917+0000 7fd12be00640 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0063fb0 0x5590d0061970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.917+0000 7fd131000640 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d00596f0 0x5590d0061430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.917+0000 7fd12be00640 1 -- 172.21.15.1:0/3144430704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0063fb0 msgr2=0x5590d0061970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.917+0000 7fd12be00640 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0063fb0 0x5590d0061970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.917+0000 7fd12be00640 1 -- 172.21.15.1:0/3144430704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d00596f0 msgr2=0x5590d0061430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.917+0000 7fd12be00640 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d00596f0 0x5590d0061430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.917+0000 7fd12be00640 1 -- 172.21.15.1:0/3144430704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590cff54510 con 0x5590d0058dd0 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.917+0000 7fd12be00640 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d0060ef0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00ed50 tx=0x7fd11c0035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccba6689105d2117 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.917+0000 7fd10ea00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:37:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.918+0000 7fd10ea00640 1 -- 172.21.15.1:0/3144430704 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c025030 con 0x5590d0058dd0 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.918+0000 7fd10ea00640 1 -- 172.21.15.1:0/3144430704 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7fd11c005260 con 0x5590d0058dd0 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.918+0000 7fd13685e740 1 -- 172.21.15.1:0/3144430704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd11c004220 con 0x5590d0058dd0 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.918+0000 7fd10ea00640 1 -- 172.21.15.1:0/3144430704 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd11c032050 con 0x5590d0058dd0 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.918+0000 7fd13685e740 10 client.26120 fetch_fsmap learned FSMap version 36 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.918+0000 7fd13685e740 1 -- 172.21.15.1:0/3144430704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5590cfcd0210 con 0x5590d0058dd0 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.918+0000 7fd10ea00640 1 -- 172.21.15.1:0/3144430704 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd11c004220 con 0x5590d0058dd0 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.918+0000 7fd10ea00640 1 -- 172.21.15.1:0/3144430704 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fd11c0036f0 con 0x5590d0058dd0 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.918+0000 7fd13685e740 10 client.26120 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.919+0000 7fd13685e740 10 client.26120 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.919+0000 7fd13685e740 20 client.26120 populate_metadata read hostname 'smithi001' 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.919+0000 7fd13685e740 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5590d0073c10 0x5590d0076000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.919+0000 7fd13685e740 4 client.26120 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.919+0000 7fd13685e740 1 -- 172.21.15.1:0/3144430704 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590cffcbf90 con 0x5590d0073c10 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.919+0000 7fd131000640 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5590d0073c10 0x5590d0076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.919+0000 7fd131000640 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5590d0073c10 0x5590d0076000 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb49557ca9f58107 server_cookie=19288ac784bc8fa0 in_seq=0 out_seq=0 2024-04-26T21:37:52.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.919+0000 7fd10ea00640 10 client.26120 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.920+0000 7fd10ea00640 1 -- 172.21.15.1:0/3144430704 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590cffcbf90 con 0x5590d0073c10 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.920+0000 7fd10ea00640 10 client.26120 handle_command_reply: tid=0 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.920+0000 7fd13685e740 1 client.26120 shutdown 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 1 -- 172.21.15.1:0/3144430704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 msgr2=0x5590d0060ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d0060ef0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00ed50 tx=0x7fd11c0035f0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 1 -- 172.21.15.1:0/3144430704 shutdown_connections 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5590d0073c10 0x5590d0076000 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0063fb0 0x5590d0061970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d00596f0 0x5590d0061430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 1 --2- 172.21.15.1:0/3144430704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d0058dd0 0x5590d0060ef0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd10ea00640 0 client.26120 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:52.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 1 -- 172.21.15.1:0/3144430704 >> 172.21.15.1:0/3144430704 conn(0x5590d00552e0 msgr2=0x5590d005b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:52.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 1 -- 172.21.15.1:0/3144430704 shutdown_connections 2024-04-26T21:37:52.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 1 -- 172.21.15.1:0/3144430704 wait complete. 2024-04-26T21:37:52.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.921+0000 7fd13685e740 20 client.26120 trim_cache size 0 max 16384 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.939+0000 7fd13685e740 1 Processor -- start 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.939+0000 7fd13685e740 1 -- start start 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.939+0000 7fd13685e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0058dd0 0x5590d006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.939+0000 7fd13685e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d005b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.939+0000 7fd13685e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0063fb0 0x5590d007b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.939+0000 7fd13685e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1c1570 con 0x5590d0058dd0 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.939+0000 7fd13685e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1bee40 con 0x5590d00596f0 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.939+0000 7fd13685e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1bccb0 con 0x5590d0063fb0 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd12be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0058dd0 0x5590d006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd131000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d005b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd12be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0058dd0 0x5590d006faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49356/0 (socket says 172.21.15.1:49356) 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd12be00640 1 -- 172.21.15.1:0/319140320 learned_addr learned my addr 172.21.15.1:0/319140320 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd131000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d005b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:49450/0 (socket says 172.21.15.1:49450) 2024-04-26T21:37:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd12be00640 1 --2- 172.21.15.1:0/319140320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0063fb0 0x5590d007b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd131000640 1 -- 172.21.15.1:0/319140320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0063fb0 msgr2=0x5590d007b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd131000640 1 --2- 172.21.15.1:0/319140320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0063fb0 0x5590d007b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd131000640 1 -- 172.21.15.1:0/319140320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0058dd0 msgr2=0x5590d006faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd131000640 1 --2- 172.21.15.1:0/319140320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0058dd0 0x5590d006faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd131000640 1 -- 172.21.15.1:0/319140320 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590cfd9c620 con 0x5590d00596f0 2024-04-26T21:37:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.940+0000 7fd12be00640 1 --2- 172.21.15.1:0/319140320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0058dd0 0x5590d006faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:37:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.941+0000 7fd131000640 1 --2- 172.21.15.1:0/319140320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d005b790 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x5590d00765a0 tx=0x7fd12000a2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42306e986a990a6d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.941+0000 7fd119600640 1 -- 172.21.15.1:0/319140320 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd120005220 con 0x5590d00596f0 2024-04-26T21:37:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.941+0000 7fd119600640 1 -- 172.21.15.1:0/319140320 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd120004c50 con 0x5590d00596f0 2024-04-26T21:37:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.941+0000 7fd119600640 1 -- 172.21.15.1:0/319140320 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd120020570 con 0x5590d00596f0 2024-04-26T21:37:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 -- 172.21.15.1:0/319140320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 msgr2=0x5590d005b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 --2- 172.21.15.1:0/319140320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d005b790 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x5590d00765a0 tx=0x7fd12000a2d0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 -- 172.21.15.1:0/319140320 shutdown_connections 2024-04-26T21:37:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 --2- 172.21.15.1:0/319140320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0063fb0 0x5590d007b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 --2- 172.21.15.1:0/319140320 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d005b790 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 --2- 172.21.15.1:0/319140320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0058dd0 0x5590d006faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 -- 172.21.15.1:0/319140320 >> 172.21.15.1:0/319140320 conn(0x5590d00552e0 msgr2=0x5590d005f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 -- 172.21.15.1:0/319140320 shutdown_connections 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 -- 172.21.15.1:0/319140320 wait complete. 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 -- start start 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.942+0000 7fd13685e740 1 Processor -- start 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd13685e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0058dd0 0x5590d006bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd13685e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d006c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd13685e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0063fb0 0x5590d005d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd13685e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1bd300 con 0x5590d0063fb0 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd13685e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1be770 con 0x5590d00596f0 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd13685e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1bd9d0 con 0x5590d0058dd0 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd13685e740 10 client.0 _ll_register_callbacks cb 0x7fd12000a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd131000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d006c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd12be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0058dd0 0x5590d006bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd131000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d006c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:49456/0 (socket says 172.21.15.1:49456) 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd131000640 1 -- 172.21.15.1:0/937660590 learned_addr learned my addr 172.21.15.1:0/937660590 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.943+0000 7fd12be00640 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0063fb0 0x5590d005d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd131000640 1 -- 172.21.15.1:0/937660590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0058dd0 msgr2=0x5590d006bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd131000640 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0058dd0 0x5590d006bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd131000640 1 -- 172.21.15.1:0/937660590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0063fb0 msgr2=0x5590d005d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd131000640 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0063fb0 0x5590d005d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd131000640 1 -- 172.21.15.1:0/937660590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590cff78cb0 con 0x5590d00596f0 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd131000640 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d006c4f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd120000c00 tx=0x7fd120004f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f21fafe1b42d062d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd10ea00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd10ea00640 1 -- 172.21.15.1:0/937660590 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd120020210 con 0x5590d00596f0 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd10ea00640 1 -- 172.21.15.1:0/937660590 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7fd1200203b0 con 0x5590d00596f0 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd13685e740 1 -- 172.21.15.1:0/937660590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd120004c50 con 0x5590d00596f0 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.944+0000 7fd10ea00640 1 -- 172.21.15.1:0/937660590 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd12003b050 con 0x5590d00596f0 2024-04-26T21:37:52.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.945+0000 7fd13685e740 10 client.26126 fetch_fsmap learned FSMap version 36 2024-04-26T21:37:52.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.945+0000 7fd13685e740 1 -- 172.21.15.1:0/937660590 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5590d00d9980 con 0x5590d00596f0 2024-04-26T21:37:52.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.945+0000 7fd10ea00640 1 -- 172.21.15.1:0/937660590 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd120004c50 con 0x5590d00596f0 2024-04-26T21:37:52.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.945+0000 7fd10ea00640 1 -- 172.21.15.1:0/937660590 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fd120030020 con 0x5590d00596f0 2024-04-26T21:37:52.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.945+0000 7fd13685e740 10 client.26126 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:37:52.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.945+0000 7fd13685e740 10 client.26126 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:37:52.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.945+0000 7fd13685e740 20 client.26126 populate_metadata read hostname 'smithi001' 2024-04-26T21:37:52.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.945+0000 7fd13685e740 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5590d00da070 0x5590d00dc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:37:52.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.945+0000 7fd13685e740 4 client.26126 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:37:52.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.945+0000 7fd13685e740 1 -- 172.21.15.1:0/937660590 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5590cffc5e00 con 0x5590d00da070 2024-04-26T21:37:52.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.946+0000 7fd12be00640 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5590d00da070 0x5590d00dc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:37:52.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.946+0000 7fd12be00640 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5590d00da070 0x5590d00dc490 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3545f385b1d6b0ae server_cookie=98968588a1da1caa in_seq=0 out_seq=0 2024-04-26T21:37:52.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:52.946+0000 7fd10ea00640 10 client.26126 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.006+0000 7fd10ea00640 1 -- 172.21.15.1:0/937660590 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5590cffc5e00 con 0x5590d00da070 2024-04-26T21:37:53.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.006+0000 7fd10ea00640 10 client.26126 handle_command_reply: tid=0 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.006+0000 7fd13685e740 1 client.26126 shutdown 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.006+0000 7fd13685e740 1 -- 172.21.15.1:0/937660590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 msgr2=0x5590d006c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.006+0000 7fd13685e740 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d006c4f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd120000c00 tx=0x7fd120004f60 comp rx=0 tx=0).stop 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.007+0000 7fd13685e740 1 -- 172.21.15.1:0/937660590 shutdown_connections 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.007+0000 7fd13685e740 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5590d00da070 0x5590d00dc490 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.007+0000 7fd13685e740 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5590d0063fb0 0x5590d005d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.007+0000 7fd13685e740 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5590d00596f0 0x5590d006c4f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.007+0000 7fd13685e740 1 --2- 172.21.15.1:0/937660590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590d0058dd0 0x5590d006bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.007+0000 7fd13685e740 1 -- 172.21.15.1:0/937660590 >> 172.21.15.1:0/937660590 conn(0x5590d00552e0 msgr2=0x5590d0053bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.007+0000 7fd10ea00640 0 client.26126 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.007+0000 7fd13685e740 1 -- 172.21.15.1:0/937660590 shutdown_connections 2024-04-26T21:37:53.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.007+0000 7fd13685e740 1 -- 172.21.15.1:0/937660590 wait complete. 2024-04-26T21:37:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.007+0000 7fd13685e740 20 client.26126 trim_cache size 0 max 16384 2024-04-26T21:37:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:37:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:37:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:37:53.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:37:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.008+0000 7fd133a00640 1 -- 172.21.15.1:0/267557890 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1140a3dc0 msgr2=0x7fd1140a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.008+0000 7fd133a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1140a3dc0 0x7fd1140a6280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd1240ba500 tx=0x7fd124005ef0 comp rx=0 tx=0).stop 2024-04-26T21:37:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.008+0000 7fd133a00640 1 -- 172.21.15.1:0/267557890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd12c115a80 msgr2=0x7fd12c1b3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:37:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.008+0000 7fd133a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd12c115a80 0x7fd12c1b3a60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd124003ba0 tx=0x7fd124003bd0 comp rx=0 tx=0).stop 2024-04-26T21:37:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.008+0000 7fd133a00640 1 -- 172.21.15.1:0/267557890 shutdown_connections 2024-04-26T21:37:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.008+0000 7fd133a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1140a3dc0 0x7fd1140a6280 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.008+0000 7fd133a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd12c115a80 0x7fd12c1b3a60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:53.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.008+0000 7fd133a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd12c10a460 0x7fd12c112480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.008+0000 7fd133a00640 1 --2- 172.21.15.1:0/267557890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd12c109b40 0x7fd12c111f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:37:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.008+0000 7fd133a00640 1 -- 172.21.15.1:0/267557890 >> 172.21.15.1:0/267557890 conn(0x7fd12c06da10 msgr2=0x7fd12c0713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:37:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.009+0000 7fd133a00640 1 -- 172.21.15.1:0/267557890 shutdown_connections 2024-04-26T21:37:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:37:53.009+0000 7fd133a00640 1 -- 172.21.15.1:0/267557890 wait complete. 2024-04-26T21:37:53.573 INFO:tasks.workunit.client.0.smithi001.stdout: 1 255724 4.43 MB/sec execute 231 sec latency 26.097 ms 2024-04-26T21:37:54.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:54 smithi001 ceph-mon[28054]: pgmap v232: 129 pgs: 129 active+clean; 150 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.5 MiB/s wr, 155 op/s 2024-04-26T21:37:54.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:54 smithi038 ceph-mon[32724]: pgmap v232: 129 pgs: 129 active+clean; 150 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.5 MiB/s wr, 155 op/s 2024-04-26T21:37:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:54 smithi186 ceph-mon[32991]: pgmap v232: 129 pgs: 129 active+clean; 150 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.5 MiB/s wr, 155 op/s 2024-04-26T21:37:54.572 INFO:tasks.workunit.client.0.smithi001.stdout: 1 256099 4.42 MB/sec execute 232 sec latency 22.255 ms 2024-04-26T21:37:55.572 INFO:tasks.workunit.client.0.smithi001.stdout: 1 257601 4.42 MB/sec execute 233 sec latency 17.257 ms 2024-04-26T21:37:56.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:56 smithi001 ceph-mon[28054]: pgmap v233: 129 pgs: 129 active+clean; 149 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 239 op/s 2024-04-26T21:37:56.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:56 smithi038 ceph-mon[32724]: pgmap v233: 129 pgs: 129 active+clean; 149 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 239 op/s 2024-04-26T21:37:56.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:56 smithi186 ceph-mon[32991]: pgmap v233: 129 pgs: 129 active+clean; 149 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.1 MiB/s wr, 239 op/s 2024-04-26T21:37:56.572 INFO:tasks.workunit.client.0.smithi001.stdout: 1 258682 4.44 MB/sec execute 234 sec latency 24.241 ms 2024-04-26T21:37:57.572 INFO:tasks.workunit.client.0.smithi001.stdout: 1 259352 4.44 MB/sec execute 235 sec latency 30.070 ms 2024-04-26T21:37:58.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:37:58 smithi001 ceph-mon[28054]: pgmap v234: 129 pgs: 129 active+clean; 149 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-26T21:37:58.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:37:58 smithi038 ceph-mon[32724]: pgmap v234: 129 pgs: 129 active+clean; 149 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-26T21:37:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:37:58 smithi186 ceph-mon[32991]: pgmap v234: 129 pgs: 129 active+clean; 149 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-26T21:37:58.572 INFO:tasks.workunit.client.0.smithi001.stdout: 1 259950 4.43 MB/sec execute 236 sec latency 25.370 ms 2024-04-26T21:37:59.573 INFO:tasks.workunit.client.0.smithi001.stdout: 1 261304 4.43 MB/sec execute 237 sec latency 21.914 ms 2024-04-26T21:38:00.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:00 smithi001 ceph-mon[28054]: pgmap v235: 129 pgs: 129 active+clean; 149 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-26T21:38:00.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:00 smithi038 ceph-mon[32724]: pgmap v235: 129 pgs: 129 active+clean; 149 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-26T21:38:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:00 smithi186 ceph-mon[32991]: pgmap v235: 129 pgs: 129 active+clean; 149 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 1.4 MiB/s wr, 150 op/s 2024-04-26T21:38:00.573 INFO:tasks.workunit.client.0.smithi001.stdout: 1 261991 4.44 MB/sec execute 238 sec latency 26.352 ms 2024-04-26T21:38:01.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 262227 4.43 MB/sec execute 239 sec latency 99.108 ms 2024-04-26T21:38:02.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:02 smithi001 ceph-mon[28054]: pgmap v236: 129 pgs: 129 active+clean; 152 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 216 op/s 2024-04-26T21:38:02.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:02 smithi038 ceph-mon[32724]: pgmap v236: 129 pgs: 129 active+clean; 152 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 216 op/s 2024-04-26T21:38:02.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:02 smithi186 ceph-mon[32991]: pgmap v236: 129 pgs: 129 active+clean; 152 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 216 op/s 2024-04-26T21:38:02.573 INFO:tasks.workunit.client.0.smithi001.stdout: 1 262740 4.43 MB/sec execute 240 sec latency 103.982 ms 2024-04-26T21:38:03.573 INFO:tasks.workunit.client.0.smithi001.stdout: 1 263095 4.41 MB/sec execute 241 sec latency 39.540 ms 2024-04-26T21:38:04.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:04 smithi001 ceph-mon[28054]: pgmap v237: 129 pgs: 129 active+clean; 152 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-04-26T21:38:04.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:04 smithi038 ceph-mon[32724]: pgmap v237: 129 pgs: 129 active+clean; 152 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-04-26T21:38:04.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:04 smithi186 ceph-mon[32991]: pgmap v237: 129 pgs: 129 active+clean; 152 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 1.3 MiB/s wr, 149 op/s 2024-04-26T21:38:04.573 INFO:tasks.workunit.client.0.smithi001.stdout: 1 263763 4.40 MB/sec execute 242 sec latency 28.704 ms 2024-04-26T21:38:05.573 INFO:tasks.workunit.client.0.smithi001.stdout: 1 265010 4.41 MB/sec execute 243 sec latency 29.277 ms 2024-04-26T21:38:06.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.161+0000 7faa35e00640 1 -- 172.21.15.1:0/2559999042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa30103910 msgr2=0x7faa30103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:06.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.161+0000 7faa35e00640 1 --2- 172.21.15.1:0/2559999042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa30103910 0x7faa30103cf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7faa24009900 tx=0x7faa2402a480 comp rx=0 tx=0).stop 2024-04-26T21:38:06.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.161+0000 7faa35e00640 1 -- 172.21.15.1:0/2559999042 shutdown_connections 2024-04-26T21:38:06.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.161+0000 7faa35e00640 1 --2- 172.21.15.1:0/2559999042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa30104230 0x7faa3010c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.161+0000 7faa35e00640 1 --2- 172.21.15.1:0/2559999042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa3010cd30 0x7faa3010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.161+0000 7faa35e00640 1 --2- 172.21.15.1:0/2559999042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa30103910 0x7faa30103cf0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.161+0000 7faa35e00640 1 -- 172.21.15.1:0/2559999042 >> 172.21.15.1:0/2559999042 conn(0x7faa300fd110 msgr2=0x7faa300ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.162+0000 7faa35e00640 1 -- 172.21.15.1:0/2559999042 shutdown_connections 2024-04-26T21:38:06.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.162+0000 7faa35e00640 1 -- 172.21.15.1:0/2559999042 wait complete. 2024-04-26T21:38:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.163+0000 7faa35e00640 1 Processor -- start 2024-04-26T21:38:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.163+0000 7faa35e00640 1 -- start start 2024-04-26T21:38:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.163+0000 7faa35e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa30103910 0x7faa30101560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.163+0000 7faa35e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa30104230 0x7faa30101aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.163+0000 7faa35e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa3010cd30 0x7faa30101fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.163+0000 7faa35e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faa30112a30 con 0x7faa3010cd30 2024-04-26T21:38:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.163+0000 7faa35e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa301128b0 con 0x7faa30103910 2024-04-26T21:38:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.163+0000 7faa35e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faa30112bb0 con 0x7faa30104230 2024-04-26T21:38:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa34a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa30103910 0x7faa30101560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa34a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa30103910 0x7faa30101560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54294/0 (socket says 172.21.15.1:54294) 2024-04-26T21:38:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa34a00640 1 -- 172.21.15.1:0/1417659496 learned_addr learned my addr 172.21.15.1:0/1417659496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa2fe00640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa30104230 0x7faa30101aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.166 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa35400640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa3010cd30 0x7faa30101fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa34a00640 1 -- 172.21.15.1:0/1417659496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa30104230 msgr2=0x7faa30101aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa34a00640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa30104230 0x7faa30101aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa34a00640 1 -- 172.21.15.1:0/1417659496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa3010cd30 msgr2=0x7faa30101fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa34a00640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa3010cd30 0x7faa30101fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa34a00640 1 -- 172.21.15.1:0/1417659496 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa301a7450 con 0x7faa30103910 2024-04-26T21:38:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.164+0000 7faa34a00640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa30103910 0x7faa30101560 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7faa24005ab0 tx=0x7faa24003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.165+0000 7faa2d600640 1 -- 172.21.15.1:0/1417659496 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2403f070 con 0x7faa30103910 2024-04-26T21:38:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.165+0000 7faa2d600640 1 -- 172.21.15.1:0/1417659496 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa24031d60 con 0x7faa30103910 2024-04-26T21:38:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.165+0000 7faa35e00640 1 -- 172.21.15.1:0/1417659496 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa301a76e0 con 0x7faa30103910 2024-04-26T21:38:06.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.165+0000 7faa2d600640 1 -- 172.21.15.1:0/1417659496 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24035030 con 0x7faa30103910 2024-04-26T21:38:06.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.165+0000 7faa35e00640 1 -- 172.21.15.1:0/1417659496 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa301a7c20 con 0x7faa30103910 2024-04-26T21:38:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.168+0000 7faa2d600640 1 -- 172.21.15.1:0/1417659496 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7faa24042070 con 0x7faa30103910 2024-04-26T21:38:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.169+0000 7faa3a1f8740 1 Processor -- start 2024-04-26T21:38:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.169+0000 7faa3a1f8740 1 -- start start 2024-04-26T21:38:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.169+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0afe0 0x56336df0b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.169+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0b900 0x56336df15c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.169+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 0x56336df185b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.169+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x56336de70380 con 0x56336df0afe0 2024-04-26T21:38:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.169+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56336de0cc30 con 0x56336df161c0 2024-04-26T21:38:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.169+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56336de6c340 con 0x56336df0b900 2024-04-26T21:38:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa2fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 0x56336df185b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa35400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0afe0 0x56336df0b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa2fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 0x56336df185b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54306/0 (socket says 172.21.15.1:54306) 2024-04-26T21:38:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa2fe00640 1 -- 172.21.15.1:0/554479986 learned_addr learned my addr 172.21.15.1:0/554479986 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa34a00640 1 --2- 172.21.15.1:0/554479986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0b900 0x56336df15c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa2fe00640 1 -- 172.21.15.1:0/554479986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0b900 msgr2=0x56336df15c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa2fe00640 1 --2- 172.21.15.1:0/554479986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0b900 0x56336df15c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa2fe00640 1 -- 172.21.15.1:0/554479986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0afe0 msgr2=0x56336df0b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa2fe00640 1 --2- 172.21.15.1:0/554479986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0afe0 0x56336df0b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa2fe00640 1 -- 172.21.15.1:0/554479986 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56336de2e940 con 0x56336df161c0 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.170+0000 7faa2fe00640 1 --2- 172.21.15.1:0/554479986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 0x56336df185b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00bc80 tx=0x7faa1c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee47da57258368b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.171+0000 7faa21c00640 1 -- 172.21.15.1:0/554479986 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c005220 con 0x56336df161c0 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.171+0000 7faa21c00640 1 -- 172.21.15.1:0/554479986 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa1c00be30 con 0x56336df161c0 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.171+0000 7faa21c00640 1 -- 172.21.15.1:0/554479986 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c011790 con 0x56336df161c0 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.171+0000 7faa2d600640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faa180a3db0 0x7faa180a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.171+0000 7faa2d600640 1 -- 172.21.15.1:0/1417659496 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7faa240f1410 con 0x7faa30103910 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa35400640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faa180a3db0 0x7faa180a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa3a1f8740 1 -- 172.21.15.1:0/554479986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 msgr2=0x56336df185b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/554479986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 0x56336df185b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7faa1c00bc80 tx=0x7faa1c0107a0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa3a1f8740 1 -- 172.21.15.1:0/554479986 shutdown_connections 2024-04-26T21:38:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/554479986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0b900 0x56336df15c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/554479986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 0x56336df185b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/554479986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0afe0 0x56336df0b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa3a1f8740 1 -- 172.21.15.1:0/554479986 >> 172.21.15.1:0/554479986 conn(0x56336df074f0 msgr2=0x56336df078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa3a1f8740 1 -- 172.21.15.1:0/554479986 shutdown_connections 2024-04-26T21:38:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa3a1f8740 1 -- 172.21.15.1:0/554479986 wait complete. 2024-04-26T21:38:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.172+0000 7faa35400640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faa180a3db0 0x7faa180a6270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faa28005fd0 tx=0x7faa28009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:06.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.173+0000 7faa3a1f8740 1 -- start start 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.173+0000 7faa3a1f8740 1 Processor -- start 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.173+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df130a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.173+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df135e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df161c0 0x56336df13b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x56336df18df0 con 0x56336df0b900 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56336ddf7c60 con 0x56336df0afe0 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56336df18f70 con 0x56336df161c0 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa3a1f8740 10 client.0 _ll_register_callbacks cb 0x56336de23e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa34a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa34a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df130a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54316/0 (socket says 172.21.15.1:54316) 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa34a00640 1 -- 172.21.15.1:0/4114952967 learned_addr learned my addr 172.21.15.1:0/4114952967 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa2fe00640 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa35400640 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df161c0 0x56336df13b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa34a00640 1 -- 172.21.15.1:0/4114952967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df161c0 msgr2=0x56336df13b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa34a00640 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df161c0 0x56336df13b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa34a00640 1 -- 172.21.15.1:0/4114952967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 msgr2=0x56336df135e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa34a00640 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df135e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.174+0000 7faa34a00640 1 -- 172.21.15.1:0/4114952967 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56336de06510 con 0x56336df0afe0 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.175+0000 7faa34a00640 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df130a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faa24031440 tx=0x7faa240b8b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af14be80397d51c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.175+0000 7faa16000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.175+0000 7faa16000640 1 -- 172.21.15.1:0/4114952967 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24049070 con 0x56336df0afe0 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.175+0000 7faa16000640 1 -- 172.21.15.1:0/4114952967 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7faa240b9560 con 0x56336df0afe0 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.175+0000 7faa3a1f8740 1 -- 172.21.15.1:0/4114952967 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1c00be30 con 0x56336df0afe0 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.175+0000 7faa16000640 1 -- 172.21.15.1:0/4114952967 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7faa240bc850 con 0x56336df0afe0 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.175+0000 7faa3a1f8740 10 client.26138 fetch_fsmap learned FSMap version 36 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.175+0000 7faa3a1f8740 1 -- 172.21.15.1:0/4114952967 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56336db82210 con 0x56336df0afe0 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.176+0000 7faa16000640 1 -- 172.21.15.1:0/4114952967 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa1c00be30 con 0x56336df0afe0 2024-04-26T21:38:06.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.176+0000 7faa16000640 1 -- 172.21.15.1:0/4114952967 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7faa240ee920 con 0x56336df0afe0 2024-04-26T21:38:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.176+0000 7faa3a1f8740 10 client.26138 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:38:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.176+0000 7faa3a1f8740 10 client.26138 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:38:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.176+0000 7faa3a1f8740 20 client.26138 populate_metadata read hostname 'smithi001' 2024-04-26T21:38:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.176+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56336df25ee0 0x56336df282f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.176+0000 7faa3a1f8740 4 client.26138 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:38:06.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.176+0000 7faa3a1f8740 1 -- 172.21.15.1:0/4114952967 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56336de7df90 con 0x56336df25ee0 2024-04-26T21:38:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.177+0000 7faa2fe00640 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56336df25ee0 0x56336df282f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.177+0000 7faa2fe00640 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56336df25ee0 0x56336df282f0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb790e1f233b0119 server_cookie=8916fc28e3b03704 in_seq=0 out_seq=0 2024-04-26T21:38:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.177+0000 7faa16000640 10 client.26138 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa16000640 1 -- 172.21.15.1:0/4114952967 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56336de7df90 con 0x56336df25ee0 2024-04-26T21:38:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa16000640 10 client.26138 handle_command_reply: tid=0 2024-04-26T21:38:06.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa3a1f8740 1 client.26138 shutdown 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa3a1f8740 1 -- 172.21.15.1:0/4114952967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 msgr2=0x56336df130a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df130a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faa24031440 tx=0x7faa240b8b30 comp rx=0 tx=0).stop 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa3a1f8740 1 -- 172.21.15.1:0/4114952967 shutdown_connections 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56336df25ee0 0x56336df282f0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df161c0 0x56336df13b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df135e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/4114952967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df130a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa16000640 0 client.26138 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.178+0000 7faa3a1f8740 1 -- 172.21.15.1:0/4114952967 >> 172.21.15.1:0/4114952967 conn(0x56336df074f0 msgr2=0x56336df0d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.179+0000 7faa3a1f8740 1 -- 172.21.15.1:0/4114952967 shutdown_connections 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.179+0000 7faa3a1f8740 1 -- 172.21.15.1:0/4114952967 wait complete. 2024-04-26T21:38:06.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.179+0000 7faa3a1f8740 20 client.26138 trim_cache size 0 max 16384 2024-04-26T21:38:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.197+0000 7faa3a1f8740 1 Processor -- start 2024-04-26T21:38:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.197+0000 7faa3a1f8740 1 -- start start 2024-04-26T21:38:06.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df21cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df064d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df161c0 0x56336df2dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b4900 con 0x56336df0b900 2024-04-26T21:38:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b21d0 con 0x56336df0afe0 2024-04-26T21:38:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b0040 con 0x56336df161c0 2024-04-26T21:38:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa2fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa34a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df21cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa2fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60764/0 (socket says 172.21.15.1:60764) 2024-04-26T21:38:06.200 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa2fe00640 1 -- 172.21.15.1:0/2963911974 learned_addr learned my addr 172.21.15.1:0/2963911974 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.198+0000 7faa35400640 1 --2- 172.21.15.1:0/2963911974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df161c0 0x56336df2dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.199+0000 7faa2fe00640 1 -- 172.21.15.1:0/2963911974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df161c0 msgr2=0x56336df2dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.199+0000 7faa2fe00640 1 --2- 172.21.15.1:0/2963911974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df161c0 0x56336df2dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.199+0000 7faa2fe00640 1 -- 172.21.15.1:0/2963911974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 msgr2=0x56336df21cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.199+0000 7faa2fe00640 1 --2- 172.21.15.1:0/2963911974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df21cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.199+0000 7faa2fe00640 1 -- 172.21.15.1:0/2963911974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56336df18920 con 0x56336df0b900 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.199+0000 7faa34a00640 1 --2- 172.21.15.1:0/2963911974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df21cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.199+0000 7faa2fe00640 1 --2- 172.21.15.1:0/2963911974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df064d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faa1c000c00 tx=0x7faa1c0056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7ecbeebf02af61 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.199+0000 7faa21200640 1 -- 172.21.15.1:0/2963911974 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c00be30 con 0x56336df0b900 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.199+0000 7faa21200640 1 -- 172.21.15.1:0/2963911974 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa1c02e020 con 0x56336df0b900 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.199+0000 7faa21200640 1 -- 172.21.15.1:0/2963911974 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c0116a0 con 0x56336df0b900 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.200+0000 7faa3a1f8740 1 -- 172.21.15.1:0/2963911974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 msgr2=0x56336df064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:06.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.200+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/2963911974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df064d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7faa1c000c00 tx=0x7faa1c0056a0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.200+0000 7faa3a1f8740 1 -- 172.21.15.1:0/2963911974 shutdown_connections 2024-04-26T21:38:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.200+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/2963911974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df161c0 0x56336df2dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.200+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/2963911974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df064d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.200+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/2963911974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df0afe0 0x56336df21cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.200+0000 7faa3a1f8740 1 -- 172.21.15.1:0/2963911974 >> 172.21.15.1:0/2963911974 conn(0x56336df074f0 msgr2=0x56336df05dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.200+0000 7faa3a1f8740 1 -- 172.21.15.1:0/2963911974 shutdown_connections 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.200+0000 7faa3a1f8740 1 -- 172.21.15.1:0/2963911974 wait complete. 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.201+0000 7faa3a1f8740 1 -- start start 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.201+0000 7faa3a1f8740 1 Processor -- start 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0afe0 0x56336df10400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df10940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa3a1f8740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 0x56336df8b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b0690 con 0x56336df0b900 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b1b00 con 0x56336df161c0 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa3a1f8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faa301b0d60 con 0x56336df0afe0 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa3a1f8740 10 client.0 _ll_register_callbacks cb 0x7faa24005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa34a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0afe0 0x56336df10400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa34a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0afe0 0x56336df10400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41922/0 (socket says 172.21.15.1:41922) 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa34a00640 1 -- 172.21.15.1:0/3699099460 learned_addr learned my addr 172.21.15.1:0/3699099460 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:06.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa35400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 0x56336df8b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa2fe00640 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df10940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa34a00640 1 -- 172.21.15.1:0/3699099460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 msgr2=0x56336df8b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa34a00640 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 0x56336df8b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa34a00640 1 -- 172.21.15.1:0/3699099460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 msgr2=0x56336df10940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa34a00640 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df10940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.202+0000 7faa34a00640 1 -- 172.21.15.1:0/3699099460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56336dc4e620 con 0x56336df0afe0 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.203+0000 7faa34a00640 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0afe0 0x56336df10400 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7faa240bb1a0 tx=0x7faa24003df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a2aa8df24f5450 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.203+0000 7faa16000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.203+0000 7faa16000640 1 -- 172.21.15.1:0/3699099460 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24049040 con 0x56336df0afe0 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.203+0000 7faa16000640 1 -- 172.21.15.1:0/3699099460 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9642+0+0 (secure 0 0 0) 0x7faa240c56d0 con 0x56336df0afe0 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.203+0000 7faa3a1f8740 1 -- 172.21.15.1:0/3699099460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1c02e020 con 0x56336df0afe0 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.203+0000 7faa16000640 1 -- 172.21.15.1:0/3699099460 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7faa240bcef0 con 0x56336df0afe0 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.203+0000 7faa3a1f8740 10 client.26119 fetch_fsmap learned FSMap version 36 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.203+0000 7faa3a1f8740 1 -- 172.21.15.1:0/3699099460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56336de2acb0 con 0x56336df0afe0 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.204+0000 7faa16000640 1 -- 172.21.15.1:0/3699099460 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa1c02e020 con 0x56336df0afe0 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.204+0000 7faa16000640 1 -- 172.21.15.1:0/3699099460 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7faa240bc340 con 0x56336df0afe0 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.204+0000 7faa3a1f8740 10 client.26119 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.204+0000 7faa3a1f8740 10 client.26119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:38:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.204+0000 7faa3a1f8740 20 client.26119 populate_metadata read hostname 'smithi001' 2024-04-26T21:38:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.204+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56336df8ce20 0x56336df8f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.204+0000 7faa3a1f8740 4 client.26119 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:38:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.204+0000 7faa3a1f8740 1 -- 172.21.15.1:0/3699099460 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56336de77e00 con 0x56336df8ce20 2024-04-26T21:38:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.204+0000 7faa2fe00640 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56336df8ce20 0x56336df8f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.205+0000 7faa2fe00640 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56336df8ce20 0x56336df8f240 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40d7fa9443a973e4 server_cookie=c14f1cd05ca01ae6 in_seq=0 out_seq=0 2024-04-26T21:38:06.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.205+0000 7faa16000640 10 client.26119 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa16000640 1 -- 172.21.15.1:0/3699099460 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56336de77e00 con 0x56336df8ce20 2024-04-26T21:38:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa16000640 10 client.26119 handle_command_reply: tid=0 2024-04-26T21:38:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 client.26119 shutdown 2024-04-26T21:38:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 -- 172.21.15.1:0/3699099460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0afe0 msgr2=0x56336df10400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0afe0 0x56336df10400 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7faa240bb1a0 tx=0x7faa24003df0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 -- 172.21.15.1:0/3699099460 shutdown_connections 2024-04-26T21:38:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56336df8ce20 0x56336df8f240 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56336df161c0 0x56336df8b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56336df0b900 0x56336df10940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 --2- 172.21.15.1:0/3699099460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56336df0afe0 0x56336df10400 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa16000640 0 client.26119 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 -- 172.21.15.1:0/3699099460 >> 172.21.15.1:0/3699099460 conn(0x56336df074f0 msgr2=0x56336df138f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 -- 172.21.15.1:0/3699099460 shutdown_connections 2024-04-26T21:38:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 1 -- 172.21.15.1:0/3699099460 wait complete. 2024-04-26T21:38:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.260+0000 7faa3a1f8740 20 client.26119 trim_cache size 0 max 16384 2024-04-26T21:38:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:38:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:38:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:38:06.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:38:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.261+0000 7faa35e00640 1 -- 172.21.15.1:0/1417659496 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faa180a3db0 msgr2=0x7faa180a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.261+0000 7faa35e00640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faa180a3db0 0x7faa180a6270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faa28005fd0 tx=0x7faa28009500 comp rx=0 tx=0).stop 2024-04-26T21:38:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.261+0000 7faa35e00640 1 -- 172.21.15.1:0/1417659496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa30103910 msgr2=0x7faa30101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.261+0000 7faa35e00640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa30103910 0x7faa30101560 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7faa24005ab0 tx=0x7faa24003c80 comp rx=0 tx=0).stop 2024-04-26T21:38:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.261+0000 7faa35e00640 1 -- 172.21.15.1:0/1417659496 shutdown_connections 2024-04-26T21:38:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.261+0000 7faa35e00640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faa180a3db0 0x7faa180a6270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.261+0000 7faa35e00640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa30104230 0x7faa30101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.261+0000 7faa35e00640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa3010cd30 0x7faa30101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.261+0000 7faa35e00640 1 --2- 172.21.15.1:0/1417659496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa30103910 0x7faa30101560 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:06.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.261+0000 7faa35e00640 1 -- 172.21.15.1:0/1417659496 >> 172.21.15.1:0/1417659496 conn(0x7faa300fd110 msgr2=0x7faa300fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:06.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.262+0000 7faa35e00640 1 -- 172.21.15.1:0/1417659496 shutdown_connections 2024-04-26T21:38:06.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:06.262+0000 7faa35e00640 1 -- 172.21.15.1:0/1417659496 wait complete. 2024-04-26T21:38:06.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:06 smithi001 ceph-mon[28054]: pgmap v238: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.9 MiB/s wr, 238 op/s 2024-04-26T21:38:06.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:06 smithi038 ceph-mon[32724]: pgmap v238: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.9 MiB/s wr, 238 op/s 2024-04-26T21:38:06.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:06 smithi186 ceph-mon[32991]: pgmap v238: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.9 MiB/s wr, 238 op/s 2024-04-26T21:38:06.573 INFO:tasks.workunit.client.0.smithi001.stdout: 1 266202 4.43 MB/sec execute 244 sec latency 45.964 ms 2024-04-26T21:38:07.573 INFO:tasks.workunit.client.0.smithi001.stdout: 1 266439 4.42 MB/sec execute 245 sec latency 83.734 ms 2024-04-26T21:38:08.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:08 smithi001 ceph-mon[28054]: pgmap v239: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-04-26T21:38:08.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:08 smithi001 ceph-mon[28054]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T21:38:08.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:08 smithi038 ceph-mon[32724]: pgmap v239: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-04-26T21:38:08.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:08 smithi038 ceph-mon[32724]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T21:38:08.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:08 smithi186 ceph-mon[32991]: pgmap v239: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 1.3 MiB/s wr, 154 op/s 2024-04-26T21:38:08.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:08 smithi186 ceph-mon[32991]: osdmap e99: 12 total, 12 up, 12 in 2024-04-26T21:38:08.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 267129 4.41 MB/sec execute 246 sec latency 28.975 ms 2024-04-26T21:38:09.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 268491 4.41 MB/sec execute 247 sec latency 28.015 ms 2024-04-26T21:38:10.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:10 smithi001 ceph-mon[28054]: pgmap v241: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-04-26T21:38:10.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:10 smithi038 ceph-mon[32724]: pgmap v241: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-04-26T21:38:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:10 smithi186 ceph-mon[32991]: pgmap v241: 129 pgs: 129 active+clean; 155 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-04-26T21:38:10.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 269593 4.43 MB/sec execute 248 sec latency 26.579 ms 2024-04-26T21:38:11.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 270012 4.42 MB/sec execute 249 sec latency 34.290 ms 2024-04-26T21:38:12.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:12 smithi038 ceph-mon[32724]: pgmap v242: 129 pgs: 129 active+clean; 158 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 1.6 MiB/s wr, 203 op/s 2024-04-26T21:38:12.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:12 smithi186 ceph-mon[32991]: pgmap v242: 129 pgs: 129 active+clean; 158 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 1.6 MiB/s wr, 203 op/s 2024-04-26T21:38:12.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:12 smithi001 ceph-mon[28054]: pgmap v242: 129 pgs: 129 active+clean; 158 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 1.6 MiB/s wr, 203 op/s 2024-04-26T21:38:12.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 270531 4.41 MB/sec execute 250 sec latency 40.817 ms 2024-04-26T21:38:13.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 272010 4.41 MB/sec execute 251 sec latency 35.611 ms 2024-04-26T21:38:14.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:14 smithi038 ceph-mon[32724]: pgmap v243: 129 pgs: 129 active+clean; 158 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 1.6 MiB/s wr, 203 op/s 2024-04-26T21:38:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:14 smithi038 ceph-mon[32724]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T21:38:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:14 smithi186 ceph-mon[32991]: pgmap v243: 129 pgs: 129 active+clean; 158 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 1.6 MiB/s wr, 203 op/s 2024-04-26T21:38:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:14 smithi186 ceph-mon[32991]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T21:38:14.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:14 smithi001 ceph-mon[28054]: pgmap v243: 129 pgs: 129 active+clean; 158 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 1.6 MiB/s wr, 203 op/s 2024-04-26T21:38:14.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:14 smithi001 ceph-mon[28054]: osdmap e100: 12 total, 12 up, 12 in 2024-04-26T21:38:14.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 272979 4.43 MB/sec execute 252 sec latency 29.399 ms 2024-04-26T21:38:15.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 273539 4.43 MB/sec execute 253 sec latency 43.305 ms 2024-04-26T21:38:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:16 smithi186 ceph-mon[32991]: pgmap v245: 129 pgs: 129 active+clean; 163 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-04-26T21:38:16.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:16 smithi001 ceph-mon[28054]: pgmap v245: 129 pgs: 129 active+clean; 163 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-04-26T21:38:16.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 274023 4.41 MB/sec execute 254 sec latency 23.435 ms 2024-04-26T21:38:16.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:16 smithi038 ceph-mon[32724]: pgmap v245: 129 pgs: 129 active+clean; 163 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 221 op/s 2024-04-26T21:38:17.575 INFO:tasks.workunit.client.0.smithi001.stdout: 1 275511 4.42 MB/sec execute 255 sec latency 43.219 ms 2024-04-26T21:38:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:18 smithi186 ceph-mon[32991]: pgmap v246: 129 pgs: 129 active+clean; 163 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 2.0 MiB/s wr, 181 op/s 2024-04-26T21:38:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:18 smithi001 ceph-mon[28054]: pgmap v246: 129 pgs: 129 active+clean; 163 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 2.0 MiB/s wr, 181 op/s 2024-04-26T21:38:18.574 INFO:tasks.workunit.client.0.smithi001.stdout: 1 276358 4.43 MB/sec execute 256 sec latency 131.916 ms 2024-04-26T21:38:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:18 smithi038 ceph-mon[32724]: pgmap v246: 129 pgs: 129 active+clean; 163 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 2.0 MiB/s wr, 181 op/s 2024-04-26T21:38:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.412+0000 7fc37a600640 1 -- 172.21.15.1:0/3958066675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3741032e0 msgr2=0x7fc37410f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.412+0000 7fc37a600640 1 --2- 172.21.15.1:0/3958066675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3741032e0 0x7fc37410f7f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009cd0 tx=0x7fc36c02a8e0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.412+0000 7fc37a600640 1 -- 172.21.15.1:0/3958066675 shutdown_connections 2024-04-26T21:38:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.412+0000 7fc37a600640 1 --2- 172.21.15.1:0/3958066675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3741032e0 0x7fc37410f7f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.412+0000 7fc37a600640 1 --2- 172.21.15.1:0/3958066675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc374102920 0x7fc374102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.412+0000 7fc37a600640 1 --2- 172.21.15.1:0/3958066675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc374108930 0x7fc374108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.412+0000 7fc37a600640 1 -- 172.21.15.1:0/3958066675 >> 172.21.15.1:0/3958066675 conn(0x7fc3740fe3f0 msgr2=0x7fc374100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.412+0000 7fc37a600640 1 -- 172.21.15.1:0/3958066675 shutdown_connections 2024-04-26T21:38:19.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.412+0000 7fc37a600640 1 -- 172.21.15.1:0/3958066675 wait complete. 2024-04-26T21:38:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.413+0000 7fc37a600640 1 Processor -- start 2024-04-26T21:38:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.413+0000 7fc37a600640 1 -- start start 2024-04-26T21:38:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc37a600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc374102920 0x7fc374074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc37a600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3741032e0 0x7fc374075170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc37a600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc374108930 0x7fc3740756b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc37a600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc374112c40 con 0x7fc3741032e0 2024-04-26T21:38:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc37a600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc374112ac0 con 0x7fc374102920 2024-04-26T21:38:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc37a600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc374112dc0 con 0x7fc374108930 2024-04-26T21:38:19.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc373400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc374102920 0x7fc374074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc372a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3741032e0 0x7fc374075170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc373400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc374102920 0x7fc374074c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:33714/0 (socket says 172.21.15.1:33714) 2024-04-26T21:38:19.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc373400640 1 -- 172.21.15.1:0/4125617438 learned_addr learned my addr 172.21.15.1:0/4125617438 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:19.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.414+0000 7fc373e00640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc374108930 0x7fc3740756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.415+0000 7fc373400640 1 -- 172.21.15.1:0/4125617438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc374108930 msgr2=0x7fc3740756b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.415+0000 7fc373400640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc374108930 0x7fc3740756b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.415+0000 7fc373400640 1 -- 172.21.15.1:0/4125617438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3741032e0 msgr2=0x7fc374075170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.415+0000 7fc373400640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3741032e0 0x7fc374075170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.415+0000 7fc373400640 1 -- 172.21.15.1:0/4125617438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3741a7640 con 0x7fc374102920 2024-04-26T21:38:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.415+0000 7fc373400640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc374102920 0x7fc374074c30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc36400e9f0 tx=0x7fc36400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.416+0000 7fc36be00640 1 -- 172.21.15.1:0/4125617438 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36400cd80 con 0x7fc374102920 2024-04-26T21:38:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.416+0000 7fc36be00640 1 -- 172.21.15.1:0/4125617438 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3640108e0 con 0x7fc374102920 2024-04-26T21:38:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.416+0000 7fc36be00640 1 -- 172.21.15.1:0/4125617438 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364018980 con 0x7fc374102920 2024-04-26T21:38:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.416+0000 7fc37a600640 1 -- 172.21.15.1:0/4125617438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3741a7930 con 0x7fc374102920 2024-04-26T21:38:19.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.416+0000 7fc37a600640 1 -- 172.21.15.1:0/4125617438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3741a7e30 con 0x7fc374102920 2024-04-26T21:38:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.420+0000 7fc37d40c740 1 Processor -- start 2024-04-26T21:38:19.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.420+0000 7fc37d40c740 1 -- start start 2024-04-26T21:38:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.421+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 0x5602f6bb91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.421+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bc3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.421+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.421+0000 7fc37d40c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5602f6b1e380 con 0x5602f6bc3fb0 2024-04-26T21:38:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.421+0000 7fc37d40c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5602f6abac30 con 0x5602f6bb8dd0 2024-04-26T21:38:19.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.421+0000 7fc37d40c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5602f6b1a340 con 0x5602f6bb96f0 2024-04-26T21:38:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.421+0000 7fc36be00640 1 -- 172.21.15.1:0/4125617438 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fc364010430 con 0x7fc374102920 2024-04-26T21:38:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.421+0000 7fc372a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.421+0000 7fc373400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bc3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.421+0000 7fc372a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38664/0 (socket says 172.21.15.1:38664) 2024-04-26T21:38:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.422+0000 7fc372a00640 1 -- 172.21.15.1:0/1007309972 learned_addr learned my addr 172.21.15.1:0/1007309972 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:19.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.422+0000 7fc373e00640 1 --2- 172.21.15.1:0/1007309972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 0x5602f6bb91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.422+0000 7fc372a00640 1 -- 172.21.15.1:0/1007309972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 msgr2=0x5602f6bc3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.422+0000 7fc372a00640 1 --2- 172.21.15.1:0/1007309972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bc3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.422+0000 7fc372a00640 1 -- 172.21.15.1:0/1007309972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 msgr2=0x5602f6bb91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.422+0000 7fc372a00640 1 --2- 172.21.15.1:0/1007309972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 0x5602f6bb91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.422+0000 7fc372a00640 1 -- 172.21.15.1:0/1007309972 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f6adc940 con 0x5602f6bc3fb0 2024-04-26T21:38:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.422+0000 7fc373400640 1 --2- 172.21.15.1:0/1007309972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bc3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:38:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.423+0000 7fc372a00640 1 --2- 172.21.15.1:0/1007309972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc63a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc360009c90 tx=0x7fc360011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32139cb702dda890 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.423+0000 7fc368c00640 1 -- 172.21.15.1:0/1007309972 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36001e070 con 0x5602f6bc3fb0 2024-04-26T21:38:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.423+0000 7fc368c00640 1 -- 172.21.15.1:0/1007309972 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc360019ae0 con 0x5602f6bc3fb0 2024-04-26T21:38:19.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.423+0000 7fc368c00640 1 -- 172.21.15.1:0/1007309972 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36000e9c0 con 0x5602f6bc3fb0 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.423+0000 7fc36be00640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc3580a3d10 0x7fc3580a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc36be00640 1 -- 172.21.15.1:0/4125617438 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7fc3640d0fe0 con 0x7fc374102920 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc373e00640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc3580a3d10 0x7fc3580a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc37d40c740 1 -- 172.21.15.1:0/1007309972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 msgr2=0x5602f6bc63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc37d40c740 1 --2- 172.21.15.1:0/1007309972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc63a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc360009c90 tx=0x7fc360011810 comp rx=0 tx=0).stop 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc37d40c740 1 -- 172.21.15.1:0/1007309972 shutdown_connections 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc37d40c740 1 --2- 172.21.15.1:0/1007309972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bc3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc37d40c740 1 --2- 172.21.15.1:0/1007309972 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc63a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc37d40c740 1 --2- 172.21.15.1:0/1007309972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 0x5602f6bb91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc37d40c740 1 -- 172.21.15.1:0/1007309972 >> 172.21.15.1:0/1007309972 conn(0x5602f6bb52e0 msgr2=0x5602f6bb56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:19.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc37d40c740 1 -- 172.21.15.1:0/1007309972 shutdown_connections 2024-04-26T21:38:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.424+0000 7fc37d40c740 1 -- 172.21.15.1:0/1007309972 wait complete. 2024-04-26T21:38:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.425+0000 7fc373e00640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc3580a3d10 0x7fc3580a61d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009430 tx=0x7fc36c0098a0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.425+0000 7fc37d40c740 1 -- start start 2024-04-26T21:38:19.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.425+0000 7fc37d40c740 1 Processor -- start 2024-04-26T21:38:19.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb96f0 0x5602f6bc1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc37d40c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5602f6bc6be0 con 0x5602f6bc3fb0 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc37d40c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5602f6aa5c60 con 0x5602f6bb96f0 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc37d40c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5602f6bc6d60 con 0x5602f6bb8dd0 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc373400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc37d40c740 10 client.0 _ll_register_callbacks cb 0x5602f6ad1e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc373400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:49142/0 (socket says 172.21.15.1:49142) 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc373400640 1 -- 172.21.15.1:0/879819695 learned_addr learned my addr 172.21.15.1:0/879819695 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.426+0000 7fc372a00640 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb96f0 0x5602f6bc1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc373e00640 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc373400640 1 -- 172.21.15.1:0/879819695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb96f0 msgr2=0x5602f6bc1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:19.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc373400640 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb96f0 0x5602f6bc1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc373400640 1 -- 172.21.15.1:0/879819695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 msgr2=0x5602f6bc19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc373400640 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc373400640 1 -- 172.21.15.1:0/879819695 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f6ab4510 con 0x5602f6bb8dd0 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc372a00640 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb96f0 0x5602f6bc1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc373400640 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc0f50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc3640ce2d0 tx=0x7fc364018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cea2a52c758ce8cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc35d600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc35d600640 1 -- 172.21.15.1:0/879819695 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364010720 con 0x5602f6bb8dd0 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc35d600640 1 -- 172.21.15.1:0/879819695 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7fc3640ce4f0 con 0x5602f6bb8dd0 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.427+0000 7fc37d40c740 1 -- 172.21.15.1:0/879819695 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc360019ae0 con 0x5602f6bb8dd0 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc35d600640 1 -- 172.21.15.1:0/879819695 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc3640ce6f0 con 0x5602f6bb8dd0 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc37d40c740 10 client.26131 fetch_fsmap learned FSMap version 36 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc37d40c740 1 -- 172.21.15.1:0/879819695 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5602f6830210 con 0x5602f6bb8dd0 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc35d600640 1 -- 172.21.15.1:0/879819695 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc360019ae0 con 0x5602f6bb8dd0 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc35d600640 1 -- 172.21.15.1:0/879819695 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fc3640a52a0 con 0x5602f6bb8dd0 2024-04-26T21:38:19.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc37d40c740 10 client.26131 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:38:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc37d40c740 10 client.26131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:38:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc37d40c740 20 client.26131 populate_metadata read hostname 'smithi001' 2024-04-26T21:38:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc37d40c740 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5602f6bd3cb0 0x5602f6bd60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc37d40c740 4 client.26131 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:38:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.428+0000 7fc37d40c740 1 -- 172.21.15.1:0/879819695 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602f6b2bf90 con 0x5602f6bd3cb0 2024-04-26T21:38:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.429+0000 7fc372a00640 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5602f6bd3cb0 0x5602f6bd60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.429+0000 7fc372a00640 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5602f6bd3cb0 0x5602f6bd60a0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d4a09273a17f6be server_cookie=d1c2464c8c24df43 in_seq=0 out_seq=0 2024-04-26T21:38:19.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.429+0000 7fc35d600640 10 client.26131 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.430+0000 7fc35d600640 1 -- 172.21.15.1:0/879819695 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602f6b2bf90 con 0x5602f6bd3cb0 2024-04-26T21:38:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.430+0000 7fc35d600640 10 client.26131 handle_command_reply: tid=0 2024-04-26T21:38:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.430+0000 7fc37d40c740 1 client.26131 shutdown 2024-04-26T21:38:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 1 -- 172.21.15.1:0/879819695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb8dd0 msgr2=0x5602f6bc0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:19.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc0f50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc3640ce2d0 tx=0x7fc364018420 comp rx=0 tx=0).stop 2024-04-26T21:38:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 1 -- 172.21.15.1:0/879819695 shutdown_connections 2024-04-26T21:38:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5602f6bd3cb0 0x5602f6bd60a0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb96f0 0x5602f6bc1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 1 --2- 172.21.15.1:0/879819695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc0f50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc35d600640 0 client.26131 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 1 -- 172.21.15.1:0/879819695 >> 172.21.15.1:0/879819695 conn(0x5602f6bb52e0 msgr2=0x5602f6bbb010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 1 -- 172.21.15.1:0/879819695 shutdown_connections 2024-04-26T21:38:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 1 -- 172.21.15.1:0/879819695 wait complete. 2024-04-26T21:38:19.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.431+0000 7fc37d40c740 20 client.26131 trim_cache size 0 max 16384 2024-04-26T21:38:19.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.453+0000 7fc37d40c740 1 Processor -- start 2024-04-26T21:38:19.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.453+0000 7fc37d40c740 1 -- start start 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 0x5602f6bcfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bbae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc37d40c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc3741b4ac0 con 0x5602f6bc3fb0 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc37d40c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc3741b2390 con 0x5602f6bb8dd0 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc37d40c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc3741b0200 con 0x5602f6bb96f0 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc373400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 0x5602f6bcfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc373400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 0x5602f6bcfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:33744/0 (socket says 172.21.15.1:33744) 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc373400640 1 -- 172.21.15.1:0/3297539221 learned_addr learned my addr 172.21.15.1:0/3297539221 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc372a00640 1 --2- 172.21.15.1:0/3297539221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bbae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc373e00640 1 --2- 172.21.15.1:0/3297539221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc372a00640 1 -- 172.21.15.1:0/3297539221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 msgr2=0x5602f6bcfaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc372a00640 1 --2- 172.21.15.1:0/3297539221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 0x5602f6bcfaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.454+0000 7fc372a00640 1 -- 172.21.15.1:0/3297539221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 msgr2=0x5602f6bc17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.455+0000 7fc372a00640 1 --2- 172.21.15.1:0/3297539221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.455+0000 7fc372a00640 1 -- 172.21.15.1:0/3297539221 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f6bd6410 con 0x5602f6bb96f0 2024-04-26T21:38:19.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.455+0000 7fc373400640 1 --2- 172.21.15.1:0/3297539221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 0x5602f6bcfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:38:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.455+0000 7fc372a00640 1 --2- 172.21.15.1:0/3297539221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bbae90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc360000c00 tx=0x7fc3600099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29941d2bcfbe5f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.455+0000 7fc35fe00640 1 -- 172.21.15.1:0/3297539221 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360029e60 con 0x5602f6bb96f0 2024-04-26T21:38:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.455+0000 7fc35fe00640 1 -- 172.21.15.1:0/3297539221 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc36001f040 con 0x5602f6bb96f0 2024-04-26T21:38:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.455+0000 7fc35fe00640 1 -- 172.21.15.1:0/3297539221 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36001e040 con 0x5602f6bb96f0 2024-04-26T21:38:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.456+0000 7fc37d40c740 1 -- 172.21.15.1:0/3297539221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 msgr2=0x5602f6bbae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.456+0000 7fc37d40c740 1 --2- 172.21.15.1:0/3297539221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bbae90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc360000c00 tx=0x7fc3600099e0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.456+0000 7fc37d40c740 1 -- 172.21.15.1:0/3297539221 shutdown_connections 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.456+0000 7fc37d40c740 1 --2- 172.21.15.1:0/3297539221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bbae90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.456+0000 7fc37d40c740 1 --2- 172.21.15.1:0/3297539221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bc3fb0 0x5602f6bc17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.456+0000 7fc37d40c740 1 --2- 172.21.15.1:0/3297539221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bb8dd0 0x5602f6bcfaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.456+0000 7fc37d40c740 1 -- 172.21.15.1:0/3297539221 >> 172.21.15.1:0/3297539221 conn(0x5602f6bb52e0 msgr2=0x5602f6bba780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.456+0000 7fc37d40c740 1 -- 172.21.15.1:0/3297539221 shutdown_connections 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.456+0000 7fc37d40c740 1 -- 172.21.15.1:0/3297539221 wait complete. 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.457+0000 7fc37d40c740 1 -- start start 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.457+0000 7fc37d40c740 1 Processor -- start 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.457+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bc8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc37d40c740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bc3fb0 0x5602f6c391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc37d40c740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc3741b0850 con 0x5602f6bb8dd0 2024-04-26T21:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc37d40c740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc3741b1cc0 con 0x5602f6bc3fb0 2024-04-26T21:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc37d40c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc3741b0f20 con 0x5602f6bb96f0 2024-04-26T21:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc37d40c740 10 client.0 _ll_register_callbacks cb 0x7fc3640d4780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc373400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc372a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bc8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc372a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bc8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:49162/0 (socket says 172.21.15.1:49162) 2024-04-26T21:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc373400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38684/0 (socket says 172.21.15.1:38684) 2024-04-26T21:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc372a00640 1 -- 172.21.15.1:0/1003304396 learned_addr learned my addr 172.21.15.1:0/1003304396 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc373e00640 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bc3fb0 0x5602f6c391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc373400640 1 -- 172.21.15.1:0/1003304396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 msgr2=0x5602f6bc8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc373400640 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bc8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc373400640 1 -- 172.21.15.1:0/1003304396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bc3fb0 msgr2=0x5602f6c391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc373400640 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bc3fb0 0x5602f6c391b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.458+0000 7fc373400640 1 -- 172.21.15.1:0/1003304396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f68fc620 con 0x5602f6bb8dd0 2024-04-26T21:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.459+0000 7fc373400640 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc7dc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc36409ca50 tx=0x7fc36400c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3fae7ce0b62b9a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.459+0000 7fc35d600640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.459+0000 7fc35d600640 1 -- 172.21.15.1:0/1003304396 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36409c600 con 0x5602f6bb8dd0 2024-04-26T21:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.459+0000 7fc35d600640 1 -- 172.21.15.1:0/1003304396 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7fc36409bf10 con 0x5602f6bb8dd0 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.459+0000 7fc37d40c740 1 -- 172.21.15.1:0/1003304396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc36001f040 con 0x5602f6bb8dd0 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.460+0000 7fc35d600640 1 -- 172.21.15.1:0/1003304396 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fc3640ce6f0 con 0x5602f6bb8dd0 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.460+0000 7fc37d40c740 10 client.15984 fetch_fsmap learned FSMap version 36 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.460+0000 7fc37d40c740 1 -- 172.21.15.1:0/1003304396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5602f6ad8cb0 con 0x5602f6bb8dd0 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.460+0000 7fc35d600640 1 -- 172.21.15.1:0/1003304396 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc36001f040 con 0x5602f6bb8dd0 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.460+0000 7fc35d600640 1 -- 172.21.15.1:0/1003304396 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fc364087110 con 0x5602f6bb8dd0 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.461+0000 7fc37d40c740 10 client.15984 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.461+0000 7fc37d40c740 10 client.15984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.461+0000 7fc37d40c740 20 client.15984 populate_metadata read hostname 'smithi001' 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.461+0000 7fc37d40c740 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5602f6c3ac70 0x5602f6c3d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.461+0000 7fc37d40c740 4 client.15984 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.461+0000 7fc37d40c740 1 -- 172.21.15.1:0/1003304396 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5602f6b25e00 con 0x5602f6c3ac70 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.461+0000 7fc372a00640 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5602f6c3ac70 0x5602f6c3d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.462+0000 7fc372a00640 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5602f6c3ac70 0x5602f6c3d060 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e03ec772f25eeac server_cookie=7046265359e58a0c in_seq=0 out_seq=0 2024-04-26T21:38:19.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.462+0000 7fc35d600640 10 client.15984 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:19.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc35d600640 1 -- 172.21.15.1:0/1003304396 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5602f6b25e00 con 0x5602f6c3ac70 2024-04-26T21:38:19.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc35d600640 10 client.15984 handle_command_reply: tid=0 2024-04-26T21:38:19.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc37d40c740 1 client.15984 shutdown 2024-04-26T21:38:19.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc37d40c740 1 -- 172.21.15.1:0/1003304396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bb8dd0 msgr2=0x5602f6bc7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc37d40c740 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc7dc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc36409ca50 tx=0x7fc36400c7d0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc37d40c740 1 -- 172.21.15.1:0/1003304396 shutdown_connections 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc37d40c740 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5602f6c3ac70 0x5602f6c3d060 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc37d40c740 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5602f6bb96f0 0x5602f6bc8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc37d40c740 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5602f6bc3fb0 0x5602f6c391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc37d40c740 1 --2- 172.21.15.1:0/1003304396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5602f6bb8dd0 0x5602f6bc7dc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc37d40c740 1 -- 172.21.15.1:0/1003304396 >> 172.21.15.1:0/1003304396 conn(0x5602f6bb52e0 msgr2=0x5602f6bbb010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.534+0000 7fc35d600640 0 client.15984 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.535+0000 7fc37d40c740 1 -- 172.21.15.1:0/1003304396 shutdown_connections 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.535+0000 7fc37d40c740 1 -- 172.21.15.1:0/1003304396 wait complete. 2024-04-26T21:38:19.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.535+0000 7fc37d40c740 20 client.15984 trim_cache size 0 max 16384 2024-04-26T21:38:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:38:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:38:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:38:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:38:19.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 -- 172.21.15.1:0/4125617438 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc3580a3d10 msgr2=0x7fc3580a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc3580a3d10 0x7fc3580a61d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009430 tx=0x7fc36c0098a0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 -- 172.21.15.1:0/4125617438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc374102920 msgr2=0x7fc374074c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc374102920 0x7fc374074c30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc36400e9f0 tx=0x7fc36400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 -- 172.21.15.1:0/4125617438 shutdown_connections 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc3580a3d10 0x7fc3580a61d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc374108930 0x7fc3740756b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3741032e0 0x7fc374075170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 --2- 172.21.15.1:0/4125617438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc374102920 0x7fc374074c30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 -- 172.21.15.1:0/4125617438 >> 172.21.15.1:0/4125617438 conn(0x7fc3740fe3f0 msgr2=0x7fc3740feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 -- 172.21.15.1:0/4125617438 shutdown_connections 2024-04-26T21:38:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:19.536+0000 7fc37a600640 1 -- 172.21.15.1:0/4125617438 wait complete. 2024-04-26T21:38:19.575 INFO:tasks.workunit.client.0.smithi001.stdout: 1 277047 4.43 MB/sec execute 257 sec latency 34.073 ms 2024-04-26T21:38:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:20 smithi186 ceph-mon[32991]: pgmap v247: 129 pgs: 129 active+clean; 163 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 1.9 MiB/s wr, 177 op/s 2024-04-26T21:38:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:20 smithi001 ceph-mon[28054]: pgmap v247: 129 pgs: 129 active+clean; 163 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 1.9 MiB/s wr, 177 op/s 2024-04-26T21:38:20.575 INFO:tasks.workunit.client.0.smithi001.stdout: 1 277506 4.42 MB/sec execute 258 sec latency 23.081 ms 2024-04-26T21:38:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:20 smithi038 ceph-mon[32724]: pgmap v247: 129 pgs: 129 active+clean; 163 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 1.9 MiB/s wr, 177 op/s 2024-04-26T21:38:21.575 INFO:tasks.workunit.client.0.smithi001.stdout: 1 278834 4.42 MB/sec execute 259 sec latency 19.172 ms 2024-04-26T21:38:22.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:22 smithi038 ceph-mon[32724]: pgmap v248: 129 pgs: 129 active+clean; 168 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-04-26T21:38:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:22 smithi186 ceph-mon[32991]: pgmap v248: 129 pgs: 129 active+clean; 168 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-04-26T21:38:22.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:22 smithi001 ceph-mon[28054]: pgmap v248: 129 pgs: 129 active+clean; 168 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-04-26T21:38:22.575 INFO:tasks.workunit.client.0.smithi001.stdout: 1 279698 4.43 MB/sec execute 260 sec latency 53.993 ms 2024-04-26T21:38:23.575 INFO:tasks.workunit.client.0.smithi001.stdout: 1 280556 4.44 MB/sec execute 261 sec latency 39.155 ms 2024-04-26T21:38:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:24 smithi186 ceph-mon[32991]: pgmap v249: 129 pgs: 129 active+clean; 168 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-04-26T21:38:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:24 smithi001 ceph-mon[28054]: pgmap v249: 129 pgs: 129 active+clean; 168 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-04-26T21:38:24.575 INFO:tasks.workunit.client.0.smithi001.stdout: 1 280987 4.42 MB/sec execute 262 sec latency 34.769 ms 2024-04-26T21:38:24.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:24 smithi038 ceph-mon[32724]: pgmap v249: 129 pgs: 129 active+clean; 168 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-04-26T21:38:25.575 INFO:tasks.workunit.client.0.smithi001.stdout: 1 282068 4.43 MB/sec execute 263 sec latency 19.361 ms 2024-04-26T21:38:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:26 smithi186 ceph-mon[32991]: pgmap v250: 129 pgs: 129 active+clean; 171 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 245 op/s 2024-04-26T21:38:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:26 smithi001 ceph-mon[28054]: pgmap v250: 129 pgs: 129 active+clean; 171 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 245 op/s 2024-04-26T21:38:26.575 INFO:tasks.workunit.client.0.smithi001.stdout: 1 283208 4.44 MB/sec execute 264 sec latency 39.478 ms 2024-04-26T21:38:26.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:26 smithi038 ceph-mon[32724]: pgmap v250: 129 pgs: 129 active+clean; 171 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 245 op/s 2024-04-26T21:38:27.576 INFO:tasks.workunit.client.0.smithi001.stdout: 1 284067 4.44 MB/sec execute 265 sec latency 25.875 ms 2024-04-26T21:38:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:28 smithi186 ceph-mon[32991]: pgmap v251: 129 pgs: 129 active+clean; 171 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.5 MiB/s wr, 156 op/s 2024-04-26T21:38:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:28 smithi001 ceph-mon[28054]: pgmap v251: 129 pgs: 129 active+clean; 171 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.5 MiB/s wr, 156 op/s 2024-04-26T21:38:28.576 INFO:tasks.workunit.client.0.smithi001.stdout: 1 284565 4.43 MB/sec execute 266 sec latency 34.198 ms 2024-04-26T21:38:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:28 smithi038 ceph-mon[32724]: pgmap v251: 129 pgs: 129 active+clean; 171 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.5 MiB/s wr, 156 op/s 2024-04-26T21:38:29.576 INFO:tasks.workunit.client.0.smithi001.stdout: 1 285515 4.43 MB/sec execute 267 sec latency 24.795 ms 2024-04-26T21:38:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:30 smithi186 ceph-mon[32991]: pgmap v252: 129 pgs: 129 active+clean; 171 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.5 MiB/s wr, 156 op/s 2024-04-26T21:38:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:30 smithi001 ceph-mon[28054]: pgmap v252: 129 pgs: 129 active+clean; 171 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.5 MiB/s wr, 156 op/s 2024-04-26T21:38:30.576 INFO:tasks.workunit.client.0.smithi001.stdout: 1 286685 4.44 MB/sec execute 268 sec latency 25.745 ms 2024-04-26T21:38:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:30 smithi038 ceph-mon[32724]: pgmap v252: 129 pgs: 129 active+clean; 171 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 1.5 MiB/s wr, 156 op/s 2024-04-26T21:38:31.576 INFO:tasks.workunit.client.0.smithi001.stdout: 1 287584 4.45 MB/sec execute 269 sec latency 25.741 ms 2024-04-26T21:38:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:32 smithi186 ceph-mon[32991]: pgmap v253: 129 pgs: 129 active+clean; 175 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-04-26T21:38:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:32 smithi001 ceph-mon[28054]: pgmap v253: 129 pgs: 129 active+clean; 175 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-04-26T21:38:32.576 INFO:tasks.workunit.client.0.smithi001.stdout: 1 287899 4.43 MB/sec execute 270 sec latency 28.957 ms 2024-04-26T21:38:32.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:32 smithi038 ceph-mon[32724]: pgmap v253: 129 pgs: 129 active+clean; 175 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-04-26T21:38:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.690+0000 7fcb63e00640 1 -- 172.21.15.1:0/3449036619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c104dc0 msgr2=0x7fcb5c1051a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.690+0000 7fcb63e00640 1 --2- 172.21.15.1:0/3449036619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c104dc0 0x7fcb5c1051a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009900 tx=0x7fcb5002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.690+0000 7fcb63e00640 1 -- 172.21.15.1:0/3449036619 shutdown_connections 2024-04-26T21:38:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.690+0000 7fcb63e00640 1 --2- 172.21.15.1:0/3449036619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb5c109e90 0x7fcb5c111a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.690+0000 7fcb63e00640 1 --2- 172.21.15.1:0/3449036619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c105770 0x7fcb5c1097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.690+0000 7fcb63e00640 1 --2- 172.21.15.1:0/3449036619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c104dc0 0x7fcb5c1051a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.690+0000 7fcb63e00640 1 -- 172.21.15.1:0/3449036619 >> 172.21.15.1:0/3449036619 conn(0x7fcb5c1005e0 msgr2=0x7fcb5c102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.690+0000 7fcb63e00640 1 -- 172.21.15.1:0/3449036619 shutdown_connections 2024-04-26T21:38:32.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.690+0000 7fcb63e00640 1 -- 172.21.15.1:0/3449036619 wait complete. 2024-04-26T21:38:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.691+0000 7fcb63e00640 1 Processor -- start 2024-04-26T21:38:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.691+0000 7fcb63e00640 1 -- start start 2024-04-26T21:38:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.692+0000 7fcb63e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c104dc0 0x7fcb5c19b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.692+0000 7fcb63e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb5c109e90 0x7fcb5c19b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.692+0000 7fcb63e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c19fbf0 0x7fcb5c1a00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.692+0000 7fcb63e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c114e40 con 0x7fcb5c104dc0 2024-04-26T21:38:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.692+0000 7fcb63e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c114cc0 con 0x7fcb5c19fbf0 2024-04-26T21:38:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.692+0000 7fcb63e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5c114fc0 con 0x7fcb5c109e90 2024-04-26T21:38:32.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.692+0000 7fcb61400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c104dc0 0x7fcb5c19b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb61400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c104dc0 0x7fcb5c19b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57064/0 (socket says 172.21.15.1:57064) 2024-04-26T21:38:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb61400640 1 -- 172.21.15.1:0/967021590 learned_addr learned my addr 172.21.15.1:0/967021590 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb61e00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c19fbf0 0x7fcb5c1a00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb60a00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb5c109e90 0x7fcb5c19b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb60a00640 1 -- 172.21.15.1:0/967021590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c19fbf0 msgr2=0x7fcb5c1a00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb60a00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c19fbf0 0x7fcb5c1a00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb60a00640 1 -- 172.21.15.1:0/967021590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c104dc0 msgr2=0x7fcb5c19b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb60a00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c104dc0 0x7fcb5c19b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb60a00640 1 -- 172.21.15.1:0/967021590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5c1a06e0 con 0x7fcb5c109e90 2024-04-26T21:38:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb61400640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c104dc0 0x7fcb5c19b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:38:32.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.693+0000 7fcb61e00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c19fbf0 0x7fcb5c1a00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:38:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.694+0000 7fcb60a00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb5c109e90 0x7fcb5c19b8c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800c940 tx=0x7fcb4800ce00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.694+0000 7fcb5a000640 1 -- 172.21.15.1:0/967021590 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48007bb0 con 0x7fcb5c109e90 2024-04-26T21:38:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.694+0000 7fcb63e00640 1 -- 172.21.15.1:0/967021590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb5c1a09d0 con 0x7fcb5c109e90 2024-04-26T21:38:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.694+0000 7fcb63e00640 1 -- 172.21.15.1:0/967021590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb5c1a81c0 con 0x7fcb5c109e90 2024-04-26T21:38:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.695+0000 7fcb5a000640 1 -- 172.21.15.1:0/967021590 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb48007d50 con 0x7fcb5c109e90 2024-04-26T21:38:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.695+0000 7fcb5a000640 1 -- 172.21.15.1:0/967021590 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48004e60 con 0x7fcb5c109e90 2024-04-26T21:38:32.698 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.698+0000 7fcb66da9740 1 Processor -- start 2024-04-26T21:38:32.699 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.698+0000 7fcb66da9740 1 -- start start 2024-04-26T21:38:32.699 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.699+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396dd10 0x55d93396e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.699+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d93396e630 0x55d933981280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.699+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9339817c0 0x55d933983bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.699+0000 7fcb66da9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d9338d3380 con 0x55d93396e630 2024-04-26T21:38:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.699+0000 7fcb66da9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d93386fc30 con 0x55d93396dd10 2024-04-26T21:38:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.699+0000 7fcb66da9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d9338cf340 con 0x55d9339817c0 2024-04-26T21:38:32.700 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.699+0000 7fcb5a000640 1 -- 172.21.15.1:0/967021590 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fcb4800f510 con 0x7fcb5c109e90 2024-04-26T21:38:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.699+0000 7fcb61e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396dd10 0x55d93396e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.699+0000 7fcb61e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396dd10 0x55d93396e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42424/0 (socket says 172.21.15.1:42424) 2024-04-26T21:38:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb61e00640 1 -- 172.21.15.1:0/1662180468 learned_addr learned my addr 172.21.15.1:0/1662180468 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb61400640 1 --2- 172.21.15.1:0/1662180468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d93396e630 0x55d933981280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb60a00640 1 --2- 172.21.15.1:0/1662180468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9339817c0 0x55d933983bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb61e00640 1 -- 172.21.15.1:0/1662180468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9339817c0 msgr2=0x55d933983bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb61e00640 1 --2- 172.21.15.1:0/1662180468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9339817c0 0x55d933983bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb61e00640 1 -- 172.21.15.1:0/1662180468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d93396e630 msgr2=0x55d933981280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb61e00640 1 --2- 172.21.15.1:0/1662180468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d93396e630 0x55d933981280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb61e00640 1 -- 172.21.15.1:0/1662180468 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d933891940 con 0x55d93396dd10 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb61400640 1 --2- 172.21.15.1:0/1662180468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d93396e630 0x55d933981280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb61e00640 1 --2- 172.21.15.1:0/1662180468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396dd10 0x55d93396e0f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fcb54010710 tx=0x7fcb54010bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7ff7f96ee27f093 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.700+0000 7fcb4e600640 1 -- 172.21.15.1:0/1662180468 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5400e8d0 con 0x55d93396dd10 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.701+0000 7fcb4e600640 1 -- 172.21.15.1:0/1662180468 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb54012450 con 0x55d93396dd10 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.701+0000 7fcb4e600640 1 -- 172.21.15.1:0/1662180468 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5401c670 con 0x55d93396dd10 2024-04-26T21:38:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.701+0000 7fcb5a000640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcb440a3e10 0x7fcb440a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.701+0000 7fcb5a000640 1 -- 172.21.15.1:0/967021590 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7fcb480d0710 con 0x7fcb5c109e90 2024-04-26T21:38:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.702+0000 7fcb66da9740 1 -- 172.21.15.1:0/1662180468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396dd10 msgr2=0x55d93396e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.702+0000 7fcb66da9740 1 --2- 172.21.15.1:0/1662180468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396dd10 0x55d93396e0f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fcb54010710 tx=0x7fcb54010bd0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.702+0000 7fcb61400640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcb440a3e10 0x7fcb440a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.702+0000 7fcb66da9740 1 -- 172.21.15.1:0/1662180468 shutdown_connections 2024-04-26T21:38:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.702+0000 7fcb66da9740 1 --2- 172.21.15.1:0/1662180468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9339817c0 0x55d933983bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.702+0000 7fcb66da9740 1 --2- 172.21.15.1:0/1662180468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d93396e630 0x55d933981280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.702+0000 7fcb66da9740 1 --2- 172.21.15.1:0/1662180468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396dd10 0x55d93396e0f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.702+0000 7fcb66da9740 1 -- 172.21.15.1:0/1662180468 >> 172.21.15.1:0/1662180468 conn(0x55d93396a2e0 msgr2=0x55d93396a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.703+0000 7fcb66da9740 1 -- 172.21.15.1:0/1662180468 shutdown_connections 2024-04-26T21:38:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.703+0000 7fcb66da9740 1 -- 172.21.15.1:0/1662180468 wait complete. 2024-04-26T21:38:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.703+0000 7fcb66da9740 1 -- start start 2024-04-26T21:38:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.704+0000 7fcb66da9740 1 Processor -- start 2024-04-26T21:38:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.704+0000 7fcb61400640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcb440a3e10 0x7fcb440a62d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcb50005ec0 tx=0x7fcb50005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.704+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.704+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93397ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.708 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.704+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb66da9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d933984390 con 0x55d9339817c0 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb66da9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d93385ac60 con 0x55d93396e630 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb66da9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d933984510 con 0x55d93396dd10 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb66da9740 10 client.0 _ll_register_callbacks cb 0x55d933886e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb61400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb61400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37840/0 (socket says 172.21.15.1:37840) 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb61400640 1 -- 172.21.15.1:0/3249954288 learned_addr learned my addr 172.21.15.1:0/3249954288 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb60a00640 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93397ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb61e00640 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb60a00640 1 -- 172.21.15.1:0/3249954288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 msgr2=0x55d93397e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb60a00640 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb60a00640 1 -- 172.21.15.1:0/3249954288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 msgr2=0x55d93397f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:32.709 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb60a00640 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb60a00640 1 -- 172.21.15.1:0/3249954288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d933869510 con 0x55d93396e630 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.705+0000 7fcb61e00640 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.706+0000 7fcb60a00640 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93397ec20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcb480cd9a0 tx=0x7fcb4809bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b2e1ef2d52da58d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.706+0000 7fcb42a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.706+0000 7fcb42a00640 1 -- 172.21.15.1:0/3249954288 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4801d040 con 0x55d93396e630 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.706+0000 7fcb42a00640 1 -- 172.21.15.1:0/3249954288 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7fcb480d2bd0 con 0x55d93396e630 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.706+0000 7fcb66da9740 1 -- 172.21.15.1:0/3249954288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb54012450 con 0x55d93396e630 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.706+0000 7fcb42a00640 1 -- 172.21.15.1:0/3249954288 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcb480a05a0 con 0x55d93396e630 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.706+0000 7fcb66da9740 10 client.26174 fetch_fsmap learned FSMap version 36 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.706+0000 7fcb66da9740 1 -- 172.21.15.1:0/3249954288 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d9335e5210 con 0x55d93396e630 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.706+0000 7fcb42a00640 1 -- 172.21.15.1:0/3249954288 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb54012450 con 0x55d93396e630 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.706+0000 7fcb42a00640 1 -- 172.21.15.1:0/3249954288 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fcb480b5020 con 0x55d93396e630 2024-04-26T21:38:32.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.707+0000 7fcb66da9740 10 client.26174 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:38:32.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.707+0000 7fcb66da9740 10 client.26174 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:38:32.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.707+0000 7fcb66da9740 20 client.26174 populate_metadata read hostname 'smithi001' 2024-04-26T21:38:32.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.707+0000 7fcb66da9740 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d933991520 0x55d933993930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.707+0000 7fcb66da9740 4 client.26174 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:38:32.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.707+0000 7fcb66da9740 1 -- 172.21.15.1:0/3249954288 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9338e0f90 con 0x55d933991520 2024-04-26T21:38:32.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.707+0000 7fcb61400640 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d933991520 0x55d933993930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.708+0000 7fcb61400640 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d933991520 0x55d933993930 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=240efe9f18a19f3f server_cookie=4987ff8c5915d47d in_seq=0 out_seq=0 2024-04-26T21:38:32.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.708+0000 7fcb42a00640 10 client.26174 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.708+0000 7fcb42a00640 1 -- 172.21.15.1:0/3249954288 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d9338e0f90 con 0x55d933991520 2024-04-26T21:38:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.708+0000 7fcb42a00640 10 client.26174 handle_command_reply: tid=0 2024-04-26T21:38:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.709+0000 7fcb66da9740 1 client.26174 shutdown 2024-04-26T21:38:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.709+0000 7fcb66da9740 1 -- 172.21.15.1:0/3249954288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 msgr2=0x55d93397ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.709+0000 7fcb66da9740 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93397ec20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fcb480cd9a0 tx=0x7fcb4809bde0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.709+0000 7fcb66da9740 1 -- 172.21.15.1:0/3249954288 shutdown_connections 2024-04-26T21:38:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.709+0000 7fcb66da9740 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d933991520 0x55d933993930 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.709+0000 7fcb66da9740 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.709+0000 7fcb66da9740 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93397ec20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.709+0000 7fcb66da9740 1 --2- 172.21.15.1:0/3249954288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.709+0000 7fcb42a00640 0 client.26174 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.709+0000 7fcb66da9740 1 -- 172.21.15.1:0/3249954288 >> 172.21.15.1:0/3249954288 conn(0x55d93396a2e0 msgr2=0x55d933970060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.710+0000 7fcb66da9740 1 -- 172.21.15.1:0/3249954288 shutdown_connections 2024-04-26T21:38:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.710+0000 7fcb66da9740 1 -- 172.21.15.1:0/3249954288 wait complete. 2024-04-26T21:38:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.710+0000 7fcb66da9740 20 client.26174 trim_cache size 0 max 16384 2024-04-26T21:38:32.743 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.743+0000 7fcb66da9740 1 Processor -- start 2024-04-26T21:38:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.743+0000 7fcb66da9740 1 -- start start 2024-04-26T21:38:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.744+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d933980e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.744+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93398d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.744+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.744+0000 7fcb66da9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c012970 con 0x55d9339817c0 2024-04-26T21:38:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.744+0000 7fcb66da9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c010240 con 0x55d93396e630 2024-04-26T21:38:32.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.744+0000 7fcb66da9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c00e0b0 con 0x55d93396dd10 2024-04-26T21:38:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.744+0000 7fcb61400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d933980e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.744+0000 7fcb61400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d933980e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37842/0 (socket says 172.21.15.1:37842) 2024-04-26T21:38:32.745 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.744+0000 7fcb61400640 1 -- 172.21.15.1:0/3466323267 learned_addr learned my addr 172.21.15.1:0/3466323267 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.745+0000 7fcb61e00640 1 --2- 172.21.15.1:0/3466323267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.745+0000 7fcb60a00640 1 --2- 172.21.15.1:0/3466323267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93398d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.745+0000 7fcb61400640 1 -- 172.21.15.1:0/3466323267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 msgr2=0x55d93398d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.745+0000 7fcb61400640 1 --2- 172.21.15.1:0/3466323267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93398d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.745+0000 7fcb61400640 1 -- 172.21.15.1:0/3466323267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 msgr2=0x55d93397d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.745+0000 7fcb61400640 1 --2- 172.21.15.1:0/3466323267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.745+0000 7fcb61400640 1 -- 172.21.15.1:0/3466323267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9336b1620 con 0x55d93396dd10 2024-04-26T21:38:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.746+0000 7fcb61400640 1 --2- 172.21.15.1:0/3466323267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d933980e10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcb5002a9e0 tx=0x7fcb5003f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4e2c5eb34d72d40 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.747+0000 7fcb4dc00640 1 -- 172.21.15.1:0/3466323267 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb50035040 con 0x55d93396dd10 2024-04-26T21:38:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.747+0000 7fcb4dc00640 1 -- 172.21.15.1:0/3466323267 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb5005a070 con 0x55d93396dd10 2024-04-26T21:38:32.747 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.747+0000 7fcb4dc00640 1 -- 172.21.15.1:0/3466323267 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5003d450 con 0x55d93396dd10 2024-04-26T21:38:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.748+0000 7fcb66da9740 1 -- 172.21.15.1:0/3466323267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 msgr2=0x55d933980e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.748+0000 7fcb66da9740 1 --2- 172.21.15.1:0/3466323267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d933980e10 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fcb5002a9e0 tx=0x7fcb5003f000 comp rx=0 tx=0).stop 2024-04-26T21:38:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.748+0000 7fcb66da9740 1 -- 172.21.15.1:0/3466323267 shutdown_connections 2024-04-26T21:38:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.748+0000 7fcb66da9740 1 --2- 172.21.15.1:0/3466323267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.748+0000 7fcb66da9740 1 --2- 172.21.15.1:0/3466323267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93398d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.748+0000 7fcb66da9740 1 --2- 172.21.15.1:0/3466323267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d933980e10 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.748+0000 7fcb66da9740 1 -- 172.21.15.1:0/3466323267 >> 172.21.15.1:0/3466323267 conn(0x55d93396a2e0 msgr2=0x55d933968bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:32.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.748+0000 7fcb66da9740 1 -- 172.21.15.1:0/3466323267 shutdown_connections 2024-04-26T21:38:32.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.748+0000 7fcb66da9740 1 -- 172.21.15.1:0/3466323267 wait complete. 2024-04-26T21:38:32.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.749+0000 7fcb66da9740 1 -- start start 2024-04-26T21:38:32.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.749+0000 7fcb66da9740 1 Processor -- start 2024-04-26T21:38:32.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.750+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.750+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93397bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.750+0000 7fcb66da9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.750+0000 7fcb66da9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c00e700 con 0x55d9339817c0 2024-04-26T21:38:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.750+0000 7fcb66da9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c00fb70 con 0x55d93396e630 2024-04-26T21:38:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.750+0000 7fcb66da9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c00edd0 con 0x55d93396dd10 2024-04-26T21:38:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.750+0000 7fcb66da9740 10 client.0 _ll_register_callbacks cb 0x7fcb48098540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:38:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.750+0000 7fcb61400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.750+0000 7fcb61400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37858/0 (socket says 172.21.15.1:37858) 2024-04-26T21:38:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.750+0000 7fcb61400640 1 -- 172.21.15.1:0/2085751866 learned_addr learned my addr 172.21.15.1:0/2085751866 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.751+0000 7fcb60a00640 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93397bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.751+0000 7fcb61e00640 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.751+0000 7fcb61400640 1 -- 172.21.15.1:0/2085751866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 msgr2=0x55d93397bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.751+0000 7fcb61400640 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93397bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.751+0000 7fcb61400640 1 -- 172.21.15.1:0/2085751866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 msgr2=0x55d93397c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.751+0000 7fcb61400640 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.751+0000 7fcb61400640 1 -- 172.21.15.1:0/2085751866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d93388dcb0 con 0x55d93396dd10 2024-04-26T21:38:32.752 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.751+0000 7fcb61e00640 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:38:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.752+0000 7fcb61400640 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397b670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcb5002aab0 tx=0x7fcb50009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=596c9876eb6a59b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.752+0000 7fcb42a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:38:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.752+0000 7fcb42a00640 1 -- 172.21.15.1:0/2085751866 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb50063070 con 0x55d93396dd10 2024-04-26T21:38:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.752+0000 7fcb42a00640 1 -- 172.21.15.1:0/2085751866 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7fcb500314a0 con 0x55d93396dd10 2024-04-26T21:38:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.752+0000 7fcb66da9740 1 -- 172.21.15.1:0/2085751866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb5005a070 con 0x55d93396dd10 2024-04-26T21:38:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb42a00640 1 -- 172.21.15.1:0/2085751866 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcb500316a0 con 0x55d93396dd10 2024-04-26T21:38:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb66da9740 10 client.26155 fetch_fsmap learned FSMap version 36 2024-04-26T21:38:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb66da9740 1 -- 172.21.15.1:0/2085751866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d9339f7e70 con 0x55d93396dd10 2024-04-26T21:38:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb42a00640 1 -- 172.21.15.1:0/2085751866 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb5005a070 con 0x55d93396dd10 2024-04-26T21:38:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb42a00640 1 -- 172.21.15.1:0/2085751866 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fcb5003dbd0 con 0x55d93396dd10 2024-04-26T21:38:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb66da9740 10 client.26155 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:38:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb66da9740 10 client.26155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:38:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb66da9740 20 client.26155 populate_metadata read hostname 'smithi001' 2024-04-26T21:38:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb66da9740 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d9339f85f0 0x55d9339faa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb66da9740 4 client.26155 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:38:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.753+0000 7fcb66da9740 1 -- 172.21.15.1:0/2085751866 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d9338dae00 con 0x55d9339f85f0 2024-04-26T21:38:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.754+0000 7fcb60a00640 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d9339f85f0 0x55d9339faa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.754+0000 7fcb60a00640 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d9339f85f0 0x55d9339faa10 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50a6a465164c8d85 server_cookie=5a0c8ac0f2a241c1 in_seq=0 out_seq=0 2024-04-26T21:38:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.754+0000 7fcb42a00640 10 client.26155 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb42a00640 1 -- 172.21.15.1:0/2085751866 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d9338dae00 con 0x55d9339f85f0 2024-04-26T21:38:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb42a00640 10 client.26155 handle_command_reply: tid=0 2024-04-26T21:38:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb66da9740 1 client.26155 shutdown 2024-04-26T21:38:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb66da9740 1 -- 172.21.15.1:0/2085751866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 msgr2=0x55d93397b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb66da9740 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397b670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcb5002aab0 tx=0x7fcb50009260 comp rx=0 tx=0).stop 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb66da9740 1 -- 172.21.15.1:0/2085751866 shutdown_connections 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb66da9740 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d9339f85f0 0x55d9339faa10 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb66da9740 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d9339817c0 0x55d93397c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb66da9740 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d93396e630 0x55d93397bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb66da9740 1 --2- 172.21.15.1:0/2085751866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d93396dd10 0x55d93397b670 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb42a00640 0 client.26155 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb66da9740 1 -- 172.21.15.1:0/2085751866 >> 172.21.15.1:0/2085751866 conn(0x55d93396a2e0 msgr2=0x55d933969440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.851+0000 7fcb66da9740 1 -- 172.21.15.1:0/2085751866 shutdown_connections 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.852+0000 7fcb66da9740 1 -- 172.21.15.1:0/2085751866 wait complete. 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.852+0000 7fcb66da9740 20 client.26155 trim_cache size 0 max 16384 2024-04-26T21:38:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:38:32.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:38:32.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:38:32.854 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 -- 172.21.15.1:0/967021590 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcb440a3e10 msgr2=0x7fcb440a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcb440a3e10 0x7fcb440a62d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcb50005ec0 tx=0x7fcb50005a20 comp rx=0 tx=0).stop 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 -- 172.21.15.1:0/967021590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb5c109e90 msgr2=0x7fcb5c19b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb5c109e90 0x7fcb5c19b8c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800c940 tx=0x7fcb4800ce00 comp rx=0 tx=0).stop 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 -- 172.21.15.1:0/967021590 shutdown_connections 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcb440a3e10 0x7fcb440a62d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb5c109e90 0x7fcb5c19b8c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb5c19fbf0 0x7fcb5c1a00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 --2- 172.21.15.1:0/967021590 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb5c104dc0 0x7fcb5c19b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 -- 172.21.15.1:0/967021590 >> 172.21.15.1:0/967021590 conn(0x7fcb5c1005e0 msgr2=0x7fcb5c1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 -- 172.21.15.1:0/967021590 shutdown_connections 2024-04-26T21:38:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:32.853+0000 7fcb63e00640 1 -- 172.21.15.1:0/967021590 wait complete. 2024-04-26T21:38:33.576 INFO:tasks.workunit.client.0.smithi001.stdout: 1 288585 4.42 MB/sec execute 271 sec latency 60.237 ms 2024-04-26T21:38:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:34 smithi186 ceph-mon[32991]: pgmap v254: 129 pgs: 129 active+clean; 175 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 1.6 MiB/s wr, 131 op/s 2024-04-26T21:38:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:34 smithi001 ceph-mon[28054]: pgmap v254: 129 pgs: 129 active+clean; 175 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 1.6 MiB/s wr, 131 op/s 2024-04-26T21:38:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:34 smithi038 ceph-mon[32724]: pgmap v254: 129 pgs: 129 active+clean; 175 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 1.6 MiB/s wr, 131 op/s 2024-04-26T21:38:34.577 INFO:tasks.workunit.client.0.smithi001.stdout: 1 289891 4.43 MB/sec execute 272 sec latency 56.913 ms 2024-04-26T21:38:35.577 INFO:tasks.workunit.client.0.smithi001.stdout: 1 291103 4.45 MB/sec execute 273 sec latency 23.554 ms 2024-04-26T21:38:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:36 smithi186 ceph-mon[32991]: pgmap v255: 129 pgs: 129 active+clean; 174 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-04-26T21:38:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:36 smithi001 ceph-mon[28054]: pgmap v255: 129 pgs: 129 active+clean; 174 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-04-26T21:38:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:36 smithi038 ceph-mon[32724]: pgmap v255: 129 pgs: 129 active+clean; 174 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2024-04-26T21:38:36.577 INFO:tasks.workunit.client.0.smithi001.stdout: 1 291353 4.44 MB/sec execute 274 sec latency 23.702 ms 2024-04-26T21:38:37.577 INFO:tasks.workunit.client.0.smithi001.stdout: 1 292096 4.43 MB/sec execute 275 sec latency 16.621 ms 2024-04-26T21:38:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:38 smithi186 ceph-mon[32991]: pgmap v256: 129 pgs: 129 active+clean; 174 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-04-26T21:38:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:38 smithi001 ceph-mon[28054]: pgmap v256: 129 pgs: 129 active+clean; 174 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-04-26T21:38:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:38 smithi038 ceph-mon[32724]: pgmap v256: 129 pgs: 129 active+clean; 174 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-04-26T21:38:38.577 INFO:tasks.workunit.client.0.smithi001.stdout: 1 293482 4.43 MB/sec execute 276 sec latency 50.961 ms 2024-04-26T21:38:39.577 INFO:tasks.workunit.client.0.smithi001.stdout: 1 294687 4.46 MB/sec execute 277 sec latency 21.393 ms 2024-04-26T21:38:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:40 smithi186 ceph-mon[32991]: pgmap v257: 129 pgs: 129 active+clean; 174 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-04-26T21:38:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:40 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:38:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:40 smithi001 ceph-mon[28054]: pgmap v257: 129 pgs: 129 active+clean; 174 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-04-26T21:38:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:40 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:38:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:40 smithi038 ceph-mon[32724]: pgmap v257: 129 pgs: 129 active+clean; 174 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 1.2 MiB/s wr, 136 op/s 2024-04-26T21:38:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:40 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:38:40.577 INFO:tasks.workunit.client.0.smithi001.stdout: 1 295204 4.44 MB/sec execute 278 sec latency 21.940 ms 2024-04-26T21:38:41.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:41 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:38:41.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:41 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:38:41.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:41 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:38:41.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:41 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:38:41.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:41 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:38:41.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:41 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:38:41.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:41 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:38:41.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:41 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:38:41.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:41 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:38:41.577 INFO:tasks.workunit.client.0.smithi001.stdout: 1 296801 4.45 MB/sec execute 279 sec latency 20.628 ms 2024-04-26T21:38:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:42 smithi186 ceph-mon[32991]: pgmap v258: 129 pgs: 129 active+clean; 178 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 199 op/s 2024-04-26T21:38:42.470 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:42 smithi001 ceph-mon[28054]: pgmap v258: 129 pgs: 129 active+clean; 178 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 199 op/s 2024-04-26T21:38:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:42 smithi038 ceph-mon[32724]: pgmap v258: 129 pgs: 129 active+clean; 178 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 199 op/s 2024-04-26T21:38:42.577 INFO:tasks.workunit.client.0.smithi001.stdout: 1 297812 4.46 MB/sec execute 280 sec latency 22.907 ms 2024-04-26T21:38:43.578 INFO:tasks.workunit.client.0.smithi001.stdout: 1 298391 4.46 MB/sec execute 281 sec latency 43.790 ms 2024-04-26T21:38:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:44 smithi186 ceph-mon[32991]: pgmap v259: 129 pgs: 129 active+clean; 178 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-04-26T21:38:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:44 smithi001 ceph-mon[28054]: pgmap v259: 129 pgs: 129 active+clean; 178 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-04-26T21:38:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:44 smithi038 ceph-mon[32724]: pgmap v259: 129 pgs: 129 active+clean; 178 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-04-26T21:38:44.578 INFO:tasks.workunit.client.0.smithi001.stdout: 1 299109 4.45 MB/sec execute 282 sec latency 29.055 ms 2024-04-26T21:38:45.578 INFO:tasks.workunit.client.0.smithi001.stdout: 1 300472 4.46 MB/sec execute 283 sec latency 29.907 ms 2024-04-26T21:38:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.004+0000 7fdafa600640 1 -- 172.21.15.1:0/1987698016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf410a970 msgr2=0x7fdaf410ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.004+0000 7fdafa600640 1 --2- 172.21.15.1:0/1987698016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf410a970 0x7fdaf410ad50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009890 tx=0x7fdae802a370 comp rx=0 tx=0).stop 2024-04-26T21:38:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.004+0000 7fdafa600640 1 -- 172.21.15.1:0/1987698016 shutdown_connections 2024-04-26T21:38:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.004+0000 7fdafa600640 1 --2- 172.21.15.1:0/1987698016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf4105320 0x7fdaf4111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.004+0000 7fdafa600640 1 --2- 172.21.15.1:0/1987698016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaf4104960 0x7fdaf4104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.004+0000 7fdafa600640 1 --2- 172.21.15.1:0/1987698016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf410a970 0x7fdaf410ad50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.004+0000 7fdafa600640 1 -- 172.21.15.1:0/1987698016 >> 172.21.15.1:0/1987698016 conn(0x7fdaf4100430 msgr2=0x7fdaf4102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.005+0000 7fdafa600640 1 -- 172.21.15.1:0/1987698016 shutdown_connections 2024-04-26T21:38:46.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.005+0000 7fdafa600640 1 -- 172.21.15.1:0/1987698016 wait complete. 2024-04-26T21:38:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.006+0000 7fdafa600640 1 Processor -- start 2024-04-26T21:38:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.006+0000 7fdafa600640 1 -- start start 2024-04-26T21:38:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.006+0000 7fdafa600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf4104960 0x7fdaf419b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.006+0000 7fdafa600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf4105320 0x7fdaf419b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.006+0000 7fdafa600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaf410a970 0x7fdaf419fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.006+0000 7fdafa600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf4114c00 con 0x7fdaf4105320 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.006+0000 7fdafa600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf4114a80 con 0x7fdaf410a970 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.006+0000 7fdafa600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf4114d80 con 0x7fdaf4104960 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.006+0000 7fdaf9c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaf410a970 0x7fdaf419fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf9c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaf410a970 0x7fdaf419fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:56556/0 (socket says 172.21.15.1:56556) 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf4104960 0x7fdaf419b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf9c00640 1 -- 172.21.15.1:0/3257455797 learned_addr learned my addr 172.21.15.1:0/3257455797 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf4105320 0x7fdaf419b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf9c00640 1 -- 172.21.15.1:0/3257455797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf4104960 msgr2=0x7fdaf419b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf9c00640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf4104960 0x7fdaf419b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf9c00640 1 -- 172.21.15.1:0/3257455797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf4105320 msgr2=0x7fdaf419b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf9c00640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf4105320 0x7fdaf419b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf9c00640 1 -- 172.21.15.1:0/3257455797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf41a01f0 con 0x7fdaf410a970 2024-04-26T21:38:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf9c00640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaf410a970 0x7fdaf419fac0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00efd0 tx=0x7fdaec00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.007+0000 7fdaf1600640 1 -- 172.21.15.1:0/3257455797 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec003b70 con 0x7fdaf410a970 2024-04-26T21:38:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.008+0000 7fdaf1600640 1 -- 172.21.15.1:0/3257455797 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdaec003d10 con 0x7fdaf410a970 2024-04-26T21:38:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.008+0000 7fdafa600640 1 -- 172.21.15.1:0/3257455797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf41a04e0 con 0x7fdaf410a970 2024-04-26T21:38:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.008+0000 7fdaf1600640 1 -- 172.21.15.1:0/3257455797 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec0108a0 con 0x7fdaf410a970 2024-04-26T21:38:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.008+0000 7fdafa600640 1 -- 172.21.15.1:0/3257455797 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf41a7d70 con 0x7fdaf410a970 2024-04-26T21:38:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.011+0000 7fdaf1600640 1 -- 172.21.15.1:0/3257455797 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fdaec004020 con 0x7fdaf410a970 2024-04-26T21:38:46.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.011+0000 7fdafeb91740 1 Processor -- start 2024-04-26T21:38:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.011+0000 7fdafeb91740 1 -- start start 2024-04-26T21:38:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.012+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df4523c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.012+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.012+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df45d1c0 0x5613df45f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.012+0000 7fdafeb91740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5613df3b7380 con 0x5613df451fe0 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.012+0000 7fdafeb91740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5613df353c30 con 0x5613df45d1c0 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.012+0000 7fdafeb91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5613df3b3340 con 0x5613df452900 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.012+0000 7fdaf9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.012+0000 7fdaf9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50094/0 (socket says 172.21.15.1:50094) 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.012+0000 7fdaf9200640 1 -- 172.21.15.1:0/1261294381 learned_addr learned my addr 172.21.15.1:0/1261294381 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.012+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/1261294381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df4523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/1261294381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df45d1c0 0x5613df45f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf9200640 1 -- 172.21.15.1:0/1261294381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df45d1c0 msgr2=0x5613df45f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf9200640 1 --2- 172.21.15.1:0/1261294381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df45d1c0 0x5613df45f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf9200640 1 -- 172.21.15.1:0/1261294381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 msgr2=0x5613df4523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf9200640 1 --2- 172.21.15.1:0/1261294381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df4523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf9200640 1 -- 172.21.15.1:0/1261294381 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613df375940 con 0x5613df452900 2024-04-26T21:38:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf1600640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdadc0a3d00 0x7fdadc0a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/1261294381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df4523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:38:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf1600640 1 -- 172.21.15.1:0/3257455797 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7fdaec0d0fe0 con 0x7fdaf410a970 2024-04-26T21:38:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf9c00640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdadc0a3d00 0x7fdadc0a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.013+0000 7fdaf9200640 1 --2- 172.21.15.1:0/1261294381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45cc80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdae802a890 tx=0x7fdae8005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ca364f2e3e4ecdd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.014+0000 7fdae6000640 1 -- 172.21.15.1:0/1261294381 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8037040 con 0x5613df452900 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.014+0000 7fdae6000640 1 -- 172.21.15.1:0/1261294381 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae8038040 con 0x5613df452900 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.014+0000 7fdae6000640 1 -- 172.21.15.1:0/1261294381 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8003850 con 0x5613df452900 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.014+0000 7fdafeb91740 1 -- 172.21.15.1:0/1261294381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 msgr2=0x5613df45cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.014+0000 7fdaf9c00640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdadc0a3d00 0x7fdadc0a61c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdaec0994b0 tx=0x7fdaec00af00 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.014+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1261294381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45cc80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdae802a890 tx=0x7fdae8005dc0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.015+0000 7fdafeb91740 1 -- 172.21.15.1:0/1261294381 shutdown_connections 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.015+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1261294381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45cc80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.015+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1261294381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df45d1c0 0x5613df45f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.015+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1261294381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df4523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.015+0000 7fdafeb91740 1 -- 172.21.15.1:0/1261294381 >> 172.21.15.1:0/1261294381 conn(0x5613df44e4f0 msgr2=0x5613df44e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.015+0000 7fdafeb91740 1 -- 172.21.15.1:0/1261294381 shutdown_connections 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.015+0000 7fdafeb91740 1 -- 172.21.15.1:0/1261294381 wait complete. 2024-04-26T21:38:46.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.015+0000 7fdafeb91740 1 -- start start 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.015+0000 7fdafeb91740 1 Processor -- start 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df45a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df452900 0x5613df45a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df45d1c0 0x5613df45aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdafeb91740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5613df45fd90 con 0x5613df451fe0 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdafeb91740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5613df33ec60 con 0x5613df452900 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdafeb91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5613df45ff10 con 0x5613df45d1c0 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdafeb91740 10 client.0 _ll_register_callbacks cb 0x5613df36ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdaf9200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df452900 0x5613df45a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdaf9200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df452900 0x5613df45a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:56576/0 (socket says 172.21.15.1:56576) 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdaf9200640 1 -- 172.21.15.1:0/1298493315 learned_addr learned my addr 172.21.15.1:0/1298493315 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df45a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.016+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df45d1c0 0x5613df45aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdaf9200640 1 -- 172.21.15.1:0/1298493315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df45d1c0 msgr2=0x5613df45aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdaf9200640 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df45d1c0 0x5613df45aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdaf9200640 1 -- 172.21.15.1:0/1298493315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 msgr2=0x5613df45a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdaf9200640 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df45a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdaf9200640 1 -- 172.21.15.1:0/1298493315 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613df34d510 con 0x5613df452900 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df45a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdaf9200640 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df452900 0x5613df45a560 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdae8035040 tx=0x7fdae8030e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b75378acb510d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdadaa00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdadaa00640 1 -- 172.21.15.1:0/1298493315 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae804b030 con 0x5613df452900 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdadaa00640 1 -- 172.21.15.1:0/1298493315 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7fdae8030660 con 0x5613df452900 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.017+0000 7fdafeb91740 1 -- 172.21.15.1:0/1298493315 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae8038040 con 0x5613df452900 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdadaa00640 1 -- 172.21.15.1:0/1298493315 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fdae8057050 con 0x5613df452900 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdafeb91740 10 client.26192 fetch_fsmap learned FSMap version 36 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdafeb91740 1 -- 172.21.15.1:0/1298493315 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5613df0c9210 con 0x5613df452900 2024-04-26T21:38:46.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdadaa00640 1 -- 172.21.15.1:0/1298493315 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae8038040 con 0x5613df452900 2024-04-26T21:38:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdadaa00640 1 -- 172.21.15.1:0/1298493315 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fdae8004b10 con 0x5613df452900 2024-04-26T21:38:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdafeb91740 10 client.26192 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:38:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdafeb91740 10 client.26192 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:38:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdafeb91740 20 client.26192 populate_metadata read hostname 'smithi001' 2024-04-26T21:38:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5613df46cc80 0x5613df46f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdafeb91740 4 client.26192 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:38:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.018+0000 7fdafeb91740 1 -- 172.21.15.1:0/1298493315 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613df3c4f90 con 0x5613df46cc80 2024-04-26T21:38:46.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.019+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5613df46cc80 0x5613df46f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.019+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5613df46cc80 0x5613df46f070 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0608f3745353c98 server_cookie=574c42e2c246dec3 in_seq=0 out_seq=0 2024-04-26T21:38:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.019+0000 7fdadaa00640 10 client.26192 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.020+0000 7fdadaa00640 1 -- 172.21.15.1:0/1298493315 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5613df3c4f90 con 0x5613df46cc80 2024-04-26T21:38:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.020+0000 7fdadaa00640 10 client.26192 handle_command_reply: tid=0 2024-04-26T21:38:46.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 client.26192 shutdown 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 -- 172.21.15.1:0/1298493315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df452900 msgr2=0x5613df45a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df452900 0x5613df45a560 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdae8035040 tx=0x7fdae8030e60 comp rx=0 tx=0).stop 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 -- 172.21.15.1:0/1298493315 shutdown_connections 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5613df46cc80 0x5613df46f070 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df45d1c0 0x5613df45aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df452900 0x5613df45a560 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1298493315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df45a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 -- 172.21.15.1:0/1298493315 >> 172.21.15.1:0/1298493315 conn(0x5613df44e4f0 msgr2=0x5613df454270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdadaa00640 0 client.26192 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 -- 172.21.15.1:0/1298493315 shutdown_connections 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 1 -- 172.21.15.1:0/1298493315 wait complete. 2024-04-26T21:38:46.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.021+0000 7fdafeb91740 20 client.26192 trim_cache size 0 max 16384 2024-04-26T21:38:46.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.038+0000 7fdafeb91740 1 Processor -- start 2024-04-26T21:38:46.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.038+0000 7fdafeb91740 1 -- start start 2024-04-26T21:38:46.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.038+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df4591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.038+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.038+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df45d1c0 0x5613df474720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.038+0000 7fdafeb91740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf41b4920 con 0x5613df451fe0 2024-04-26T21:38:46.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.038+0000 7fdafeb91740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf41b21f0 con 0x5613df45d1c0 2024-04-26T21:38:46.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.038+0000 7fdafeb91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf41b0060 con 0x5613df452900 2024-04-26T21:38:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf3e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df4591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50106/0 (socket says 172.21.15.1:50106) 2024-04-26T21:38:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf9200640 1 -- 172.21.15.1:0/1657810800 learned_addr learned my addr 172.21.15.1:0/1657810800 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/1657810800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df45d1c0 0x5613df474720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf9200640 1 -- 172.21.15.1:0/1657810800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df45d1c0 msgr2=0x5613df474720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf9200640 1 --2- 172.21.15.1:0/1657810800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df45d1c0 0x5613df474720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf9200640 1 -- 172.21.15.1:0/1657810800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 msgr2=0x5613df4591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf9200640 1 --2- 172.21.15.1:0/1657810800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df4591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf9200640 1 -- 172.21.15.1:0/1657810800 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613df195620 con 0x5613df452900 2024-04-26T21:38:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.039+0000 7fdaf9200640 1 --2- 172.21.15.1:0/1657810800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45be50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdae8030e30 tx=0x7fdae8005d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e2c9b6e4c70431c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdae5600640 1 -- 172.21.15.1:0/1657810800 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae804b020 con 0x5613df452900 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdae5600640 1 -- 172.21.15.1:0/1657810800 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae8004430 con 0x5613df452900 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdae5600640 1 -- 172.21.15.1:0/1657810800 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8037040 con 0x5613df452900 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdafeb91740 1 -- 172.21.15.1:0/1657810800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 msgr2=0x5613df45be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1657810800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45be50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fdae8030e30 tx=0x7fdae8005d40 comp rx=0 tx=0).stop 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdafeb91740 1 -- 172.21.15.1:0/1657810800 shutdown_connections 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1657810800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df45be50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1657810800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df45d1c0 0x5613df474720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdafeb91740 1 --2- 172.21.15.1:0/1657810800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df451fe0 0x5613df4591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdafeb91740 1 -- 172.21.15.1:0/1657810800 >> 172.21.15.1:0/1657810800 conn(0x5613df44e4f0 msgr2=0x5613df45b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdafeb91740 1 -- 172.21.15.1:0/1657810800 shutdown_connections 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.040+0000 7fdafeb91740 1 -- 172.21.15.1:0/1657810800 wait complete. 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.041+0000 7fdafeb91740 1 -- start start 2024-04-26T21:38:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.041+0000 7fdafeb91740 1 Processor -- start 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df451fe0 0x5613df4566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df456be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdafeb91740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df45d1c0 0x5613df464190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdafeb91740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf41b06b0 con 0x5613df45d1c0 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdafeb91740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf41b1b20 con 0x5613df451fe0 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdafeb91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf41b0d80 con 0x5613df452900 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdafeb91740 10 client.0 _ll_register_callbacks cb 0x7fdaec0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdaf9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df456be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdaf9200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df456be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50112/0 (socket says 172.21.15.1:50112) 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdaf9200640 1 -- 172.21.15.1:0/2445498243 learned_addr learned my addr 172.21.15.1:0/2445498243 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df451fe0 0x5613df4566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df45d1c0 0x5613df464190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdaf9200640 1 -- 172.21.15.1:0/2445498243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df451fe0 msgr2=0x5613df4566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdaf9200640 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df451fe0 0x5613df4566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdaf9200640 1 -- 172.21.15.1:0/2445498243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df45d1c0 msgr2=0x5613df464190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdaf9200640 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df45d1c0 0x5613df464190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.042+0000 7fdaf9200640 1 -- 172.21.15.1:0/2445498243 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613df371cb0 con 0x5613df452900 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.043+0000 7fdaf9200640 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df456be0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdae802a890 tx=0x7fdae80094d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d975f12f368675c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.043+0000 7fdadaa00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.043+0000 7fdadaa00640 1 -- 172.21.15.1:0/2445498243 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8037040 con 0x5613df452900 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.043+0000 7fdadaa00640 1 -- 172.21.15.1:0/2445498243 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7fdae80048c0 con 0x5613df452900 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.043+0000 7fdafeb91740 1 -- 172.21.15.1:0/2445498243 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae8004430 con 0x5613df452900 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.043+0000 7fdadaa00640 1 -- 172.21.15.1:0/2445498243 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fdae804b040 con 0x5613df452900 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.043+0000 7fdafeb91740 10 client.26173 fetch_fsmap learned FSMap version 36 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.043+0000 7fdafeb91740 1 -- 172.21.15.1:0/2445498243 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5613df465720 con 0x5613df452900 2024-04-26T21:38:46.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.043+0000 7fdadaa00640 1 -- 172.21.15.1:0/2445498243 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdae8004430 con 0x5613df452900 2024-04-26T21:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.044+0000 7fdadaa00640 1 -- 172.21.15.1:0/2445498243 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fdae8004f60 con 0x5613df452900 2024-04-26T21:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.044+0000 7fdafeb91740 10 client.26173 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.044+0000 7fdafeb91740 10 client.26173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.044+0000 7fdafeb91740 20 client.26173 populate_metadata read hostname 'smithi001' 2024-04-26T21:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.044+0000 7fdafeb91740 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5613df4d2f90 0x5613df4d5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.044+0000 7fdafeb91740 4 client.26173 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.044+0000 7fdafeb91740 1 -- 172.21.15.1:0/2445498243 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5613df3bee00 con 0x5613df4d2f90 2024-04-26T21:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.044+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5613df4d2f90 0x5613df4d5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.044+0000 7fdaf3e00640 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5613df4d2f90 0x5613df4d5380 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7e7da0d4d9cba3c server_cookie=6607d4be002269d3 in_seq=0 out_seq=0 2024-04-26T21:38:46.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.044+0000 7fdadaa00640 10 client.26173 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.104+0000 7fdadaa00640 1 -- 172.21.15.1:0/2445498243 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5613df3bee00 con 0x5613df4d2f90 2024-04-26T21:38:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.104+0000 7fdadaa00640 10 client.26173 handle_command_reply: tid=0 2024-04-26T21:38:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 client.26173 shutdown 2024-04-26T21:38:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 -- 172.21.15.1:0/2445498243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 msgr2=0x5613df456be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df456be0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdae802a890 tx=0x7fdae80094d0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 -- 172.21.15.1:0/2445498243 shutdown_connections 2024-04-26T21:38:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5613df4d2f90 0x5613df4d5380 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5613df452900 0x5613df456be0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5613df45d1c0 0x5613df464190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdadaa00640 0 client.26173 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 --2- 172.21.15.1:0/2445498243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5613df451fe0 0x5613df4566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 -- 172.21.15.1:0/2445498243 >> 172.21.15.1:0/2445498243 conn(0x5613df44e4f0 msgr2=0x5613df454220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 -- 172.21.15.1:0/2445498243 shutdown_connections 2024-04-26T21:38:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 1 -- 172.21.15.1:0/2445498243 wait complete. 2024-04-26T21:38:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.105+0000 7fdafeb91740 20 client.26173 trim_cache size 0 max 16384 2024-04-26T21:38:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:38:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:38:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:38:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:38:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 -- 172.21.15.1:0/3257455797 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdadc0a3d00 msgr2=0x7fdadc0a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdadc0a3d00 0x7fdadc0a61c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdaec0994b0 tx=0x7fdaec00af00 comp rx=0 tx=0).stop 2024-04-26T21:38:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 -- 172.21.15.1:0/3257455797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaf410a970 msgr2=0x7fdaf419fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaf410a970 0x7fdaf419fac0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00efd0 tx=0x7fdaec00c6a0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 -- 172.21.15.1:0/3257455797 shutdown_connections 2024-04-26T21:38:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdadc0a3d00 0x7fdadc0a61c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdaf410a970 0x7fdaf419fac0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdaf4105320 0x7fdaf419b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 --2- 172.21.15.1:0/3257455797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf4104960 0x7fdaf419b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 -- 172.21.15.1:0/3257455797 >> 172.21.15.1:0/3257455797 conn(0x7fdaf4100430 msgr2=0x7fdaf4100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 -- 172.21.15.1:0/3257455797 shutdown_connections 2024-04-26T21:38:46.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:46.106+0000 7fdafa600640 1 -- 172.21.15.1:0/3257455797 wait complete. 2024-04-26T21:38:46.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:46 smithi186 ceph-mon[32991]: pgmap v260: 129 pgs: 129 active+clean; 184 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 198 op/s 2024-04-26T21:38:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:46 smithi001 ceph-mon[28054]: pgmap v260: 129 pgs: 129 active+clean; 184 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 198 op/s 2024-04-26T21:38:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:46 smithi038 ceph-mon[32724]: pgmap v260: 129 pgs: 129 active+clean; 184 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 198 op/s 2024-04-26T21:38:46.578 INFO:tasks.workunit.client.0.smithi001.stdout: 1 301755 4.48 MB/sec execute 284 sec latency 30.818 ms 2024-04-26T21:38:47.578 INFO:tasks.workunit.client.0.smithi001.stdout: 1 302002 4.47 MB/sec execute 285 sec latency 22.756 ms 2024-04-26T21:38:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:48 smithi186 ceph-mon[32991]: pgmap v261: 129 pgs: 129 active+clean; 184 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-04-26T21:38:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:48 smithi001 ceph-mon[28054]: pgmap v261: 129 pgs: 129 active+clean; 184 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-04-26T21:38:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:48 smithi038 ceph-mon[32724]: pgmap v261: 129 pgs: 129 active+clean; 184 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-04-26T21:38:48.578 INFO:tasks.workunit.client.0.smithi001.stdout: 1 302743 4.46 MB/sec execute 286 sec latency 90.788 ms 2024-04-26T21:38:49.578 INFO:tasks.workunit.client.0.smithi001.stdout: 1 304290 4.47 MB/sec execute 287 sec latency 31.475 ms 2024-04-26T21:38:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:50 smithi186 ceph-mon[32991]: pgmap v262: 129 pgs: 129 active+clean; 184 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-04-26T21:38:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:50 smithi001 ceph-mon[28054]: pgmap v262: 129 pgs: 129 active+clean; 184 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-04-26T21:38:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:50 smithi038 ceph-mon[32724]: pgmap v262: 129 pgs: 129 active+clean; 184 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 1.4 MiB/s wr, 126 op/s 2024-04-26T21:38:50.578 INFO:tasks.workunit.client.0.smithi001.stdout: 1 305343 4.49 MB/sec execute 288 sec latency 22.674 ms 2024-04-26T21:38:51.579 INFO:tasks.workunit.client.0.smithi001.stdout: 1 305642 4.47 MB/sec execute 289 sec latency 24.588 ms 2024-04-26T21:38:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:52 smithi186 ceph-mon[32991]: pgmap v263: 129 pgs: 129 active+clean; 185 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-04-26T21:38:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:52 smithi001 ceph-mon[28054]: pgmap v263: 129 pgs: 129 active+clean; 185 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-04-26T21:38:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:52 smithi038 ceph-mon[32724]: pgmap v263: 129 pgs: 129 active+clean; 185 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-04-26T21:38:52.579 INFO:tasks.workunit.client.0.smithi001.stdout: 1 306327 4.46 MB/sec execute 290 sec latency 33.674 ms 2024-04-26T21:38:53.579 INFO:tasks.workunit.client.0.smithi001.stdout: 1 307661 4.47 MB/sec execute 291 sec latency 59.269 ms 2024-04-26T21:38:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:54 smithi186 ceph-mon[32991]: pgmap v264: 129 pgs: 129 active+clean; 185 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-04-26T21:38:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:54 smithi001 ceph-mon[28054]: pgmap v264: 129 pgs: 129 active+clean; 185 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-04-26T21:38:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:54 smithi038 ceph-mon[32724]: pgmap v264: 129 pgs: 129 active+clean; 185 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-04-26T21:38:54.579 INFO:tasks.workunit.client.0.smithi001.stdout: 1 308821 4.49 MB/sec execute 292 sec latency 26.621 ms 2024-04-26T21:38:55.579 INFO:tasks.workunit.client.0.smithi001.stdout: 1 309048 4.48 MB/sec execute 293 sec latency 30.069 ms 2024-04-26T21:38:56.457 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:56 smithi186 ceph-mon[32991]: pgmap v265: 129 pgs: 129 active+clean; 189 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 221 op/s 2024-04-26T21:38:56.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:56 smithi001 ceph-mon[28054]: pgmap v265: 129 pgs: 129 active+clean; 189 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 221 op/s 2024-04-26T21:38:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:56 smithi038 ceph-mon[32724]: pgmap v265: 129 pgs: 129 active+clean; 189 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 221 op/s 2024-04-26T21:38:56.579 INFO:tasks.workunit.client.0.smithi001.stdout: 1 309612 4.46 MB/sec execute 294 sec latency 25.801 ms 2024-04-26T21:38:57.579 INFO:tasks.workunit.client.0.smithi001.stdout: 1 310982 4.47 MB/sec execute 295 sec latency 26.243 ms 2024-04-26T21:38:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:38:58 smithi186 ceph-mon[32991]: pgmap v266: 129 pgs: 129 active+clean; 189 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-04-26T21:38:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:38:58 smithi001 ceph-mon[28054]: pgmap v266: 129 pgs: 129 active+clean; 189 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-04-26T21:38:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:38:58 smithi038 ceph-mon[32724]: pgmap v266: 129 pgs: 129 active+clean; 189 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-04-26T21:38:58.579 INFO:tasks.workunit.client.0.smithi001.stdout: 1 311800 4.48 MB/sec execute 296 sec latency 38.194 ms 2024-04-26T21:38:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.286+0000 7f22afc00640 1 -- 172.21.15.1:0/571498986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22a8103ba0 msgr2=0x7f22a8103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.286+0000 7f22afc00640 1 --2- 172.21.15.1:0/571498986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22a8103ba0 0x7f22a8103f80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f229c009900 tx=0x7f229c02a480 comp rx=0 tx=0).stop 2024-04-26T21:38:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.286+0000 7f22afc00640 1 -- 172.21.15.1:0/571498986 shutdown_connections 2024-04-26T21:38:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.286+0000 7f22afc00640 1 --2- 172.21.15.1:0/571498986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a81044c0 0x7f22a810c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.286+0000 7f22afc00640 1 --2- 172.21.15.1:0/571498986 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22a810cf70 0x7f22a810f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.286+0000 7f22afc00640 1 --2- 172.21.15.1:0/571498986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22a8103ba0 0x7f22a8103f80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.286+0000 7f22afc00640 1 -- 172.21.15.1:0/571498986 >> 172.21.15.1:0/571498986 conn(0x7f22a8078950 msgr2=0x7f22a8078d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.286+0000 7f22afc00640 1 -- 172.21.15.1:0/571498986 shutdown_connections 2024-04-26T21:38:59.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.286+0000 7f22afc00640 1 -- 172.21.15.1:0/571498986 wait complete. 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.287+0000 7f22afc00640 1 Processor -- start 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22afc00640 1 -- start start 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22afc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22a8103ba0 0x7f22a819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22afc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22a81044c0 0x7f22a819b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22afc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a810cf70 0x7f22a819fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22afc00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22a8112c00 con 0x7f22a81044c0 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22afc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22a8112a80 con 0x7f22a8103ba0 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22afc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22a8112d80 con 0x7f22a810cf70 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22ad200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22a8103ba0 0x7f22a819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22ad200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22a8103ba0 0x7f22a819b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39334/0 (socket says 172.21.15.1:39334) 2024-04-26T21:38:59.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22ad200640 1 -- 172.21.15.1:0/3875592427 learned_addr learned my addr 172.21.15.1:0/3875592427 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22adc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a810cf70 0x7f22a819fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.288+0000 7f22a7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22a81044c0 0x7f22a819b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.289+0000 7f22ad200640 1 -- 172.21.15.1:0/3875592427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a810cf70 msgr2=0x7f22a819fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.289+0000 7f22ad200640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a810cf70 0x7f22a819fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.289+0000 7f22ad200640 1 -- 172.21.15.1:0/3875592427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22a81044c0 msgr2=0x7f22a819b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.289+0000 7f22ad200640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22a81044c0 0x7f22a819b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.289+0000 7f22ad200640 1 -- 172.21.15.1:0/3875592427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a81a0300 con 0x7f22a8103ba0 2024-04-26T21:38:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.289+0000 7f22ad200640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22a8103ba0 0x7f22a819b310 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f229c005ab0 tx=0x7f229c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.290+0000 7f22a5600640 1 -- 172.21.15.1:0/3875592427 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c03f070 con 0x7f22a8103ba0 2024-04-26T21:38:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.290+0000 7f22a5600640 1 -- 172.21.15.1:0/3875592427 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f229c031b60 con 0x7f22a8103ba0 2024-04-26T21:38:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.290+0000 7f22afc00640 1 -- 172.21.15.1:0/3875592427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a81a0590 con 0x7f22a8103ba0 2024-04-26T21:38:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.290+0000 7f22a5600640 1 -- 172.21.15.1:0/3875592427 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c035030 con 0x7f22a8103ba0 2024-04-26T21:38:59.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.290+0000 7f22afc00640 1 -- 172.21.15.1:0/3875592427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a81a7e80 con 0x7f22a8103ba0 2024-04-26T21:38:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.294+0000 7f22a5600640 1 -- 172.21.15.1:0/3875592427 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f229c042070 con 0x7f22a8103ba0 2024-04-26T21:38:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.294+0000 7f22b2a66740 1 Processor -- start 2024-04-26T21:38:59.294 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.294+0000 7f22b2a66740 1 -- start start 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.294+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a89dd0 0x555751a8a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.294+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a94a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.294+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 0x555751a973a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.294+0000 7f22b2a66740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5557519ef380 con 0x555751a89dd0 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.294+0000 7f22b2a66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55575198bc30 con 0x555751a8a6f0 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.294+0000 7f22b2a66740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5557519eb340 con 0x555751a94fb0 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22adc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a89dd0 0x555751a8a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22adc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a89dd0 0x555751a8a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41734/0 (socket says 172.21.15.1:41734) 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22adc00640 1 -- 172.21.15.1:0/1339888291 learned_addr learned my addr 172.21.15.1:0/1339888291 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22a7e00640 1 --2- 172.21.15.1:0/1339888291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 0x555751a973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22ad200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a94a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22a7e00640 1 -- 172.21.15.1:0/1339888291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 msgr2=0x555751a94a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22a7e00640 1 --2- 172.21.15.1:0/1339888291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a94a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22a7e00640 1 -- 172.21.15.1:0/1339888291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a89dd0 msgr2=0x555751a8a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22a7e00640 1 --2- 172.21.15.1:0/1339888291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a89dd0 0x555751a8a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22a7e00640 1 -- 172.21.15.1:0/1339888291 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557519ad940 con 0x555751a94fb0 2024-04-26T21:38:59.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.295+0000 7f22ad200640 1 --2- 172.21.15.1:0/1339888291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a94a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.296+0000 7f22a7e00640 1 --2- 172.21.15.1:0/1339888291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 0x555751a973a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2294009bf0 tx=0x7f2294011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47e85539acc65f7a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.296+0000 7f22a5600640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22900a3d10 0x7f22900a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.296+0000 7f229a000640 1 -- 172.21.15.1:0/1339888291 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294004050 con 0x555751a94fb0 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.296+0000 7f22a5600640 1 -- 172.21.15.1:0/3875592427 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7f229c0f0f20 con 0x7f22a8103ba0 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.296+0000 7f229a000640 1 -- 172.21.15.1:0/1339888291 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22940054d0 con 0x555751a94fb0 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.296+0000 7f229a000640 1 -- 172.21.15.1:0/1339888291 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294004d10 con 0x555751a94fb0 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.296+0000 7f22adc00640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22900a3d10 0x7f22900a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.297+0000 7f22b2a66740 1 -- 172.21.15.1:0/1339888291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 msgr2=0x555751a973a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.297+0000 7f22b2a66740 1 --2- 172.21.15.1:0/1339888291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 0x555751a973a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2294009bf0 tx=0x7f2294011810 comp rx=0 tx=0).stop 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.297+0000 7f22adc00640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22900a3d10 0x7f22900a61d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f22a0011630 tx=0x7f22a0009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.297+0000 7f22b2a66740 1 -- 172.21.15.1:0/1339888291 shutdown_connections 2024-04-26T21:38:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.297+0000 7f22b2a66740 1 --2- 172.21.15.1:0/1339888291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 0x555751a973a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.297+0000 7f22b2a66740 1 --2- 172.21.15.1:0/1339888291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a94a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.297+0000 7f22b2a66740 1 --2- 172.21.15.1:0/1339888291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a89dd0 0x555751a8a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.297+0000 7f22b2a66740 1 -- 172.21.15.1:0/1339888291 >> 172.21.15.1:0/1339888291 conn(0x555751a862e0 msgr2=0x555751a866c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.297+0000 7f22b2a66740 1 -- 172.21.15.1:0/1339888291 shutdown_connections 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.297+0000 7f22b2a66740 1 -- 172.21.15.1:0/1339888291 wait complete. 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.298+0000 7f22b2a66740 1 -- start start 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.298+0000 7f22b2a66740 1 Processor -- start 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 0x555751a91ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 0x555751a92430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a94fb0 0x555751a92970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22b2a66740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x555751a97be0 con 0x555751a8a6f0 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22b2a66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x555751976c60 con 0x555751a94fb0 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22b2a66740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555751a97d60 con 0x555751a89dd0 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22b2a66740 10 client.0 _ll_register_callbacks cb 0x5557519a2e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22ad200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 0x555751a91ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22a7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 0x555751a92430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22ad200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 0x555751a91ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:40456/0 (socket says 172.21.15.1:40456) 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22ad200640 1 -- 172.21.15.1:0/2299530038 learned_addr learned my addr 172.21.15.1:0/2299530038 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.299+0000 7f22adc00640 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a94fb0 0x555751a92970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.300+0000 7f22ad200640 1 -- 172.21.15.1:0/2299530038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a94fb0 msgr2=0x555751a92970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.300+0000 7f22ad200640 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a94fb0 0x555751a92970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.300+0000 7f22ad200640 1 -- 172.21.15.1:0/2299530038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 msgr2=0x555751a92430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.300+0000 7f22ad200640 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 0x555751a92430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.300+0000 7f22ad200640 1 -- 172.21.15.1:0/2299530038 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555751985510 con 0x555751a89dd0 2024-04-26T21:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.300+0000 7f22a7e00640 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 0x555751a92430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.300+0000 7f22ad200640 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 0x555751a91ef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f229c0bcaa0 tx=0x7f229c0b8e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fc1ade0a9f88365 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.300+0000 7f228ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:38:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.300+0000 7f228ea00640 1 -- 172.21.15.1:0/2299530038 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c049070 con 0x555751a89dd0 2024-04-26T21:38:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.301+0000 7f22b2a66740 1 -- 172.21.15.1:0/2299530038 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22940054d0 con 0x555751a89dd0 2024-04-26T21:38:59.305 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.301+0000 7f228ea00640 1 -- 172.21.15.1:0/2299530038 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7f229c0b9180 con 0x555751a89dd0 2024-04-26T21:38:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.301+0000 7f228ea00640 1 -- 172.21.15.1:0/2299530038 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f229c0dd0e0 con 0x555751a89dd0 2024-04-26T21:38:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.301+0000 7f22b2a66740 10 client.26185 fetch_fsmap learned FSMap version 36 2024-04-26T21:38:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.301+0000 7f22b2a66740 1 -- 172.21.15.1:0/2299530038 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555751701210 con 0x555751a89dd0 2024-04-26T21:38:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.301+0000 7f228ea00640 1 -- 172.21.15.1:0/2299530038 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22940054d0 con 0x555751a89dd0 2024-04-26T21:38:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.301+0000 7f228ea00640 1 -- 172.21.15.1:0/2299530038 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f229c03b460 con 0x555751a89dd0 2024-04-26T21:38:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.301+0000 7f22b2a66740 10 client.26185 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:38:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.301+0000 7f22b2a66740 10 client.26185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:38:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.301+0000 7f22b2a66740 20 client.26185 populate_metadata read hostname 'smithi001' 2024-04-26T21:38:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.302+0000 7f22b2a66740 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555751aa4c60 0x555751aa7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.307 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.302+0000 7f22b2a66740 4 client.26185 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.302+0000 7f22b2a66740 1 -- 172.21.15.1:0/2299530038 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557519fcf90 con 0x555751aa4c60 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.302+0000 7f22a7e00640 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555751aa4c60 0x555751aa7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.302+0000 7f22a7e00640 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555751aa4c60 0x555751aa7070 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=973aede85ec92560 server_cookie=124875b65d2ebea9 in_seq=0 out_seq=0 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.302+0000 7f228ea00640 10 client.26185 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.303+0000 7f228ea00640 1 -- 172.21.15.1:0/2299530038 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5557519fcf90 con 0x555751aa4c60 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.303+0000 7f228ea00640 10 client.26185 handle_command_reply: tid=0 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.303+0000 7f22b2a66740 1 client.26185 shutdown 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 1 -- 172.21.15.1:0/2299530038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 msgr2=0x555751a91ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 0x555751a91ef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f229c0bcaa0 tx=0x7f229c0b8e70 comp rx=0 tx=0).stop 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 1 -- 172.21.15.1:0/2299530038 shutdown_connections 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555751aa4c60 0x555751aa7070 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a94fb0 0x555751a92970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 0x555751a92430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.308 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 1 --2- 172.21.15.1:0/2299530038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 0x555751a91ef0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 1 -- 172.21.15.1:0/2299530038 >> 172.21.15.1:0/2299530038 conn(0x555751a862e0 msgr2=0x555751a8c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f228ea00640 0 client.26185 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 1 -- 172.21.15.1:0/2299530038 shutdown_connections 2024-04-26T21:38:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 1 -- 172.21.15.1:0/2299530038 wait complete. 2024-04-26T21:38:59.309 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.304+0000 7f22b2a66740 20 client.26185 trim_cache size 0 max 16384 2024-04-26T21:38:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.324+0000 7f22b2a66740 1 Processor -- start 2024-04-26T21:38:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22b2a66740 1 -- start start 2024-04-26T21:38:59.325 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 0x555751aa0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a8c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a94fb0 0x555751aac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22b2a66740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81b4970 con 0x555751a94fb0 2024-04-26T21:38:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22b2a66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81b2240 con 0x555751a8a6f0 2024-04-26T21:38:59.326 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22b2a66740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81b00b0 con 0x555751a89dd0 2024-04-26T21:38:59.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22a7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a8c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22a7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a8c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39372/0 (socket says 172.21.15.1:39372) 2024-04-26T21:38:59.334 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22a7e00640 1 -- 172.21.15.1:0/1608328592 learned_addr learned my addr 172.21.15.1:0/1608328592 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22adc00640 1 --2- 172.21.15.1:0/1608328592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a94fb0 0x555751aac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.325+0000 7f22ad200640 1 --2- 172.21.15.1:0/1608328592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 0x555751aa0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.326+0000 7f22a7e00640 1 -- 172.21.15.1:0/1608328592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 msgr2=0x555751aa0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.326+0000 7f22a7e00640 1 --2- 172.21.15.1:0/1608328592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 0x555751aa0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.326+0000 7f22a7e00640 1 -- 172.21.15.1:0/1608328592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a94fb0 msgr2=0x555751aac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.326+0000 7f22a7e00640 1 --2- 172.21.15.1:0/1608328592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a94fb0 0x555751aac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.326+0000 7f22a7e00640 1 -- 172.21.15.1:0/1608328592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557517cd620 con 0x555751a8a6f0 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.326+0000 7f22a7e00640 1 --2- 172.21.15.1:0/1608328592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a8c790 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2294011d10 tx=0x7f2294009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4fbd1bd6d22043c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.326+0000 7f2299600640 1 -- 172.21.15.1:0/1608328592 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294009da0 con 0x555751a8a6f0 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.327+0000 7f2299600640 1 -- 172.21.15.1:0/1608328592 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2294005b90 con 0x555751a8a6f0 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.327+0000 7f2299600640 1 -- 172.21.15.1:0/1608328592 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294003850 con 0x555751a8a6f0 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.327+0000 7f22b2a66740 1 -- 172.21.15.1:0/1608328592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 msgr2=0x555751a8c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:59.335 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.327+0000 7f22b2a66740 1 --2- 172.21.15.1:0/1608328592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a8c790 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2294011d10 tx=0x7f2294009680 comp rx=0 tx=0).stop 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.327+0000 7f22b2a66740 1 -- 172.21.15.1:0/1608328592 shutdown_connections 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.327+0000 7f22b2a66740 1 --2- 172.21.15.1:0/1608328592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a94fb0 0x555751aac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.327+0000 7f22b2a66740 1 --2- 172.21.15.1:0/1608328592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a8a6f0 0x555751a8c790 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.327+0000 7f22b2a66740 1 --2- 172.21.15.1:0/1608328592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a89dd0 0x555751aa0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.327+0000 7f22b2a66740 1 -- 172.21.15.1:0/1608328592 >> 172.21.15.1:0/1608328592 conn(0x555751a862e0 msgr2=0x555751a92740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.328+0000 7f22b2a66740 1 -- 172.21.15.1:0/1608328592 shutdown_connections 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.328+0000 7f22b2a66740 1 -- 172.21.15.1:0/1608328592 wait complete. 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.328+0000 7f22b2a66740 1 -- start start 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.328+0000 7f22b2a66740 1 Processor -- start 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a89dd0 0x555751a8eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 0x555751a8f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22b2a66740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 0x555751a8f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.336 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22b2a66740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81b0700 con 0x555751a8a6f0 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22b2a66740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81b1b70 con 0x555751a89dd0 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22b2a66740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81b0dd0 con 0x555751a94fb0 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22b2a66740 10 client.0 _ll_register_callbacks cb 0x7f229c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22ad200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a89dd0 0x555751a8eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22ad200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a89dd0 0x555751a8eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39386/0 (socket says 172.21.15.1:39386) 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22ad200640 1 -- 172.21.15.1:0/3477987878 learned_addr learned my addr 172.21.15.1:0/3477987878 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22a7e00640 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 0x555751a8f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22adc00640 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 0x555751a8f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22ad200640 1 -- 172.21.15.1:0/3477987878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 msgr2=0x555751a8f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22ad200640 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 0x555751a8f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22ad200640 1 -- 172.21.15.1:0/3477987878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 msgr2=0x555751a8f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.329+0000 7f22ad200640 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 0x555751a8f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.330+0000 7f22ad200640 1 -- 172.21.15.1:0/3477987878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557519a9cb0 con 0x555751a89dd0 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.330+0000 7f22adc00640 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 0x555751a8f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.330+0000 7f22ad200640 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a89dd0 0x555751a8eb60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f229c0badc0 tx=0x7f229c005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6f824953e76911f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.330+0000 7f228ea00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.330+0000 7f228ea00640 1 -- 172.21.15.1:0/3477987878 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c049040 con 0x555751a89dd0 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.330+0000 7f228ea00640 1 -- 172.21.15.1:0/3477987878 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9642+0+0 (secure 0 0 0) 0x7f229c0bc030 con 0x555751a89dd0 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.330+0000 7f22b2a66740 1 -- 172.21.15.1:0/3477987878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2294005b90 con 0x555751a89dd0 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.330+0000 7f228ea00640 1 -- 172.21.15.1:0/3477987878 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f229c0c56d0 con 0x555751a89dd0 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.330+0000 7f22b2a66740 10 client.26216 fetch_fsmap learned FSMap version 36 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.330+0000 7f22b2a66740 1 -- 172.21.15.1:0/3477987878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555751b0b260 con 0x555751a89dd0 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.331+0000 7f228ea00640 1 -- 172.21.15.1:0/3477987878 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2294005b90 con 0x555751a89dd0 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.331+0000 7f228ea00640 1 -- 172.21.15.1:0/3477987878 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f229c0f40d0 con 0x555751a89dd0 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.331+0000 7f22b2a66740 10 client.26216 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.331+0000 7f22b2a66740 10 client.26216 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:38:59.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.331+0000 7f22b2a66740 20 client.26216 populate_metadata read hostname 'smithi001' 2024-04-26T21:38:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.331+0000 7f22b2a66740 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555751b0b7f0 0x555751b0dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:38:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.331+0000 7f22b2a66740 4 client.26216 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:38:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.331+0000 7f22b2a66740 1 -- 172.21.15.1:0/3477987878 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5557519f6e00 con 0x555751b0b7f0 2024-04-26T21:38:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.331+0000 7f22a7e00640 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555751b0b7f0 0x555751b0dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:38:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.332+0000 7f22a7e00640 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555751b0b7f0 0x555751b0dbe0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f48cfea6d714de42 server_cookie=82b630660370811b in_seq=0 out_seq=0 2024-04-26T21:38:59.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.332+0000 7f228ea00640 10 client.26216 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.408+0000 7f228ea00640 1 -- 172.21.15.1:0/3477987878 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5557519f6e00 con 0x555751b0b7f0 2024-04-26T21:38:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.408+0000 7f228ea00640 10 client.26216 handle_command_reply: tid=0 2024-04-26T21:38:59.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.408+0000 7f22b2a66740 1 client.26216 shutdown 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.408+0000 7f22b2a66740 1 -- 172.21.15.1:0/3477987878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a89dd0 msgr2=0x555751a8eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.408+0000 7f22b2a66740 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a89dd0 0x555751a8eb60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f229c0badc0 tx=0x7f229c005c90 comp rx=0 tx=0).stop 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.409+0000 7f22b2a66740 1 -- 172.21.15.1:0/3477987878 shutdown_connections 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.409+0000 7f22b2a66740 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x555751b0b7f0 0x555751b0dbe0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.409+0000 7f22b2a66740 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555751a94fb0 0x555751a8f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.409+0000 7f22b2a66740 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x555751a8a6f0 0x555751a8f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.409+0000 7f22b2a66740 1 --2- 172.21.15.1:0/3477987878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555751a89dd0 0x555751a8eb60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.409+0000 7f228ea00640 0 client.26216 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.409+0000 7f22b2a66740 1 -- 172.21.15.1:0/3477987878 >> 172.21.15.1:0/3477987878 conn(0x555751a862e0 msgr2=0x555751a91eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.409+0000 7f22b2a66740 1 -- 172.21.15.1:0/3477987878 shutdown_connections 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.409+0000 7f22b2a66740 1 -- 172.21.15.1:0/3477987878 wait complete. 2024-04-26T21:38:59.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.409+0000 7f22b2a66740 20 client.26216 trim_cache size 0 max 16384 2024-04-26T21:38:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:38:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:38:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:38:59.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 -- 172.21.15.1:0/3875592427 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22900a3d10 msgr2=0x7f22900a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22900a3d10 0x7f22900a61d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f22a0011630 tx=0x7f22a0009500 comp rx=0 tx=0).stop 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 -- 172.21.15.1:0/3875592427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22a8103ba0 msgr2=0x7f22a819b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22a8103ba0 0x7f22a819b310 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f229c005ab0 tx=0x7f229c003c10 comp rx=0 tx=0).stop 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 -- 172.21.15.1:0/3875592427 shutdown_connections 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22900a3d10 0x7f22900a61d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a810cf70 0x7f22a819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22a81044c0 0x7f22a819b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 --2- 172.21.15.1:0/3875592427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22a8103ba0 0x7f22a819b310 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 -- 172.21.15.1:0/3875592427 >> 172.21.15.1:0/3875592427 conn(0x7f22a8078950 msgr2=0x7f22a8108df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:38:59.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 -- 172.21.15.1:0/3875592427 shutdown_connections 2024-04-26T21:38:59.413 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:38:59.410+0000 7f22afc00640 1 -- 172.21.15.1:0/3875592427 wait complete. 2024-04-26T21:38:59.580 INFO:tasks.workunit.client.0.smithi001.stdout: 1 312527 4.48 MB/sec execute 297 sec latency 62.112 ms 2024-04-26T21:39:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:00 smithi186 ceph-mon[32991]: pgmap v267: 129 pgs: 129 active+clean; 189 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-04-26T21:39:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:00 smithi001 ceph-mon[28054]: pgmap v267: 129 pgs: 129 active+clean; 189 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-04-26T21:39:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:00 smithi038 ceph-mon[32724]: pgmap v267: 129 pgs: 129 active+clean; 189 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-04-26T21:39:00.580 INFO:tasks.workunit.client.0.smithi001.stdout: 1 312946 4.47 MB/sec execute 298 sec latency 33.619 ms 2024-04-26T21:39:01.580 INFO:tasks.workunit.client.0.smithi001.stdout: 1 313187 4.46 MB/sec execute 299 sec latency 86.127 ms 2024-04-26T21:39:02.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:02 smithi186 ceph-mon[32991]: pgmap v268: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 234 op/s 2024-04-26T21:39:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:02 smithi001 ceph-mon[28054]: pgmap v268: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 234 op/s 2024-04-26T21:39:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:02 smithi038 ceph-mon[32724]: pgmap v268: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 234 op/s 2024-04-26T21:39:02.580 INFO:tasks.workunit.client.0.smithi001.stdout: 1 314227 4.45 MB/sec execute 300 sec latency 100.688 ms 2024-04-26T21:39:03.580 INFO:tasks.workunit.client.0.smithi001.stdout: 1 314922 4.46 MB/sec execute 301 sec latency 111.787 ms 2024-04-26T21:39:04.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:04 smithi186 ceph-mon[32991]: pgmap v269: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.4 MiB/s wr, 140 op/s 2024-04-26T21:39:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:04 smithi001 ceph-mon[28054]: pgmap v269: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.4 MiB/s wr, 140 op/s 2024-04-26T21:39:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:04 smithi038 ceph-mon[32724]: pgmap v269: 129 pgs: 129 active+clean; 194 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.4 MiB/s wr, 140 op/s 2024-04-26T21:39:04.580 INFO:tasks.workunit.client.0.smithi001.stdout: 1 315980 4.47 MB/sec execute 302 sec latency 21.651 ms 2024-04-26T21:39:05.580 INFO:tasks.workunit.client.0.smithi001.stdout: 1 316252 4.45 MB/sec execute 303 sec latency 24.783 ms 2024-04-26T21:39:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:06 smithi001 ceph-mon[28054]: pgmap v270: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-04-26T21:39:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:06 smithi038 ceph-mon[32724]: pgmap v270: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-04-26T21:39:06.580 INFO:tasks.workunit.client.0.smithi001.stdout: 1 317234 4.46 MB/sec execute 304 sec latency 24.147 ms 2024-04-26T21:39:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:06 smithi186 ceph-mon[32991]: pgmap v270: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-04-26T21:39:07.580 INFO:tasks.workunit.client.0.smithi001.stdout: 1 318477 4.46 MB/sec execute 305 sec latency 39.693 ms 2024-04-26T21:39:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:08 smithi001 ceph-mon[28054]: pgmap v271: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.3 MiB/s wr, 172 op/s 2024-04-26T21:39:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:08 smithi001 ceph-mon[28054]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T21:39:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:08 smithi038 ceph-mon[32724]: pgmap v271: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.3 MiB/s wr, 172 op/s 2024-04-26T21:39:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:08 smithi038 ceph-mon[32724]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T21:39:08.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 319518 4.47 MB/sec execute 306 sec latency 23.328 ms 2024-04-26T21:39:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:08 smithi186 ceph-mon[32991]: pgmap v271: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.3 MiB/s wr, 172 op/s 2024-04-26T21:39:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:08 smithi186 ceph-mon[32991]: osdmap e101: 12 total, 12 up, 12 in 2024-04-26T21:39:09.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:09 smithi001 ceph-mon[28054]: pgmap v273: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.6 MiB/s wr, 206 op/s 2024-04-26T21:39:09.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:09 smithi038 ceph-mon[32724]: pgmap v273: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.6 MiB/s wr, 206 op/s 2024-04-26T21:39:09.581 INFO:tasks.workunit.client.0.smithi001.stdout: 1 319991 4.46 MB/sec execute 307 sec latency 44.877 ms 2024-04-26T21:39:09.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:09 smithi186 ceph-mon[32991]: pgmap v273: 129 pgs: 129 active+clean; 195 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 1.6 MiB/s wr, 206 op/s 2024-04-26T21:39:10.581 INFO:tasks.workunit.client.0.smithi001.stdout: 1 321138 4.46 MB/sec execute 308 sec latency 22.917 ms 2024-04-26T21:39:11.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 322413 4.47 MB/sec execute 309 sec latency 37.175 ms 2024-04-26T21:39:12.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:11 smithi186 ceph-mon[32991]: pgmap v274: 129 pgs: 129 active+clean; 198 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-26T21:39:12.271 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:11 smithi001 ceph-mon[28054]: pgmap v274: 129 pgs: 129 active+clean; 198 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-26T21:39:12.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:11 smithi038 ceph-mon[32724]: pgmap v274: 129 pgs: 129 active+clean; 198 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-26T21:39:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.572+0000 7fa2dcc00640 1 -- 172.21.15.1:0/1545880662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2d8107820 msgr2=0x7fa2d8107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.572+0000 7fa2dcc00640 1 --2- 172.21.15.1:0/1545880662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2d8107820 0x7fa2d8107c00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009900 tx=0x7fa2c802a3f0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.572+0000 7fa2dcc00640 1 -- 172.21.15.1:0/1545880662 shutdown_connections 2024-04-26T21:39:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.572+0000 7fa2dcc00640 1 --2- 172.21.15.1:0/1545880662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d8101810 0x7fa2d8101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.572+0000 7fa2dcc00640 1 --2- 172.21.15.1:0/1545880662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2d81021d0 0x7fa2d810e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.572+0000 7fa2dcc00640 1 --2- 172.21.15.1:0/1545880662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2d8107820 0x7fa2d8107c00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.572+0000 7fa2dcc00640 1 -- 172.21.15.1:0/1545880662 >> 172.21.15.1:0/1545880662 conn(0x7fa2d8078950 msgr2=0x7fa2d8078d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.573+0000 7fa2dcc00640 1 -- 172.21.15.1:0/1545880662 shutdown_connections 2024-04-26T21:39:12.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.573+0000 7fa2dcc00640 1 -- 172.21.15.1:0/1545880662 wait complete. 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.574+0000 7fa2dcc00640 1 Processor -- start 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.574+0000 7fa2dcc00640 1 -- start start 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.574+0000 7fa2dcc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d8101810 0x7fa2d80fe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.574+0000 7fa2dcc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2d81021d0 0x7fa2d80fed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.574+0000 7fa2dcc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2d8107820 0x7fa2d80ff250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.574+0000 7fa2dcc00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8111b30 con 0x7fa2d81021d0 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.574+0000 7fa2dcc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81119b0 con 0x7fa2d8107820 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.574+0000 7fa2dcc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d8111cb0 con 0x7fa2d8101810 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.575+0000 7fa2d6800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2d8107820 0x7fa2d80ff250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.575+0000 7fa2d6800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2d8107820 0x7fa2d80ff250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53402/0 (socket says 172.21.15.1:53402) 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.575+0000 7fa2d6800640 1 -- 172.21.15.1:0/2950476575 learned_addr learned my addr 172.21.15.1:0/2950476575 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.575+0000 7fa2d5400640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2d81021d0 0x7fa2d80fed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.575+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d8101810 0x7fa2d80fe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.575+0000 7fa2d6800640 1 -- 172.21.15.1:0/2950476575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d8101810 msgr2=0x7fa2d80fe7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.575+0000 7fa2d6800640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d8101810 0x7fa2d80fe7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.575+0000 7fa2d6800640 1 -- 172.21.15.1:0/2950476575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2d81021d0 msgr2=0x7fa2d80fed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.575+0000 7fa2d6800640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2d81021d0 0x7fa2d80fed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.575+0000 7fa2d6800640 1 -- 172.21.15.1:0/2950476575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d81a76e0 con 0x7fa2d8107820 2024-04-26T21:39:12.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.576+0000 7fa2d6800640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2d8107820 0x7fa2d80ff250 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009fd0 tx=0x7fa2d000b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.576+0000 7fa2cea00640 1 -- 172.21.15.1:0/2950476575 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0017070 con 0x7fa2d8107820 2024-04-26T21:39:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.576+0000 7fa2cea00640 1 -- 172.21.15.1:0/2950476575 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2d0003e20 con 0x7fa2d8107820 2024-04-26T21:39:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.576+0000 7fa2cea00640 1 -- 172.21.15.1:0/2950476575 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0010460 con 0x7fa2d8107820 2024-04-26T21:39:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.576+0000 7fa2dcc00640 1 -- 172.21.15.1:0/2950476575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d81a79d0 con 0x7fa2d8107820 2024-04-26T21:39:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.576+0000 7fa2dcc00640 1 -- 172.21.15.1:0/2950476575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d81a7ed0 con 0x7fa2d8107820 2024-04-26T21:39:12.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.581+0000 7fa2cea00640 1 -- 172.21.15.1:0/2950476575 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa2d0003970 con 0x7fa2d8107820 2024-04-26T21:39:12.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.581+0000 7fa2dfa0b740 1 Processor -- start 2024-04-26T21:39:12.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 323155 4.48 MB/sec execute 310 sec latency 27.853 ms 2024-04-26T21:39:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.581+0000 7fa2dfa0b740 1 -- start start 2024-04-26T21:39:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 0x5593af3471b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af3476f0 0x5593af351a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af351fb0 0x5593af3543a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5593af2ac380 con 0x5593af3476f0 2024-04-26T21:39:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5593af248c30 con 0x5593af351fb0 2024-04-26T21:39:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5593af2a8340 con 0x5593af346dd0 2024-04-26T21:39:12.584 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2d5400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af351fb0 0x5593af3543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2d5400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af351fb0 0x5593af3543a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53406/0 (socket says 172.21.15.1:53406) 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2d5400640 1 -- 172.21.15.1:0/1526355526 learned_addr learned my addr 172.21.15.1:0/1526355526 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/1526355526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af3476f0 0x5593af351a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.582+0000 7fa2d5400640 1 --2- 172.21.15.1:0/1526355526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 0x5593af3471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2d5400640 1 -- 172.21.15.1:0/1526355526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af351fb0 msgr2=0x5593af3543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2d5400640 1 --2- 172.21.15.1:0/1526355526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af351fb0 0x5593af3543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2d5400640 1 -- 172.21.15.1:0/1526355526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af3476f0 msgr2=0x5593af351a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2d5400640 1 --2- 172.21.15.1:0/1526355526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af3476f0 0x5593af351a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2d5400640 1 -- 172.21.15.1:0/1526355526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593af26a940 con 0x5593af346dd0 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2d5400640 1 --2- 172.21.15.1:0/1526355526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 0x5593af3471b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa2c40117e0 tx=0x7fa2c4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a00b7c6c23ffeca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2cea00640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa2bc0a3d90 0x7fa2bc0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2c3000640 1 -- 172.21.15.1:0/1526355526 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4011e20 con 0x5593af346dd0 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2c3000640 1 -- 172.21.15.1:0/1526355526 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c400f7c0 con 0x5593af346dd0 2024-04-26T21:39:12.585 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2c3000640 1 -- 172.21.15.1:0/1526355526 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c40037f0 con 0x5593af346dd0 2024-04-26T21:39:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.583+0000 7fa2cea00640 1 -- 172.21.15.1:0/2950476575 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9730+0+0 (secure 0 0 0) 0x7fa2d00d08d0 con 0x7fa2d8107820 2024-04-26T21:39:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.584+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa2bc0a3d90 0x7fa2bc0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.584+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/1526355526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 msgr2=0x5593af3471b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.584+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/1526355526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 0x5593af3471b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa2c40117e0 tx=0x7fa2c4005f90 comp rx=0 tx=0).stop 2024-04-26T21:39:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.585+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa2bc0a3d90 0x7fa2bc0a6250 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8000c00 tx=0x7fa2c8005e80 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.585+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/1526355526 shutdown_connections 2024-04-26T21:39:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.585+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/1526355526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af351fb0 0x5593af3543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.585+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/1526355526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af3476f0 0x5593af351a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.585+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/1526355526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 0x5593af3471b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.585+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/1526355526 >> 172.21.15.1:0/1526355526 conn(0x5593af3432e0 msgr2=0x5593af3436c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:12.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.585+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/1526355526 shutdown_connections 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.585+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/1526355526 wait complete. 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.585+0000 7fa2dfa0b740 1 -- start start 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.586+0000 7fa2dfa0b740 1 Processor -- start 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.586+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af346dd0 0x5593af34ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.586+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 0x5593af34f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.586+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af351fb0 0x5593af34f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.586+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5593af354be0 con 0x5593af346dd0 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.586+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5593af233c60 con 0x5593af3476f0 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.586+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5593af354d60 con 0x5593af351fb0 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.586+0000 7fa2dfa0b740 10 client.0 _ll_register_callbacks cb 0x5593af25fe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d5e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af346dd0 0x5593af34ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d5e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af346dd0 0x5593af34ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50870/0 (socket says 172.21.15.1:50870) 2024-04-26T21:39:12.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d5e00640 1 -- 172.21.15.1:0/3737874501 learned_addr learned my addr 172.21.15.1:0/3737874501 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d6800640 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af351fb0 0x5593af34f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d5400640 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 0x5593af34f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d6800640 1 -- 172.21.15.1:0/3737874501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 msgr2=0x5593af34f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d6800640 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 0x5593af34f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d6800640 1 -- 172.21.15.1:0/3737874501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af346dd0 msgr2=0x5593af34ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d6800640 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af346dd0 0x5593af34ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d6800640 1 -- 172.21.15.1:0/3737874501 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593af242510 con 0x5593af351fb0 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af346dd0 0x5593af34ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2d6800640 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af351fb0 0x5593af34f990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa2d00cdb60 tx=0x7fa2d0003c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b64f13bd2a02f91 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.587+0000 7fa2b7400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.588+0000 7fa2b7400640 1 -- 172.21.15.1:0/3737874501 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0020070 con 0x5593af351fb0 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.588+0000 7fa2b7400640 1 -- 172.21.15.1:0/3737874501 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9730+0+0 (secure 0 0 0) 0x7fa2d0098ad0 con 0x5593af351fb0 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.588+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/3737874501 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c400f7c0 con 0x5593af351fb0 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.588+0000 7fa2b7400640 1 -- 172.21.15.1:0/3737874501 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa2d00a0760 con 0x5593af351fb0 2024-04-26T21:39:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.588+0000 7fa2dfa0b740 10 client.26203 fetch_fsmap learned FSMap version 36 2024-04-26T21:39:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.588+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/3737874501 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5593aefbe210 con 0x5593af351fb0 2024-04-26T21:39:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.588+0000 7fa2b7400640 1 -- 172.21.15.1:0/3737874501 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c400f7c0 con 0x5593af351fb0 2024-04-26T21:39:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.588+0000 7fa2b7400640 1 -- 172.21.15.1:0/3737874501 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fa2d009b9b0 con 0x5593af351fb0 2024-04-26T21:39:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.589+0000 7fa2dfa0b740 10 client.26203 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:39:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.589+0000 7fa2dfa0b740 10 client.26203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:39:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.589+0000 7fa2dfa0b740 20 client.26203 populate_metadata read hostname 'smithi001' 2024-04-26T21:39:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.589+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5593af361d10 0x5593af364100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.589+0000 7fa2dfa0b740 4 client.26203 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:39:12.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.589+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/3737874501 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593af2b9f90 con 0x5593af361d10 2024-04-26T21:39:12.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.589+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5593af361d10 0x5593af364100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.590+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5593af361d10 0x5593af364100 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9394b461414c6b9c server_cookie=acf50ee9e57acc3e in_seq=0 out_seq=0 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.590+0000 7fa2b7400640 10 client.26203 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.590+0000 7fa2b7400640 1 -- 172.21.15.1:0/3737874501 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5593af2b9f90 con 0x5593af361d10 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.590+0000 7fa2b7400640 10 client.26203 handle_command_reply: tid=0 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.590+0000 7fa2dfa0b740 1 client.26203 shutdown 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/3737874501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af351fb0 msgr2=0x5593af34f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af351fb0 0x5593af34f990 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa2d00cdb60 tx=0x7fa2d0003c60 comp rx=0 tx=0).stop 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/3737874501 shutdown_connections 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5593af361d10 0x5593af364100 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af351fb0 0x5593af34f990 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 0x5593af34f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/3737874501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af346dd0 0x5593af34ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2b7400640 0 client.26203 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:12.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/3737874501 >> 172.21.15.1:0/3737874501 conn(0x5593af3432e0 msgr2=0x5593af349060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/3737874501 shutdown_connections 2024-04-26T21:39:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/3737874501 wait complete. 2024-04-26T21:39:12.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.591+0000 7fa2dfa0b740 20 client.26203 trim_cache size 0 max 16384 2024-04-26T21:39:12.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.610+0000 7fa2dfa0b740 1 Processor -- start 2024-04-26T21:39:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.610+0000 7fa2dfa0b740 1 -- start start 2024-04-26T21:39:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.610+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 0x5593af35daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.610+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 0x5593af3497e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.610+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af351fb0 0x5593af350d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.610+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b4af0 con 0x5593af351fb0 2024-04-26T21:39:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.610+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b23c0 con 0x5593af3476f0 2024-04-26T21:39:12.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.610+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b0230 con 0x5593af346dd0 2024-04-26T21:39:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 0x5593af3497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 0x5593af35daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d6800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af351fb0 0x5593af350d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 0x5593af3497e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53424/0 (socket says 172.21.15.1:53424) 2024-04-26T21:39:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5400640 1 -- 172.21.15.1:0/780539002 learned_addr learned my addr 172.21.15.1:0/780539002 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5400640 1 -- 172.21.15.1:0/780539002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 msgr2=0x5593af35daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5400640 1 --2- 172.21.15.1:0/780539002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 0x5593af35daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5400640 1 -- 172.21.15.1:0/780539002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af351fb0 msgr2=0x5593af350d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5400640 1 --2- 172.21.15.1:0/780539002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af351fb0 0x5593af350d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5400640 1 -- 172.21.15.1:0/780539002 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593af347520 con 0x5593af3476f0 2024-04-26T21:39:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/780539002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 0x5593af35daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:39:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.611+0000 7fa2d5400640 1 --2- 172.21.15.1:0/780539002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 0x5593af3497e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4009e40 tx=0x7fa2c401a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37b508e1bde3845 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.612+0000 7fa2c2600640 1 -- 172.21.15.1:0/780539002 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c40052b0 con 0x5593af3476f0 2024-04-26T21:39:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.612+0000 7fa2c2600640 1 -- 172.21.15.1:0/780539002 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c4009e70 con 0x5593af3476f0 2024-04-26T21:39:12.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.612+0000 7fa2c2600640 1 -- 172.21.15.1:0/780539002 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c4003850 con 0x5593af3476f0 2024-04-26T21:39:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.612+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/780539002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 msgr2=0x5593af3497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.612+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/780539002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 0x5593af3497e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4009e40 tx=0x7fa2c401a040 comp rx=0 tx=0).stop 2024-04-26T21:39:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.613+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/780539002 shutdown_connections 2024-04-26T21:39:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.613+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/780539002 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af351fb0 0x5593af350d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.613+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/780539002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af3476f0 0x5593af3497e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.613+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/780539002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af346dd0 0x5593af35daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.613+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/780539002 >> 172.21.15.1:0/780539002 conn(0x5593af3432e0 msgr2=0x5593af350630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.613+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/780539002 shutdown_connections 2024-04-26T21:39:12.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.613+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/780539002 wait complete. 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.613+0000 7fa2dfa0b740 1 -- start start 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.613+0000 7fa2dfa0b740 1 Processor -- start 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af346dd0 0x5593af355920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af3476f0 0x5593af355e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2dfa0b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af351fb0 0x5593af3563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b0880 con 0x5593af351fb0 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b1cf0 con 0x5593af346dd0 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2dfa0b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d81b0f50 con 0x5593af3476f0 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2dfa0b740 10 client.0 _ll_register_callbacks cb 0x7fa2d000bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2d5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af346dd0 0x5593af355920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2d5e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af346dd0 0x5593af355920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53436/0 (socket says 172.21.15.1:53436) 2024-04-26T21:39:12.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2d5400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af3476f0 0x5593af355e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2d5e00640 1 -- 172.21.15.1:0/2273371219 learned_addr learned my addr 172.21.15.1:0/2273371219 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.614+0000 7fa2d6800640 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af351fb0 0x5593af3563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.615+0000 7fa2d5e00640 1 -- 172.21.15.1:0/2273371219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af3476f0 msgr2=0x5593af355e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.615+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af3476f0 0x5593af355e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.615+0000 7fa2d5e00640 1 -- 172.21.15.1:0/2273371219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af351fb0 msgr2=0x5593af3563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.615+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af351fb0 0x5593af3563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.615+0000 7fa2d5e00640 1 -- 172.21.15.1:0/2273371219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593af08a620 con 0x5593af346dd0 2024-04-26T21:39:12.618 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.615+0000 7fa2d5e00640 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af346dd0 0x5593af355920 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8006010 tx=0x7fa2c8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b290d6c1bbe2d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.615+0000 7fa2b7400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.615+0000 7fa2b7400640 1 -- 172.21.15.1:0/2273371219 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8035040 con 0x5593af346dd0 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.615+0000 7fa2b7400640 1 -- 172.21.15.1:0/2273371219 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9730+0+0 (secure 0 0 0) 0x7fa2c80547d0 con 0x5593af346dd0 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.615+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/2273371219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2c4009e70 con 0x5593af346dd0 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2b7400640 1 -- 172.21.15.1:0/2273371219 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fa2c80035f0 con 0x5593af346dd0 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2dfa0b740 10 client.26234 fetch_fsmap learned FSMap version 36 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/2273371219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5593af266cb0 con 0x5593af346dd0 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2b7400640 1 -- 172.21.15.1:0/2273371219 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2c4009e70 con 0x5593af346dd0 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2b7400640 1 -- 172.21.15.1:0/2273371219 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fa2c805b070 con 0x5593af346dd0 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2dfa0b740 10 client.26234 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:39:12.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2dfa0b740 10 client.26234 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:39:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2dfa0b740 20 client.26234 populate_metadata read hostname 'smithi001' 2024-04-26T21:39:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5593af3c8870 0x5593af3cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2dfa0b740 4 client.26234 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:39:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.616+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/2273371219 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5593af2b3e00 con 0x5593af3c8870 2024-04-26T21:39:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.617+0000 7fa2d5400640 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5593af3c8870 0x5593af3cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.617+0000 7fa2d5400640 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5593af3c8870 0x5593af3cac90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b0484e4df505908 server_cookie=30938ada0ab46624 in_seq=0 out_seq=0 2024-04-26T21:39:12.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.617+0000 7fa2b7400640 10 client.26234 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2b7400640 1 -- 172.21.15.1:0/2273371219 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5593af2b3e00 con 0x5593af3c8870 2024-04-26T21:39:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2b7400640 10 client.26234 handle_command_reply: tid=0 2024-04-26T21:39:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2dfa0b740 1 client.26234 shutdown 2024-04-26T21:39:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/2273371219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af346dd0 msgr2=0x5593af355920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af346dd0 0x5593af355920 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8006010 tx=0x7fa2c8009260 comp rx=0 tx=0).stop 2024-04-26T21:39:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/2273371219 shutdown_connections 2024-04-26T21:39:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5593af3c8870 0x5593af3cac90 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593af3476f0 0x5593af355e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5593af351fb0 0x5593af3563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2dfa0b740 1 --2- 172.21.15.1:0/2273371219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5593af346dd0 0x5593af355920 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/2273371219 >> 172.21.15.1:0/2273371219 conn(0x5593af3432e0 msgr2=0x5593af34f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2b7400640 0 client.26234 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.679+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/2273371219 shutdown_connections 2024-04-26T21:39:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.680+0000 7fa2dfa0b740 1 -- 172.21.15.1:0/2273371219 wait complete. 2024-04-26T21:39:12.681 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.680+0000 7fa2dfa0b740 20 client.26234 trim_cache size 0 max 16384 2024-04-26T21:39:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:39:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:39:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:39:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:39:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.680+0000 7fa2dcc00640 1 -- 172.21.15.1:0/2950476575 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa2bc0a3d90 msgr2=0x7fa2bc0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:12.682 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.680+0000 7fa2dcc00640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa2bc0a3d90 0x7fa2bc0a6250 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8000c00 tx=0x7fa2c8005e80 comp rx=0 tx=0).stop 2024-04-26T21:39:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.680+0000 7fa2dcc00640 1 -- 172.21.15.1:0/2950476575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2d8107820 msgr2=0x7fa2d80ff250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.680+0000 7fa2dcc00640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2d8107820 0x7fa2d80ff250 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa2d0009fd0 tx=0x7fa2d000b260 comp rx=0 tx=0).stop 2024-04-26T21:39:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.681+0000 7fa2dcc00640 1 -- 172.21.15.1:0/2950476575 shutdown_connections 2024-04-26T21:39:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.681+0000 7fa2dcc00640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa2bc0a3d90 0x7fa2bc0a6250 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.681+0000 7fa2dcc00640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2d8107820 0x7fa2d80ff250 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.681+0000 7fa2dcc00640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2d81021d0 0x7fa2d80fed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.681+0000 7fa2dcc00640 1 --2- 172.21.15.1:0/2950476575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa2d8101810 0x7fa2d80fe7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.681+0000 7fa2dcc00640 1 -- 172.21.15.1:0/2950476575 >> 172.21.15.1:0/2950476575 conn(0x7fa2d8078950 msgr2=0x7fa2d806a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.681+0000 7fa2dcc00640 1 -- 172.21.15.1:0/2950476575 shutdown_connections 2024-04-26T21:39:12.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:12.681+0000 7fa2dcc00640 1 -- 172.21.15.1:0/2950476575 wait complete. 2024-04-26T21:39:13.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 323773 4.47 MB/sec execute 311 sec latency 59.132 ms 2024-04-26T21:39:14.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:13 smithi186 ceph-mon[32991]: pgmap v275: 129 pgs: 129 active+clean; 198 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-26T21:39:14.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:13 smithi001 ceph-mon[28054]: pgmap v275: 129 pgs: 129 active+clean; 198 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-26T21:39:14.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:13 smithi038 ceph-mon[32724]: pgmap v275: 129 pgs: 129 active+clean; 198 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-26T21:39:14.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 325237 4.47 MB/sec execute 312 sec latency 38.080 ms 2024-04-26T21:39:15.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:14 smithi001 ceph-mon[28054]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T21:39:15.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:14 smithi038 ceph-mon[32724]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T21:39:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:14 smithi186 ceph-mon[32991]: osdmap e102: 12 total, 12 up, 12 in 2024-04-26T21:39:15.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 326426 4.49 MB/sec execute 313 sec latency 27.367 ms 2024-04-26T21:39:16.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:15 smithi001 ceph-mon[28054]: pgmap v277: 129 pgs: 129 active+clean; 200 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 233 op/s 2024-04-26T21:39:16.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:15 smithi038 ceph-mon[32724]: pgmap v277: 129 pgs: 129 active+clean; 200 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 233 op/s 2024-04-26T21:39:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:15 smithi186 ceph-mon[32991]: pgmap v277: 129 pgs: 129 active+clean; 200 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 233 op/s 2024-04-26T21:39:16.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 326793 4.48 MB/sec execute 314 sec latency 33.287 ms 2024-04-26T21:39:17.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 327445 4.47 MB/sec execute 315 sec latency 61.902 ms 2024-04-26T21:39:18.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:17 smithi001 ceph-mon[28054]: pgmap v278: 129 pgs: 129 active+clean; 200 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 194 op/s 2024-04-26T21:39:18.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:17 smithi038 ceph-mon[32724]: pgmap v278: 129 pgs: 129 active+clean; 200 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 194 op/s 2024-04-26T21:39:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:17 smithi186 ceph-mon[32991]: pgmap v278: 129 pgs: 129 active+clean; 200 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 194 op/s 2024-04-26T21:39:18.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 328897 4.48 MB/sec execute 316 sec latency 36.781 ms 2024-04-26T21:39:19.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 330107 4.50 MB/sec execute 317 sec latency 40.577 ms 2024-04-26T21:39:20.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:19 smithi001 ceph-mon[28054]: pgmap v279: 129 pgs: 129 active+clean; 200 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 187 op/s 2024-04-26T21:39:20.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:19 smithi038 ceph-mon[32724]: pgmap v279: 129 pgs: 129 active+clean; 200 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 187 op/s 2024-04-26T21:39:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:19 smithi186 ceph-mon[32991]: pgmap v279: 129 pgs: 129 active+clean; 200 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 187 op/s 2024-04-26T21:39:20.582 INFO:tasks.workunit.client.0.smithi001.stdout: 1 330547 4.48 MB/sec execute 318 sec latency 23.219 ms 2024-04-26T21:39:21.583 INFO:tasks.workunit.client.0.smithi001.stdout: 1 331789 4.49 MB/sec execute 319 sec latency 18.321 ms 2024-04-26T21:39:22.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:21 smithi001 ceph-mon[28054]: pgmap v280: 129 pgs: 129 active+clean; 204 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-04-26T21:39:22.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:21 smithi038 ceph-mon[32724]: pgmap v280: 129 pgs: 129 active+clean; 204 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-04-26T21:39:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:21 smithi186 ceph-mon[32991]: pgmap v280: 129 pgs: 129 active+clean; 204 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-04-26T21:39:22.583 INFO:tasks.workunit.client.0.smithi001.stdout: 1 333152 4.50 MB/sec execute 320 sec latency 20.483 ms 2024-04-26T21:39:23.583 INFO:tasks.workunit.client.0.smithi001.stdout: 1 333823 4.50 MB/sec execute 321 sec latency 21.632 ms 2024-04-26T21:39:24.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:23 smithi001 ceph-mon[28054]: pgmap v281: 129 pgs: 129 active+clean; 204 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-04-26T21:39:24.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:23 smithi038 ceph-mon[32724]: pgmap v281: 129 pgs: 129 active+clean; 204 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-04-26T21:39:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:23 smithi186 ceph-mon[32991]: pgmap v281: 129 pgs: 129 active+clean; 204 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 190 op/s 2024-04-26T21:39:24.583 INFO:tasks.workunit.client.0.smithi001.stdout: 1 334497 4.49 MB/sec execute 322 sec latency 27.163 ms 2024-04-26T21:39:25.583 INFO:tasks.workunit.client.0.smithi001.stdout: 1 336003 4.50 MB/sec execute 323 sec latency 22.570 ms 2024-04-26T21:39:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.832+0000 7f151a800640 1 -- 172.21.15.1:0/2452041290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f151410ac10 msgr2=0x7f151410aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.832+0000 7f151a800640 1 --2- 172.21.15.1:0/2452041290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f151410ac10 0x7f151410aff0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1504009930 tx=0x7f150402a440 comp rx=0 tx=0).stop 2024-04-26T21:39:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.832+0000 7f151a800640 1 -- 172.21.15.1:0/2452041290 shutdown_connections 2024-04-26T21:39:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.832+0000 7f151a800640 1 --2- 172.21.15.1:0/2452041290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1514104b50 0x7f1514104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.832+0000 7f151a800640 1 --2- 172.21.15.1:0/2452041290 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1514105510 0x7f15141119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.832+0000 7f151a800640 1 --2- 172.21.15.1:0/2452041290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f151410ac10 0x7f151410aff0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.832+0000 7f151a800640 1 -- 172.21.15.1:0/2452041290 >> 172.21.15.1:0/2452041290 conn(0x7f1514100620 msgr2=0x7f1514102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.832+0000 7f151a800640 1 -- 172.21.15.1:0/2452041290 shutdown_connections 2024-04-26T21:39:25.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.832+0000 7f151a800640 1 -- 172.21.15.1:0/2452041290 wait complete. 2024-04-26T21:39:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.833+0000 7f151a800640 1 Processor -- start 2024-04-26T21:39:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.833+0000 7f151a800640 1 -- start start 2024-04-26T21:39:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f151a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1514104b50 0x7f151419b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f151a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1514105510 0x7f151419b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f151a800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151410ac10 0x7f151419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f151a800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1514114e50 con 0x7f151410ac10 2024-04-26T21:39:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f151a800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1514114cd0 con 0x7f1514105510 2024-04-26T21:39:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f151a800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1514114fd0 con 0x7f1514104b50 2024-04-26T21:39:25.836 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f1513e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151410ac10 0x7f151419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f150aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1514105510 0x7f151419b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f150aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1514105510 0x7f151419b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:38308/0 (socket says 172.21.15.1:38308) 2024-04-26T21:39:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f150aa00640 1 -- 172.21.15.1:0/3648916044 learned_addr learned my addr 172.21.15.1:0/3648916044 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.834+0000 7f1513400640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1514104b50 0x7f151419b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f150aa00640 1 -- 172.21.15.1:0/3648916044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1514104b50 msgr2=0x7f151419b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f150aa00640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1514104b50 0x7f151419b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f150aa00640 1 -- 172.21.15.1:0/3648916044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151410ac10 msgr2=0x7f151419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f150aa00640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151410ac10 0x7f151419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f150aa00640 1 -- 172.21.15.1:0/3648916044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15141a0340 con 0x7f1514105510 2024-04-26T21:39:25.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f150aa00640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1514105510 0x7f151419b890 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00e9f0 tx=0x7f14fc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:25.838 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f1510c00640 1 -- 172.21.15.1:0/3648916044 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc00cd80 con 0x7f1514105510 2024-04-26T21:39:25.838 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f151a800640 1 -- 172.21.15.1:0/3648916044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15141a0630 con 0x7f1514105510 2024-04-26T21:39:25.838 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f1510c00640 1 -- 172.21.15.1:0/3648916044 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14fc0054d0 con 0x7f1514105510 2024-04-26T21:39:25.838 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f1510c00640 1 -- 172.21.15.1:0/3648916044 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc010690 con 0x7f1514105510 2024-04-26T21:39:25.838 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.835+0000 7f151a800640 1 -- 172.21.15.1:0/3648916044 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15141a7ec0 con 0x7f1514105510 2024-04-26T21:39:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.839+0000 7f1510c00640 1 -- 172.21.15.1:0/3648916044 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f14fc004e90 con 0x7f1514105510 2024-04-26T21:39:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.839+0000 7f151d799740 1 Processor -- start 2024-04-26T21:39:25.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.839+0000 7f151d799740 1 -- start start 2024-04-26T21:39:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.839+0000 7f151d799740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f99dd0 0x559f28f9a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.839+0000 7f151d799740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f9a6f0 0x559f28fa4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.839+0000 7f151d799740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28fa4fb0 0x559f28fa73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.839+0000 7f151d799740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x559f28eff380 con 0x559f28f99dd0 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.839+0000 7f151d799740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559f28e9bc30 con 0x559f28f9a6f0 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.839+0000 7f151d799740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559f28efb340 con 0x559f28fa4fb0 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f9a6f0 0x559f28fa4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f99dd0 0x559f28f9a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f9a6f0 0x559f28fa4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:38314/0 (socket says 172.21.15.1:38314) 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513400640 1 -- 172.21.15.1:0/1855322423 learned_addr learned my addr 172.21.15.1:0/1855322423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f150aa00640 1 --2- 172.21.15.1:0/1855322423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28fa4fb0 0x559f28fa73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513400640 1 -- 172.21.15.1:0/1855322423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28fa4fb0 msgr2=0x559f28fa73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513400640 1 --2- 172.21.15.1:0/1855322423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28fa4fb0 0x559f28fa73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513400640 1 -- 172.21.15.1:0/1855322423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f99dd0 msgr2=0x559f28f9a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513400640 1 --2- 172.21.15.1:0/1855322423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f99dd0 0x559f28f9a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513400640 1 -- 172.21.15.1:0/1855322423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f28ebd940 con 0x559f28f9a6f0 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513e00640 1 --2- 172.21.15.1:0/1855322423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f99dd0 0x559f28f9a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:39:25.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.840+0000 7f1513400640 1 --2- 172.21.15.1:0/1855322423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f9a6f0 0x559f28fa4a70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1504006010 tx=0x7f1504004170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf853c592a686214 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.841+0000 7f1510c00640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f14f40a3d10 0x7f14f40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.841+0000 7f1510c00640 1 -- 172.21.15.1:0/3648916044 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7f14fc0d10b0 con 0x7f1514105510 2024-04-26T21:39:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.841+0000 7f1513e00640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f14f40a3d10 0x7f14f40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.841+0000 7f1508c00640 1 -- 172.21.15.1:0/1855322423 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504035040 con 0x559f28f9a6f0 2024-04-26T21:39:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.841+0000 7f1508c00640 1 -- 172.21.15.1:0/1855322423 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15040054f0 con 0x559f28f9a6f0 2024-04-26T21:39:25.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.841+0000 7f1508c00640 1 -- 172.21.15.1:0/1855322423 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504003850 con 0x559f28f9a6f0 2024-04-26T21:39:25.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f151d799740 1 -- 172.21.15.1:0/1855322423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f9a6f0 msgr2=0x559f28fa4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:25.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f151d799740 1 --2- 172.21.15.1:0/1855322423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f9a6f0 0x559f28fa4a70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1504006010 tx=0x7f1504004170 comp rx=0 tx=0).stop 2024-04-26T21:39:25.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f1513e00640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f14f40a3d10 0x7f14f40a61d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f150c005fd0 tx=0x7f150c0096a0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:25.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f151d799740 1 -- 172.21.15.1:0/1855322423 shutdown_connections 2024-04-26T21:39:25.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f151d799740 1 --2- 172.21.15.1:0/1855322423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28fa4fb0 0x559f28fa73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f151d799740 1 --2- 172.21.15.1:0/1855322423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f9a6f0 0x559f28fa4a70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f151d799740 1 --2- 172.21.15.1:0/1855322423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f99dd0 0x559f28f9a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f151d799740 1 -- 172.21.15.1:0/1855322423 >> 172.21.15.1:0/1855322423 conn(0x559f28f962e0 msgr2=0x559f28f966c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f151d799740 1 -- 172.21.15.1:0/1855322423 shutdown_connections 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f151d799740 1 -- 172.21.15.1:0/1855322423 wait complete. 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.842+0000 7f151d799740 1 -- start start 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.843+0000 7f151d799740 1 Processor -- start 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.843+0000 7f151d799740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 0x559f28fa1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.843+0000 7f151d799740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 0x559f28fa24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.843+0000 7f151d799740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28fa4fb0 0x559f28fa2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.843+0000 7f151d799740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x559f28fa7b80 con 0x559f28fa4fb0 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.843+0000 7f151d799740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559f28e86c60 con 0x559f28f99dd0 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.843+0000 7f151d799740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559f28fa7d00 con 0x559f28f9a6f0 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.843+0000 7f151d799740 10 client.0 _ll_register_callbacks cb 0x559f28eb2e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f150aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 0x559f28fa24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f1513e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28fa4fb0 0x559f28fa2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f1513400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 0x559f28fa1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f150aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 0x559f28fa24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51354/0 (socket says 172.21.15.1:51354) 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f150aa00640 1 -- 172.21.15.1:0/2018138048 learned_addr learned my addr 172.21.15.1:0/2018138048 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f150aa00640 1 -- 172.21.15.1:0/2018138048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 msgr2=0x559f28fa1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f150aa00640 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 0x559f28fa1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f150aa00640 1 -- 172.21.15.1:0/2018138048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28fa4fb0 msgr2=0x559f28fa2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f150aa00640 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28fa4fb0 0x559f28fa2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f150aa00640 1 -- 172.21.15.1:0/2018138048 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f28e95510 con 0x559f28f9a6f0 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f150aa00640 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 0x559f28fa24e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0ce3a0 tx=0x7f14fc098ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45048012c0d7ca3d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f1501600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.844+0000 7f1501600640 1 -- 172.21.15.1:0/2018138048 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc005180 con 0x559f28f9a6f0 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f1501600640 1 -- 172.21.15.1:0/2018138048 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7f14fc0d4cd0 con 0x559f28f9a6f0 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f151d799740 1 -- 172.21.15.1:0/2018138048 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15040054f0 con 0x559f28f9a6f0 2024-04-26T21:39:25.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f1501600640 1 -- 172.21.15.1:0/2018138048 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f14fc005680 con 0x559f28f9a6f0 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f151d799740 10 client.26221 fetch_fsmap learned FSMap version 36 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f151d799740 1 -- 172.21.15.1:0/2018138048 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559f28c11210 con 0x559f28f9a6f0 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f1501600640 1 -- 172.21.15.1:0/2018138048 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15040054f0 con 0x559f28f9a6f0 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f1501600640 1 -- 172.21.15.1:0/2018138048 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f14fc0ce5c0 con 0x559f28f9a6f0 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f151d799740 10 client.26221 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f151d799740 10 client.26221 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f151d799740 20 client.26221 populate_metadata read hostname 'smithi001' 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f151d799740 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559f28fb4cf0 0x559f28fb70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f151d799740 4 client.26221 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.845+0000 7f151d799740 1 -- 172.21.15.1:0/2018138048 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f28f0cf90 con 0x559f28fb4cf0 2024-04-26T21:39:25.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.846+0000 7f1513400640 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559f28fb4cf0 0x559f28fb70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.846+0000 7f1513400640 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559f28fb4cf0 0x559f28fb70e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6480c5e0f2b7bee server_cookie=d59d3449b13666ff in_seq=0 out_seq=0 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.846+0000 7f1501600640 10 client.26221 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.847+0000 7f1501600640 1 -- 172.21.15.1:0/2018138048 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559f28f0cf90 con 0x559f28fb4cf0 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.847+0000 7f1501600640 10 client.26221 handle_command_reply: tid=0 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.847+0000 7f151d799740 1 client.26221 shutdown 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 1 -- 172.21.15.1:0/2018138048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 msgr2=0x559f28fa24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 0x559f28fa24e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0ce3a0 tx=0x7f14fc098ef0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 1 -- 172.21.15.1:0/2018138048 shutdown_connections 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559f28fb4cf0 0x559f28fb70e0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 0x559f28fa24e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28fa4fb0 0x559f28fa2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 1 --2- 172.21.15.1:0/2018138048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 0x559f28fa1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 1 -- 172.21.15.1:0/2018138048 >> 172.21.15.1:0/2018138048 conn(0x559f28f962e0 msgr2=0x559f28f9c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f1501600640 0 client.26221 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 1 -- 172.21.15.1:0/2018138048 shutdown_connections 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 1 -- 172.21.15.1:0/2018138048 wait complete. 2024-04-26T21:39:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.848+0000 7f151d799740 20 client.26221 trim_cache size 0 max 16384 2024-04-26T21:39:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.866+0000 7f151d799740 1 Processor -- start 2024-04-26T21:39:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.866+0000 7f151d799740 1 -- start start 2024-04-26T21:39:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.866+0000 7f151d799740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f99dd0 0x559f28fb0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.866+0000 7f151d799740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f9a6f0 0x559f28fbc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.866+0000 7f151d799740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28fa4fb0 0x559f28fbcef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.866+0000 7f151d799740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f15141b49e0 con 0x559f28f9a6f0 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.866+0000 7f151d799740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15141b22b0 con 0x559f28fa4fb0 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.866+0000 7f151d799740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15141b0120 con 0x559f28f99dd0 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28fa4fb0 0x559f28fbcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28fa4fb0 0x559f28fbcef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:38336/0 (socket says 172.21.15.1:38336) 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513e00640 1 -- 172.21.15.1:0/3134174006 learned_addr learned my addr 172.21.15.1:0/3134174006 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f150aa00640 1 --2- 172.21.15.1:0/3134174006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f9a6f0 0x559f28fbc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513400640 1 --2- 172.21.15.1:0/3134174006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f99dd0 0x559f28fb0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513e00640 1 -- 172.21.15.1:0/3134174006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f99dd0 msgr2=0x559f28fb0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513e00640 1 --2- 172.21.15.1:0/3134174006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f99dd0 0x559f28fb0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513e00640 1 -- 172.21.15.1:0/3134174006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f9a6f0 msgr2=0x559f28fbc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513e00640 1 --2- 172.21.15.1:0/3134174006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f9a6f0 0x559f28fbc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513e00640 1 -- 172.21.15.1:0/3134174006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f28cdd620 con 0x559f28fa4fb0 2024-04-26T21:39:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513400640 1 --2- 172.21.15.1:0/3134174006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f99dd0 0x559f28fb0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:39:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.867+0000 7f1513e00640 1 --2- 172.21.15.1:0/3134174006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28fa4fb0 0x559f28fbcef0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f150c004010 tx=0x7f150c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e45ccb697f0499c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.868+0000 7f1503e00640 1 -- 172.21.15.1:0/3134174006 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c011500 con 0x559f28fa4fb0 2024-04-26T21:39:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.868+0000 7f1503e00640 1 -- 172.21.15.1:0/3134174006 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f150c011b50 con 0x559f28fa4fb0 2024-04-26T21:39:25.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.868+0000 7f1503e00640 1 -- 172.21.15.1:0/3134174006 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150c029cb0 con 0x559f28fa4fb0 2024-04-26T21:39:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 -- 172.21.15.1:0/3134174006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28fa4fb0 msgr2=0x559f28fbcef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 --2- 172.21.15.1:0/3134174006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28fa4fb0 0x559f28fbcef0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f150c004010 tx=0x7f150c019000 comp rx=0 tx=0).stop 2024-04-26T21:39:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 -- 172.21.15.1:0/3134174006 shutdown_connections 2024-04-26T21:39:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 --2- 172.21.15.1:0/3134174006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28fa4fb0 0x559f28fbcef0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 --2- 172.21.15.1:0/3134174006 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28f9a6f0 0x559f28fbc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 --2- 172.21.15.1:0/3134174006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f99dd0 0x559f28fb0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 -- 172.21.15.1:0/3134174006 >> 172.21.15.1:0/3134174006 conn(0x559f28f962e0 msgr2=0x559f28f9d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 -- 172.21.15.1:0/3134174006 shutdown_connections 2024-04-26T21:39:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 -- 172.21.15.1:0/3134174006 wait complete. 2024-04-26T21:39:25.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 -- start start 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.869+0000 7f151d799740 1 Processor -- start 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f151d799740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 0x559f28f9ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f151d799740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 0x559f28f9f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f151d799740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28fa4fb0 0x559f2901a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f151d799740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f15141b0770 con 0x559f28fa4fb0 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f151d799740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15141b1be0 con 0x559f28f99dd0 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f151d799740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15141b0e40 con 0x559f28f9a6f0 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f151d799740 10 client.0 _ll_register_callbacks cb 0x7f14fc0d4850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f1513400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 0x559f28f9ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f1513e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28fa4fb0 0x559f2901a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f150aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 0x559f28f9f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f1513400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 0x559f28f9ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:38352/0 (socket says 172.21.15.1:38352) 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.870+0000 7f1513400640 1 -- 172.21.15.1:0/1232839077 learned_addr learned my addr 172.21.15.1:0/1232839077 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:25.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f1513400640 1 -- 172.21.15.1:0/1232839077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 msgr2=0x559f28f9f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f1513400640 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 0x559f28f9f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f1513400640 1 -- 172.21.15.1:0/1232839077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28fa4fb0 msgr2=0x559f2901a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f1513400640 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28fa4fb0 0x559f2901a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f1513400640 1 -- 172.21.15.1:0/1232839077 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f28eb9cb0 con 0x559f28f99dd0 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f1513400640 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 0x559f28f9ef00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f150402aa30 tx=0x7f150402ab80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5185bc13000ff53 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f1501600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f1501600640 1 -- 172.21.15.1:0/1232839077 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1504035040 con 0x559f28f99dd0 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f1501600640 1 -- 172.21.15.1:0/1232839077 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7f1504004c70 con 0x559f28f99dd0 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f151d799740 1 -- 172.21.15.1:0/1232839077 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f150c011b50 con 0x559f28f99dd0 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.871+0000 7f1501600640 1 -- 172.21.15.1:0/1232839077 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f15040044a0 con 0x559f28f99dd0 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f151d799740 10 client.26252 fetch_fsmap learned FSMap version 36 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f151d799740 1 -- 172.21.15.1:0/1232839077 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559f2901b600 con 0x559f28f99dd0 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f1501600640 1 -- 172.21.15.1:0/1232839077 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f150c011b50 con 0x559f28f99dd0 2024-04-26T21:39:25.876 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f1501600640 1 -- 172.21.15.1:0/1232839077 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f15040039d0 con 0x559f28f99dd0 2024-04-26T21:39:25.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f151d799740 10 client.26252 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:39:25.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f151d799740 10 client.26252 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:39:25.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f151d799740 20 client.26252 populate_metadata read hostname 'smithi001' 2024-04-26T21:39:25.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f151d799740 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559f2901b940 0x559f2901dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:25.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f151d799740 4 client.26252 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:39:25.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f151d799740 1 -- 172.21.15.1:0/1232839077 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559f28f06e00 con 0x559f2901b940 2024-04-26T21:39:25.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.872+0000 7f150aa00640 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559f2901b940 0x559f2901dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:25.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.873+0000 7f150aa00640 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559f2901b940 0x559f2901dd30 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=908eef22edb32df server_cookie=cb26854fef5dd9ff in_seq=0 out_seq=0 2024-04-26T21:39:25.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.873+0000 7f1501600640 10 client.26252 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f1501600640 1 -- 172.21.15.1:0/1232839077 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559f28f06e00 con 0x559f2901b940 2024-04-26T21:39:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f1501600640 10 client.26252 handle_command_reply: tid=0 2024-04-26T21:39:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 client.26252 shutdown 2024-04-26T21:39:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 -- 172.21.15.1:0/1232839077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 msgr2=0x559f28f9ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 0x559f28f9ef00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f150402aa30 tx=0x7f150402ab80 comp rx=0 tx=0).stop 2024-04-26T21:39:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 -- 172.21.15.1:0/1232839077 shutdown_connections 2024-04-26T21:39:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559f2901b940 0x559f2901dd30 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f28f9a6f0 0x559f28f9f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559f28fa4fb0 0x559f2901a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f1501600640 0 client.26252 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 --2- 172.21.15.1:0/1232839077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559f28f99dd0 0x559f28f9ef00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 -- 172.21.15.1:0/1232839077 >> 172.21.15.1:0/1232839077 conn(0x559f28f962e0 msgr2=0x559f28fa1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 -- 172.21.15.1:0/1232839077 shutdown_connections 2024-04-26T21:39:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 1 -- 172.21.15.1:0/1232839077 wait complete. 2024-04-26T21:39:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.935+0000 7f151d799740 20 client.26252 trim_cache size 0 max 16384 2024-04-26T21:39:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:39:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:39:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:39:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:39:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 -- 172.21.15.1:0/3648916044 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f14f40a3d10 msgr2=0x7f14f40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f14f40a3d10 0x7f14f40a61d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f150c005fd0 tx=0x7f150c0096a0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 -- 172.21.15.1:0/3648916044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1514105510 msgr2=0x7f151419b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1514105510 0x7f151419b890 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f14fc00e9f0 tx=0x7f14fc00edf0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 -- 172.21.15.1:0/3648916044 shutdown_connections 2024-04-26T21:39:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f14f40a3d10 0x7f14f40a61d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f151410ac10 0x7f151419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1514105510 0x7f151419b890 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 --2- 172.21.15.1:0/3648916044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1514104b50 0x7f151419b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 -- 172.21.15.1:0/3648916044 >> 172.21.15.1:0/3648916044 conn(0x7f1514100620 msgr2=0x7f1514101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 -- 172.21.15.1:0/3648916044 shutdown_connections 2024-04-26T21:39:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:25.936+0000 7f151a800640 1 -- 172.21.15.1:0/3648916044 wait complete. 2024-04-26T21:39:26.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:25 smithi001 ceph-mon[28054]: pgmap v282: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 253 op/s 2024-04-26T21:39:26.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:25 smithi038 ceph-mon[32724]: pgmap v282: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 253 op/s 2024-04-26T21:39:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:25 smithi186 ceph-mon[32991]: pgmap v282: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 253 op/s 2024-04-26T21:39:26.583 INFO:tasks.workunit.client.0.smithi001.stdout: 1 337220 4.52 MB/sec execute 324 sec latency 41.904 ms 2024-04-26T21:39:27.583 INFO:tasks.workunit.client.0.smithi001.stdout: 1 337548 4.51 MB/sec execute 325 sec latency 20.282 ms 2024-04-26T21:39:28.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:28 smithi001 ceph-mon[28054]: pgmap v283: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-04-26T21:39:28.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:28 smithi038 ceph-mon[32724]: pgmap v283: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-04-26T21:39:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:28 smithi186 ceph-mon[32991]: pgmap v283: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-04-26T21:39:28.583 INFO:tasks.workunit.client.0.smithi001.stdout: 1 338627 4.51 MB/sec execute 326 sec latency 22.342 ms 2024-04-26T21:39:29.583 INFO:tasks.workunit.client.0.smithi001.stdout: 1 340184 4.52 MB/sec execute 327 sec latency 28.007 ms 2024-04-26T21:39:30.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:30 smithi001 ceph-mon[28054]: pgmap v284: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-04-26T21:39:30.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:30 smithi038 ceph-mon[32724]: pgmap v284: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-04-26T21:39:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:30 smithi186 ceph-mon[32991]: pgmap v284: 129 pgs: 129 active+clean; 211 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 137 op/s 2024-04-26T21:39:30.584 INFO:tasks.workunit.client.0.smithi001.stdout: 1 340904 4.52 MB/sec execute 328 sec latency 26.306 ms 2024-04-26T21:39:31.584 INFO:tasks.workunit.client.0.smithi001.stdout: 1 341402 4.51 MB/sec execute 329 sec latency 19.000 ms 2024-04-26T21:39:32.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:32 smithi001 ceph-mon[28054]: pgmap v285: 129 pgs: 129 active+clean; 213 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-04-26T21:39:32.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:32 smithi038 ceph-mon[32724]: pgmap v285: 129 pgs: 129 active+clean; 213 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-04-26T21:39:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:32 smithi186 ceph-mon[32991]: pgmap v285: 129 pgs: 129 active+clean; 213 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 235 op/s 2024-04-26T21:39:32.584 INFO:tasks.workunit.client.0.smithi001.stdout: 1 342908 4.52 MB/sec execute 330 sec latency 22.259 ms 2024-04-26T21:39:33.584 INFO:tasks.workunit.client.0.smithi001.stdout: 1 344047 4.53 MB/sec execute 331 sec latency 22.025 ms 2024-04-26T21:39:34.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:34 smithi001 ceph-mon[28054]: pgmap v286: 129 pgs: 129 active+clean; 213 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-26T21:39:34.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:34 smithi038 ceph-mon[32724]: pgmap v286: 129 pgs: 129 active+clean; 213 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-26T21:39:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:34 smithi186 ceph-mon[32991]: pgmap v286: 129 pgs: 129 active+clean; 213 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-04-26T21:39:34.586 INFO:tasks.workunit.client.0.smithi001.stdout: 1 344536 4.53 MB/sec execute 332 sec latency 44.344 ms 2024-04-26T21:39:35.584 INFO:tasks.workunit.client.0.smithi001.stdout: 1 345318 4.52 MB/sec execute 333 sec latency 22.675 ms 2024-04-26T21:39:36.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:36 smithi001 ceph-mon[28054]: pgmap v287: 129 pgs: 129 active+clean; 216 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 233 op/s 2024-04-26T21:39:36.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:36 smithi038 ceph-mon[32724]: pgmap v287: 129 pgs: 129 active+clean; 216 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 233 op/s 2024-04-26T21:39:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:36 smithi186 ceph-mon[32991]: pgmap v287: 129 pgs: 129 active+clean; 216 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 233 op/s 2024-04-26T21:39:36.584 INFO:tasks.workunit.client.0.smithi001.stdout: 1 346829 4.53 MB/sec execute 334 sec latency 21.993 ms 2024-04-26T21:39:37.584 INFO:tasks.workunit.client.0.smithi001.stdout: 1 347885 4.54 MB/sec execute 335 sec latency 26.940 ms 2024-04-26T21:39:38.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:38 smithi001 ceph-mon[28054]: pgmap v288: 129 pgs: 129 active+clean; 216 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-26T21:39:38.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:38 smithi038 ceph-mon[32724]: pgmap v288: 129 pgs: 129 active+clean; 216 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-26T21:39:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:38 smithi186 ceph-mon[32991]: pgmap v288: 129 pgs: 129 active+clean; 216 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-26T21:39:38.585 INFO:tasks.workunit.client.0.smithi001.stdout: 1 348203 4.53 MB/sec execute 336 sec latency 22.148 ms 2024-04-26T21:39:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.095+0000 7f7da2000640 1 -- 172.21.15.1:0/67357025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d9c101830 msgr2=0x7f7d9c101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.095+0000 7f7da2000640 1 --2- 172.21.15.1:0/67357025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d9c101830 0x7f7d9c101cb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009860 tx=0x7f7d8402a340 comp rx=0 tx=0).stop 2024-04-26T21:39:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.095+0000 7f7da2000640 1 -- 172.21.15.1:0/67357025 shutdown_connections 2024-04-26T21:39:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.095+0000 7f7da2000640 1 --2- 172.21.15.1:0/67357025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9c1021f0 0x7f7d9c10e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.095+0000 7f7da2000640 1 --2- 172.21.15.1:0/67357025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d9c101830 0x7f7d9c101cb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.095+0000 7f7da2000640 1 --2- 172.21.15.1:0/67357025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c107840 0x7f7d9c107c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.095+0000 7f7da2000640 1 -- 172.21.15.1:0/67357025 >> 172.21.15.1:0/67357025 conn(0x7f7d9c078930 msgr2=0x7f7d9c078d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.095+0000 7f7da2000640 1 -- 172.21.15.1:0/67357025 shutdown_connections 2024-04-26T21:39:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.095+0000 7f7da2000640 1 -- 172.21.15.1:0/67357025 wait complete. 2024-04-26T21:39:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.096+0000 7f7da2000640 1 Processor -- start 2024-04-26T21:39:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.097+0000 7f7da2000640 1 -- start start 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.097+0000 7f7da2000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9c101830 0x7f7d9c199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.097+0000 7f7da2000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c1021f0 0x7f7d9c199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.097+0000 7f7da2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d9c107840 0x7f7d9c19d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.097+0000 7f7da2000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c111ac0 con 0x7f7d9c1021f0 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.097+0000 7f7da2000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c111940 con 0x7f7d9c107840 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.097+0000 7f7da2000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c111c40 con 0x7f7d9c101830 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.097+0000 7f7d92a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c1021f0 0x7f7d9c199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.097+0000 7f7d92a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c1021f0 0x7f7d9c199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35808/0 (socket says 172.21.15.1:35808) 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d92a00640 1 -- 172.21.15.1:0/93613055 learned_addr learned my addr 172.21.15.1:0/93613055 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d9be00640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d9c107840 0x7f7d9c19d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d9b400640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9c101830 0x7f7d9c199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d9be00640 1 -- 172.21.15.1:0/93613055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9c101830 msgr2=0x7f7d9c199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d9be00640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9c101830 0x7f7d9c199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d9be00640 1 -- 172.21.15.1:0/93613055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c1021f0 msgr2=0x7f7d9c199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d9be00640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c1021f0 0x7f7d9c199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d9be00640 1 -- 172.21.15.1:0/93613055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9c19e100 con 0x7f7d9c107840 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d92a00640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c1021f0 0x7f7d9c199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d9b400640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9c101830 0x7f7d9c199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.098+0000 7f7d9be00640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d9c107840 0x7f7d9c19d9d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7d94009fd0 tx=0x7f7d9400eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.099+0000 7f7d98c00640 1 -- 172.21.15.1:0/93613055 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d94004220 con 0x7f7d9c107840 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.099+0000 7f7da2000640 1 -- 172.21.15.1:0/93613055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d9c19e3f0 con 0x7f7d9c107840 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.099+0000 7f7d98c00640 1 -- 172.21.15.1:0/93613055 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d940043c0 con 0x7f7d9c107840 2024-04-26T21:39:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.099+0000 7f7d98c00640 1 -- 172.21.15.1:0/93613055 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d94005650 con 0x7f7d9c107840 2024-04-26T21:39:39.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.099+0000 7f7da2000640 1 -- 172.21.15.1:0/93613055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d9c1a5c80 con 0x7f7d9c107840 2024-04-26T21:39:39.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.102+0000 7f7d98c00640 1 -- 172.21.15.1:0/93613055 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f7d94004a00 con 0x7f7d9c107840 2024-04-26T21:39:39.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.102+0000 7f7da4d7f740 1 Processor -- start 2024-04-26T21:39:39.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.103+0000 7f7da4d7f740 1 -- start start 2024-04-26T21:39:39.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.103+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3453cf0 0x55aac34540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.103+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3462e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.103+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac34633c0 0x55aac34657b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.103+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55aac33b9380 con 0x55aac3453cf0 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.103+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55aac3355c30 con 0x55aac3454610 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.103+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aac33b5340 con 0x55aac34633c0 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.103+0000 7f7d9b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3462e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.103+0000 7f7d92a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3453cf0 0x55aac34540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.103+0000 7f7d9b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3462e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42718/0 (socket says 172.21.15.1:42718) 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d9b400640 1 -- 172.21.15.1:0/3429213967 learned_addr learned my addr 172.21.15.1:0/3429213967 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d92a00640 1 --2- 172.21.15.1:0/3429213967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac34633c0 0x55aac34657b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d98c00640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7d800a3d90 0x7f7d800a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d9b400640 1 -- 172.21.15.1:0/3429213967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac34633c0 msgr2=0x55aac34657b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d9b400640 1 --2- 172.21.15.1:0/3429213967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac34633c0 0x55aac34657b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d9b400640 1 -- 172.21.15.1:0/3429213967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3453cf0 msgr2=0x55aac34540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:39.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d9b400640 1 --2- 172.21.15.1:0/3429213967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3453cf0 0x55aac34540d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d9b400640 1 -- 172.21.15.1:0/3429213967 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aac3377940 con 0x55aac3454610 2024-04-26T21:39:39.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d92a00640 1 --2- 172.21.15.1:0/3429213967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3453cf0 0x55aac34540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:39:39.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d98c00640 1 -- 172.21.15.1:0/93613055 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7f7d940d0af0 con 0x7f7d9c107840 2024-04-26T21:39:39.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d9b400640 1 --2- 172.21.15.1:0/3429213967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3462e80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c005f40 tx=0x7f7d8c011cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f44b25fa122bb4fc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:39.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.104+0000 7f7d9be00640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7d800a3d90 0x7f7d800a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.105+0000 7f7d8be00640 1 -- 172.21.15.1:0/3429213967 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c011220 con 0x55aac3454610 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.105+0000 7f7d8be00640 1 -- 172.21.15.1:0/3429213967 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d8c00f9a0 con 0x55aac3454610 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.105+0000 7f7d8be00640 1 -- 172.21.15.1:0/3429213967 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c004540 con 0x55aac3454610 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3429213967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 msgr2=0x55aac3462e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/3429213967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3462e80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c005f40 tx=0x7f7d8c011cb0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7d9be00640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7d800a3d90 0x7f7d800a6250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7d940cdde0 tx=0x7f7d9400cf10 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3429213967 shutdown_connections 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/3429213967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac34633c0 0x55aac34657b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/3429213967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3462e80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/3429213967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3453cf0 0x55aac34540d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3429213967 >> 172.21.15.1:0/3429213967 conn(0x55aac34502e0 msgr2=0x55aac34506c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3429213967 shutdown_connections 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3429213967 wait complete. 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.106+0000 7f7da4d7f740 1 -- start start 2024-04-26T21:39:39.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.107+0000 7f7da4d7f740 1 Processor -- start 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.107+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac3460300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.107+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3460840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.107+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac34633c0 0x55aac3460d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.107+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55aac3465ff0 con 0x55aac34633c0 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.107+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55aac3340c60 con 0x55aac3454610 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.107+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55aac3466170 con 0x55aac3453cf0 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.107+0000 7f7da4d7f740 10 client.0 _ll_register_callbacks cb 0x55aac336ce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d9b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3460840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d9b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3460840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42724/0 (socket says 172.21.15.1:42724) 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d9b400640 1 -- 172.21.15.1:0/2312116647 learned_addr learned my addr 172.21.15.1:0/2312116647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d92a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac3460300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d92a00640 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac34633c0 0x55aac3460d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d9b400640 1 -- 172.21.15.1:0/2312116647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 msgr2=0x55aac3460300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:39.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d9b400640 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac3460300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d9b400640 1 -- 172.21.15.1:0/2312116647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac34633c0 msgr2=0x55aac3460d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d9b400640 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac34633c0 0x55aac3460d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d9b400640 1 -- 172.21.15.1:0/2312116647 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aac334f510 con 0x55aac3454610 2024-04-26T21:39:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d92a00640 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac3460300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:39:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d9b400640 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3460840 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c005c10 tx=0x7f7d8c011270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e48bc7f897bfb3e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.108+0000 7f7d88c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:39:39.113 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.109+0000 7f7d88c00640 1 -- 172.21.15.1:0/2312116647 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c01e040 con 0x55aac3454610 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.109+0000 7f7da4d7f740 1 -- 172.21.15.1:0/2312116647 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d8c00f9a0 con 0x55aac3454610 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.109+0000 7f7d88c00640 1 -- 172.21.15.1:0/2312116647 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7f7d8c004c40 con 0x55aac3454610 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.109+0000 7f7d88c00640 1 -- 172.21.15.1:0/2312116647 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7d8c031050 con 0x55aac3454610 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.109+0000 7f7da4d7f740 10 client.26264 fetch_fsmap learned FSMap version 36 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.109+0000 7f7da4d7f740 1 -- 172.21.15.1:0/2312116647 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55aac30cb210 con 0x55aac3454610 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.109+0000 7f7d88c00640 1 -- 172.21.15.1:0/2312116647 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d8c00f9a0 con 0x55aac3454610 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.110+0000 7f7d88c00640 1 -- 172.21.15.1:0/2312116647 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f7d8c018070 con 0x55aac3454610 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.110+0000 7f7da4d7f740 10 client.26264 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.110+0000 7f7da4d7f740 10 client.26264 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.110+0000 7f7da4d7f740 20 client.26264 populate_metadata read hostname 'smithi001' 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.110+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aac3473000 0x55aac34753f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.110+0000 7f7da4d7f740 4 client.26264 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:39:39.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.110+0000 7f7da4d7f740 1 -- 172.21.15.1:0/2312116647 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aac33c6f90 con 0x55aac3473000 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.110+0000 7f7d92a00640 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aac3473000 0x55aac34753f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.110+0000 7f7d92a00640 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aac3473000 0x55aac34753f0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cce88ec83c4f910 server_cookie=5e7f887193b12e7c in_seq=0 out_seq=0 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.110+0000 7f7d88c00640 10 client.26264 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.111+0000 7f7d88c00640 1 -- 172.21.15.1:0/2312116647 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aac33c6f90 con 0x55aac3473000 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.111+0000 7f7d88c00640 10 client.26264 handle_command_reply: tid=0 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.111+0000 7f7da4d7f740 1 client.26264 shutdown 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 1 -- 172.21.15.1:0/2312116647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 msgr2=0x55aac3460840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3460840 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c005c10 tx=0x7f7d8c011270 comp rx=0 tx=0).stop 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 1 -- 172.21.15.1:0/2312116647 shutdown_connections 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aac3473000 0x55aac34753f0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac34633c0 0x55aac3460d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac3460840 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7d88c00640 0 client.26264 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:39.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/2312116647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac3460300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 1 -- 172.21.15.1:0/2312116647 >> 172.21.15.1:0/2312116647 conn(0x55aac34502e0 msgr2=0x55aac3456150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 1 -- 172.21.15.1:0/2312116647 shutdown_connections 2024-04-26T21:39:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 1 -- 172.21.15.1:0/2312116647 wait complete. 2024-04-26T21:39:39.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.112+0000 7f7da4d7f740 20 client.26264 trim_cache size 0 max 16384 2024-04-26T21:39:39.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.129+0000 7f7da4d7f740 1 Processor -- start 2024-04-26T21:39:39.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.129+0000 7f7da4d7f740 1 -- start start 2024-04-26T21:39:39.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac346eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac347abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac34633c0 0x55aac347b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c1b0740 con 0x55aac34633c0 2024-04-26T21:39:39.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c1ae010 con 0x55aac3454610 2024-04-26T21:39:39.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c1abe80 con 0x55aac3453cf0 2024-04-26T21:39:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d9b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac347abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.132 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d92a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac346eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d9b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac347abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42736/0 (socket says 172.21.15.1:42736) 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d92a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac346eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:48074/0 (socket says 172.21.15.1:48074) 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d9b400640 1 -- 172.21.15.1:0/1449240235 learned_addr learned my addr 172.21.15.1:0/1449240235 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d92a00640 1 --2- 172.21.15.1:0/1449240235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac34633c0 0x55aac347b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d92a00640 1 -- 172.21.15.1:0/1449240235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 msgr2=0x55aac347abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d92a00640 1 --2- 172.21.15.1:0/1449240235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac347abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d92a00640 1 -- 172.21.15.1:0/1449240235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac34633c0 msgr2=0x55aac347b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d92a00640 1 --2- 172.21.15.1:0/1449240235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac34633c0 0x55aac347b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d92a00640 1 -- 172.21.15.1:0/1449240235 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aac3475760 con 0x55aac3453cf0 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.130+0000 7f7d9b400640 1 --2- 172.21.15.1:0/1449240235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac347abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.131+0000 7f7d92a00640 1 --2- 172.21.15.1:0/1449240235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac346eeb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d840057c0 tx=0x7f7d84005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c6342e6bfef539c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.131+0000 7f7d8b400640 1 -- 172.21.15.1:0/1449240235 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d84037040 con 0x55aac3453cf0 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.131+0000 7f7d8b400640 1 -- 172.21.15.1:0/1449240235 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d8404abd0 con 0x55aac3453cf0 2024-04-26T21:39:39.133 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.131+0000 7f7d8b400640 1 -- 172.21.15.1:0/1449240235 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d84032be0 con 0x55aac3453cf0 2024-04-26T21:39:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.132+0000 7f7da4d7f740 1 -- 172.21.15.1:0/1449240235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 msgr2=0x55aac346eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.132+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/1449240235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac346eeb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d840057c0 tx=0x7f7d84005a90 comp rx=0 tx=0).stop 2024-04-26T21:39:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.132+0000 7f7da4d7f740 1 -- 172.21.15.1:0/1449240235 shutdown_connections 2024-04-26T21:39:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.132+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/1449240235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac34633c0 0x55aac347b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.132+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/1449240235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3454610 0x55aac347abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.132+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/1449240235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac3453cf0 0x55aac346eeb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.132+0000 7f7da4d7f740 1 -- 172.21.15.1:0/1449240235 >> 172.21.15.1:0/1449240235 conn(0x55aac34502e0 msgr2=0x55aac345ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.132+0000 7f7da4d7f740 1 -- 172.21.15.1:0/1449240235 shutdown_connections 2024-04-26T21:39:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.132+0000 7f7da4d7f740 1 -- 172.21.15.1:0/1449240235 wait complete. 2024-04-26T21:39:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.132+0000 7f7da4d7f740 1 -- start start 2024-04-26T21:39:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.133+0000 7f7da4d7f740 1 Processor -- start 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.133+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3453cf0 0x55aac346b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.133+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3454610 0x55aac345cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.133+0000 7f7da4d7f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac34633c0 0x55aac345d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.133+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c1ac4d0 con 0x55aac3454610 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.133+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c1ad940 con 0x55aac3453cf0 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.133+0000 7f7da4d7f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9c1acba0 con 0x55aac34633c0 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.133+0000 7f7da4d7f740 10 client.0 _ll_register_callbacks cb 0x7f7d8c0116a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d9b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3454610 0x55aac345cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d92a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3453cf0 0x55aac346b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d9b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3454610 0x55aac345cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35830/0 (socket says 172.21.15.1:35830) 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d92a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3453cf0 0x55aac346b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42738/0 (socket says 172.21.15.1:42738) 2024-04-26T21:39:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d9b400640 1 -- 172.21.15.1:0/3088864693 learned_addr learned my addr 172.21.15.1:0/3088864693 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d92a00640 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac34633c0 0x55aac345d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d92a00640 1 -- 172.21.15.1:0/3088864693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac34633c0 msgr2=0x55aac345d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d92a00640 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac34633c0 0x55aac345d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d92a00640 1 -- 172.21.15.1:0/3088864693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3454610 msgr2=0x55aac345cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d92a00640 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3454610 0x55aac345cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d92a00640 1 -- 172.21.15.1:0/3088864693 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aac3197620 con 0x55aac3453cf0 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d9b400640 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3454610 0x55aac345cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d92a00640 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3453cf0 0x55aac346b5b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7d840057c0 tx=0x7f7d84005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bef5397aaf66740 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.134+0000 7f7d88c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.135+0000 7f7d88c00640 1 -- 172.21.15.1:0/3088864693 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d84037070 con 0x55aac3453cf0 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.135+0000 7f7d88c00640 1 -- 172.21.15.1:0/3088864693 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7f7d84032b60 con 0x55aac3453cf0 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.135+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3088864693 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d8404abd0 con 0x55aac3453cf0 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.135+0000 7f7d88c00640 1 -- 172.21.15.1:0/3088864693 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7d8404ae10 con 0x55aac3453cf0 2024-04-26T21:39:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.135+0000 7f7da4d7f740 10 client.26270 fetch_fsmap learned FSMap version 36 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.135+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3088864693 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55aac3373cb0 con 0x55aac3453cf0 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.135+0000 7f7d88c00640 1 -- 172.21.15.1:0/3088864693 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d8404abd0 con 0x55aac3453cf0 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.136+0000 7f7d88c00640 1 -- 172.21.15.1:0/3088864693 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f7d84040820 con 0x55aac3453cf0 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.136+0000 7f7da4d7f740 10 client.26270 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.136+0000 7f7da4d7f740 10 client.26270 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.136+0000 7f7da4d7f740 20 client.26270 populate_metadata read hostname 'smithi001' 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.136+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aac34d93f0 0x55aac34db7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.136+0000 7f7da4d7f740 4 client.26270 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.136+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3088864693 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aac33c0e00 con 0x55aac34d93f0 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.136+0000 7f7d9b400640 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aac34d93f0 0x55aac34db7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.137+0000 7f7d9b400640 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aac34d93f0 0x55aac34db7e0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15a9fbeabf0c6ace server_cookie=c7daabac2b120135 in_seq=0 out_seq=0 2024-04-26T21:39:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.137+0000 7f7d88c00640 10 client.26270 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.207+0000 7f7d88c00640 1 -- 172.21.15.1:0/3088864693 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aac33c0e00 con 0x55aac34d93f0 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.207+0000 7f7d88c00640 10 client.26270 handle_command_reply: tid=0 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.207+0000 7f7da4d7f740 1 client.26270 shutdown 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3088864693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3453cf0 msgr2=0x55aac346b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3453cf0 0x55aac346b5b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7d840057c0 tx=0x7f7d84005f90 comp rx=0 tx=0).stop 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3088864693 shutdown_connections 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55aac34d93f0 0x55aac34db7e0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55aac34633c0 0x55aac345d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55aac3454610 0x55aac345cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 1 --2- 172.21.15.1:0/3088864693 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55aac3453cf0 0x55aac346b5b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7d88c00640 0 client.26270 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3088864693 >> 172.21.15.1:0/3088864693 conn(0x55aac34502e0 msgr2=0x55aac34602a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3088864693 shutdown_connections 2024-04-26T21:39:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 1 -- 172.21.15.1:0/3088864693 wait complete. 2024-04-26T21:39:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.208+0000 7f7da4d7f740 20 client.26270 trim_cache size 0 max 16384 2024-04-26T21:39:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:39:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:39:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:39:39.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:39:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 -- 172.21.15.1:0/93613055 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7d800a3d90 msgr2=0x7f7d800a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7d800a3d90 0x7f7d800a6250 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7d940cdde0 tx=0x7f7d9400cf10 comp rx=0 tx=0).stop 2024-04-26T21:39:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 -- 172.21.15.1:0/93613055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d9c107840 msgr2=0x7f7d9c19d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d9c107840 0x7f7d9c19d9d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7d94009fd0 tx=0x7f7d9400eba0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 -- 172.21.15.1:0/93613055 shutdown_connections 2024-04-26T21:39:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7d800a3d90 0x7f7d800a6250 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d9c107840 0x7f7d9c19d9d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d9c1021f0 0x7f7d9c199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 --2- 172.21.15.1:0/93613055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d9c101830 0x7f7d9c199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 -- 172.21.15.1:0/93613055 >> 172.21.15.1:0/93613055 conn(0x7f7d9c078930 msgr2=0x7f7d9c100ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.209+0000 7f7da2000640 1 -- 172.21.15.1:0/93613055 shutdown_connections 2024-04-26T21:39:39.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:39.210+0000 7f7da2000640 1 -- 172.21.15.1:0/93613055 wait complete. 2024-04-26T21:39:39.585 INFO:tasks.workunit.client.0.smithi001.stdout: 1 349314 4.53 MB/sec execute 337 sec latency 38.849 ms 2024-04-26T21:39:40.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:40 smithi001 ceph-mon[28054]: pgmap v289: 129 pgs: 129 active+clean; 216 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-26T21:39:40.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:40 smithi038 ceph-mon[32724]: pgmap v289: 129 pgs: 129 active+clean; 216 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-26T21:39:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:40 smithi186 ceph-mon[32991]: pgmap v289: 129 pgs: 129 active+clean; 216 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.5 MiB/s wr, 161 op/s 2024-04-26T21:39:40.585 INFO:tasks.workunit.client.0.smithi001.stdout: 1 350792 4.54 MB/sec execute 338 sec latency 28.960 ms 2024-04-26T21:39:41.585 INFO:tasks.workunit.client.0.smithi001.stdout: 1 351550 4.55 MB/sec execute 339 sec latency 25.828 ms 2024-04-26T21:39:42.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:42 smithi001 ceph-mon[28054]: pgmap v290: 129 pgs: 129 active+clean; 223 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 254 op/s 2024-04-26T21:39:42.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:42 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:39:42.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:42 smithi038 ceph-mon[32724]: pgmap v290: 129 pgs: 129 active+clean; 223 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 254 op/s 2024-04-26T21:39:42.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:42 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:39:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:42 smithi186 ceph-mon[32991]: pgmap v290: 129 pgs: 129 active+clean; 223 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 254 op/s 2024-04-26T21:39:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:42 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:39:42.585 INFO:tasks.workunit.client.0.smithi001.stdout: 1 352147 4.53 MB/sec execute 340 sec latency 27.339 ms 2024-04-26T21:39:43.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:43 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:39:43.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:43 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:39:43.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:43 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:39:43.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:43 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:39:43.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:43 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:39:43.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:43 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:39:43.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:43 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:39:43.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:43 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:39:43.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:43 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:39:43.585 INFO:tasks.workunit.client.0.smithi001.stdout: 1 353601 4.54 MB/sec execute 341 sec latency 32.274 ms 2024-04-26T21:39:44.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:44 smithi001 ceph-mon[28054]: pgmap v291: 129 pgs: 129 active+clean; 223 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-04-26T21:39:44.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:44 smithi038 ceph-mon[32724]: pgmap v291: 129 pgs: 129 active+clean; 223 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-04-26T21:39:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:44 smithi186 ceph-mon[32991]: pgmap v291: 129 pgs: 129 active+clean; 223 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.4 MiB/s wr, 156 op/s 2024-04-26T21:39:44.585 INFO:tasks.workunit.client.0.smithi001.stdout: 1 354647 4.56 MB/sec execute 342 sec latency 54.625 ms 2024-04-26T21:39:45.585 INFO:tasks.workunit.client.0.smithi001.stdout: 1 355123 4.55 MB/sec execute 343 sec latency 35.640 ms 2024-04-26T21:39:46.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:46 smithi001 ceph-mon[28054]: pgmap v292: 129 pgs: 129 active+clean; 227 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 228 op/s 2024-04-26T21:39:46.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:46 smithi038 ceph-mon[32724]: pgmap v292: 129 pgs: 129 active+clean; 227 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 228 op/s 2024-04-26T21:39:46.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:46 smithi186 ceph-mon[32991]: pgmap v292: 129 pgs: 129 active+clean; 227 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.1 MiB/s wr, 228 op/s 2024-04-26T21:39:46.586 INFO:tasks.workunit.client.0.smithi001.stdout: 1 355769 4.54 MB/sec execute 344 sec latency 25.423 ms 2024-04-26T21:39:47.586 INFO:tasks.workunit.client.0.smithi001.stdout: 1 357102 4.54 MB/sec execute 345 sec latency 29.337 ms 2024-04-26T21:39:48.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:48 smithi001 ceph-mon[28054]: pgmap v293: 129 pgs: 129 active+clean; 227 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-26T21:39:48.330 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:48 smithi038 ceph-mon[32724]: pgmap v293: 129 pgs: 129 active+clean; 227 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-26T21:39:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:48 smithi186 ceph-mon[32991]: pgmap v293: 129 pgs: 129 active+clean; 227 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-26T21:39:48.586 INFO:tasks.workunit.client.0.smithi001.stdout: 1 358076 4.55 MB/sec execute 346 sec latency 25.640 ms 2024-04-26T21:39:49.586 INFO:tasks.workunit.client.0.smithi001.stdout: 1 358616 4.55 MB/sec execute 347 sec latency 34.320 ms 2024-04-26T21:39:50.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:50 smithi001 ceph-mon[28054]: pgmap v294: 129 pgs: 129 active+clean; 227 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-26T21:39:50.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:50 smithi038 ceph-mon[32724]: pgmap v294: 129 pgs: 129 active+clean; 227 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-26T21:39:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:50 smithi186 ceph-mon[32991]: pgmap v294: 129 pgs: 129 active+clean; 227 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-04-26T21:39:50.586 INFO:tasks.workunit.client.0.smithi001.stdout: 1 359308 4.54 MB/sec execute 348 sec latency 62.662 ms 2024-04-26T21:39:51.586 INFO:tasks.workunit.client.0.smithi001.stdout: 1 360679 4.54 MB/sec execute 349 sec latency 33.277 ms 2024-04-26T21:39:52.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:52 smithi001 ceph-mon[28054]: pgmap v295: 129 pgs: 129 active+clean; 233 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-04-26T21:39:52.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:52 smithi038 ceph-mon[32724]: pgmap v295: 129 pgs: 129 active+clean; 233 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-04-26T21:39:52.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.371+0000 7f9a23e00640 1 -- 172.21.15.1:0/1311609114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a1c1032d0 msgr2=0x7f9a1c10f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:52.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.371+0000 7f9a23e00640 1 --2- 172.21.15.1:0/1311609114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a1c1032d0 0x7f9a1c10f7e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9a14009900 tx=0x7f9a1402a3f0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.371+0000 7f9a23e00640 1 -- 172.21.15.1:0/1311609114 shutdown_connections 2024-04-26T21:39:52.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.371+0000 7f9a23e00640 1 --2- 172.21.15.1:0/1311609114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1c102910 0x7f9a1c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.371+0000 7f9a23e00640 1 --2- 172.21.15.1:0/1311609114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a1c1032d0 0x7f9a1c10f7e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.371+0000 7f9a23e00640 1 --2- 172.21.15.1:0/1311609114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a1c108920 0x7f9a1c108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.371+0000 7f9a23e00640 1 -- 172.21.15.1:0/1311609114 >> 172.21.15.1:0/1311609114 conn(0x7f9a1c0fe400 msgr2=0x7f9a1c100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.371+0000 7f9a23e00640 1 -- 172.21.15.1:0/1311609114 shutdown_connections 2024-04-26T21:39:52.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.371+0000 7f9a23e00640 1 -- 172.21.15.1:0/1311609114 wait complete. 2024-04-26T21:39:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.372+0000 7f9a23e00640 1 Processor -- start 2024-04-26T21:39:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.372+0000 7f9a23e00640 1 -- start start 2024-04-26T21:39:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a23e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a1c102910 0x7f9a1c19b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a23e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1c1032d0 0x7f9a1c19b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a23e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a1c108920 0x7f9a1c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a23e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c112be0 con 0x7f9a1c102910 2024-04-26T21:39:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a23e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c112a60 con 0x7f9a1c108920 2024-04-26T21:39:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a23e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c112d60 con 0x7f9a1c1032d0 2024-04-26T21:39:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a21400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a1c102910 0x7f9a1c19b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a21e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a1c108920 0x7f9a1c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a21e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a1c108920 0x7f9a1c19fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59576/0 (socket says 172.21.15.1:59576) 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a21e00640 1 -- 172.21.15.1:0/4104047633 learned_addr learned my addr 172.21.15.1:0/4104047633 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a21400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a1c102910 0x7f9a1c19b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55636/0 (socket says 172.21.15.1:55636) 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.373+0000 7f9a20a00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1c1032d0 0x7f9a1c19b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a21e00640 1 -- 172.21.15.1:0/4104047633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1c1032d0 msgr2=0x7f9a1c19b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a21e00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1c1032d0 0x7f9a1c19b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a21e00640 1 -- 172.21.15.1:0/4104047633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a1c102910 msgr2=0x7f9a1c19b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a21e00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a1c102910 0x7f9a1c19b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a21e00640 1 -- 172.21.15.1:0/4104047633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1c1a02b0 con 0x7f9a1c108920 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a21400640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a1c102910 0x7f9a1c19b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a21e00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a1c108920 0x7f9a1c19fb80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9a14005ab0 tx=0x7f9a14004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:52.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a1a000640 1 -- 172.21.15.1:0/4104047633 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1403f030 con 0x7f9a1c108920 2024-04-26T21:39:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a1a000640 1 -- 172.21.15.1:0/4104047633 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a14004440 con 0x7f9a1c108920 2024-04-26T21:39:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a23e00640 1 -- 172.21.15.1:0/4104047633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a1c1a0540 con 0x7f9a1c108920 2024-04-26T21:39:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a1a000640 1 -- 172.21.15.1:0/4104047633 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14035070 con 0x7f9a1c108920 2024-04-26T21:39:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.374+0000 7f9a23e00640 1 -- 172.21.15.1:0/4104047633 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a1c1a7e30 con 0x7f9a1c108920 2024-04-26T21:39:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.378+0000 7f9a26beb740 1 Processor -- start 2024-04-26T21:39:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.378+0000 7f9a1a000640 1 -- 172.21.15.1:0/4104047633 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f9a14042030 con 0x7f9a1c108920 2024-04-26T21:39:52.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a26beb740 1 -- start start 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 0x55b007e551b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e5fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 0x55b007e623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a26beb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b007dba380 con 0x55b007e5ffb0 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a26beb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b007d56c30 con 0x55b007e556f0 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a26beb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b007db6340 con 0x55b007e54dd0 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a21400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e5fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a21400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e5fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59582/0 (socket says 172.21.15.1:59582) 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a21400640 1 -- 172.21.15.1:0/424016897 learned_addr learned my addr 172.21.15.1:0/424016897 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.379+0000 7f9a20a00640 1 --2- 172.21.15.1:0/424016897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 0x55b007e551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a20a00640 1 --2- 172.21.15.1:0/424016897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 0x55b007e623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a21400640 1 -- 172.21.15.1:0/424016897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 msgr2=0x55b007e551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a21400640 1 --2- 172.21.15.1:0/424016897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 0x55b007e551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a21400640 1 -- 172.21.15.1:0/424016897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 msgr2=0x55b007e623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a21400640 1 --2- 172.21.15.1:0/424016897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 0x55b007e623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a21400640 1 -- 172.21.15.1:0/424016897 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b007d78940 con 0x55b007e556f0 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a20a00640 1 --2- 172.21.15.1:0/424016897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 0x55b007e623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a21400640 1 --2- 172.21.15.1:0/424016897 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e5fa70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9a10005f40 tx=0x7f9a10011cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28dad912cc57e40d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a1a000640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a040a3d90 0x7f9a040a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a0e600640 1 -- 172.21.15.1:0/424016897 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10011220 con 0x55b007e556f0 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a1a000640 1 -- 172.21.15.1:0/4104047633 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7f9a140f0e00 con 0x7f9a1c108920 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a0e600640 1 -- 172.21.15.1:0/424016897 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a1000f9a0 con 0x55b007e556f0 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a0e600640 1 -- 172.21.15.1:0/424016897 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10004570 con 0x55b007e556f0 2024-04-26T21:39:52.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.380+0000 7f9a20a00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a040a3d90 0x7f9a040a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.381+0000 7f9a26beb740 1 -- 172.21.15.1:0/424016897 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 msgr2=0x55b007e5fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.381+0000 7f9a26beb740 1 --2- 172.21.15.1:0/424016897 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e5fa70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9a10005f40 tx=0x7f9a10011cb0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.381+0000 7f9a20a00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a040a3d90 0x7f9a040a6250 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55b007e52700 tx=0x7f9a08004730 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.381+0000 7f9a26beb740 1 -- 172.21.15.1:0/424016897 shutdown_connections 2024-04-26T21:39:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.381+0000 7f9a26beb740 1 --2- 172.21.15.1:0/424016897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 0x55b007e623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.381+0000 7f9a26beb740 1 --2- 172.21.15.1:0/424016897 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e5fa70 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.381+0000 7f9a26beb740 1 --2- 172.21.15.1:0/424016897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 0x55b007e551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.381+0000 7f9a26beb740 1 -- 172.21.15.1:0/424016897 >> 172.21.15.1:0/424016897 conn(0x55b007e512e0 msgr2=0x55b007e516c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.381+0000 7f9a26beb740 1 -- 172.21.15.1:0/424016897 shutdown_connections 2024-04-26T21:39:52.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.381+0000 7f9a26beb740 1 -- 172.21.15.1:0/424016897 wait complete. 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.382+0000 7f9a26beb740 1 -- start start 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.382+0000 7f9a26beb740 1 Processor -- start 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e54dd0 0x55b007e5ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e556f0 0x55b007e5d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 0x55b007e5d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a26beb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b007e62be0 con 0x55b007e5ffb0 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a26beb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b007d41c60 con 0x55b007e54dd0 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a26beb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b007e62d60 con 0x55b007e556f0 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a26beb740 10 client.0 _ll_register_callbacks cb 0x55b007d6de50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a21400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e54dd0 0x55b007e5ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a21400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e54dd0 0x55b007e5ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59592/0 (socket says 172.21.15.1:59592) 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a21e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 0x55b007e5d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a21400640 1 -- 172.21.15.1:0/2734552675 learned_addr learned my addr 172.21.15.1:0/2734552675 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:52.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.383+0000 7f9a20a00640 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e556f0 0x55b007e5d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a21400640 1 -- 172.21.15.1:0/2734552675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e556f0 msgr2=0x55b007e5d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a21400640 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e556f0 0x55b007e5d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a21400640 1 -- 172.21.15.1:0/2734552675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 msgr2=0x55b007e5d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a21400640 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 0x55b007e5d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a21400640 1 -- 172.21.15.1:0/2734552675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b007d50510 con 0x55b007e54dd0 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a21e00640 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 0x55b007e5d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a21400640 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e54dd0 0x55b007e5ce30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9a10005f10 tx=0x7f9a10011270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51f2b56862fe445d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a02a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a02a00640 1 -- 172.21.15.1:0/2734552675 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1001e040 con 0x55b007e54dd0 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a02a00640 1 -- 172.21.15.1:0/2734552675 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7f9a10004c40 con 0x55b007e54dd0 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.384+0000 7f9a26beb740 1 -- 172.21.15.1:0/2734552675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a1000f9a0 con 0x55b007e54dd0 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a02a00640 1 -- 172.21.15.1:0/2734552675 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9a10031050 con 0x55b007e54dd0 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a26beb740 10 client.26282 fetch_fsmap learned FSMap version 36 2024-04-26T21:39:52.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a26beb740 1 -- 172.21.15.1:0/2734552675 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b007acc210 con 0x55b007e54dd0 2024-04-26T21:39:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a02a00640 1 -- 172.21.15.1:0/2734552675 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a1000f9a0 con 0x55b007e54dd0 2024-04-26T21:39:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a02a00640 1 -- 172.21.15.1:0/2734552675 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f9a10018070 con 0x55b007e54dd0 2024-04-26T21:39:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a26beb740 10 client.26282 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:39:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a26beb740 10 client.26282 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:39:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a26beb740 20 client.26282 populate_metadata read hostname 'smithi001' 2024-04-26T21:39:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b007e6fb10 0x55b007e71f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a26beb740 4 client.26282 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:39:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.385+0000 7f9a26beb740 1 -- 172.21.15.1:0/2734552675 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b007dc7f90 con 0x55b007e6fb10 2024-04-26T21:39:52.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.386+0000 7f9a20a00640 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b007e6fb10 0x55b007e71f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.386+0000 7f9a20a00640 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b007e6fb10 0x55b007e71f00 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57b575a8b5aade6b server_cookie=86403e4cae757cdd in_seq=0 out_seq=0 2024-04-26T21:39:52.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.386+0000 7f9a02a00640 10 client.26282 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:52.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.387+0000 7f9a02a00640 1 -- 172.21.15.1:0/2734552675 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b007dc7f90 con 0x55b007e6fb10 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.387+0000 7f9a02a00640 10 client.26282 handle_command_reply: tid=0 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.387+0000 7f9a26beb740 1 client.26282 shutdown 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.387+0000 7f9a26beb740 1 -- 172.21.15.1:0/2734552675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e54dd0 msgr2=0x55b007e5ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.387+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e54dd0 0x55b007e5ce30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9a10005f10 tx=0x7f9a10011270 comp rx=0 tx=0).stop 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.388+0000 7f9a26beb740 1 -- 172.21.15.1:0/2734552675 shutdown_connections 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.388+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b007e6fb10 0x55b007e71f00 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.388+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e556f0 0x55b007e5d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.388+0000 7f9a02a00640 0 client.26282 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.388+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e5ffb0 0x55b007e5d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.388+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2734552675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e54dd0 0x55b007e5ce30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.388+0000 7f9a26beb740 1 -- 172.21.15.1:0/2734552675 >> 172.21.15.1:0/2734552675 conn(0x55b007e512e0 msgr2=0x55b007e57010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.388+0000 7f9a26beb740 1 -- 172.21.15.1:0/2734552675 shutdown_connections 2024-04-26T21:39:52.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.388+0000 7f9a26beb740 1 -- 172.21.15.1:0/2734552675 wait complete. 2024-04-26T21:39:52.392 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.388+0000 7f9a26beb740 20 client.26282 trim_cache size 0 max 16384 2024-04-26T21:39:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a26beb740 1 Processor -- start 2024-04-26T21:39:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a26beb740 1 -- start start 2024-04-26T21:39:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e54dd0 0x55b007e5bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e56e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e5ffb0 0x55b007e77700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a26beb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c1b4a20 con 0x55b007e54dd0 2024-04-26T21:39:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a26beb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c1b22f0 con 0x55b007e556f0 2024-04-26T21:39:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a26beb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c1b0160 con 0x55b007e5ffb0 2024-04-26T21:39:52.408 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a20a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e56e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a20a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e56e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59606/0 (socket says 172.21.15.1:59606) 2024-04-26T21:39:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a20a00640 1 -- 172.21.15.1:0/1687597921 learned_addr learned my addr 172.21.15.1:0/1687597921 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.407+0000 7f9a21400640 1 --2- 172.21.15.1:0/1687597921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e54dd0 0x55b007e5bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.408+0000 7f9a21e00640 1 --2- 172.21.15.1:0/1687597921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e5ffb0 0x55b007e77700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.408+0000 7f9a20a00640 1 -- 172.21.15.1:0/1687597921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e5ffb0 msgr2=0x55b007e77700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:52.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.408+0000 7f9a20a00640 1 --2- 172.21.15.1:0/1687597921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e5ffb0 0x55b007e77700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.408+0000 7f9a20a00640 1 -- 172.21.15.1:0/1687597921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e54dd0 msgr2=0x55b007e5bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.408+0000 7f9a20a00640 1 --2- 172.21.15.1:0/1687597921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e54dd0 0x55b007e5bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.408+0000 7f9a20a00640 1 -- 172.21.15.1:0/1687597921 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b007b98620 con 0x55b007e556f0 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.408+0000 7f9a21e00640 1 --2- 172.21.15.1:0/1687597921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e5ffb0 0x55b007e77700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.408+0000 7f9a20a00640 1 --2- 172.21.15.1:0/1687597921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e56e90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x55b007e52470 tx=0x7f9a08018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=952defbca0f7268d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.409+0000 7f9a0dc00640 1 -- 172.21.15.1:0/1687597921 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a08004310 con 0x55b007e556f0 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.409+0000 7f9a0dc00640 1 -- 172.21.15.1:0/1687597921 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a080044b0 con 0x55b007e556f0 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.409+0000 7f9a0dc00640 1 -- 172.21.15.1:0/1687597921 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0802a680 con 0x55b007e556f0 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 -- 172.21.15.1:0/1687597921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 msgr2=0x55b007e56e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 --2- 172.21.15.1:0/1687597921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e56e90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x55b007e52470 tx=0x7f9a08018000 comp rx=0 tx=0).stop 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 -- 172.21.15.1:0/1687597921 shutdown_connections 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 --2- 172.21.15.1:0/1687597921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e5ffb0 0x55b007e77700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 --2- 172.21.15.1:0/1687597921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e556f0 0x55b007e56e90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 --2- 172.21.15.1:0/1687597921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e54dd0 0x55b007e5bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 -- 172.21.15.1:0/1687597921 >> 172.21.15.1:0/1687597921 conn(0x55b007e512e0 msgr2=0x55b007e56780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 -- 172.21.15.1:0/1687597921 shutdown_connections 2024-04-26T21:39:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 -- 172.21.15.1:0/1687597921 wait complete. 2024-04-26T21:39:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 -- start start 2024-04-26T21:39:52.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.410+0000 7f9a26beb740 1 Processor -- start 2024-04-26T21:39:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 0x55b007e594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e556f0 0x55b007e59a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a26beb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e5ffb0 0x55b007e66fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a26beb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c1b07b0 con 0x55b007e556f0 2024-04-26T21:39:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a26beb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c1b1c20 con 0x55b007e5ffb0 2024-04-26T21:39:52.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a26beb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1c1b0e80 con 0x55b007e54dd0 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a26beb740 10 client.0 _ll_register_callbacks cb 0x7f9a100116a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a21400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 0x55b007e594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a20a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e556f0 0x55b007e59a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a21400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 0x55b007e594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:57858/0 (socket says 172.21.15.1:57858) 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a21400640 1 -- 172.21.15.1:0/2217116529 learned_addr learned my addr 172.21.15.1:0/2217116529 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a21e00640 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e5ffb0 0x55b007e66fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a21400640 1 -- 172.21.15.1:0/2217116529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e5ffb0 msgr2=0x55b007e66fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.411+0000 7f9a21400640 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e5ffb0 0x55b007e66fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.412+0000 7f9a21400640 1 -- 172.21.15.1:0/2217116529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e556f0 msgr2=0x55b007e59a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.412+0000 7f9a21400640 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e556f0 0x55b007e59a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.412+0000 7f9a21400640 1 -- 172.21.15.1:0/2217116529 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b007d74cb0 con 0x55b007e54dd0 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.412+0000 7f9a21400640 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 0x55b007e594d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a10005d70 tx=0x7f9a1000ffc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6645c7af455bf6bc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.412+0000 7f9a02a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:39:52.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.412+0000 7f9a02a00640 1 -- 172.21.15.1:0/2217116529 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1003a060 con 0x55b007e54dd0 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.412+0000 7f9a02a00640 1 -- 172.21.15.1:0/2217116529 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7f9a10004340 con 0x55b007e54dd0 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.412+0000 7f9a26beb740 1 -- 172.21.15.1:0/2217116529 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a080044b0 con 0x55b007e54dd0 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a02a00640 1 -- 172.21.15.1:0/2217116529 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9a10011460 con 0x55b007e54dd0 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a26beb740 10 client.26263 fetch_fsmap learned FSMap version 36 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a26beb740 1 -- 172.21.15.1:0/2217116529 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b007e684a0 con 0x55b007e54dd0 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a02a00640 1 -- 172.21.15.1:0/2217116529 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a080044b0 con 0x55b007e54dd0 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a02a00640 1 -- 172.21.15.1:0/2217116529 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f9a10018040 con 0x55b007e54dd0 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a26beb740 10 client.26263 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a26beb740 10 client.26263 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a26beb740 20 client.26263 populate_metadata read hostname 'smithi001' 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b007ed6030 0x55b007ed8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a26beb740 4 client.26263 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.413+0000 7f9a26beb740 1 -- 172.21.15.1:0/2217116529 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b007dc1e00 con 0x55b007ed6030 2024-04-26T21:39:52.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.414+0000 7f9a20a00640 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b007ed6030 0x55b007ed8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:39:52.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.414+0000 7f9a20a00640 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b007ed6030 0x55b007ed8420 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2936d7937a2b78b6 server_cookie=66f365accf645b35 in_seq=0 out_seq=0 2024-04-26T21:39:52.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.414+0000 7f9a02a00640 10 client.26263 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:52 smithi186 ceph-mon[32991]: pgmap v295: 129 pgs: 129 active+clean; 233 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 227 op/s 2024-04-26T21:39:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.480+0000 7f9a02a00640 1 -- 172.21.15.1:0/2217116529 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b007dc1e00 con 0x55b007ed6030 2024-04-26T21:39:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.480+0000 7f9a02a00640 10 client.26263 handle_command_reply: tid=0 2024-04-26T21:39:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.480+0000 7f9a26beb740 1 client.26263 shutdown 2024-04-26T21:39:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 1 -- 172.21.15.1:0/2217116529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 msgr2=0x55b007e594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 0x55b007e594d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9a10005d70 tx=0x7f9a1000ffc0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 1 -- 172.21.15.1:0/2217116529 shutdown_connections 2024-04-26T21:39:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b007ed6030 0x55b007ed8420 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b007e5ffb0 0x55b007e66fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b007e556f0 0x55b007e59a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 1 --2- 172.21.15.1:0/2217116529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b007e54dd0 0x55b007e594d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a02a00640 0 client.26263 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:39:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 1 -- 172.21.15.1:0/2217116529 >> 172.21.15.1:0/2217116529 conn(0x55b007e512e0 msgr2=0x55b007e57010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 1 -- 172.21.15.1:0/2217116529 shutdown_connections 2024-04-26T21:39:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 1 -- 172.21.15.1:0/2217116529 wait complete. 2024-04-26T21:39:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.481+0000 7f9a26beb740 20 client.26263 trim_cache size 0 max 16384 2024-04-26T21:39:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:39:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:39:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:39:52.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:39:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 -- 172.21.15.1:0/4104047633 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a040a3d90 msgr2=0x7f9a040a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a040a3d90 0x7f9a040a6250 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55b007e52700 tx=0x7f9a08004730 comp rx=0 tx=0).stop 2024-04-26T21:39:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 -- 172.21.15.1:0/4104047633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a1c108920 msgr2=0x7f9a1c19fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:39:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a1c108920 0x7f9a1c19fb80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f9a14005ab0 tx=0x7f9a14004040 comp rx=0 tx=0).stop 2024-04-26T21:39:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 -- 172.21.15.1:0/4104047633 shutdown_connections 2024-04-26T21:39:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a040a3d90 0x7f9a040a6250 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1c1032d0 0x7f9a1c19b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a1c108920 0x7f9a1c19fb80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 --2- 172.21.15.1:0/4104047633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a1c102910 0x7f9a1c19b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:39:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 -- 172.21.15.1:0/4104047633 >> 172.21.15.1:0/4104047633 conn(0x7f9a1c0fe400 msgr2=0x7f9a1c10e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:39:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 -- 172.21.15.1:0/4104047633 shutdown_connections 2024-04-26T21:39:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:39:52.482+0000 7f9a23e00640 1 -- 172.21.15.1:0/4104047633 wait complete. 2024-04-26T21:39:52.586 INFO:tasks.workunit.client.0.smithi001.stdout: 1 361789 4.56 MB/sec execute 350 sec latency 43.539 ms 2024-04-26T21:39:53.586 INFO:tasks.workunit.client.0.smithi001.stdout: 1 362186 4.55 MB/sec execute 351 sec latency 27.115 ms 2024-04-26T21:39:54.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:54 smithi001 ceph-mon[28054]: pgmap v296: 129 pgs: 129 active+clean; 233 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-04-26T21:39:54.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:54 smithi038 ceph-mon[32724]: pgmap v296: 129 pgs: 129 active+clean; 233 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-04-26T21:39:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:54 smithi186 ceph-mon[32991]: pgmap v296: 129 pgs: 129 active+clean; 233 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-04-26T21:39:54.586 INFO:tasks.workunit.client.0.smithi001.stdout: 1 362601 4.54 MB/sec execute 352 sec latency 24.422 ms 2024-04-26T21:39:55.587 INFO:tasks.workunit.client.0.smithi001.stdout: 1 364152 4.55 MB/sec execute 353 sec latency 38.960 ms 2024-04-26T21:39:56.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:56 smithi038 ceph-mon[32724]: pgmap v297: 129 pgs: 129 active+clean; 229 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 216 op/s 2024-04-26T21:39:56.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:56 smithi186 ceph-mon[32991]: pgmap v297: 129 pgs: 129 active+clean; 229 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 216 op/s 2024-04-26T21:39:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:56 smithi001 ceph-mon[28054]: pgmap v297: 129 pgs: 129 active+clean; 229 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 216 op/s 2024-04-26T21:39:56.587 INFO:tasks.workunit.client.0.smithi001.stdout: 1 365001 4.55 MB/sec execute 354 sec latency 26.664 ms 2024-04-26T21:39:57.587 INFO:tasks.workunit.client.0.smithi001.stdout: 1 365654 4.55 MB/sec execute 355 sec latency 25.887 ms 2024-04-26T21:39:58.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:39:58 smithi038 ceph-mon[32724]: pgmap v298: 129 pgs: 129 active+clean; 229 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-26T21:39:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:39:58 smithi186 ceph-mon[32991]: pgmap v298: 129 pgs: 129 active+clean; 229 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-26T21:39:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:39:58 smithi001 ceph-mon[28054]: pgmap v298: 129 pgs: 129 active+clean; 229 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-26T21:39:58.587 INFO:tasks.workunit.client.0.smithi001.stdout: 1 365951 4.54 MB/sec execute 356 sec latency 25.599 ms 2024-04-26T21:39:59.587 INFO:tasks.workunit.client.0.smithi001.stdout: 1 366765 4.54 MB/sec execute 357 sec latency 25.176 ms 2024-04-26T21:40:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:00 smithi186 ceph-mon[32991]: pgmap v299: 129 pgs: 129 active+clean; 229 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-26T21:40:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:00 smithi186 ceph-mon[32991]: overall HEALTH_OK 2024-04-26T21:40:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:00 smithi001 ceph-mon[28054]: pgmap v299: 129 pgs: 129 active+clean; 229 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-26T21:40:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:00 smithi001 ceph-mon[28054]: overall HEALTH_OK 2024-04-26T21:40:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:00 smithi038 ceph-mon[32724]: pgmap v299: 129 pgs: 129 active+clean; 229 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-04-26T21:40:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:00 smithi038 ceph-mon[32724]: overall HEALTH_OK 2024-04-26T21:40:00.587 INFO:tasks.workunit.client.0.smithi001.stdout: 1 367946 4.54 MB/sec execute 358 sec latency 42.360 ms 2024-04-26T21:40:01.587 INFO:tasks.workunit.client.0.smithi001.stdout: 1 368594 4.54 MB/sec execute 359 sec latency 99.904 ms 2024-04-26T21:40:02.457 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:02 smithi186 ceph-mon[32991]: pgmap v300: 129 pgs: 129 active+clean; 231 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 203 op/s 2024-04-26T21:40:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:02 smithi001 ceph-mon[28054]: pgmap v300: 129 pgs: 129 active+clean; 231 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 203 op/s 2024-04-26T21:40:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:02 smithi038 ceph-mon[32724]: pgmap v300: 129 pgs: 129 active+clean; 231 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 203 op/s 2024-04-26T21:40:02.587 INFO:tasks.workunit.client.0.smithi001.stdout: 1 369054 4.54 MB/sec execute 360 sec latency 88.018 ms 2024-04-26T21:40:03.587 INFO:tasks.workunit.client.0.smithi001.stdout: 1 369305 4.53 MB/sec execute 361 sec latency 113.873 ms 2024-04-26T21:40:04.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:04 smithi186 ceph-mon[32991]: pgmap v301: 129 pgs: 129 active+clean; 231 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-26T21:40:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:04 smithi001 ceph-mon[28054]: pgmap v301: 129 pgs: 129 active+clean; 231 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-26T21:40:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:04 smithi038 ceph-mon[32724]: pgmap v301: 129 pgs: 129 active+clean; 231 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.1 MiB/s wr, 141 op/s 2024-04-26T21:40:04.587 INFO:tasks.workunit.client.0.smithi001.stdout: 1 369807 4.52 MB/sec execute 362 sec latency 25.056 ms 2024-04-26T21:40:05.588 INFO:tasks.workunit.client.0.smithi001.stdout: 1 371244 4.53 MB/sec execute 363 sec latency 38.257 ms 2024-04-26T21:40:05.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.640+0000 7fbfc4c00640 1 -- 172.21.15.1:0/2135881908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfc010cf70 msgr2=0x7fbfc010f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:05.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.640+0000 7fbfc4c00640 1 --2- 172.21.15.1:0/2135881908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfc010cf70 0x7fbfc010f3c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009ca0 tx=0x7fbfb802a950 comp rx=0 tx=0).stop 2024-04-26T21:40:05.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.640+0000 7fbfc4c00640 1 -- 172.21.15.1:0/2135881908 shutdown_connections 2024-04-26T21:40:05.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.640+0000 7fbfc4c00640 1 --2- 172.21.15.1:0/2135881908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfc010cf70 0x7fbfc010f3c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.640+0000 7fbfc4c00640 1 --2- 172.21.15.1:0/2135881908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfc01044c0 0x7fbfc010c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.640+0000 7fbfc4c00640 1 --2- 172.21.15.1:0/2135881908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc0103ba0 0x7fbfc0103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.640+0000 7fbfc4c00640 1 -- 172.21.15.1:0/2135881908 >> 172.21.15.1:0/2135881908 conn(0x7fbfc0078980 msgr2=0x7fbfc0078d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:05.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.640+0000 7fbfc4c00640 1 -- 172.21.15.1:0/2135881908 shutdown_connections 2024-04-26T21:40:05.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.641+0000 7fbfc4c00640 1 -- 172.21.15.1:0/2135881908 wait complete. 2024-04-26T21:40:05.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfc4c00640 1 Processor -- start 2024-04-26T21:40:05.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfc4c00640 1 -- start start 2024-04-26T21:40:05.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfc4c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfc0103ba0 0x7fbfc01990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfc4c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc01044c0 0x7fbfc0199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfc4c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfc010cf70 0x7fbfc019d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfc4c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc0112c10 con 0x7fbfc01044c0 2024-04-26T21:40:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfc4c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc0112a90 con 0x7fbfc010cf70 2024-04-26T21:40:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfc4c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc0112d90 con 0x7fbfc0103ba0 2024-04-26T21:40:05.645 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfbde00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfc0103ba0 0x7fbfc01990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfbde00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfc0103ba0 0x7fbfc01990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37910/0 (socket says 172.21.15.1:37910) 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.642+0000 7fbfbde00640 1 -- 172.21.15.1:0/3669682542 learned_addr learned my addr 172.21.15.1:0/3669682542 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.643+0000 7fbfbd400640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc01044c0 0x7fbfc0199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.643+0000 7fbfbe800640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfc010cf70 0x7fbfc019d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.643+0000 7fbfbde00640 1 -- 172.21.15.1:0/3669682542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfc010cf70 msgr2=0x7fbfc019d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.643+0000 7fbfbde00640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfc010cf70 0x7fbfc019d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.643+0000 7fbfbde00640 1 -- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc01044c0 msgr2=0x7fbfc0199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.643+0000 7fbfbde00640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc01044c0 0x7fbfc0199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.643+0000 7fbfbde00640 1 -- 172.21.15.1:0/3669682542 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc019e0e0 con 0x7fbfc0103ba0 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.643+0000 7fbfbd400640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc01044c0 0x7fbfc0199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:40:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.643+0000 7fbfbde00640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfc0103ba0 0x7fbfc01990f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000e9f0 tx=0x7fbfb000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.644+0000 7fbfb6a00640 1 -- 172.21.15.1:0/3669682542 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb000cd80 con 0x7fbfc0103ba0 2024-04-26T21:40:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.644+0000 7fbfb6a00640 1 -- 172.21.15.1:0/3669682542 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb00108e0 con 0x7fbfc0103ba0 2024-04-26T21:40:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.644+0000 7fbfb6a00640 1 -- 172.21.15.1:0/3669682542 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0018980 con 0x7fbfc0103ba0 2024-04-26T21:40:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.644+0000 7fbfc4c00640 1 -- 172.21.15.1:0/3669682542 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc019e3d0 con 0x7fbfc0103ba0 2024-04-26T21:40:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.644+0000 7fbfc4c00640 1 -- 172.21.15.1:0/3669682542 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc01a5c60 con 0x7fbfc0103ba0 2024-04-26T21:40:05.647 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.647+0000 7fbfb6a00640 1 -- 172.21.15.1:0/3669682542 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fbfb0010430 con 0x7fbfc0103ba0 2024-04-26T21:40:05.649 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.648+0000 7fbfc79e7740 1 Processor -- start 2024-04-26T21:40:05.649 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.649+0000 7fbfc79e7740 1 -- start start 2024-04-26T21:40:05.650 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.649+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b5a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.649+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b68e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.649+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b6b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.649+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d0e4abf380 con 0x55d0e4b693c0 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.649+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d0e4a5bc30 con 0x55d0e4b59cd0 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.649+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d0e4abb340 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.649+0000 7fbfbe800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b5a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfbe800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b5a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53692/0 (socket says 172.21.15.1:53692) 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfbe800640 1 -- 172.21.15.1:0/69833716 learned_addr learned my addr 172.21.15.1:0/69833716 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfb6a00640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbfa40a3d10 0x7fbfa40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfbde00640 1 --2- 172.21.15.1:0/69833716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b68e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfbd400640 1 --2- 172.21.15.1:0/69833716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b6b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfb6a00640 1 -- 172.21.15.1:0/3669682542 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7fbfb00d0ff0 con 0x7fbfc0103ba0 2024-04-26T21:40:05.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfbe800640 1 -- 172.21.15.1:0/69833716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 msgr2=0x55d0e4b68e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfbe800640 1 --2- 172.21.15.1:0/69833716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b68e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfbe800640 1 -- 172.21.15.1:0/69833716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 msgr2=0x55d0e4b6b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfbe800640 1 --2- 172.21.15.1:0/69833716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b6b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfbe800640 1 -- 172.21.15.1:0/69833716 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0e4a7d940 con 0x55d0e4b59cd0 2024-04-26T21:40:05.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.650+0000 7fbfbe800640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbfa40a3d10 0x7fbfa40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.651+0000 7fbfbe800640 1 --2- 172.21.15.1:0/69833716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b5a0b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009ca0 tx=0x7fbfb802f410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd1a70d25163b739 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.651+0000 7fbfab000640 1 -- 172.21.15.1:0/69833716 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8034040 con 0x55d0e4b59cd0 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.651+0000 7fbfab000640 1 -- 172.21.15.1:0/69833716 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb8004040 con 0x55d0e4b59cd0 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.651+0000 7fbfab000640 1 -- 172.21.15.1:0/69833716 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8046690 con 0x55d0e4b59cd0 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.651+0000 7fbfbe800640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbfa40a3d10 0x7fbfa40a61d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbfb804d000 tx=0x7fbfb804e000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.652+0000 7fbfc79e7740 1 -- 172.21.15.1:0/69833716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 msgr2=0x55d0e4b5a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.652+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/69833716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b5a0b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009ca0 tx=0x7fbfb802f410 comp rx=0 tx=0).stop 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.652+0000 7fbfc79e7740 1 -- 172.21.15.1:0/69833716 shutdown_connections 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.652+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/69833716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b68e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.652+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/69833716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b6b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.652+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/69833716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b5a0b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.652+0000 7fbfc79e7740 1 -- 172.21.15.1:0/69833716 >> 172.21.15.1:0/69833716 conn(0x55d0e4b562e0 msgr2=0x55d0e4b566c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.652+0000 7fbfc79e7740 1 -- 172.21.15.1:0/69833716 shutdown_connections 2024-04-26T21:40:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.652+0000 7fbfc79e7740 1 -- 172.21.15.1:0/69833716 wait complete. 2024-04-26T21:40:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.653+0000 7fbfc79e7740 1 -- start start 2024-04-26T21:40:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.653+0000 7fbfc79e7740 1 Processor -- start 2024-04-26T21:40:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b66310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b66850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b66d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d0e4b6bff0 con 0x55d0e4b693c0 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d0e4a46c60 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d0e4b6c170 con 0x55d0e4b59cd0 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfc79e7740 10 client.0 _ll_register_callbacks cb 0x55d0e4a72e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfbd400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b66850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfbd400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b66850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53696/0 (socket says 172.21.15.1:53696) 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfbde00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b66310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfbd400640 1 -- 172.21.15.1:0/1980416355 learned_addr learned my addr 172.21.15.1:0/1980416355 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.654+0000 7fbfbe800640 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b66d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.655+0000 7fbfbd400640 1 -- 172.21.15.1:0/1980416355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 msgr2=0x55d0e4b66310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.655+0000 7fbfbd400640 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b66310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.655+0000 7fbfbd400640 1 -- 172.21.15.1:0/1980416355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 msgr2=0x55d0e4b66d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.655+0000 7fbfbd400640 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b66d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.655+0000 7fbfbd400640 1 -- 172.21.15.1:0/1980416355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0e4a55510 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.655+0000 7fbfbde00640 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b66310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.655+0000 7fbfbd400640 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b66850 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbfac013510 tx=0x7fbfac0139d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b671a1ece2b7ddd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.655+0000 7fbf9f400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.656+0000 7fbf9f400640 1 -- 172.21.15.1:0/1980416355 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac011970 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.656+0000 7fbfc79e7740 1 -- 172.21.15.1:0/1980416355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfb8004040 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.656+0000 7fbf9f400640 1 -- 172.21.15.1:0/1980416355 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7fbfac011b10 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.656+0000 7fbf9f400640 1 -- 172.21.15.1:0/1980416355 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbfac028050 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.656+0000 7fbfc79e7740 10 client.26300 fetch_fsmap learned FSMap version 36 2024-04-26T21:40:05.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.656+0000 7fbfc79e7740 1 -- 172.21.15.1:0/1980416355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d0e47d1210 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.657+0000 7fbf9f400640 1 -- 172.21.15.1:0/1980416355 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb8004040 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.657+0000 7fbf9f400640 1 -- 172.21.15.1:0/1980416355 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fbfac01d3f0 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.657+0000 7fbfc79e7740 10 client.26300 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:40:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.657+0000 7fbfc79e7740 10 client.26300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:40:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.657+0000 7fbfc79e7740 20 client.26300 populate_metadata read hostname 'smithi001' 2024-04-26T21:40:05.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.657+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d0e4b791b0 0x55d0e4b7b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.657+0000 7fbfc79e7740 4 client.26300 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.657+0000 7fbfc79e7740 1 -- 172.21.15.1:0/1980416355 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0e4accf90 con 0x55d0e4b791b0 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.657+0000 7fbfbde00640 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d0e4b791b0 0x55d0e4b7b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.658+0000 7fbfbde00640 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d0e4b791b0 0x55d0e4b7b600 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1fca17b99ac9a24 server_cookie=6a860c9863336318 in_seq=0 out_seq=0 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.658+0000 7fbf9f400640 10 client.26300 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbf9f400640 1 -- 172.21.15.1:0/1980416355 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d0e4accf90 con 0x55d0e4b791b0 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbf9f400640 10 client.26300 handle_command_reply: tid=0 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbfc79e7740 1 client.26300 shutdown 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbfc79e7740 1 -- 172.21.15.1:0/1980416355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b5a5f0 msgr2=0x55d0e4b66850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b66850 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbfac013510 tx=0x7fbfac0139d0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbfc79e7740 1 -- 172.21.15.1:0/1980416355 shutdown_connections 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d0e4b791b0 0x55d0e4b7b600 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b66d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b66850 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/1980416355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b66310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbfc79e7740 1 -- 172.21.15.1:0/1980416355 >> 172.21.15.1:0/1980416355 conn(0x55d0e4b562e0 msgr2=0x55d0e4b5c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbf9f400640 0 client.26300 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.659+0000 7fbfc79e7740 1 -- 172.21.15.1:0/1980416355 shutdown_connections 2024-04-26T21:40:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.660+0000 7fbfc79e7740 1 -- 172.21.15.1:0/1980416355 wait complete. 2024-04-26T21:40:05.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.660+0000 7fbfc79e7740 20 client.26300 trim_cache size 0 max 16384 2024-04-26T21:40:05.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.678+0000 7fbfc79e7740 1 Processor -- start 2024-04-26T21:40:05.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.679+0000 7fbfc79e7740 1 -- start start 2024-04-26T21:40:05.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.679+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b74eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.683 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.679+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b80f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.684 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.679+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b81450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.684 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.679+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc01b0660 con 0x55d0e4b693c0 2024-04-26T21:40:05.684 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.679+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc01adf30 con 0x55d0e4b59cd0 2024-04-26T21:40:05.684 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.679+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc01abda0 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.684 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.679+0000 7fbfbde00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b74eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.684 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.679+0000 7fbfbde00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b74eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:53698/0 (socket says 172.21.15.1:53698) 2024-04-26T21:40:05.684 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.679+0000 7fbfbde00640 1 -- 172.21.15.1:0/3101041726 learned_addr learned my addr 172.21.15.1:0/3101041726 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:05.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.680+0000 7fbfbd400640 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b80f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.680+0000 7fbfbe800640 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b81450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.680+0000 7fbfbde00640 1 -- 172.21.15.1:0/3101041726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 msgr2=0x55d0e4b80f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:05.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.680+0000 7fbfbde00640 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b80f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.680+0000 7fbfbde00640 1 -- 172.21.15.1:0/3101041726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 msgr2=0x55d0e4b81450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:05.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.680+0000 7fbfbde00640 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b81450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.680+0000 7fbfbde00640 1 -- 172.21.15.1:0/3101041726 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0e489d620 con 0x55d0e4b59cd0 2024-04-26T21:40:05.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.680+0000 7fbfbd400640 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b80f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:40:05.685 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.680+0000 7fbfbe800640 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b81450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:40:05.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.680+0000 7fbfbde00640 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b74eb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000c6a0 tx=0x7fbfb00bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=224e31e7ee2862b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:05.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.681+0000 7fbfaa600640 1 -- 172.21.15.1:0/3101041726 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00a0ee0 con 0x55d0e4b59cd0 2024-04-26T21:40:05.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.681+0000 7fbfaa600640 1 -- 172.21.15.1:0/3101041726 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb00a57a0 con 0x55d0e4b59cd0 2024-04-26T21:40:05.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.681+0000 7fbfaa600640 1 -- 172.21.15.1:0/3101041726 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb00991a0 con 0x55d0e4b59cd0 2024-04-26T21:40:05.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.682+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3101041726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 msgr2=0x55d0e4b74eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:05.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.682+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b74eb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000c6a0 tx=0x7fbfb00bd040 comp rx=0 tx=0).stop 2024-04-26T21:40:05.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.682+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3101041726 shutdown_connections 2024-04-26T21:40:05.686 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.682+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b80f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.687 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.682+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b693c0 0x55d0e4b81450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.687 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.682+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/3101041726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b74eb0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.687 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.682+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3101041726 >> 172.21.15.1:0/3101041726 conn(0x55d0e4b562e0 msgr2=0x55d0e4b64c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:05.687 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.682+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3101041726 shutdown_connections 2024-04-26T21:40:05.687 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.682+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3101041726 wait complete. 2024-04-26T21:40:05.687 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.683+0000 7fbfc79e7740 1 -- start start 2024-04-26T21:40:05.687 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.683+0000 7fbfc79e7740 1 Processor -- start 2024-04-26T21:40:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.683+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b63040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.683+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b63580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.683+0000 7fbfc79e7740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b693c0 0x55d0e4b63ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc01ac3f0 con 0x55d0e4b5a5f0 2024-04-26T21:40:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc01ad860 con 0x55d0e4b693c0 2024-04-26T21:40:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfc79e7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc01acac0 con 0x55d0e4b59cd0 2024-04-26T21:40:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfc79e7740 10 client.0 _ll_register_callbacks cb 0x7fbfac009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:40:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbde00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b63040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbde00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b63040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37956/0 (socket says 172.21.15.1:37956) 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbe800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b693c0 0x55d0e4b63ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbde00640 1 -- 172.21.15.1:0/3204154393 learned_addr learned my addr 172.21.15.1:0/3204154393 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbd400640 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b63580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbde00640 1 -- 172.21.15.1:0/3204154393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b693c0 msgr2=0x55d0e4b63ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbde00640 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b693c0 0x55d0e4b63ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbde00640 1 -- 172.21.15.1:0/3204154393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b5a5f0 msgr2=0x55d0e4b63580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbde00640 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b63580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbde00640 1 -- 172.21.15.1:0/3204154393 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0e4a79cb0 con 0x55d0e4b59cd0 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbe800640 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b693c0 0x55d0e4b63ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.684+0000 7fbfbd400640 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b63580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.685+0000 7fbfbde00640 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b63040 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00bd010 tx=0x7fbfb0010b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd6c054527ef3825 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.685+0000 7fbf9f400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:40:05.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.685+0000 7fbf9f400640 1 -- 172.21.15.1:0/3204154393 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb009de60 con 0x55d0e4b59cd0 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.685+0000 7fbf9f400640 1 -- 172.21.15.1:0/3204154393 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9642+0+0 (secure 0 0 0) 0x7fbfb00ce560 con 0x55d0e4b59cd0 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.685+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3204154393 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfb00a57a0 con 0x55d0e4b59cd0 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.685+0000 7fbf9f400640 1 -- 172.21.15.1:0/3204154393 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbfb00b1350 con 0x55d0e4b59cd0 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.685+0000 7fbfc79e7740 10 client.26281 fetch_fsmap learned FSMap version 36 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.685+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3204154393 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d0e4b5ca10 con 0x55d0e4b59cd0 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.686+0000 7fbf9f400640 1 -- 172.21.15.1:0/3204154393 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfb00a57a0 con 0x55d0e4b59cd0 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.686+0000 7fbf9f400640 1 -- 172.21.15.1:0/3204154393 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7fbfb00a53a0 con 0x55d0e4b59cd0 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.686+0000 7fbfc79e7740 10 client.26281 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.686+0000 7fbfc79e7740 10 client.26281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.686+0000 7fbfc79e7740 20 client.26281 populate_metadata read hostname 'smithi001' 2024-04-26T21:40:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.686+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d0e4b5d0e0 0x55d0e4b5f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.686+0000 7fbfc79e7740 4 client.26281 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:40:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.686+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3204154393 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d0e4ac6e00 con 0x55d0e4b5d0e0 2024-04-26T21:40:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.686+0000 7fbfbd400640 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d0e4b5d0e0 0x55d0e4b5f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.687+0000 7fbfbd400640 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d0e4b5d0e0 0x55d0e4b5f500 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c4237912d93f39c server_cookie=cf63385038829c57 in_seq=0 out_seq=0 2024-04-26T21:40:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.687+0000 7fbf9f400640 10 client.26281 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.754+0000 7fbf9f400640 1 -- 172.21.15.1:0/3204154393 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d0e4ac6e00 con 0x55d0e4b5d0e0 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.754+0000 7fbf9f400640 10 client.26281 handle_command_reply: tid=0 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.754+0000 7fbfc79e7740 1 client.26281 shutdown 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3204154393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 msgr2=0x55d0e4b63040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b63040 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbfb00bd010 tx=0x7fbfb0010b10 comp rx=0 tx=0).stop 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3204154393 shutdown_connections 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d0e4b5d0e0 0x55d0e4b5f500 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d0e4b693c0 0x55d0e4b63ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d0e4b5a5f0 0x55d0e4b63580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 1 --2- 172.21.15.1:0/3204154393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0e4b59cd0 0x55d0e4b63040 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbf9f400640 0 client.26281 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3204154393 >> 172.21.15.1:0/3204154393 conn(0x55d0e4b562e0 msgr2=0x55d0e4b54bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3204154393 shutdown_connections 2024-04-26T21:40:05.756 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 1 -- 172.21.15.1:0/3204154393 wait complete. 2024-04-26T21:40:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.755+0000 7fbfc79e7740 20 client.26281 trim_cache size 0 max 16384 2024-04-26T21:40:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:40:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:40:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:40:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:40:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 -- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbfa40a3d10 msgr2=0x7fbfa40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbfa40a3d10 0x7fbfa40a61d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbfb804d000 tx=0x7fbfb804e000 comp rx=0 tx=0).stop 2024-04-26T21:40:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 -- 172.21.15.1:0/3669682542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfc0103ba0 msgr2=0x7fbfc01990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfc0103ba0 0x7fbfc01990f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbfb000e9f0 tx=0x7fbfb000eeb0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 -- 172.21.15.1:0/3669682542 shutdown_connections 2024-04-26T21:40:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbfa40a3d10 0x7fbfa40a61d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbfc010cf70 0x7fbfc019d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfc01044c0 0x7fbfc0199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 --2- 172.21.15.1:0/3669682542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfc0103ba0 0x7fbfc01990f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 -- 172.21.15.1:0/3669682542 >> 172.21.15.1:0/3669682542 conn(0x7fbfc0078980 msgr2=0x7fbfc00fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 -- 172.21.15.1:0/3669682542 shutdown_connections 2024-04-26T21:40:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:05.756+0000 7fbfc4c00640 1 -- 172.21.15.1:0/3669682542 wait complete. 2024-04-26T21:40:06.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:06 smithi186 ceph-mon[32991]: pgmap v302: 129 pgs: 129 active+clean; 234 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 235 op/s 2024-04-26T21:40:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:06 smithi001 ceph-mon[28054]: pgmap v302: 129 pgs: 129 active+clean; 234 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 235 op/s 2024-04-26T21:40:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:06 smithi038 ceph-mon[32724]: pgmap v302: 129 pgs: 129 active+clean; 234 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 235 op/s 2024-04-26T21:40:06.588 INFO:tasks.workunit.client.0.smithi001.stdout: 1 372436 4.54 MB/sec execute 364 sec latency 29.169 ms 2024-04-26T21:40:07.588 INFO:tasks.workunit.client.0.smithi001.stdout: 1 372820 4.54 MB/sec execute 365 sec latency 37.077 ms 2024-04-26T21:40:08.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:08 smithi186 ceph-mon[32991]: pgmap v303: 129 pgs: 129 active+clean; 234 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-04-26T21:40:08.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:08 smithi186 ceph-mon[32991]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T21:40:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:08 smithi001 ceph-mon[28054]: pgmap v303: 129 pgs: 129 active+clean; 234 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-04-26T21:40:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:08 smithi001 ceph-mon[28054]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T21:40:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:08 smithi038 ceph-mon[32724]: pgmap v303: 129 pgs: 129 active+clean; 234 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 1.4 MiB/s wr, 152 op/s 2024-04-26T21:40:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:08 smithi038 ceph-mon[32724]: osdmap e103: 12 total, 12 up, 12 in 2024-04-26T21:40:08.588 INFO:tasks.workunit.client.0.smithi001.stdout: 1 373477 4.53 MB/sec execute 366 sec latency 26.718 ms 2024-04-26T21:40:09.588 INFO:tasks.workunit.client.0.smithi001.stdout: 1 374861 4.53 MB/sec execute 367 sec latency 22.080 ms 2024-04-26T21:40:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:10 smithi186 ceph-mon[32991]: pgmap v305: 129 pgs: 129 active+clean; 234 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 1.7 MiB/s wr, 183 op/s 2024-04-26T21:40:10.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:10 smithi001 ceph-mon[28054]: pgmap v305: 129 pgs: 129 active+clean; 234 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 1.7 MiB/s wr, 183 op/s 2024-04-26T21:40:10.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:10 smithi038 ceph-mon[32724]: pgmap v305: 129 pgs: 129 active+clean; 234 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 1.7 MiB/s wr, 183 op/s 2024-04-26T21:40:10.588 INFO:tasks.workunit.client.0.smithi001.stdout: 1 375958 4.55 MB/sec execute 368 sec latency 42.296 ms 2024-04-26T21:40:11.588 INFO:tasks.workunit.client.0.smithi001.stdout: 1 376413 4.54 MB/sec execute 369 sec latency 63.120 ms 2024-04-26T21:40:12.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:12 smithi186 ceph-mon[32991]: pgmap v306: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.0 MiB/s wr, 218 op/s 2024-04-26T21:40:12.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:12 smithi001 ceph-mon[28054]: pgmap v306: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.0 MiB/s wr, 218 op/s 2024-04-26T21:40:12.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:12 smithi038 ceph-mon[32724]: pgmap v306: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.0 MiB/s wr, 218 op/s 2024-04-26T21:40:12.588 INFO:tasks.workunit.client.0.smithi001.stdout: 1 376568 4.53 MB/sec execute 370 sec latency 617.031 ms 2024-04-26T21:40:13.589 INFO:tasks.workunit.client.0.smithi001.stdout: 1 376568 4.51 MB/sec execute 371 sec latency 1617.176 ms 2024-04-26T21:40:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:14 smithi186 ceph-mon[32991]: pgmap v307: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.0 MiB/s wr, 218 op/s 2024-04-26T21:40:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:14 smithi186 ceph-mon[32991]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T21:40:14.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:14 smithi001 ceph-mon[28054]: pgmap v307: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.0 MiB/s wr, 218 op/s 2024-04-26T21:40:14.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:14 smithi001 ceph-mon[28054]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T21:40:14.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:14 smithi038 ceph-mon[32724]: pgmap v307: 129 pgs: 129 active+clean; 237 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 2.0 MiB/s wr, 218 op/s 2024-04-26T21:40:14.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:14 smithi038 ceph-mon[32724]: osdmap e104: 12 total, 12 up, 12 in 2024-04-26T21:40:14.589 INFO:tasks.workunit.client.0.smithi001.stdout: 1 376568 4.50 MB/sec execute 372 sec latency 2617.323 ms 2024-04-26T21:40:15.589 INFO:tasks.workunit.client.0.smithi001.stdout: 1 376568 4.49 MB/sec execute 373 sec latency 3617.475 ms 2024-04-26T21:40:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:16 smithi186 ceph-mon[32991]: pgmap v309: 129 pgs: 129 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 196 op/s 2024-04-26T21:40:16.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:16 smithi001 ceph-mon[28054]: pgmap v309: 129 pgs: 129 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 196 op/s 2024-04-26T21:40:16.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:16 smithi038 ceph-mon[32724]: pgmap v309: 129 pgs: 129 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.9 MiB/s wr, 196 op/s 2024-04-26T21:40:16.589 INFO:tasks.workunit.client.0.smithi001.stdout: 1 376568 4.48 MB/sec execute 374 sec latency 4617.678 ms 2024-04-26T21:40:17.589 INFO:tasks.workunit.client.0.smithi001.stdout: 1 376568 4.47 MB/sec execute 375 sec latency 5617.821 ms 2024-04-26T21:40:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:18 smithi186 ceph-mon[32991]: pgmap v310: 129 pgs: 129 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-26T21:40:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:18 smithi001 ceph-mon[28054]: pgmap v310: 129 pgs: 129 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-26T21:40:18.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:18 smithi038 ceph-mon[32724]: pgmap v310: 129 pgs: 129 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.6 MiB/s wr, 160 op/s 2024-04-26T21:40:18.589 INFO:tasks.workunit.client.0.smithi001.stdout: 1 376568 4.45 MB/sec execute 376 sec latency 6617.961 ms 2024-04-26T21:40:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.909+0000 7f64dc200640 1 -- 172.21.15.1:0/1820745809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64d406b010 msgr2=0x7f64d406b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.909+0000 7f64dc200640 1 --2- 172.21.15.1:0/1820745809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64d406b010 0x7f64d406b3f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f64c8009ca0 tx=0x7f64c802a950 comp rx=0 tx=0).stop 2024-04-26T21:40:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.909+0000 7f64dc200640 1 -- 172.21.15.1:0/1820745809 shutdown_connections 2024-04-26T21:40:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.909+0000 7f64dc200640 1 --2- 172.21.15.1:0/1820745809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d406b930 0x7f64d410ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.909+0000 7f64dc200640 1 --2- 172.21.15.1:0/1820745809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64d410f340 0x7f64d4111730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.909+0000 7f64dc200640 1 --2- 172.21.15.1:0/1820745809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64d406b010 0x7f64d406b3f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.909+0000 7f64dc200640 1 -- 172.21.15.1:0/1820745809 >> 172.21.15.1:0/1820745809 conn(0x7f64d4078930 msgr2=0x7f64d4078d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.909+0000 7f64dc200640 1 -- 172.21.15.1:0/1820745809 shutdown_connections 2024-04-26T21:40:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.909+0000 7f64dc200640 1 -- 172.21.15.1:0/1820745809 wait complete. 2024-04-26T21:40:18.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.910+0000 7f64dc200640 1 Processor -- start 2024-04-26T21:40:18.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.911+0000 7f64dc200640 1 -- start start 2024-04-26T21:40:18.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.911+0000 7f64dc200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64d406b010 0x7f64d419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.911+0000 7f64dc200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d406b930 0x7f64d419b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.911+0000 7f64dc200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64d410f340 0x7f64d419fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.911+0000 7f64dc200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f64d4114e10 con 0x7f64d410f340 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.911+0000 7f64dc200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f64d4114c90 con 0x7f64d406b010 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.911+0000 7f64dc200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64d4114f90 con 0x7f64d406b930 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64da200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64d410f340 0x7f64d419fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64d9800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64d406b010 0x7f64d419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64da200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64d410f340 0x7f64d419fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44910/0 (socket says 172.21.15.1:44910) 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64da200640 1 -- 172.21.15.1:0/190370584 learned_addr learned my addr 172.21.15.1:0/190370584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64d8e00640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d406b930 0x7f64d419b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64d9800640 1 -- 172.21.15.1:0/190370584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d406b930 msgr2=0x7f64d419b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64d9800640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d406b930 0x7f64d419b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64d9800640 1 -- 172.21.15.1:0/190370584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64d410f340 msgr2=0x7f64d419fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64d9800640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64d410f340 0x7f64d419fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64d9800640 1 -- 172.21.15.1:0/190370584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d41a0280 con 0x7f64d406b010 2024-04-26T21:40:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.912+0000 7f64da200640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64d410f340 0x7f64d419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.913+0000 7f64d9800640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64d406b010 0x7f64d419b290 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f64c8005e30 tx=0x7f64c8003c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.913+0000 7f64d2000640 1 -- 172.21.15.1:0/190370584 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8034040 con 0x7f64d406b010 2024-04-26T21:40:18.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.913+0000 7f64dc200640 1 -- 172.21.15.1:0/190370584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d41a0510 con 0x7f64d406b010 2024-04-26T21:40:18.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.913+0000 7f64d2000640 1 -- 172.21.15.1:0/190370584 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64c8030850 con 0x7f64d406b010 2024-04-26T21:40:18.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.913+0000 7f64d2000640 1 -- 172.21.15.1:0/190370584 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c8004620 con 0x7f64d406b010 2024-04-26T21:40:18.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.913+0000 7f64dc200640 1 -- 172.21.15.1:0/190370584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d41a7e00 con 0x7f64d406b010 2024-04-26T21:40:18.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.917+0000 7f64df007740 1 Processor -- start 2024-04-26T21:40:18.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.917+0000 7f64df007740 1 -- start start 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.917+0000 7f64d2000640 1 -- 172.21.15.1:0/190370584 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f64c8037020 con 0x7f64d406b010 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.917+0000 7f64df007740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927e9dd0 0x55b9927ea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.917+0000 7f64df007740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927ea6f0 0x55b9927f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.917+0000 7f64df007740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 0x55b9927f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.917+0000 7f64df007740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b99274f380 con 0x55b9927ea6f0 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.917+0000 7f64df007740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b9926ebc30 con 0x55b9927e9dd0 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.917+0000 7f64df007740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b99274b340 con 0x55b9927f4fb0 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64d9800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927ea6f0 0x55b9927f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64d9800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927ea6f0 0x55b9927f4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44926/0 (socket says 172.21.15.1:44926) 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64d9800640 1 -- 172.21.15.1:0/2149005152 learned_addr learned my addr 172.21.15.1:0/2149005152 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64da200640 1 --2- 172.21.15.1:0/2149005152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927e9dd0 0x55b9927ea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64d8e00640 1 --2- 172.21.15.1:0/2149005152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 0x55b9927f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64da200640 1 -- 172.21.15.1:0/2149005152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 msgr2=0x55b9927f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:18.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64da200640 1 --2- 172.21.15.1:0/2149005152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 0x55b9927f73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64da200640 1 -- 172.21.15.1:0/2149005152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927ea6f0 msgr2=0x55b9927f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64da200640 1 --2- 172.21.15.1:0/2149005152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927ea6f0 0x55b9927f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64da200640 1 -- 172.21.15.1:0/2149005152 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b99270d940 con 0x55b9927e9dd0 2024-04-26T21:40:18.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.918+0000 7f64d8e00640 1 --2- 172.21.15.1:0/2149005152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 0x55b9927f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.919+0000 7f64da200640 1 --2- 172.21.15.1:0/2149005152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927e9dd0 0x55b9927ea1b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f64cc011540 tx=0x7f64cc011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cca77650da0ea08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.919+0000 7f64c6a00640 1 -- 172.21.15.1:0/2149005152 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc00f680 con 0x55b9927e9dd0 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.919+0000 7f64c6a00640 1 -- 172.21.15.1:0/2149005152 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64cc00fcd0 con 0x55b9927e9dd0 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.919+0000 7f64c6a00640 1 -- 172.21.15.1:0/2149005152 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc019c90 con 0x55b9927e9dd0 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.919+0000 7f64d2000640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f64bc0a3d10 0x7f64bc0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64d2000640 1 -- 172.21.15.1:0/190370584 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f64c80f1860 con 0x7f64d406b010 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64df007740 1 -- 172.21.15.1:0/2149005152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927e9dd0 msgr2=0x55b9927ea1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64df007740 1 --2- 172.21.15.1:0/2149005152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927e9dd0 0x55b9927ea1b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f64cc011540 tx=0x7f64cc011a00 comp rx=0 tx=0).stop 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64d9800640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f64bc0a3d10 0x7f64bc0a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64df007740 1 -- 172.21.15.1:0/2149005152 shutdown_connections 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64df007740 1 --2- 172.21.15.1:0/2149005152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 0x55b9927f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64df007740 1 --2- 172.21.15.1:0/2149005152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927ea6f0 0x55b9927f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64df007740 1 --2- 172.21.15.1:0/2149005152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927e9dd0 0x55b9927ea1b0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64df007740 1 -- 172.21.15.1:0/2149005152 >> 172.21.15.1:0/2149005152 conn(0x55b9927e62e0 msgr2=0x55b9927e66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64df007740 1 -- 172.21.15.1:0/2149005152 shutdown_connections 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64df007740 1 -- 172.21.15.1:0/2149005152 wait complete. 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64df007740 1 -- start start 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.920+0000 7f64d9800640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f64bc0a3d10 0x7f64bc0a61d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f64c802ae70 tx=0x7f64c80f7530 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.921+0000 7f64df007740 1 Processor -- start 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.921+0000 7f64df007740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.921+0000 7f64df007740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927ea6f0 0x55b9927f2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.921+0000 7f64df007740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927f4fb0 0x55b9927f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.921+0000 7f64df007740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b9927f7be0 con 0x55b9927e9dd0 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.921+0000 7f64df007740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b9926d6c60 con 0x55b9927f4fb0 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.921+0000 7f64df007740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b9927f7d60 con 0x55b9927ea6f0 2024-04-26T21:40:18.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.921+0000 7f64df007740 10 client.0 _ll_register_callbacks cb 0x55b992702e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.921+0000 7f64da200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.921+0000 7f64da200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44938/0 (socket says 172.21.15.1:44938) 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.922+0000 7f64da200640 1 -- 172.21.15.1:0/2166661515 learned_addr learned my addr 172.21.15.1:0/2166661515 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.922+0000 7f64d8e00640 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927ea6f0 0x55b9927f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.922+0000 7f64da200640 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927f4fb0 0x55b9927f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.922+0000 7f64da200640 1 -- 172.21.15.1:0/2166661515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927ea6f0 msgr2=0x55b9927f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.922+0000 7f64da200640 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927ea6f0 0x55b9927f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.922+0000 7f64da200640 1 -- 172.21.15.1:0/2166661515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927f4fb0 msgr2=0x55b9927f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.922+0000 7f64da200640 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927f4fb0 0x55b9927f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.922+0000 7f64da200640 1 -- 172.21.15.1:0/2166661515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9926e5510 con 0x55b9927e9dd0 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.922+0000 7f64da200640 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f1f30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f64cc004570 tx=0x7f64cc0198d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1a896847fd98d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.922+0000 7f64bb400640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.923+0000 7f64bb400640 1 -- 172.21.15.1:0/2166661515 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc00fe80 con 0x55b9927e9dd0 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.923+0000 7f64bb400640 1 -- 172.21.15.1:0/2166661515 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f64cc016070 con 0x55b9927e9dd0 2024-04-26T21:40:18.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.923+0000 7f64df007740 1 -- 172.21.15.1:0/2166661515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64cc00fcd0 con 0x55b9927e9dd0 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.923+0000 7f64bb400640 1 -- 172.21.15.1:0/2166661515 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f64cc033050 con 0x55b9927e9dd0 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.923+0000 7f64df007740 10 client.16128 fetch_fsmap learned FSMap version 36 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.923+0000 7f64df007740 1 -- 172.21.15.1:0/2166661515 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b992461210 con 0x55b9927e9dd0 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.923+0000 7f64bb400640 1 -- 172.21.15.1:0/2166661515 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64cc00fcd0 con 0x55b9927e9dd0 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.924+0000 7f64bb400640 1 -- 172.21.15.1:0/2166661515 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f64cc025040 con 0x55b9927e9dd0 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.924+0000 7f64df007740 10 client.16128 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.924+0000 7f64df007740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.924+0000 7f64df007740 20 client.16128 populate_metadata read hostname 'smithi001' 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.924+0000 7f64df007740 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b992804c60 0x55b992807050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.924+0000 7f64df007740 4 client.16128 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.924+0000 7f64df007740 1 -- 172.21.15.1:0/2166661515 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b99275cf90 con 0x55b992804c60 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.924+0000 7f64d8e00640 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b992804c60 0x55b992807050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.925+0000 7f64d8e00640 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b992804c60 0x55b992807050 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82037078d78e7d3c server_cookie=8869a1617c1fe227 in_seq=0 out_seq=0 2024-04-26T21:40:18.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.925+0000 7f64bb400640 10 client.16128 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:18.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64bb400640 1 -- 172.21.15.1:0/2166661515 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b99275cf90 con 0x55b992804c60 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64bb400640 10 client.16128 handle_command_reply: tid=0 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 client.16128 shutdown 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 -- 172.21.15.1:0/2166661515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 msgr2=0x55b9927f1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f1f30 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f64cc004570 tx=0x7f64cc0198d0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 -- 172.21.15.1:0/2166661515 shutdown_connections 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b992804c60 0x55b992807050 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927ea6f0 0x55b9927f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927f4fb0 0x55b9927f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 --2- 172.21.15.1:0/2166661515 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f1f30 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64bb400640 0 client.16128 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 -- 172.21.15.1:0/2166661515 >> 172.21.15.1:0/2166661515 conn(0x55b9927e62e0 msgr2=0x55b9927ec060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 -- 172.21.15.1:0/2166661515 shutdown_connections 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 1 -- 172.21.15.1:0/2166661515 wait complete. 2024-04-26T21:40:18.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.926+0000 7f64df007740 20 client.16128 trim_cache size 0 max 16384 2024-04-26T21:40:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.943+0000 7f64df007740 1 Processor -- start 2024-04-26T21:40:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.943+0000 7f64df007740 1 -- start start 2024-04-26T21:40:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.943+0000 7f64df007740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.943+0000 7f64df007740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927ea6f0 0x55b992800aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.943+0000 7f64df007740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 0x55b9927e52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.943+0000 7f64df007740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41b49b0 con 0x55b9927e9dd0 2024-04-26T21:40:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.943+0000 7f64df007740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41b2280 con 0x55b9927ea6f0 2024-04-26T21:40:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.943+0000 7f64df007740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41b00f0 con 0x55b9927f4fb0 2024-04-26T21:40:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.943+0000 7f64da200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64da200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44940/0 (socket says 172.21.15.1:44940) 2024-04-26T21:40:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64da200640 1 -- 172.21.15.1:0/3951190740 learned_addr learned my addr 172.21.15.1:0/3951190740 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64d8e00640 1 --2- 172.21.15.1:0/3951190740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927ea6f0 0x55b992800aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64da200640 1 --2- 172.21.15.1:0/3951190740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 0x55b9927e52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64da200640 1 -- 172.21.15.1:0/3951190740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 msgr2=0x55b9927e52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64da200640 1 --2- 172.21.15.1:0/3951190740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 0x55b9927e52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64da200640 1 -- 172.21.15.1:0/3951190740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927ea6f0 msgr2=0x55b992800aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64da200640 1 --2- 172.21.15.1:0/3951190740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927ea6f0 0x55b992800aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64da200640 1 -- 172.21.15.1:0/3951190740 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b99252d620 con 0x55b9927e9dd0 2024-04-26T21:40:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64d8e00640 1 --2- 172.21.15.1:0/3951190740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927ea6f0 0x55b992800aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.944+0000 7f64da200640 1 --2- 172.21.15.1:0/3951190740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f4660 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f64cc019980 tx=0x7f64cc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d9e4595ed6214f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64c6000640 1 -- 172.21.15.1:0/3951190740 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc009d20 con 0x55b9927e9dd0 2024-04-26T21:40:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64c6000640 1 -- 172.21.15.1:0/3951190740 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64cc005630 con 0x55b9927e9dd0 2024-04-26T21:40:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64c6000640 1 -- 172.21.15.1:0/3951190740 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc019e20 con 0x55b9927e9dd0 2024-04-26T21:40:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64df007740 1 -- 172.21.15.1:0/3951190740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 msgr2=0x55b9927f4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64df007740 1 --2- 172.21.15.1:0/3951190740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f4660 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f64cc019980 tx=0x7f64cc005f90 comp rx=0 tx=0).stop 2024-04-26T21:40:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64df007740 1 -- 172.21.15.1:0/3951190740 shutdown_connections 2024-04-26T21:40:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64df007740 1 --2- 172.21.15.1:0/3951190740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927f4fb0 0x55b9927e52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64df007740 1 --2- 172.21.15.1:0/3951190740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927ea6f0 0x55b992800aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64df007740 1 --2- 172.21.15.1:0/3951190740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927f4660 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64df007740 1 -- 172.21.15.1:0/3951190740 >> 172.21.15.1:0/3951190740 conn(0x55b9927e62e0 msgr2=0x55b9927e4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.945+0000 7f64df007740 1 -- 172.21.15.1:0/3951190740 shutdown_connections 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.946+0000 7f64df007740 1 -- 172.21.15.1:0/3951190740 wait complete. 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.946+0000 7f64df007740 1 -- start start 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.946+0000 7f64df007740 1 Processor -- start 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.946+0000 7f64df007740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927eec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.946+0000 7f64df007740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927ea6f0 0x55b9927ef160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.946+0000 7f64df007740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927f4fb0 0x55b9927ef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64df007740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41b0740 con 0x55b9927e9dd0 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64df007740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41b1bb0 con 0x55b9927f4fb0 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64df007740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64d41b0e10 con 0x55b9927ea6f0 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64df007740 10 client.0 _ll_register_callbacks cb 0x7f64c80f7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64da200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927eec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64da200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927eec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44942/0 (socket says 172.21.15.1:44942) 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64da200640 1 -- 172.21.15.1:0/3951527158 learned_addr learned my addr 172.21.15.1:0/3951527158 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64d8e00640 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927ea6f0 0x55b9927ef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64da200640 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927f4fb0 0x55b9927ef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64da200640 1 -- 172.21.15.1:0/3951527158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927ea6f0 msgr2=0x55b9927ef160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64da200640 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927ea6f0 0x55b9927ef160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64da200640 1 -- 172.21.15.1:0/3951527158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927f4fb0 msgr2=0x55b9927ef6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64da200640 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927f4fb0 0x55b9927ef6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64da200640 1 -- 172.21.15.1:0/3951527158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b992709cb0 con 0x55b9927e9dd0 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.947+0000 7f64da200640 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927eec20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f64cc000c00 tx=0x7f64cc0057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2a47d4b0fe4655 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.948+0000 7f64bb400640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.948+0000 7f64bb400640 1 -- 172.21.15.1:0/3951527158 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc019dc0 con 0x55b9927e9dd0 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.948+0000 7f64bb400640 1 -- 172.21.15.1:0/3951527158 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f64cc011050 con 0x55b9927e9dd0 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.948+0000 7f64df007740 1 -- 172.21.15.1:0/3951527158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64cc005630 con 0x55b9927e9dd0 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.948+0000 7f64bb400640 1 -- 172.21.15.1:0/3951527158 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f64cc009470 con 0x55b9927e9dd0 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.948+0000 7f64df007740 10 client.16134 fetch_fsmap learned FSMap version 36 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.948+0000 7f64df007740 1 -- 172.21.15.1:0/3951527158 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55b99286b0c0 con 0x55b9927e9dd0 2024-04-26T21:40:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.948+0000 7f64bb400640 1 -- 172.21.15.1:0/3951527158 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64cc005630 con 0x55b9927e9dd0 2024-04-26T21:40:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.949+0000 7f64bb400640 1 -- 172.21.15.1:0/3951527158 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f64cc0035f0 con 0x55b9927e9dd0 2024-04-26T21:40:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.949+0000 7f64df007740 10 client.16134 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:40:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.949+0000 7f64df007740 10 client.16134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:40:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.949+0000 7f64df007740 20 client.16134 populate_metadata read hostname 'smithi001' 2024-04-26T21:40:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.949+0000 7f64df007740 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b99286b610 0x55b99286da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.949+0000 7f64df007740 4 client.16134 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:40:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.949+0000 7f64df007740 1 -- 172.21.15.1:0/3951527158 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b992756e00 con 0x55b99286b610 2024-04-26T21:40:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.949+0000 7f64d8e00640 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b99286b610 0x55b99286da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.950+0000 7f64d8e00640 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b99286b610 0x55b99286da30 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8df9e465ea5a54c server_cookie=803fa2df22bf808e in_seq=0 out_seq=0 2024-04-26T21:40:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:18.950+0000 7f64bb400640 10 client.16134 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.000+0000 7f64bb400640 1 -- 172.21.15.1:0/3951527158 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b992756e00 con 0x55b99286b610 2024-04-26T21:40:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.000+0000 7f64bb400640 10 client.16134 handle_command_reply: tid=0 2024-04-26T21:40:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.000+0000 7f64df007740 1 client.16134 shutdown 2024-04-26T21:40:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.001+0000 7f64df007740 1 -- 172.21.15.1:0/3951527158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 msgr2=0x55b9927eec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.001+0000 7f64df007740 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927eec20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f64cc000c00 tx=0x7f64cc0057e0 comp rx=0 tx=0).stop 2024-04-26T21:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.002+0000 7f64df007740 1 -- 172.21.15.1:0/3951527158 shutdown_connections 2024-04-26T21:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.002+0000 7f64df007740 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b99286b610 0x55b99286da30 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.002+0000 7f64df007740 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b9927ea6f0 0x55b9927ef160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.002+0000 7f64df007740 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9927f4fb0 0x55b9927ef6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.002+0000 7f64df007740 1 --2- 172.21.15.1:0/3951527158 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b9927e9dd0 0x55b9927eec20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.002+0000 7f64bb400640 0 client.16134 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.002+0000 7f64df007740 1 -- 172.21.15.1:0/3951527158 >> 172.21.15.1:0/3951527158 conn(0x55b9927e62e0 msgr2=0x55b9927ebfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.002+0000 7f64df007740 1 -- 172.21.15.1:0/3951527158 shutdown_connections 2024-04-26T21:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.002+0000 7f64df007740 1 -- 172.21.15.1:0/3951527158 wait complete. 2024-04-26T21:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.002+0000 7f64df007740 20 client.16134 trim_cache size 0 max 16384 2024-04-26T21:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:40:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.003+0000 7f64dc200640 1 -- 172.21.15.1:0/190370584 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f64bc0a3d10 msgr2=0x7f64bc0a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.003+0000 7f64dc200640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f64bc0a3d10 0x7f64bc0a61d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f64c802ae70 tx=0x7f64c80f7530 comp rx=0 tx=0).stop 2024-04-26T21:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.003+0000 7f64dc200640 1 -- 172.21.15.1:0/190370584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64d406b010 msgr2=0x7f64d419b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.003+0000 7f64dc200640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64d406b010 0x7f64d419b290 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f64c8005e30 tx=0x7f64c8003c90 comp rx=0 tx=0).stop 2024-04-26T21:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.003+0000 7f64dc200640 1 -- 172.21.15.1:0/190370584 shutdown_connections 2024-04-26T21:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.003+0000 7f64dc200640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f64bc0a3d10 0x7f64bc0a61d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.003+0000 7f64dc200640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64d406b930 0x7f64d419b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.003+0000 7f64dc200640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64d410f340 0x7f64d419fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.003+0000 7f64dc200640 1 --2- 172.21.15.1:0/190370584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64d406b010 0x7f64d419b290 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.003+0000 7f64dc200640 1 -- 172.21.15.1:0/190370584 >> 172.21.15.1:0/190370584 conn(0x7f64d4078930 msgr2=0x7f64d4110840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.004+0000 7f64dc200640 1 -- 172.21.15.1:0/190370584 shutdown_connections 2024-04-26T21:40:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:19.004+0000 7f64dc200640 1 -- 172.21.15.1:0/190370584 wait complete. 2024-04-26T21:40:19.589 INFO:tasks.workunit.client.0.smithi001.stdout: 1 377067 4.45 MB/sec execute 377 sec latency 7044.292 ms 2024-04-26T21:40:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:20 smithi186 ceph-mon[32991]: pgmap v311: 129 pgs: 129 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-04-26T21:40:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:20 smithi001 ceph-mon[28054]: pgmap v311: 129 pgs: 129 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-04-26T21:40:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:20 smithi038 ceph-mon[32724]: pgmap v311: 129 pgs: 129 active+clean; 239 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 1.6 MiB/s wr, 157 op/s 2024-04-26T21:40:20.589 INFO:tasks.workunit.client.0.smithi001.stdout: 1 378015 4.44 MB/sec execute 378 sec latency 64.784 ms 2024-04-26T21:40:21.590 INFO:tasks.workunit.client.0.smithi001.stdout: 1 379011 4.45 MB/sec execute 379 sec latency 34.002 ms 2024-04-26T21:40:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:22 smithi186 ceph-mon[32991]: pgmap v312: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 103 op/s 2024-04-26T21:40:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:22 smithi001 ceph-mon[28054]: pgmap v312: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 103 op/s 2024-04-26T21:40:22.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:22 smithi038 ceph-mon[32724]: pgmap v312: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 103 op/s 2024-04-26T21:40:22.590 INFO:tasks.workunit.client.0.smithi001.stdout: 1 379854 4.46 MB/sec execute 380 sec latency 29.779 ms 2024-04-26T21:40:23.590 INFO:tasks.workunit.client.0.smithi001.stdout: 1 380156 4.45 MB/sec execute 381 sec latency 26.830 ms 2024-04-26T21:40:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:24 smithi186 ceph-mon[32991]: pgmap v313: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 103 op/s 2024-04-26T21:40:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:24 smithi001 ceph-mon[28054]: pgmap v313: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 103 op/s 2024-04-26T21:40:24.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:24 smithi038 ceph-mon[32724]: pgmap v313: 129 pgs: 129 active+clean; 242 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 1.5 MiB/s wr, 103 op/s 2024-04-26T21:40:24.590 INFO:tasks.workunit.client.0.smithi001.stdout: 1 380847 4.44 MB/sec execute 382 sec latency 30.099 ms 2024-04-26T21:40:25.590 INFO:tasks.workunit.client.0.smithi001.stdout: 1 382005 4.44 MB/sec execute 383 sec latency 49.043 ms 2024-04-26T21:40:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:26 smithi186 ceph-mon[32991]: pgmap v314: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-04-26T21:40:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:26 smithi001 ceph-mon[28054]: pgmap v314: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-04-26T21:40:26.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:26 smithi038 ceph-mon[32724]: pgmap v314: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 2.0 MiB/s wr, 156 op/s 2024-04-26T21:40:26.590 INFO:tasks.workunit.client.0.smithi001.stdout: 1 383168 4.46 MB/sec execute 384 sec latency 25.624 ms 2024-04-26T21:40:27.590 INFO:tasks.workunit.client.0.smithi001.stdout: 1 383507 4.45 MB/sec execute 385 sec latency 28.798 ms 2024-04-26T21:40:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:28 smithi186 ceph-mon[32991]: pgmap v315: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-26T21:40:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:28 smithi001 ceph-mon[28054]: pgmap v315: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-26T21:40:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:28 smithi038 ceph-mon[32724]: pgmap v315: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-26T21:40:28.590 INFO:tasks.workunit.client.0.smithi001.stdout: 1 383997 4.44 MB/sec execute 386 sec latency 24.103 ms 2024-04-26T21:40:29.590 INFO:tasks.workunit.client.0.smithi001.stdout: 1 384796 4.44 MB/sec execute 387 sec latency 35.426 ms 2024-04-26T21:40:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:30 smithi186 ceph-mon[32991]: pgmap v316: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-26T21:40:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:30 smithi001 ceph-mon[28054]: pgmap v316: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-26T21:40:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:30 smithi038 ceph-mon[32724]: pgmap v316: 129 pgs: 129 active+clean; 245 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 1.3 MiB/s wr, 99 op/s 2024-04-26T21:40:30.590 INFO:tasks.workunit.client.0.smithi001.stdout: 1 385736 4.44 MB/sec execute 388 sec latency 27.970 ms 2024-04-26T21:40:31.591 INFO:tasks.workunit.client.0.smithi001.stdout: 1 386933 4.45 MB/sec execute 389 sec latency 36.878 ms 2024-04-26T21:40:32.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.163+0000 7f19faa00640 1 -- 172.21.15.1:0/2362953905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f40755f0 msgr2=0x7f19f41133e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.163+0000 7f19faa00640 1 --2- 172.21.15.1:0/2362953905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f40755f0 0x7f19f41133e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009900 tx=0x7f19e002a480 comp rx=0 tx=0).stop 2024-04-26T21:40:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.163+0000 7f19faa00640 1 -- 172.21.15.1:0/2362953905 shutdown_connections 2024-04-26T21:40:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.163+0000 7f19faa00640 1 --2- 172.21.15.1:0/2362953905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f40755f0 0x7f19f41133e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.163+0000 7f19faa00640 1 --2- 172.21.15.1:0/2362953905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19f4113920 0x7f19f4115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.163+0000 7f19faa00640 1 --2- 172.21.15.1:0/2362953905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f4074cd0 0x7f19f40750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.163+0000 7f19faa00640 1 -- 172.21.15.1:0/2362953905 >> 172.21.15.1:0/2362953905 conn(0x7f19f40fe3c0 msgr2=0x7f19f41007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.163+0000 7f19faa00640 1 -- 172.21.15.1:0/2362953905 shutdown_connections 2024-04-26T21:40:32.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.163+0000 7f19faa00640 1 -- 172.21.15.1:0/2362953905 wait complete. 2024-04-26T21:40:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.164+0000 7f19faa00640 1 Processor -- start 2024-04-26T21:40:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.164+0000 7f19faa00640 1 -- start start 2024-04-26T21:40:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.164+0000 7f19faa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f4074cd0 0x7f19f41a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.164+0000 7f19faa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19f40755f0 0x7f19f41a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19faa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f4113920 0x7f19f419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19faa00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f19f41191f0 con 0x7f19f4113920 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19faa00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f19f4119070 con 0x7f19f40755f0 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19faa00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f19f4119370 con 0x7f19f4074cd0 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19f40755f0 0x7f19f41a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f4074cd0 0x7f19f41a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19f40755f0 0x7f19f41a5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40856/0 (socket says 172.21.15.1:40856) 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f8a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f4113920 0x7f19f419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f3400640 1 -- 172.21.15.1:0/2546711608 learned_addr learned my addr 172.21.15.1:0/2546711608 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f3e00640 1 -- 172.21.15.1:0/2546711608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19f40755f0 msgr2=0x7f19f41a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f3e00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19f40755f0 0x7f19f41a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f3e00640 1 -- 172.21.15.1:0/2546711608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f4113920 msgr2=0x7f19f419f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f3e00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f4113920 0x7f19f419f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f3e00640 1 -- 172.21.15.1:0/2546711608 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f419fd70 con 0x7f19f4074cd0 2024-04-26T21:40:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.165+0000 7f19f3400640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19f40755f0 0x7f19f41a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.166+0000 7f19f3e00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f4074cd0 0x7f19f41a5440 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f19e400e9f0 tx=0x7f19e400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.166+0000 7f19f0c00640 1 -- 172.21.15.1:0/2546711608 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e400ccc0 con 0x7f19f4074cd0 2024-04-26T21:40:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.166+0000 7f19f0c00640 1 -- 172.21.15.1:0/2546711608 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19e40108e0 con 0x7f19f4074cd0 2024-04-26T21:40:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.166+0000 7f19faa00640 1 -- 172.21.15.1:0/2546711608 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19f41a0060 con 0x7f19f4074cd0 2024-04-26T21:40:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.166+0000 7f19f0c00640 1 -- 172.21.15.1:0/2546711608 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e4018880 con 0x7f19f4074cd0 2024-04-26T21:40:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.166+0000 7f19faa00640 1 -- 172.21.15.1:0/2546711608 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19f41ac300 con 0x7f19f4074cd0 2024-04-26T21:40:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.171+0000 7f19fd522740 1 Processor -- start 2024-04-26T21:40:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.171+0000 7f19fd522740 1 -- start start 2024-04-26T21:40:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19f0c00640 1 -- 172.21.15.1:0/2546711608 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f19e4010430 con 0x7f19f4074cd0 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19fd522740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623e9fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19fd522740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea05a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19fd522740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 0x55623ea083a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19fd522740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55623e960380 con 0x55623ea05fb0 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19fd522740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55623e8fcc30 con 0x55623e9fb6f0 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19fd522740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55623e95c340 con 0x55623e9fadd0 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19f8a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623e9fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19f8a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623e9fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:39182/0 (socket says 172.21.15.1:39182) 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19f8a00640 1 -- 172.21.15.1:0/4233097473 learned_addr learned my addr 172.21.15.1:0/4233097473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19f3e00640 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea05a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.172+0000 7f19f3400640 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 0x55623ea083a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.173+0000 7f19f8a00640 1 -- 172.21.15.1:0/4233097473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 msgr2=0x55623ea05a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.173+0000 7f19f8a00640 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea05a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.173+0000 7f19f8a00640 1 -- 172.21.15.1:0/4233097473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 msgr2=0x55623ea083a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.173+0000 7f19f8a00640 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 0x55623ea083a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.173+0000 7f19f8a00640 1 -- 172.21.15.1:0/4233097473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55623e91e940 con 0x55623e9fadd0 2024-04-26T21:40:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.173+0000 7f19f3e00640 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea05a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:40:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.173+0000 7f19f3400640 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 0x55623ea083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.173+0000 7f19f8a00640 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623e9fb1b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f19ec009f30 tx=0x7f19ec0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb4a4e3c298766e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.174+0000 7f19e9200640 1 -- 172.21.15.1:0/4233097473 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ec004d20 con 0x55623e9fadd0 2024-04-26T21:40:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.174+0000 7f19e9200640 1 -- 172.21.15.1:0/4233097473 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19ec004050 con 0x55623e9fadd0 2024-04-26T21:40:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.174+0000 7f19e9200640 1 -- 172.21.15.1:0/4233097473 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ec005280 con 0x55623e9fadd0 2024-04-26T21:40:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.174+0000 7f19f0c00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f19d80a3d10 0x7f19d80a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.174+0000 7f19f0c00640 1 -- 172.21.15.1:0/2546711608 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f19e40d0fc0 con 0x7f19f4074cd0 2024-04-26T21:40:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.174+0000 7f19f3e00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f19d80a3d10 0x7f19d80a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.175+0000 7f19fd522740 1 -- 172.21.15.1:0/4233097473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 msgr2=0x55623e9fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.175+0000 7f19fd522740 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623e9fb1b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f19ec009f30 tx=0x7f19ec0118d0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.175+0000 7f19fd522740 1 -- 172.21.15.1:0/4233097473 shutdown_connections 2024-04-26T21:40:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.175+0000 7f19fd522740 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 0x55623ea083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.175+0000 7f19fd522740 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea05a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.175+0000 7f19fd522740 1 --2- 172.21.15.1:0/4233097473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623e9fb1b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.175+0000 7f19fd522740 1 -- 172.21.15.1:0/4233097473 >> 172.21.15.1:0/4233097473 conn(0x55623e9f72e0 msgr2=0x55623e9f76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.175+0000 7f19fd522740 1 -- 172.21.15.1:0/4233097473 shutdown_connections 2024-04-26T21:40:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.175+0000 7f19fd522740 1 -- 172.21.15.1:0/4233097473 wait complete. 2024-04-26T21:40:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.175+0000 7f19f3e00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f19d80a3d10 0x7f19d80a61d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55623e9fc370 tx=0x7f19e4009700 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.176+0000 7f19fd522740 1 -- start start 2024-04-26T21:40:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.176+0000 7f19fd522740 1 Processor -- start 2024-04-26T21:40:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.176+0000 7f19fd522740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fadd0 0x55623ea02ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19fd522740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fb6f0 0x55623ea03400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19fd522740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623ea05fb0 0x55623ea03940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19fd522740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55623ea08c30 con 0x55623e9fadd0 2024-04-26T21:40:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19fd522740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55623e8e7c60 con 0x55623ea05fb0 2024-04-26T21:40:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19fd522740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55623ea08db0 con 0x55623e9fb6f0 2024-04-26T21:40:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19fd522740 10 client.0 _ll_register_callbacks cb 0x55623e913e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:40:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19f3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fb6f0 0x55623ea03400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19f3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fb6f0 0x55623ea03400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:39190/0 (socket says 172.21.15.1:39190) 2024-04-26T21:40:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19f3400640 1 -- 172.21.15.1:0/1888627715 learned_addr learned my addr 172.21.15.1:0/1888627715 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19f8a00640 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fadd0 0x55623ea02ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.177+0000 7f19f8a00640 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623ea05fb0 0x55623ea03940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.178+0000 7f19f3400640 1 -- 172.21.15.1:0/1888627715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623ea05fb0 msgr2=0x55623ea03940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.178+0000 7f19f3400640 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623ea05fb0 0x55623ea03940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.178+0000 7f19f3400640 1 -- 172.21.15.1:0/1888627715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fadd0 msgr2=0x55623ea02ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.178+0000 7f19f3400640 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fadd0 0x55623ea02ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.178+0000 7f19f3400640 1 -- 172.21.15.1:0/1888627715 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55623e8f6510 con 0x55623e9fb6f0 2024-04-26T21:40:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.178+0000 7f19f8a00640 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fadd0 0x55623ea02ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.178+0000 7f19f3400640 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fb6f0 0x55623ea03400 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f19e002aa70 tx=0x7f19e0003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25ae016d6f081c1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.178+0000 7f19dd600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:40:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.179+0000 7f19dd600640 1 -- 172.21.15.1:0/1888627715 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e0035040 con 0x55623e9fb6f0 2024-04-26T21:40:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.179+0000 7f19dd600640 1 -- 172.21.15.1:0/1888627715 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f19e0005020 con 0x55623e9fb6f0 2024-04-26T21:40:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.179+0000 7f19fd522740 1 -- 172.21.15.1:0/1888627715 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19ec004050 con 0x55623e9fb6f0 2024-04-26T21:40:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.179+0000 7f19dd600640 1 -- 172.21.15.1:0/1888627715 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f19e0003e60 con 0x55623e9fb6f0 2024-04-26T21:40:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.179+0000 7f19fd522740 10 client.26311 fetch_fsmap learned FSMap version 36 2024-04-26T21:40:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.179+0000 7f19fd522740 1 -- 172.21.15.1:0/1888627715 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55623e672210 con 0x55623e9fb6f0 2024-04-26T21:40:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.179+0000 7f19dd600640 1 -- 172.21.15.1:0/1888627715 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19ec004050 con 0x55623e9fb6f0 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.180+0000 7f19dd600640 1 -- 172.21.15.1:0/1888627715 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f19e00314a0 con 0x55623e9fb6f0 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.180+0000 7f19fd522740 10 client.26311 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.180+0000 7f19fd522740 10 client.26311 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.180+0000 7f19fd522740 20 client.26311 populate_metadata read hostname 'smithi001' 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.180+0000 7f19fd522740 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55623ea15d20 0x55623ea18110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.180+0000 7f19fd522740 4 client.26311 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.180+0000 7f19fd522740 1 -- 172.21.15.1:0/1888627715 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55623e96df90 con 0x55623ea15d20 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.181+0000 7f19f8a00640 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55623ea15d20 0x55623ea18110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.181+0000 7f19f8a00640 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55623ea15d20 0x55623ea18110 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=829f1b4d2cf85b3 server_cookie=38328251974a1d00 in_seq=0 out_seq=0 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.181+0000 7f19dd600640 10 client.26311 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:32.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.182+0000 7f19dd600640 1 -- 172.21.15.1:0/1888627715 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55623e96df90 con 0x55623ea15d20 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.182+0000 7f19dd600640 10 client.26311 handle_command_reply: tid=0 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.182+0000 7f19fd522740 1 client.26311 shutdown 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.182+0000 7f19fd522740 1 -- 172.21.15.1:0/1888627715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fb6f0 msgr2=0x55623ea03400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.182+0000 7f19fd522740 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fb6f0 0x55623ea03400 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f19e002aa70 tx=0x7f19e0003820 comp rx=0 tx=0).stop 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.183+0000 7f19fd522740 1 -- 172.21.15.1:0/1888627715 shutdown_connections 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.183+0000 7f19fd522740 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55623ea15d20 0x55623ea18110 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.183+0000 7f19fd522740 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fb6f0 0x55623ea03400 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.183+0000 7f19fd522740 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623ea05fb0 0x55623ea03940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.183+0000 7f19fd522740 1 --2- 172.21.15.1:0/1888627715 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fadd0 0x55623ea02ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.183+0000 7f19fd522740 1 -- 172.21.15.1:0/1888627715 >> 172.21.15.1:0/1888627715 conn(0x55623e9f72e0 msgr2=0x55623e9fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.183+0000 7f19dd600640 0 client.26311 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.183+0000 7f19fd522740 1 -- 172.21.15.1:0/1888627715 shutdown_connections 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.183+0000 7f19fd522740 1 -- 172.21.15.1:0/1888627715 wait complete. 2024-04-26T21:40:32.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.183+0000 7f19fd522740 20 client.26311 trim_cache size 0 max 16384 2024-04-26T21:40:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.201+0000 7f19fd522740 1 Processor -- start 2024-04-26T21:40:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.201+0000 7f19fd522740 1 -- start start 2024-04-26T21:40:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.201+0000 7f19fd522740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623ea11af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.201+0000 7f19fd522740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fb6f0 0x55623ea04cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.201+0000 7f19fd522740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623ea05fb0 0x55623ea1d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.201+0000 7f19fd522740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f19f41b8f00 con 0x55623e9fb6f0 2024-04-26T21:40:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.201+0000 7f19fd522740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f19f41b67d0 con 0x55623ea05fb0 2024-04-26T21:40:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.201+0000 7f19fd522740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f19f41b4640 con 0x55623e9fadd0 2024-04-26T21:40:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.201+0000 7f19f8a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623ea11af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.201+0000 7f19f8a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623ea11af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:39200/0 (socket says 172.21.15.1:39200) 2024-04-26T21:40:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.202+0000 7f19f8a00640 1 -- 172.21.15.1:0/1787196932 learned_addr learned my addr 172.21.15.1:0/1787196932 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.202+0000 7f19f3400640 1 --2- 172.21.15.1:0/1787196932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fb6f0 0x55623ea04cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.202+0000 7f19f8a00640 1 --2- 172.21.15.1:0/1787196932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623ea05fb0 0x55623ea1d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.202+0000 7f19f8a00640 1 -- 172.21.15.1:0/1787196932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623ea05fb0 msgr2=0x55623ea1d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.202+0000 7f19f8a00640 1 --2- 172.21.15.1:0/1787196932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623ea05fb0 0x55623ea1d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.202+0000 7f19f8a00640 1 -- 172.21.15.1:0/1787196932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fb6f0 msgr2=0x55623ea04cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.202+0000 7f19f8a00640 1 --2- 172.21.15.1:0/1787196932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fb6f0 0x55623ea04cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.202+0000 7f19f8a00640 1 -- 172.21.15.1:0/1787196932 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55623e9f2940 con 0x55623e9fadd0 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.202+0000 7f19f3400640 1 --2- 172.21.15.1:0/1787196932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fb6f0 0x55623ea04cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.202+0000 7f19f8a00640 1 --2- 172.21.15.1:0/1787196932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623ea11af0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f19ec003fd0 tx=0x7f19ec0096c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae1a7aeae09088b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.203+0000 7f19dfe00640 1 -- 172.21.15.1:0/1787196932 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ec02e070 con 0x55623e9fadd0 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.203+0000 7f19dfe00640 1 -- 172.21.15.1:0/1787196932 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19ec004700 con 0x55623e9fadd0 2024-04-26T21:40:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.203+0000 7f19dfe00640 1 -- 172.21.15.1:0/1787196932 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ec02f8c0 con 0x55623e9fadd0 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.203+0000 7f19fd522740 1 -- 172.21.15.1:0/1787196932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 msgr2=0x55623ea11af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.203+0000 7f19fd522740 1 --2- 172.21.15.1:0/1787196932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623ea11af0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f19ec003fd0 tx=0x7f19ec0096c0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.204+0000 7f19fd522740 1 -- 172.21.15.1:0/1787196932 shutdown_connections 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.204+0000 7f19fd522740 1 --2- 172.21.15.1:0/1787196932 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623ea05fb0 0x55623ea1d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.204+0000 7f19fd522740 1 --2- 172.21.15.1:0/1787196932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623e9fb6f0 0x55623ea04cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.204+0000 7f19fd522740 1 --2- 172.21.15.1:0/1787196932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623ea11af0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.204+0000 7f19fd522740 1 -- 172.21.15.1:0/1787196932 >> 172.21.15.1:0/1787196932 conn(0x55623e9f72e0 msgr2=0x55623ea045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.204+0000 7f19fd522740 1 -- 172.21.15.1:0/1787196932 shutdown_connections 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.204+0000 7f19fd522740 1 -- 172.21.15.1:0/1787196932 wait complete. 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.204+0000 7f19fd522740 1 -- start start 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.204+0000 7f19fd522740 1 Processor -- start 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19fd522740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623e9ffbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19fd522740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea000f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19fd522740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 0x55623ea00630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19fd522740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f19f41b4c90 con 0x55623ea05fb0 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19fd522740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f19f41b6100 con 0x55623e9fb6f0 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19fd522740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f19f41b5360 con 0x55623e9fadd0 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19fd522740 10 client.0 _ll_register_callbacks cb 0x7f19e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40902/0 (socket says 172.21.15.1:40902) 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f3400640 1 -- 172.21.15.1:0/206594277 learned_addr learned my addr 172.21.15.1:0/206594277 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f8a00640 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623e9ffbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f8a00640 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 0x55623ea00630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f3400640 1 -- 172.21.15.1:0/206594277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 msgr2=0x55623e9ffbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f3400640 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623e9ffbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f3400640 1 -- 172.21.15.1:0/206594277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 msgr2=0x55623ea00630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f3400640 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 0x55623ea00630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f3400640 1 -- 172.21.15.1:0/206594277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55623ea08710 con 0x55623e9fb6f0 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.205+0000 7f19f8a00640 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 0x55623ea00630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.206+0000 7f19f3400640 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea000f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55623ea02720 tx=0x7f19e00058b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb68c7cf6aa5b4d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.206+0000 7f19dd600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.206+0000 7f19dd600640 1 -- 172.21.15.1:0/206594277 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e0035040 con 0x55623e9fb6f0 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.206+0000 7f19fd522740 1 -- 172.21.15.1:0/206594277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f19ec004700 con 0x55623e9fb6f0 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.206+0000 7f19dd600640 1 -- 172.21.15.1:0/206594277 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f19e0003c60 con 0x55623e9fb6f0 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.206+0000 7f19dd600640 1 -- 172.21.15.1:0/206594277 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f19e005e050 con 0x55623e9fb6f0 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.206+0000 7f19fd522740 10 client.26342 fetch_fsmap learned FSMap version 36 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.206+0000 7f19fd522740 1 -- 172.21.15.1:0/206594277 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55623e73e620 con 0x55623e9fb6f0 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.207+0000 7f19dd600640 1 -- 172.21.15.1:0/206594277 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19ec004700 con 0x55623e9fb6f0 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.207+0000 7f19dd600640 1 -- 172.21.15.1:0/206594277 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f19e0004830 con 0x55623e9fb6f0 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.207+0000 7f19fd522740 10 client.26342 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.207+0000 7f19fd522740 10 client.26342 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:40:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.207+0000 7f19fd522740 20 client.26342 populate_metadata read hostname 'smithi001' 2024-04-26T21:40:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.207+0000 7f19fd522740 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55623ea7c990 0x55623ea7edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.207+0000 7f19fd522740 4 client.26342 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:40:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.207+0000 7f19fd522740 1 -- 172.21.15.1:0/206594277 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55623e967e00 con 0x55623ea7c990 2024-04-26T21:40:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.207+0000 7f19f8a00640 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55623ea7c990 0x55623ea7edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.208+0000 7f19f8a00640 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55623ea7c990 0x55623ea7edb0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e37179bad01b5ad8 server_cookie=1e8a50871b30fea6 in_seq=0 out_seq=0 2024-04-26T21:40:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.208+0000 7f19dd600640 10 client.26342 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:32.271 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.268+0000 7f19dd600640 1 -- 172.21.15.1:0/206594277 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55623e967e00 con 0x55623ea7c990 2024-04-26T21:40:32.271 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.268+0000 7f19dd600640 10 client.26342 handle_command_reply: tid=0 2024-04-26T21:40:32.271 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.268+0000 7f19fd522740 1 client.26342 shutdown 2024-04-26T21:40:32.271 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 1 -- 172.21.15.1:0/206594277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 msgr2=0x55623ea000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:32.271 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea000f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55623ea02720 tx=0x7f19e00058b0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.271 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 1 -- 172.21.15.1:0/206594277 shutdown_connections 2024-04-26T21:40:32.271 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55623ea7c990 0x55623ea7edb0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.271 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55623ea05fb0 0x55623ea00630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.271 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55623e9fb6f0 0x55623ea000f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19dd600640 0 client.26342 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 1 --2- 172.21.15.1:0/206594277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55623e9fadd0 0x55623e9ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 1 -- 172.21.15.1:0/206594277 >> 172.21.15.1:0/206594277 conn(0x55623e9f72e0 msgr2=0x55623ea03650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 1 -- 172.21.15.1:0/206594277 shutdown_connections 2024-04-26T21:40:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 1 -- 172.21.15.1:0/206594277 wait complete. 2024-04-26T21:40:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.269+0000 7f19fd522740 20 client.26342 trim_cache size 0 max 16384 2024-04-26T21:40:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:40:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:40:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:40:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:40:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 -- 172.21.15.1:0/2546711608 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f19d80a3d10 msgr2=0x7f19d80a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f19d80a3d10 0x7f19d80a61d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55623e9fc370 tx=0x7f19e4009700 comp rx=0 tx=0).stop 2024-04-26T21:40:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 -- 172.21.15.1:0/2546711608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f4074cd0 msgr2=0x7f19f41a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f4074cd0 0x7f19f41a5440 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f19e400e9f0 tx=0x7f19e400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 -- 172.21.15.1:0/2546711608 shutdown_connections 2024-04-26T21:40:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f19d80a3d10 0x7f19d80a61d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19f4113920 0x7f19f419f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19f40755f0 0x7f19f41a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 --2- 172.21.15.1:0/2546711608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19f4074cd0 0x7f19f41a5440 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 -- 172.21.15.1:0/2546711608 >> 172.21.15.1:0/2546711608 conn(0x7f19f40fe3c0 msgr2=0x7f19f40ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 -- 172.21.15.1:0/2546711608 shutdown_connections 2024-04-26T21:40:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:32.270+0000 7f19faa00640 1 -- 172.21.15.1:0/2546711608 wait complete. 2024-04-26T21:40:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:32 smithi186 ceph-mon[32991]: pgmap v317: 129 pgs: 129 active+clean; 250 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 155 op/s 2024-04-26T21:40:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:32 smithi001 ceph-mon[28054]: pgmap v317: 129 pgs: 129 active+clean; 250 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 155 op/s 2024-04-26T21:40:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:32 smithi038 ceph-mon[32724]: pgmap v317: 129 pgs: 129 active+clean; 250 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.0 MiB/s wr, 155 op/s 2024-04-26T21:40:32.591 INFO:tasks.workunit.client.0.smithi001.stdout: 1 387399 4.44 MB/sec execute 390 sec latency 43.672 ms 2024-04-26T21:40:33.591 INFO:tasks.workunit.client.0.smithi001.stdout: 1 388195 4.44 MB/sec execute 391 sec latency 23.197 ms 2024-04-26T21:40:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:34 smithi186 ceph-mon[32991]: pgmap v318: 129 pgs: 129 active+clean; 250 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-04-26T21:40:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:34 smithi001 ceph-mon[28054]: pgmap v318: 129 pgs: 129 active+clean; 250 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-04-26T21:40:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:34 smithi038 ceph-mon[32724]: pgmap v318: 129 pgs: 129 active+clean; 250 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 1.4 MiB/s wr, 112 op/s 2024-04-26T21:40:34.591 INFO:tasks.workunit.client.0.smithi001.stdout: 1 389229 4.44 MB/sec execute 392 sec latency 38.722 ms 2024-04-26T21:40:35.591 INFO:tasks.workunit.client.0.smithi001.stdout: 1 390435 4.46 MB/sec execute 393 sec latency 25.427 ms 2024-04-26T21:40:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:36 smithi186 ceph-mon[32991]: pgmap v319: 129 pgs: 129 active+clean; 254 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-04-26T21:40:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:36 smithi001 ceph-mon[28054]: pgmap v319: 129 pgs: 129 active+clean; 254 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-04-26T21:40:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:36 smithi038 ceph-mon[32724]: pgmap v319: 129 pgs: 129 active+clean; 254 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 193 op/s 2024-04-26T21:40:36.592 INFO:tasks.workunit.client.0.smithi001.stdout: 1 390729 4.44 MB/sec execute 394 sec latency 31.218 ms 2024-04-26T21:40:37.591 INFO:tasks.workunit.client.0.smithi001.stdout: 1 391452 4.44 MB/sec execute 395 sec latency 19.125 ms 2024-04-26T21:40:38.457 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:38 smithi186 ceph-mon[32991]: pgmap v320: 129 pgs: 129 active+clean; 254 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-04-26T21:40:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:38 smithi001 ceph-mon[28054]: pgmap v320: 129 pgs: 129 active+clean; 254 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-04-26T21:40:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:38 smithi038 ceph-mon[32724]: pgmap v320: 129 pgs: 129 active+clean; 254 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-04-26T21:40:38.591 INFO:tasks.workunit.client.0.smithi001.stdout: 1 392615 4.44 MB/sec execute 396 sec latency 26.718 ms 2024-04-26T21:40:39.591 INFO:tasks.workunit.client.0.smithi001.stdout: 1 393567 4.45 MB/sec execute 397 sec latency 29.453 ms 2024-04-26T21:40:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:40 smithi186 ceph-mon[32991]: pgmap v321: 129 pgs: 129 active+clean; 254 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-04-26T21:40:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:40 smithi001 ceph-mon[28054]: pgmap v321: 129 pgs: 129 active+clean; 254 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-04-26T21:40:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:40 smithi038 ceph-mon[32724]: pgmap v321: 129 pgs: 129 active+clean; 254 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 1.4 MiB/s wr, 137 op/s 2024-04-26T21:40:40.591 INFO:tasks.workunit.client.0.smithi001.stdout: 1 394139 4.45 MB/sec execute 398 sec latency 34.132 ms 2024-04-26T21:40:41.592 INFO:tasks.workunit.client.0.smithi001.stdout: 1 394707 4.44 MB/sec execute 399 sec latency 72.154 ms 2024-04-26T21:40:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:42 smithi186 ceph-mon[32991]: pgmap v322: 129 pgs: 129 active+clean; 258 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-26T21:40:42.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:42 smithi001 ceph-mon[28054]: pgmap v322: 129 pgs: 129 active+clean; 258 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-26T21:40:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:42 smithi038 ceph-mon[32724]: pgmap v322: 129 pgs: 129 active+clean; 258 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 197 op/s 2024-04-26T21:40:42.592 INFO:tasks.workunit.client.0.smithi001.stdout: 1 395967 4.44 MB/sec execute 400 sec latency 22.203 ms 2024-04-26T21:40:43.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:43 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:40:43.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:43 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:40:43.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:43 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:40:43.592 INFO:tasks.workunit.client.0.smithi001.stdout: 1 397103 4.45 MB/sec execute 401 sec latency 24.927 ms 2024-04-26T21:40:44.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:44 smithi001 ceph-mon[28054]: pgmap v323: 129 pgs: 129 active+clean; 258 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-04-26T21:40:44.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:44 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:44 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:44 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:44 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:44 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:44 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:44 smithi038 ceph-mon[32724]: pgmap v323: 129 pgs: 129 active+clean; 258 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-04-26T21:40:44.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:44 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:44 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:44 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:44 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:44 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:44 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:44 smithi186 ceph-mon[32991]: pgmap v323: 129 pgs: 129 active+clean; 258 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 1.3 MiB/s wr, 141 op/s 2024-04-26T21:40:44.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:44 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:44 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:44 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:44 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:44 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.584 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:44 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:44.592 INFO:tasks.workunit.client.0.smithi001.stdout: 1 397728 4.45 MB/sec execute 402 sec latency 19.777 ms 2024-04-26T21:40:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.441+0000 7f59af200640 1 -- 172.21.15.1:0/2776123310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a810f150 msgr2=0x7f59a81115a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.441+0000 7f59af200640 1 --2- 172.21.15.1:0/2776123310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a810f150 0x7f59a81115a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f59a0009ca0 tx=0x7f59a002a950 comp rx=0 tx=0).stop 2024-04-26T21:40:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.441+0000 7f59af200640 1 -- 172.21.15.1:0/2776123310 shutdown_connections 2024-04-26T21:40:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.441+0000 7f59af200640 1 --2- 172.21.15.1:0/2776123310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a810f150 0x7f59a81115a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.441+0000 7f59af200640 1 --2- 172.21.15.1:0/2776123310 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59a8075680 0x7f59a810ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.441+0000 7f59af200640 1 --2- 172.21.15.1:0/2776123310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59a8074d60 0x7f59a8075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.441+0000 7f59af200640 1 -- 172.21.15.1:0/2776123310 >> 172.21.15.1:0/2776123310 conn(0x7f59a80fe420 msgr2=0x7f59a8100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.441+0000 7f59af200640 1 -- 172.21.15.1:0/2776123310 shutdown_connections 2024-04-26T21:40:45.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.441+0000 7f59af200640 1 -- 172.21.15.1:0/2776123310 wait complete. 2024-04-26T21:40:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.442+0000 7f59af200640 1 Processor -- start 2024-04-26T21:40:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.442+0000 7f59af200640 1 -- start start 2024-04-26T21:40:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f59af200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a8074d60 0x7f59a819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f59af200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59a8075680 0x7f59a819b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f59af200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59a810f150 0x7f59a819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f59af200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f59a8114df0 con 0x7f59a810f150 2024-04-26T21:40:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f59af200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f59a8114c70 con 0x7f59a8074d60 2024-04-26T21:40:45.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f59af200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59a8114f70 con 0x7f59a8075680 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f599fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59a8075680 0x7f59a819b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f59a7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a8074d60 0x7f59a819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f599fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59a8075680 0x7f59a819b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33574/0 (socket says 172.21.15.1:33574) 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f599fe00640 1 -- 172.21.15.1:0/3948956454 learned_addr learned my addr 172.21.15.1:0/3948956454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f59ad200640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59a810f150 0x7f59a819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.443+0000 7f599fe00640 1 -- 172.21.15.1:0/3948956454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a8074d60 msgr2=0x7f59a819b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f599fe00640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a8074d60 0x7f59a819b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f599fe00640 1 -- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59a810f150 msgr2=0x7f59a819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f599fe00640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59a810f150 0x7f59a819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f599fe00640 1 -- 172.21.15.1:0/3948956454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a81a0320 con 0x7f59a8075680 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f59ad200640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59a810f150 0x7f59a819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f59a7e00640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a8074d60 0x7f59a819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:45.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f599fe00640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59a8075680 0x7f59a819b870 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f599000e9f0 tx=0x7f599000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f59a5600640 1 -- 172.21.15.1:0/3948956454 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599000cd80 con 0x7f59a8075680 2024-04-26T21:40:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f59a5600640 1 -- 172.21.15.1:0/3948956454 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59900108e0 con 0x7f59a8075680 2024-04-26T21:40:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f59a5600640 1 -- 172.21.15.1:0/3948956454 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5990018980 con 0x7f59a8075680 2024-04-26T21:40:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f59af200640 1 -- 172.21.15.1:0/3948956454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a81a0610 con 0x7f59a8075680 2024-04-26T21:40:45.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.444+0000 7f59af200640 1 -- 172.21.15.1:0/3948956454 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a81a7ea0 con 0x7f59a8075680 2024-04-26T21:40:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.448+0000 7f59b1feb740 1 Processor -- start 2024-04-26T21:40:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.448+0000 7f59a5600640 1 -- 172.21.15.1:0/3948956454 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f5990010430 con 0x7f59a8075680 2024-04-26T21:40:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.448+0000 7f59b1feb740 1 -- start start 2024-04-26T21:40:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a4d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a57a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a5a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59b1feb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560e259b2380 con 0x560e25a57fb0 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59b1feb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560e2594ec30 con 0x560e25a4cdd0 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59b1feb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560e259ae340 con 0x560e25a4d6f0 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59a7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a57a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59ad200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a4d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59a7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a57a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33590/0 (socket says 172.21.15.1:33590) 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59a7e00640 1 -- 172.21.15.1:0/4252805525 learned_addr learned my addr 172.21.15.1:0/4252805525 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f599fe00640 1 --2- 172.21.15.1:0/4252805525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a5a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59a7e00640 1 -- 172.21.15.1:0/4252805525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 msgr2=0x560e25a4d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59a7e00640 1 --2- 172.21.15.1:0/4252805525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a4d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59a7e00640 1 -- 172.21.15.1:0/4252805525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 msgr2=0x560e25a5a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59a7e00640 1 --2- 172.21.15.1:0/4252805525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a5a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.449+0000 7f59a7e00640 1 -- 172.21.15.1:0/4252805525 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e25970940 con 0x560e25a4d6f0 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.450+0000 7f59a5600640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f598c0a3d90 0x7f598c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.450+0000 7f59a7e00640 1 --2- 172.21.15.1:0/4252805525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a57a70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5998010480 tx=0x7f5998010940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ceb933b6871679c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.450+0000 7f599d600640 1 -- 172.21.15.1:0/4252805525 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998011760 con 0x560e25a4d6f0 2024-04-26T21:40:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.450+0000 7f59a5600640 1 -- 172.21.15.1:0/3948956454 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f59900d0ee0 con 0x7f59a8075680 2024-04-26T21:40:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.450+0000 7f599d600640 1 -- 172.21.15.1:0/4252805525 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5998011db0 con 0x560e25a4d6f0 2024-04-26T21:40:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.450+0000 7f599d600640 1 -- 172.21.15.1:0/4252805525 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599800e430 con 0x560e25a4d6f0 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.450+0000 7f59ad200640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f598c0a3d90 0x7f598c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.451+0000 7f59b1feb740 1 -- 172.21.15.1:0/4252805525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 msgr2=0x560e25a57a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.451+0000 7f59b1feb740 1 --2- 172.21.15.1:0/4252805525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a57a70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5998010480 tx=0x7f5998010940 comp rx=0 tx=0).stop 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.451+0000 7f59ad200640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f598c0a3d90 0x7f598c0a6250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59a0005fa0 tx=0x7f59a0004a40 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.451+0000 7f59b1feb740 1 -- 172.21.15.1:0/4252805525 shutdown_connections 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.451+0000 7f59b1feb740 1 --2- 172.21.15.1:0/4252805525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a57a70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.451+0000 7f59b1feb740 1 --2- 172.21.15.1:0/4252805525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a5a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.451+0000 7f59b1feb740 1 --2- 172.21.15.1:0/4252805525 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a4d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.451+0000 7f59b1feb740 1 -- 172.21.15.1:0/4252805525 >> 172.21.15.1:0/4252805525 conn(0x560e25a492e0 msgr2=0x560e25a496c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.452+0000 7f59b1feb740 1 -- 172.21.15.1:0/4252805525 shutdown_connections 2024-04-26T21:40:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.452+0000 7f59b1feb740 1 -- 172.21.15.1:0/4252805525 wait complete. 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.452+0000 7f59b1feb740 1 -- start start 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.452+0000 7f59b1feb740 1 Processor -- start 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a54eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a4d6f0 0x560e25a553f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a57fb0 0x560e25a55930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59b1feb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560e25a5abe0 con 0x560e25a4d6f0 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59b1feb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x560e25939c60 con 0x560e25a4cdd0 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59b1feb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560e25a5ad60 con 0x560e25a57fb0 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59b1feb740 10 client.0 _ll_register_callbacks cb 0x560e25965e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59a7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a54eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59a7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a54eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:48524/0 (socket says 172.21.15.1:48524) 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59a7e00640 1 -- 172.21.15.1:0/1371777963 learned_addr learned my addr 172.21.15.1:0/1371777963 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:45.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.453+0000 7f59ad200640 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a57fb0 0x560e25a55930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.454+0000 7f599fe00640 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a4d6f0 0x560e25a553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.454+0000 7f59a7e00640 1 -- 172.21.15.1:0/1371777963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a57fb0 msgr2=0x560e25a55930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.454+0000 7f59a7e00640 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a57fb0 0x560e25a55930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.454+0000 7f59a7e00640 1 -- 172.21.15.1:0/1371777963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a4d6f0 msgr2=0x560e25a553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.454+0000 7f59a7e00640 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a4d6f0 0x560e25a553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.454+0000 7f59a7e00640 1 -- 172.21.15.1:0/1371777963 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e25948510 con 0x560e25a4cdd0 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.454+0000 7f59a7e00640 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a54eb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f599801caf0 tx=0x7f599801de90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3536e91d7b2a34e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.454+0000 7f5996000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.455+0000 7f5996000640 1 -- 172.21.15.1:0/1371777963 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998011560 con 0x560e25a4cdd0 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.455+0000 7f5996000640 1 -- 172.21.15.1:0/1371777963 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f5998011700 con 0x560e25a4cdd0 2024-04-26T21:40:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.455+0000 7f59b1feb740 1 -- 172.21.15.1:0/1371777963 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5998011db0 con 0x560e25a4cdd0 2024-04-26T21:40:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.455+0000 7f5996000640 1 -- 172.21.15.1:0/1371777963 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5998034050 con 0x560e25a4cdd0 2024-04-26T21:40:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.455+0000 7f59b1feb740 10 client.26354 fetch_fsmap learned FSMap version 36 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.455+0000 7f59b1feb740 1 -- 172.21.15.1:0/1371777963 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560e256c4210 con 0x560e25a4cdd0 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.455+0000 7f5996000640 1 -- 172.21.15.1:0/1371777963 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5998011db0 con 0x560e25a4cdd0 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.456+0000 7f5996000640 1 -- 172.21.15.1:0/1371777963 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f599802f030 con 0x560e25a4cdd0 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.456+0000 7f59b1feb740 10 client.26354 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.456+0000 7f59b1feb740 10 client.26354 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.456+0000 7f59b1feb740 20 client.26354 populate_metadata read hostname 'smithi001' 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.456+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560e25a67c60 0x560e25a6a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.456+0000 7f59b1feb740 4 client.26354 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.456+0000 7f59b1feb740 1 -- 172.21.15.1:0/1371777963 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e259bff90 con 0x560e25a67c60 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.456+0000 7f599fe00640 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560e25a67c60 0x560e25a6a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.457+0000 7f599fe00640 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560e25a67c60 0x560e25a6a050 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef6af17aee7f8646 server_cookie=f9a532a332ae1ff9 in_seq=0 out_seq=0 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.457+0000 7f5996000640 10 client.26354 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.457+0000 7f5996000640 1 -- 172.21.15.1:0/1371777963 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e259bff90 con 0x560e25a67c60 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.457+0000 7f5996000640 10 client.26354 handle_command_reply: tid=0 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.457+0000 7f59b1feb740 1 client.26354 shutdown 2024-04-26T21:40:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 1 -- 172.21.15.1:0/1371777963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 msgr2=0x560e25a54eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a54eb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f599801caf0 tx=0x7f599801de90 comp rx=0 tx=0).stop 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 1 -- 172.21.15.1:0/1371777963 shutdown_connections 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560e25a67c60 0x560e25a6a050 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a57fb0 0x560e25a55930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a4d6f0 0x560e25a553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1371777963 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a54eb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 1 -- 172.21.15.1:0/1371777963 >> 172.21.15.1:0/1371777963 conn(0x560e25a492e0 msgr2=0x560e25a4f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f5996000640 0 client.26354 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 1 -- 172.21.15.1:0/1371777963 shutdown_connections 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 1 -- 172.21.15.1:0/1371777963 wait complete. 2024-04-26T21:40:45.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.458+0000 7f59b1feb740 20 client.26354 trim_cache size 0 max 16384 2024-04-26T21:40:45.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.476+0000 7f59b1feb740 1 Processor -- start 2024-04-26T21:40:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.476+0000 7f59b1feb740 1 -- start start 2024-04-26T21:40:45.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a63aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a4ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a6f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f59b1feb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f59a81b4ad0 con 0x560e25a57fb0 2024-04-26T21:40:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f59b1feb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f59a81b23a0 con 0x560e25a4cdd0 2024-04-26T21:40:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f59b1feb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59a81b0210 con 0x560e25a4d6f0 2024-04-26T21:40:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f59a7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a63aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f59ad200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a6f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f599fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a4ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f59a7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a63aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:48538/0 (socket says 172.21.15.1:48538) 2024-04-26T21:40:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f59a7e00640 1 -- 172.21.15.1:0/2055262221 learned_addr learned my addr 172.21.15.1:0/2055262221 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.477+0000 7f599fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a4ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33612/0 (socket says 172.21.15.1:33612) 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.478+0000 7f59a7e00640 1 -- 172.21.15.1:0/2055262221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 msgr2=0x560e25a4ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.478+0000 7f59a7e00640 1 --2- 172.21.15.1:0/2055262221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a4ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.478+0000 7f59a7e00640 1 -- 172.21.15.1:0/2055262221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 msgr2=0x560e25a6f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.478+0000 7f59a7e00640 1 --2- 172.21.15.1:0/2055262221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a6f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.478+0000 7f59a7e00640 1 -- 172.21.15.1:0/2055262221 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e25790620 con 0x560e25a4cdd0 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.478+0000 7f599fe00640 1 --2- 172.21.15.1:0/2055262221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a4ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.478+0000 7f59a7e00640 1 --2- 172.21.15.1:0/2055262221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a63aa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5998027920 tx=0x7f5998027ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adc7f5550e5a2904 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.478+0000 7f599cc00640 1 -- 172.21.15.1:0/2055262221 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599803c070 con 0x560e25a4cdd0 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.478+0000 7f599cc00640 1 -- 172.21.15.1:0/2055262221 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5998005a70 con 0x560e25a4cdd0 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.478+0000 7f599cc00640 1 -- 172.21.15.1:0/2055262221 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5998005040 con 0x560e25a4cdd0 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.479+0000 7f59b1feb740 1 -- 172.21.15.1:0/2055262221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 msgr2=0x560e25a63aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.479+0000 7f59b1feb740 1 --2- 172.21.15.1:0/2055262221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a63aa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5998027920 tx=0x7f5998027ca0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.479+0000 7f59b1feb740 1 -- 172.21.15.1:0/2055262221 shutdown_connections 2024-04-26T21:40:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.479+0000 7f59b1feb740 1 --2- 172.21.15.1:0/2055262221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4d6f0 0x560e25a4ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.479+0000 7f59b1feb740 1 --2- 172.21.15.1:0/2055262221 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a6f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.479+0000 7f59b1feb740 1 --2- 172.21.15.1:0/2055262221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4cdd0 0x560e25a63aa0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.479+0000 7f59b1feb740 1 -- 172.21.15.1:0/2055262221 >> 172.21.15.1:0/2055262221 conn(0x560e25a492e0 msgr2=0x560e25a4e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.479+0000 7f59b1feb740 1 -- 172.21.15.1:0/2055262221 shutdown_connections 2024-04-26T21:40:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.480+0000 7f59b1feb740 1 -- 172.21.15.1:0/2055262221 wait complete. 2024-04-26T21:40:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.480+0000 7f59b1feb740 1 -- start start 2024-04-26T21:40:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.480+0000 7f59b1feb740 1 Processor -- start 2024-04-26T21:40:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4cdd0 0x560e25a51bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4d6f0 0x560e25a52130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f59b1feb740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a52670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f59b1feb740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f59a81b0860 con 0x560e25a57fb0 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f59b1feb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f59a81b1cd0 con 0x560e25a4d6f0 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f59b1feb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59a81b0f30 con 0x560e25a4cdd0 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f59b1feb740 10 client.0 _ll_register_callbacks cb 0x7f5990005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f59a7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4cdd0 0x560e25a51bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f599fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4d6f0 0x560e25a52130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f59a7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4cdd0 0x560e25a51bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33626/0 (socket says 172.21.15.1:33626) 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.481+0000 7f59a7e00640 1 -- 172.21.15.1:0/1673597560 learned_addr learned my addr 172.21.15.1:0/1673597560 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.482+0000 7f59ad200640 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a52670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.482+0000 7f59a7e00640 1 -- 172.21.15.1:0/1673597560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4d6f0 msgr2=0x560e25a52130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.482+0000 7f59a7e00640 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4d6f0 0x560e25a52130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.482+0000 7f59a7e00640 1 -- 172.21.15.1:0/1673597560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 msgr2=0x560e25a52670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.482+0000 7f59a7e00640 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a52670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.482+0000 7f59a7e00640 1 -- 172.21.15.1:0/1673597560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2596ccb0 con 0x560e25a4cdd0 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.482+0000 7f599fe00640 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4d6f0 0x560e25a52130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.482+0000 7f59a7e00640 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4cdd0 0x560e25a51bf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5998005d00 tx=0x7f5998027610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed488394324e8a4f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.482+0000 7f5996000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.483+0000 7f5996000640 1 -- 172.21.15.1:0/1673597560 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599803c040 con 0x560e25a4cdd0 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.483+0000 7f5996000640 1 -- 172.21.15.1:0/1673597560 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f599801cc40 con 0x560e25a4cdd0 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.483+0000 7f59b1feb740 1 -- 172.21.15.1:0/1673597560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5998005a70 con 0x560e25a4cdd0 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.483+0000 7f5996000640 1 -- 172.21.15.1:0/1673597560 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5998005870 con 0x560e25a4cdd0 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.483+0000 7f59b1feb740 10 client.26335 fetch_fsmap learned FSMap version 36 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.483+0000 7f59b1feb740 1 -- 172.21.15.1:0/1673597560 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560e25ad68d0 con 0x560e25a4cdd0 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.483+0000 7f5996000640 1 -- 172.21.15.1:0/1673597560 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5998005a70 con 0x560e25a4cdd0 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.483+0000 7f5996000640 1 -- 172.21.15.1:0/1673597560 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f599802f070 con 0x560e25a4cdd0 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.484+0000 7f59b1feb740 10 client.26335 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.484+0000 7f59b1feb740 10 client.26335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.484+0000 7f59b1feb740 20 client.26335 populate_metadata read hostname 'smithi001' 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.484+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560e25ad6f80 0x560e25ad9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.484+0000 7f59b1feb740 4 client.26335 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:40:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.484+0000 7f59b1feb740 1 -- 172.21.15.1:0/1673597560 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560e259b9e00 con 0x560e25ad6f80 2024-04-26T21:40:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.484+0000 7f599fe00640 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560e25ad6f80 0x560e25ad9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.485+0000 7f599fe00640 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560e25ad6f80 0x560e25ad9370 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95f003cd2955a73f server_cookie=387442e04fd4f39a in_seq=0 out_seq=0 2024-04-26T21:40:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.485+0000 7f5996000640 10 client.26335 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:45.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.587+0000 7f5996000640 1 -- 172.21.15.1:0/1673597560 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560e259b9e00 con 0x560e25ad6f80 2024-04-26T21:40:45.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.587+0000 7f5996000640 10 client.26335 handle_command_reply: tid=0 2024-04-26T21:40:45.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.588+0000 7f59b1feb740 1 client.26335 shutdown 2024-04-26T21:40:45.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.588+0000 7f59b1feb740 1 -- 172.21.15.1:0/1673597560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4cdd0 msgr2=0x560e25a51bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:45.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.588+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4cdd0 0x560e25a51bf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5998005d00 tx=0x7f5998027610 comp rx=0 tx=0).stop 2024-04-26T21:40:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.589+0000 7f59b1feb740 1 -- 172.21.15.1:0/1673597560 shutdown_connections 2024-04-26T21:40:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.589+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560e25ad6f80 0x560e25ad9370 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.589+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560e25a57fb0 0x560e25a52670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.589+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560e25a4d6f0 0x560e25a52130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.589+0000 7f59b1feb740 1 --2- 172.21.15.1:0/1673597560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560e25a4cdd0 0x560e25a51bf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.589+0000 7f5996000640 0 client.26335 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.589+0000 7f59b1feb740 1 -- 172.21.15.1:0/1673597560 >> 172.21.15.1:0/1673597560 conn(0x560e25a492e0 msgr2=0x560e25a4f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.590+0000 7f59b1feb740 1 -- 172.21.15.1:0/1673597560 shutdown_connections 2024-04-26T21:40:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.590+0000 7f59b1feb740 1 -- 172.21.15.1:0/1673597560 wait complete. 2024-04-26T21:40:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.591+0000 7f59b1feb740 20 client.26335 trim_cache size 0 max 16384 2024-04-26T21:40:45.592 INFO:tasks.workunit.client.0.smithi001.stdout: 1 398451 4.45 MB/sec execute 403 sec latency 54.784 ms 2024-04-26T21:40:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:40:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:40:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:40:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:40:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.593+0000 7f59af200640 1 -- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f598c0a3d90 msgr2=0x7f598c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.593+0000 7f59af200640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f598c0a3d90 0x7f598c0a6250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59a0005fa0 tx=0x7f59a0004a40 comp rx=0 tx=0).stop 2024-04-26T21:40:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.593+0000 7f59af200640 1 -- 172.21.15.1:0/3948956454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59a8075680 msgr2=0x7f59a819b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.593+0000 7f59af200640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59a8075680 0x7f59a819b870 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f599000e9f0 tx=0x7f599000eeb0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.593+0000 7f59af200640 1 -- 172.21.15.1:0/3948956454 shutdown_connections 2024-04-26T21:40:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.593+0000 7f59af200640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f598c0a3d90 0x7f598c0a6250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.593+0000 7f59af200640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59a8075680 0x7f59a819b870 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.593+0000 7f59af200640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59a810f150 0x7f59a819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.593+0000 7f59af200640 1 --2- 172.21.15.1:0/3948956454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a8074d60 0x7f59a819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.593+0000 7f59af200640 1 -- 172.21.15.1:0/3948956454 >> 172.21.15.1:0/3948956454 conn(0x7f59a80fe420 msgr2=0x7f59a80ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.594+0000 7f59af200640 1 -- 172.21.15.1:0/3948956454 shutdown_connections 2024-04-26T21:40:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:45.594+0000 7f59af200640 1 -- 172.21.15.1:0/3948956454 wait complete. 2024-04-26T21:40:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:46 smithi001 ceph-mon[28054]: pgmap v324: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 206 op/s 2024-04-26T21:40:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:46 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:46 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:46 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:46 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:46 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:46 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:46 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:46 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:46 smithi038 ceph-mon[32724]: pgmap v324: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 206 op/s 2024-04-26T21:40:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:46 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:46 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:46 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:46 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:46 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:46 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:46 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:46 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.592 INFO:tasks.workunit.client.0.smithi001.stdout: 1 399727 4.45 MB/sec execute 404 sec latency 27.146 ms 2024-04-26T21:40:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:46 smithi186 ceph-mon[32991]: pgmap v324: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 206 op/s 2024-04-26T21:40:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:46 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:46 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:46 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:46 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:46.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:46 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:46 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:46 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:46.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:46 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:47 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T21:40:47.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:47 smithi001 ceph-mon[28054]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:40:47.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:47 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:47 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:47 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:47 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:47 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:47.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:47 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:47.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:47 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T21:40:47.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:47 smithi038 ceph-mon[32724]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:40:47.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:47 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:47 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:47 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:47 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:47 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:47.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:47 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:47.592 INFO:tasks.workunit.client.0.smithi001.stdout: 1 401058 4.46 MB/sec execute 405 sec latency 18.651 ms 2024-04-26T21:40:47.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:47 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T21:40:47.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:47 smithi186 ceph-mon[32991]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:40:47.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:47 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:47 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:47 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:47 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:47.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:47 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:47.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:47 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:48 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T21:40:48.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:48 smithi001 ceph-mon[28054]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:40:48.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:48 smithi001 ceph-mon[28054]: pgmap v325: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-04-26T21:40:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:48 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T21:40:48.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:48 smithi038 ceph-mon[32724]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:40:48.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:48 smithi038 ceph-mon[32724]: pgmap v325: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-04-26T21:40:48.592 INFO:tasks.workunit.client.0.smithi001.stdout: 1 401326 4.45 MB/sec execute 406 sec latency 27.050 ms 2024-04-26T21:40:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:48 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T21:40:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:48 smithi186 ceph-mon[32991]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:40:48.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:48 smithi186 ceph-mon[32991]: pgmap v325: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-04-26T21:40:49.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:49 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:49 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:49 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:49 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:49 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:40:49.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:49 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:40:49.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:49 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:49.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:49 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:49 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:49 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:49 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:49 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:40:49.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:49 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:40:49.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:49 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:49.592 INFO:tasks.workunit.client.0.smithi001.stdout: 1 401931 4.45 MB/sec execute 407 sec latency 27.654 ms 2024-04-26T21:40:49.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:49 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:49 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:49 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:49 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:40:49.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:49 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:40:49.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:49 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:40:49.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:49 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:40:50.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:50 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi001 to 177.5M 2024-04-26T21:40:50.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:50 smithi001 ceph-mon[28054]: Unable to set osd_memory_target on smithi001 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:40:50.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:50 smithi001 ceph-mon[28054]: pgmap v326: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-04-26T21:40:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:50 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi001 to 177.5M 2024-04-26T21:40:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:50 smithi038 ceph-mon[32724]: Unable to set osd_memory_target on smithi001 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:40:50.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:50 smithi038 ceph-mon[32724]: pgmap v326: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-04-26T21:40:50.593 INFO:tasks.workunit.client.0.smithi001.stdout: 1 402837 4.45 MB/sec execute 408 sec latency 23.104 ms 2024-04-26T21:40:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:50 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi001 to 177.5M 2024-04-26T21:40:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:50 smithi186 ceph-mon[32991]: Unable to set osd_memory_target on smithi001 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:40:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:50 smithi186 ceph-mon[32991]: pgmap v326: 129 pgs: 129 active+clean; 262 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 1.4 MiB/s wr, 125 op/s 2024-04-26T21:40:51.593 INFO:tasks.workunit.client.0.smithi001.stdout: 1 403777 4.45 MB/sec execute 409 sec latency 44.436 ms 2024-04-26T21:40:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:52 smithi001 ceph-mon[28054]: pgmap v327: 129 pgs: 129 active+clean; 263 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 213 op/s 2024-04-26T21:40:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:52 smithi038 ceph-mon[32724]: pgmap v327: 129 pgs: 129 active+clean; 263 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 213 op/s 2024-04-26T21:40:52.593 INFO:tasks.workunit.client.0.smithi001.stdout: 1 404656 4.46 MB/sec execute 410 sec latency 43.192 ms 2024-04-26T21:40:52.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:52 smithi186 ceph-mon[32991]: pgmap v327: 129 pgs: 129 active+clean; 263 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 213 op/s 2024-04-26T21:40:53.593 INFO:tasks.workunit.client.0.smithi001.stdout: 1 404903 4.45 MB/sec execute 411 sec latency 26.560 ms 2024-04-26T21:40:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:54 smithi001 ceph-mon[28054]: pgmap v328: 129 pgs: 129 active+clean; 263 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T21:40:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:54 smithi038 ceph-mon[32724]: pgmap v328: 129 pgs: 129 active+clean; 263 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T21:40:54.593 INFO:tasks.workunit.client.0.smithi001.stdout: 1 405556 4.44 MB/sec execute 412 sec latency 37.319 ms 2024-04-26T21:40:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:54 smithi186 ceph-mon[32991]: pgmap v328: 129 pgs: 129 active+clean; 263 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 153 op/s 2024-04-26T21:40:55.593 INFO:tasks.workunit.client.0.smithi001.stdout: 1 406600 4.44 MB/sec execute 413 sec latency 28.161 ms 2024-04-26T21:40:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:56 smithi001 ceph-mon[28054]: pgmap v329: 129 pgs: 129 active+clean; 267 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-26T21:40:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:56 smithi038 ceph-mon[32724]: pgmap v329: 129 pgs: 129 active+clean; 267 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-26T21:40:56.593 INFO:tasks.workunit.client.0.smithi001.stdout: 1 407398 4.44 MB/sec execute 414 sec latency 51.991 ms 2024-04-26T21:40:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:56 smithi186 ceph-mon[32991]: pgmap v329: 129 pgs: 129 active+clean; 267 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 200 op/s 2024-04-26T21:40:57.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:57 smithi001 ceph-mon[28054]: pgmap v330: 129 pgs: 129 active+clean; 267 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 940 KiB/s wr, 136 op/s 2024-04-26T21:40:57.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:57 smithi038 ceph-mon[32724]: pgmap v330: 129 pgs: 129 active+clean; 267 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 940 KiB/s wr, 136 op/s 2024-04-26T21:40:57.593 INFO:tasks.workunit.client.0.smithi001.stdout: 1 408224 4.45 MB/sec execute 415 sec latency 46.249 ms 2024-04-26T21:40:57.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:57 smithi186 ceph-mon[32991]: pgmap v330: 129 pgs: 129 active+clean; 267 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 940 KiB/s wr, 136 op/s 2024-04-26T21:40:58.594 INFO:tasks.workunit.client.0.smithi001.stdout: 1 408454 4.44 MB/sec execute 416 sec latency 29.882 ms 2024-04-26T21:40:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.766+0000 7f0567200640 1 -- 172.21.15.1:0/2855196381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0568101520 msgr2=0x7f0568101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.766+0000 7f0567200640 1 --2- 172.21.15.1:0/2855196381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0568101520 0x7f0568101900 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0558009900 tx=0x7f055802a3f0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.766+0000 7f0567200640 1 -- 172.21.15.1:0/2855196381 shutdown_connections 2024-04-26T21:40:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.766+0000 7f0567200640 1 --2- 172.21.15.1:0/2855196381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0568101ed0 0x7f056810eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.766+0000 7f0567200640 1 --2- 172.21.15.1:0/2855196381 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f056810f0d0 0x7f05681114c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.766+0000 7f0567200640 1 --2- 172.21.15.1:0/2855196381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0568101520 0x7f0568101900 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.766+0000 7f0567200640 1 -- 172.21.15.1:0/2855196381 >> 172.21.15.1:0/2855196381 conn(0x7f05680fd110 msgr2=0x7f05680ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.766+0000 7f0567200640 1 -- 172.21.15.1:0/2855196381 shutdown_connections 2024-04-26T21:40:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.766+0000 7f0567200640 1 -- 172.21.15.1:0/2855196381 wait complete. 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.767+0000 7f0567200640 1 Processor -- start 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f0567200640 1 -- start start 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f0567200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0568101520 0x7f056819b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f0567200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0568101ed0 0x7f056819b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f0567200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f056810f0d0 0x7f056819fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f0567200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0568114c40 con 0x7f056810f0d0 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f0567200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0568114ac0 con 0x7f0568101520 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f0567200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0568114dc0 con 0x7f0568101ed0 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f055d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0568101ed0 0x7f056819b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f0565e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0568101520 0x7f056819b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.769 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f0566800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f056810f0d0 0x7f056819fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.770 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f055d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0568101ed0 0x7f056819b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:56520/0 (socket says 172.21.15.1:56520) 2024-04-26T21:40:58.770 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.768+0000 7f055d400640 1 -- 172.21.15.1:0/1899207057 learned_addr learned my addr 172.21.15.1:0/1899207057 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.769+0000 7f0565e00640 1 -- 172.21.15.1:0/1899207057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0568101ed0 msgr2=0x7f056819b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.769+0000 7f0565e00640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0568101ed0 0x7f056819b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.769+0000 7f0565e00640 1 -- 172.21.15.1:0/1899207057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f056810f0d0 msgr2=0x7f056819fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.769+0000 7f0565e00640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f056810f0d0 0x7f056819fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.769+0000 7f0565e00640 1 -- 172.21.15.1:0/1899207057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05681a0190 con 0x7f0568101520 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.769+0000 7f0565e00640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0568101520 0x7f056819b1a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f055802a9e0 tx=0x7f0558003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.770+0000 7f055f400640 1 -- 172.21.15.1:0/1899207057 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055803f070 con 0x7f0568101520 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.770+0000 7f055f400640 1 -- 172.21.15.1:0/1899207057 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0558031d60 con 0x7f0568101520 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.770+0000 7f055f400640 1 -- 172.21.15.1:0/1899207057 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558035030 con 0x7f0568101520 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.770+0000 7f0567200640 1 -- 172.21.15.1:0/1899207057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05681a0420 con 0x7f0568101520 2024-04-26T21:40:58.771 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.770+0000 7f0567200640 1 -- 172.21.15.1:0/1899207057 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05681a0890 con 0x7f0568101520 2024-04-26T21:40:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.774+0000 7f055f400640 1 -- 172.21.15.1:0/1899207057 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0558042070 con 0x7f0568101520 2024-04-26T21:40:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.774+0000 7f056ffb9740 1 Processor -- start 2024-04-26T21:40:58.774 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.774+0000 7f056ffb9740 1 -- start start 2024-04-26T21:40:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.774+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a363c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.774+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a36900 0x55ddc7a40c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a411c0 0x55ddc7a435b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f056ffb9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ddc799b380 con 0x55ddc7a35fe0 2024-04-26T21:40:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f056ffb9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7937c30 con 0x55ddc7a36900 2024-04-26T21:40:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f056ffb9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7997340 con 0x55ddc7a411c0 2024-04-26T21:40:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f0565e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a36900 0x55ddc7a40c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f0566800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a363c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f0565e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a36900 0x55ddc7a40c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43800/0 (socket says 172.21.15.1:43800) 2024-04-26T21:40:58.776 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f0565e00640 1 -- 172.21.15.1:0/4097242847 learned_addr learned my addr 172.21.15.1:0/4097242847 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f055d400640 1 --2- 172.21.15.1:0/4097242847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a411c0 0x55ddc7a435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f0565e00640 1 -- 172.21.15.1:0/4097242847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a411c0 msgr2=0x55ddc7a435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f0565e00640 1 --2- 172.21.15.1:0/4097242847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a411c0 0x55ddc7a435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f0565e00640 1 -- 172.21.15.1:0/4097242847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a35fe0 msgr2=0x55ddc7a363c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f0565e00640 1 --2- 172.21.15.1:0/4097242847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a363c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.775+0000 7f0565e00640 1 -- 172.21.15.1:0/4097242847 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddc7959940 con 0x55ddc7a36900 2024-04-26T21:40:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.776+0000 7f0565e00640 1 --2- 172.21.15.1:0/4097242847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a36900 0x55ddc7a40c80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f05580ee120 tx=0x7f05580f9220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db1d2fae45d98530 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.776+0000 7f055f400640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f054c0a3d80 0x7f054c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.776+0000 7f055f400640 1 -- 172.21.15.1:0/1899207057 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f05580f0f50 con 0x7f0568101520 2024-04-26T21:40:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.776+0000 7f0566800640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f054c0a3d80 0x7f054c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.776+0000 7f0557400640 1 -- 172.21.15.1:0/4097242847 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05580b8d80 con 0x55ddc7a36900 2024-04-26T21:40:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.776+0000 7f0557400640 1 -- 172.21.15.1:0/4097242847 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05580fe070 con 0x55ddc7a36900 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.776+0000 7f0557400640 1 -- 172.21.15.1:0/4097242847 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05580bc440 con 0x55ddc7a36900 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.777+0000 7f056ffb9740 1 -- 172.21.15.1:0/4097242847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a36900 msgr2=0x55ddc7a40c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.777+0000 7f0566800640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f054c0a3d80 0x7f054c0a6240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f056806bec0 tx=0x7f05600096a0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.777+0000 7f056ffb9740 1 --2- 172.21.15.1:0/4097242847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a36900 0x55ddc7a40c80 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f05580ee120 tx=0x7f05580f9220 comp rx=0 tx=0).stop 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.777+0000 7f056ffb9740 1 -- 172.21.15.1:0/4097242847 shutdown_connections 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.777+0000 7f056ffb9740 1 --2- 172.21.15.1:0/4097242847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a411c0 0x55ddc7a435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.777+0000 7f056ffb9740 1 --2- 172.21.15.1:0/4097242847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a36900 0x55ddc7a40c80 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.777+0000 7f056ffb9740 1 --2- 172.21.15.1:0/4097242847 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a363c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.777+0000 7f056ffb9740 1 -- 172.21.15.1:0/4097242847 >> 172.21.15.1:0/4097242847 conn(0x55ddc7a324f0 msgr2=0x55ddc7a328d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.777+0000 7f056ffb9740 1 -- 172.21.15.1:0/4097242847 shutdown_connections 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.777+0000 7f056ffb9740 1 -- 172.21.15.1:0/4097242847 wait complete. 2024-04-26T21:40:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.778+0000 7f056ffb9740 1 -- start start 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.778+0000 7f056ffb9740 1 Processor -- start 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a3e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a36900 0x55ddc7a3e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a411c0 0x55ddc7a3eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f056ffb9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7a43bb0 con 0x55ddc7a411c0 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f056ffb9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7922c60 con 0x55ddc7a35fe0 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f056ffb9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ddc7a43d30 con 0x55ddc7a36900 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f056ffb9740 10 client.0 _ll_register_callbacks cb 0x55ddc794ee50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f0565e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a3e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f0565e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a3e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43816/0 (socket says 172.21.15.1:43816) 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f0565e00640 1 -- 172.21.15.1:0/718564837 learned_addr learned my addr 172.21.15.1:0/718564837 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f055d400640 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a36900 0x55ddc7a3e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f0566800640 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a411c0 0x55ddc7a3eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f0565e00640 1 -- 172.21.15.1:0/718564837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a36900 msgr2=0x55ddc7a3e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f0565e00640 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a36900 0x55ddc7a3e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f0565e00640 1 -- 172.21.15.1:0/718564837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a411c0 msgr2=0x55ddc7a3eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f0565e00640 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a411c0 0x55ddc7a3eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.779+0000 7f0565e00640 1 -- 172.21.15.1:0/718564837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddc7931510 con 0x55ddc7a35fe0 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.780+0000 7f0565e00640 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a3e0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f05580c5170 tx=0x7f05580045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b99b3d88975f6e88 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.780+0000 7f054be00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.780+0000 7f054be00640 1 -- 172.21.15.1:0/718564837 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558049070 con 0x55ddc7a35fe0 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.780+0000 7f054be00640 1 -- 172.21.15.1:0/718564837 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f05580b9150 con 0x55ddc7a35fe0 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.780+0000 7f056ffb9740 1 -- 172.21.15.1:0/718564837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05580fe070 con 0x55ddc7a35fe0 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.780+0000 7f054be00640 1 -- 172.21.15.1:0/718564837 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f05580bc620 con 0x55ddc7a35fe0 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.780+0000 7f056ffb9740 10 client.26372 fetch_fsmap learned FSMap version 36 2024-04-26T21:40:58.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.780+0000 7f056ffb9740 1 -- 172.21.15.1:0/718564837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ddc76ad210 con 0x55ddc7a35fe0 2024-04-26T21:40:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.781+0000 7f054be00640 1 -- 172.21.15.1:0/718564837 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05580fe070 con 0x55ddc7a35fe0 2024-04-26T21:40:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.781+0000 7f054be00640 1 -- 172.21.15.1:0/718564837 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f055803b460 con 0x55ddc7a35fe0 2024-04-26T21:40:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.781+0000 7f056ffb9740 10 client.26372 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:40:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.781+0000 7f056ffb9740 10 client.26372 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:40:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.781+0000 7f056ffb9740 20 client.26372 populate_metadata read hostname 'smithi001' 2024-04-26T21:40:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.781+0000 7f056ffb9740 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ddc7a50a80 0x55ddc7a52e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.781+0000 7f056ffb9740 4 client.26372 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:40:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.781+0000 7f056ffb9740 1 -- 172.21.15.1:0/718564837 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddc79a8f90 con 0x55ddc7a50a80 2024-04-26T21:40:58.785 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.781+0000 7f055d400640 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ddc7a50a80 0x55ddc7a52e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.782+0000 7f055d400640 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ddc7a50a80 0x55ddc7a52e70 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66bb56098f63bd3 server_cookie=156e7c25bb7871a in_seq=0 out_seq=0 2024-04-26T21:40:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.782+0000 7f054be00640 10 client.26372 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:58.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.782+0000 7f054be00640 1 -- 172.21.15.1:0/718564837 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ddc79a8f90 con 0x55ddc7a50a80 2024-04-26T21:40:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.782+0000 7f054be00640 10 client.26372 handle_command_reply: tid=0 2024-04-26T21:40:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 client.26372 shutdown 2024-04-26T21:40:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 -- 172.21.15.1:0/718564837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 msgr2=0x55ddc7a3e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a3e0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f05580c5170 tx=0x7f05580045e0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 -- 172.21.15.1:0/718564837 shutdown_connections 2024-04-26T21:40:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ddc7a50a80 0x55ddc7a52e70 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a36900 0x55ddc7a3e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a411c0 0x55ddc7a3eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 --2- 172.21.15.1:0/718564837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a3e0c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f054be00640 0 client.26372 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:58.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 -- 172.21.15.1:0/718564837 >> 172.21.15.1:0/718564837 conn(0x55ddc7a324f0 msgr2=0x55ddc7a38270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:58.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 -- 172.21.15.1:0/718564837 shutdown_connections 2024-04-26T21:40:58.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 1 -- 172.21.15.1:0/718564837 wait complete. 2024-04-26T21:40:58.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.783+0000 7f056ffb9740 20 client.26372 trim_cache size 0 max 16384 2024-04-26T21:40:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.801+0000 7f056ffb9740 1 Processor -- start 2024-04-26T21:40:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.801+0000 7f056ffb9740 1 -- start start 2024-04-26T21:40:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a4c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a36900 0x55ddc7a3d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a411c0 0x55ddc7a314d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f056ffb9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f05681b4940 con 0x55ddc7a36900 2024-04-26T21:40:58.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f056ffb9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f05681b2210 con 0x55ddc7a411c0 2024-04-26T21:40:58.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f056ffb9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f05681b0080 con 0x55ddc7a35fe0 2024-04-26T21:40:58.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f055d400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a36900 0x55ddc7a3d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f0566800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a411c0 0x55ddc7a314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f0566800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a411c0 0x55ddc7a314d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43824/0 (socket says 172.21.15.1:43824) 2024-04-26T21:40:58.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f0566800640 1 -- 172.21.15.1:0/765658130 learned_addr learned my addr 172.21.15.1:0/765658130 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:58.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f0565e00640 1 --2- 172.21.15.1:0/765658130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a4c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f055d400640 1 -- 172.21.15.1:0/765658130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a35fe0 msgr2=0x55ddc7a4c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f055d400640 1 --2- 172.21.15.1:0/765658130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a4c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f055d400640 1 -- 172.21.15.1:0/765658130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a411c0 msgr2=0x55ddc7a314d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f055d400640 1 --2- 172.21.15.1:0/765658130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a411c0 0x55ddc7a314d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.802+0000 7f055d400640 1 -- 172.21.15.1:0/765658130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddc7779620 con 0x55ddc7a36900 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.803+0000 7f055d400640 1 --2- 172.21.15.1:0/765658130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a36900 0x55ddc7a3d260 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x55ddc7a33910 tx=0x7f055000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f231852759a732 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.803+0000 7f0556a00640 1 -- 172.21.15.1:0/765658130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550003dc0 con 0x55ddc7a36900 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.803+0000 7f0556a00640 1 -- 172.21.15.1:0/765658130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0550020660 con 0x55ddc7a36900 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.803+0000 7f0556a00640 1 -- 172.21.15.1:0/765658130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0550011c30 con 0x55ddc7a36900 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.804+0000 7f056ffb9740 1 -- 172.21.15.1:0/765658130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a36900 msgr2=0x55ddc7a3d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.804+0000 7f056ffb9740 1 --2- 172.21.15.1:0/765658130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a36900 0x55ddc7a3d260 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x55ddc7a33910 tx=0x7f055000b480 comp rx=0 tx=0).stop 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.804+0000 7f056ffb9740 1 -- 172.21.15.1:0/765658130 shutdown_connections 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.804+0000 7f056ffb9740 1 --2- 172.21.15.1:0/765658130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a411c0 0x55ddc7a314d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.804+0000 7f056ffb9740 1 --2- 172.21.15.1:0/765658130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a36900 0x55ddc7a3d260 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.804+0000 7f056ffb9740 1 --2- 172.21.15.1:0/765658130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a4c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.804+0000 7f056ffb9740 1 -- 172.21.15.1:0/765658130 >> 172.21.15.1:0/765658130 conn(0x55ddc7a324f0 msgr2=0x55ddc7a30dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.805+0000 7f056ffb9740 1 -- 172.21.15.1:0/765658130 shutdown_connections 2024-04-26T21:40:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.805+0000 7f056ffb9740 1 -- 172.21.15.1:0/765658130 wait complete. 2024-04-26T21:40:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.805+0000 7f056ffb9740 1 -- start start 2024-04-26T21:40:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.805+0000 7f056ffb9740 1 Processor -- start 2024-04-26T21:40:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a492d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a36900 0x55ddc7a3ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f056ffb9740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a411c0 0x55ddc7a3b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f056ffb9740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f05681b06d0 con 0x55ddc7a411c0 2024-04-26T21:40:58.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f056ffb9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f05681b1b40 con 0x55ddc7a35fe0 2024-04-26T21:40:58.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f056ffb9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f05681b0da0 con 0x55ddc7a36900 2024-04-26T21:40:58.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f056ffb9740 10 client.0 _ll_register_callbacks cb 0x7f0560009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:40:58.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f0565e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f0565e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a492d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43826/0 (socket says 172.21.15.1:43826) 2024-04-26T21:40:58.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f0565e00640 1 -- 172.21.15.1:0/401638387 learned_addr learned my addr 172.21.15.1:0/401638387 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:40:58.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f055d400640 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a36900 0x55ddc7a3ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.806+0000 7f0566800640 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a411c0 0x55ddc7a3b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f0565e00640 1 -- 172.21.15.1:0/401638387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a36900 msgr2=0x55ddc7a3ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:58.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f0565e00640 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a36900 0x55ddc7a3ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f0565e00640 1 -- 172.21.15.1:0/401638387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a411c0 msgr2=0x55ddc7a3b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:40:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f0565e00640 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a411c0 0x55ddc7a3b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f0565e00640 1 -- 172.21.15.1:0/401638387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddc7955cb0 con 0x55ddc7a35fe0 2024-04-26T21:40:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f0565e00640 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a492d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f055802a910 tx=0x7f05580f9220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce7f8e7876ca3ca3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:40:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f054be00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:40:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f054be00640 1 -- 172.21.15.1:0/401638387 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0558049070 con 0x55ddc7a35fe0 2024-04-26T21:40:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f054be00640 1 -- 172.21.15.1:0/401638387 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(104..104 src has 1..104) ==== 9642+0+0 (secure 0 0 0) 0x7f05580ee460 con 0x55ddc7a35fe0 2024-04-26T21:40:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f056ffb9740 1 -- 172.21.15.1:0/401638387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0550020660 con 0x55ddc7a35fe0 2024-04-26T21:40:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f054be00640 1 -- 172.21.15.1:0/401638387 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f05580bc8b0 con 0x55ddc7a35fe0 2024-04-26T21:40:58.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f056ffb9740 10 client.26378 fetch_fsmap learned FSMap version 36 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.807+0000 7f056ffb9740 1 -- 172.21.15.1:0/401638387 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ddc7ab6c70 con 0x55ddc7a35fe0 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.808+0000 7f054be00640 1 -- 172.21.15.1:0/401638387 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0550020660 con 0x55ddc7a35fe0 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.808+0000 7f054be00640 1 -- 172.21.15.1:0/401638387 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f05580f9c60 con 0x55ddc7a35fe0 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.808+0000 7f056ffb9740 10 client.26378 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.808+0000 7f056ffb9740 10 client.26378 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.808+0000 7f056ffb9740 20 client.26378 populate_metadata read hostname 'smithi001' 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.808+0000 7f056ffb9740 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ddc7ab7140 0x55ddc7ab9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.808+0000 7f056ffb9740 4 client.26378 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.808+0000 7f056ffb9740 1 -- 172.21.15.1:0/401638387 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ddc79a2e00 con 0x55ddc7ab7140 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.808+0000 7f055d400640 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ddc7ab7140 0x55ddc7ab9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.809+0000 7f055d400640 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ddc7ab7140 0x55ddc7ab9530 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a033c3af4c1bcf1a server_cookie=636dbbbbe4cae4ee in_seq=0 out_seq=0 2024-04-26T21:40:58.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.809+0000 7f054be00640 10 client.26378 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f054be00640 1 -- 172.21.15.1:0/401638387 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ddc79a2e00 con 0x55ddc7ab7140 2024-04-26T21:40:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f054be00640 10 client.26378 handle_command_reply: tid=0 2024-04-26T21:40:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f056ffb9740 1 client.26378 shutdown 2024-04-26T21:40:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f056ffb9740 1 -- 172.21.15.1:0/401638387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 msgr2=0x55ddc7a492d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f056ffb9740 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a492d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f055802a910 tx=0x7f05580f9220 comp rx=0 tx=0).stop 2024-04-26T21:40:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f056ffb9740 1 -- 172.21.15.1:0/401638387 shutdown_connections 2024-04-26T21:40:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f056ffb9740 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ddc7ab7140 0x55ddc7ab9530 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f056ffb9740 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ddc7a36900 0x55ddc7a3ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f056ffb9740 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ddc7a411c0 0x55ddc7a3b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f054be00640 0 client.26378 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:40:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f056ffb9740 1 --2- 172.21.15.1:0/401638387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ddc7a35fe0 0x55ddc7a492d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.876+0000 7f056ffb9740 1 -- 172.21.15.1:0/401638387 >> 172.21.15.1:0/401638387 conn(0x55ddc7a324f0 msgr2=0x55ddc7a40000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.877+0000 7f056ffb9740 1 -- 172.21.15.1:0/401638387 shutdown_connections 2024-04-26T21:40:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.877+0000 7f056ffb9740 1 -- 172.21.15.1:0/401638387 wait complete. 2024-04-26T21:40:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.877+0000 7f056ffb9740 20 client.26378 trim_cache size 0 max 16384 2024-04-26T21:40:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:40:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:40:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:40:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:40:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.878+0000 7f0567200640 1 -- 172.21.15.1:0/1899207057 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f054c0a3d80 msgr2=0x7f054c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.878+0000 7f0567200640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f054c0a3d80 0x7f054c0a6240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f056806bec0 tx=0x7f05600096a0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.878+0000 7f0567200640 1 -- 172.21.15.1:0/1899207057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0568101520 msgr2=0x7f056819b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:40:58.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.878+0000 7f0567200640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0568101520 0x7f056819b1a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f055802a9e0 tx=0x7f0558003cd0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.878+0000 7f0567200640 1 -- 172.21.15.1:0/1899207057 shutdown_connections 2024-04-26T21:40:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.878+0000 7f0567200640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f054c0a3d80 0x7f054c0a6240 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.878+0000 7f0567200640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0568101ed0 0x7f056819b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.878+0000 7f0567200640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f056810f0d0 0x7f056819fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.878+0000 7f0567200640 1 --2- 172.21.15.1:0/1899207057 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0568101520 0x7f056819b1a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:40:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.878+0000 7f0567200640 1 -- 172.21.15.1:0/1899207057 >> 172.21.15.1:0/1899207057 conn(0x7f05680fd110 msgr2=0x7f05680ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:40:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.879+0000 7f0567200640 1 -- 172.21.15.1:0/1899207057 shutdown_connections 2024-04-26T21:40:58.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:40:58.879+0000 7f0567200640 1 -- 172.21.15.1:0/1899207057 wait complete. 2024-04-26T21:40:59.594 INFO:tasks.workunit.client.0.smithi001.stdout: 1 409137 4.43 MB/sec execute 417 sec latency 39.880 ms 2024-04-26T21:41:00.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:40:59 smithi001 ceph-mon[28054]: pgmap v331: 129 pgs: 129 active+clean; 267 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 940 KiB/s wr, 135 op/s 2024-04-26T21:41:00.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:40:59 smithi038 ceph-mon[32724]: pgmap v331: 129 pgs: 129 active+clean; 267 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 940 KiB/s wr, 135 op/s 2024-04-26T21:41:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:40:59 smithi186 ceph-mon[32991]: pgmap v331: 129 pgs: 129 active+clean; 267 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 940 KiB/s wr, 135 op/s 2024-04-26T21:41:00.594 INFO:tasks.workunit.client.0.smithi001.stdout: 1 410040 4.43 MB/sec execute 418 sec latency 41.372 ms 2024-04-26T21:41:01.594 INFO:tasks.workunit.client.0.smithi001.stdout: 1 410408 4.42 MB/sec execute 419 sec latency 151.145 ms 2024-04-26T21:41:02.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:01 smithi001 ceph-mon[28054]: pgmap v332: 129 pgs: 129 active+clean; 270 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-04-26T21:41:02.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:01 smithi038 ceph-mon[32724]: pgmap v332: 129 pgs: 129 active+clean; 270 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-04-26T21:41:02.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:01 smithi186 ceph-mon[32991]: pgmap v332: 129 pgs: 129 active+clean; 270 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-04-26T21:41:02.595 INFO:tasks.workunit.client.0.smithi001.stdout: 1 411570 4.44 MB/sec execute 420 sec latency 36.990 ms 2024-04-26T21:41:03.595 INFO:tasks.workunit.client.0.smithi001.stdout: 1 411886 4.43 MB/sec execute 421 sec latency 30.135 ms 2024-04-26T21:41:04.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:03 smithi001 ceph-mon[28054]: pgmap v333: 129 pgs: 129 active+clean; 270 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-26T21:41:04.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:03 smithi038 ceph-mon[32724]: pgmap v333: 129 pgs: 129 active+clean; 270 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-26T21:41:04.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:03 smithi186 ceph-mon[32991]: pgmap v333: 129 pgs: 129 active+clean; 270 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 1.2 MiB/s wr, 122 op/s 2024-04-26T21:41:04.595 INFO:tasks.workunit.client.0.smithi001.stdout: 1 412314 4.42 MB/sec execute 422 sec latency 24.575 ms 2024-04-26T21:41:05.595 INFO:tasks.workunit.client.0.smithi001.stdout: 1 413150 4.42 MB/sec execute 423 sec latency 23.648 ms 2024-04-26T21:41:06.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:05 smithi001 ceph-mon[28054]: pgmap v334: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-04-26T21:41:06.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:05 smithi038 ceph-mon[32724]: pgmap v334: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-04-26T21:41:06.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:05 smithi186 ceph-mon[32991]: pgmap v334: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 198 op/s 2024-04-26T21:41:06.595 INFO:tasks.workunit.client.0.smithi001.stdout: 1 414211 4.43 MB/sec execute 424 sec latency 27.603 ms 2024-04-26T21:41:07.595 INFO:tasks.workunit.client.0.smithi001.stdout: 1 415110 4.43 MB/sec execute 425 sec latency 45.650 ms 2024-04-26T21:41:08.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:07 smithi001 ceph-mon[28054]: pgmap v335: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-26T21:41:08.360 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:07 smithi001 ceph-mon[28054]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T21:41:08.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:07 smithi038 ceph-mon[32724]: pgmap v335: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-26T21:41:08.361 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:07 smithi038 ceph-mon[32724]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T21:41:08.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:07 smithi186 ceph-mon[32991]: pgmap v335: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 1.5 MiB/s wr, 151 op/s 2024-04-26T21:41:08.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:07 smithi186 ceph-mon[32991]: osdmap e105: 12 total, 12 up, 12 in 2024-04-26T21:41:08.595 INFO:tasks.workunit.client.0.smithi001.stdout: 1 415467 4.42 MB/sec execute 426 sec latency 29.170 ms 2024-04-26T21:41:09.595 INFO:tasks.workunit.client.0.smithi001.stdout: 1 416020 4.42 MB/sec execute 427 sec latency 23.724 ms 2024-04-26T21:41:10.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:09 smithi001 ceph-mon[28054]: pgmap v337: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-26T21:41:10.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:09 smithi038 ceph-mon[32724]: pgmap v337: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-26T21:41:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:09 smithi186 ceph-mon[32991]: pgmap v337: 129 pgs: 129 active+clean; 274 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 1.8 MiB/s wr, 181 op/s 2024-04-26T21:41:10.596 INFO:tasks.workunit.client.0.smithi001.stdout: 1 416901 4.42 MB/sec execute 428 sec latency 25.340 ms 2024-04-26T21:41:11.596 INFO:tasks.workunit.client.0.smithi001.stdout: 1 417818 4.42 MB/sec execute 429 sec latency 71.873 ms 2024-04-26T21:41:12.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.054+0000 7f166c200640 1 -- 172.21.15.1:0/999166490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1664102910 msgr2=0x7f1664102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.054+0000 7f166c200640 1 --2- 172.21.15.1:0/999166490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1664102910 0x7f1664102d90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1650009900 tx=0x7f165002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.054+0000 7f166c200640 1 -- 172.21.15.1:0/999166490 shutdown_connections 2024-04-26T21:41:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.054+0000 7f166c200640 1 --2- 172.21.15.1:0/999166490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16641032d0 0x7f166410f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.054+0000 7f166c200640 1 --2- 172.21.15.1:0/999166490 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1664102910 0x7f1664102d90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.054+0000 7f166c200640 1 --2- 172.21.15.1:0/999166490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1664108920 0x7f1664108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.054+0000 7f166c200640 1 -- 172.21.15.1:0/999166490 >> 172.21.15.1:0/999166490 conn(0x7f16640fe3c0 msgr2=0x7f16641007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.054+0000 7f166c200640 1 -- 172.21.15.1:0/999166490 shutdown_connections 2024-04-26T21:41:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.054+0000 7f166c200640 1 -- 172.21.15.1:0/999166490 wait complete. 2024-04-26T21:41:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.055+0000 7f166c200640 1 Processor -- start 2024-04-26T21:41:12.056 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.055+0000 7f166c200640 1 -- start start 2024-04-26T21:41:12.056 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.056+0000 7f166c200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1664102910 0x7f166419b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.056+0000 7f166c200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16641032d0 0x7f166419b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.056+0000 7f166c200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1664108920 0x7f166419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.056+0000 7f166c200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1664112be0 con 0x7f1664102910 2024-04-26T21:41:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.056+0000 7f166c200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1664112a60 con 0x7f16641032d0 2024-04-26T21:41:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.056+0000 7f166c200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1664112d60 con 0x7f1664108920 2024-04-26T21:41:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.056+0000 7f1668e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16641032d0 0x7f166419b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.057 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.056+0000 7f166a200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1664108920 0x7f166419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.056+0000 7f1668e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16641032d0 0x7f166419b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40576/0 (socket says 172.21.15.1:40576) 2024-04-26T21:41:12.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.056+0000 7f1668e00640 1 -- 172.21.15.1:0/2296301071 learned_addr learned my addr 172.21.15.1:0/2296301071 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:12.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.057+0000 7f1669800640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1664102910 0x7f166419b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.057+0000 7f166a200640 1 -- 172.21.15.1:0/2296301071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16641032d0 msgr2=0x7f166419b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:12.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.057+0000 7f166a200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16641032d0 0x7f166419b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.057+0000 7f166a200640 1 -- 172.21.15.1:0/2296301071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1664102910 msgr2=0x7f166419b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:12.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.057+0000 7f166a200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1664102910 0x7f166419b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.057+0000 7f166a200640 1 -- 172.21.15.1:0/2296301071 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16641a0310 con 0x7f1664108920 2024-04-26T21:41:12.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.057+0000 7f166a200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1664108920 0x7f166419fbe0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f165c009fd0 tx=0x7f165c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:12.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.058+0000 7f1662000640 1 -- 172.21.15.1:0/2296301071 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c00ce20 con 0x7f1664108920 2024-04-26T21:41:12.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.058+0000 7f1662000640 1 -- 172.21.15.1:0/2296301071 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f165c005020 con 0x7f1664108920 2024-04-26T21:41:12.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.058+0000 7f1662000640 1 -- 172.21.15.1:0/2296301071 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c010690 con 0x7f1664108920 2024-04-26T21:41:12.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.058+0000 7f166c200640 1 -- 172.21.15.1:0/2296301071 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16641a0600 con 0x7f1664108920 2024-04-26T21:41:12.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.058+0000 7f166c200640 1 -- 172.21.15.1:0/2296301071 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16641a7e90 con 0x7f1664108920 2024-04-26T21:41:12.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.061+0000 7f1662000640 1 -- 172.21.15.1:0/2296301071 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f165c0054d0 con 0x7f1664108920 2024-04-26T21:41:12.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.062+0000 7f166f1ba740 1 Processor -- start 2024-04-26T21:41:12.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.062+0000 7f166f1ba740 1 -- start start 2024-04-26T21:41:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.062+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 0x559cf1e5c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.062+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e66a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.062+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 0x559cf1e693a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.062+0000 7f166f1ba740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x559cf1dc1380 con 0x559cf1e5bdd0 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.062+0000 7f166f1ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559cf1d5dc30 con 0x559cf1e66fb0 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.062+0000 7f166f1ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559cf1dbd340 con 0x559cf1e5c6f0 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1669800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e66a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1669800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e66a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33634/0 (socket says 172.21.15.1:33634) 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1669800640 1 -- 172.21.15.1:0/4171215928 learned_addr learned my addr 172.21.15.1:0/4171215928 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1668e00640 1 --2- 172.21.15.1:0/4171215928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 0x559cf1e5c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1662000640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f164c0a3d90 0x7f164c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1668e00640 1 --2- 172.21.15.1:0/4171215928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 0x559cf1e693a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1669800640 1 -- 172.21.15.1:0/4171215928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 msgr2=0x559cf1e693a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1669800640 1 --2- 172.21.15.1:0/4171215928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 0x559cf1e693a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1669800640 1 -- 172.21.15.1:0/4171215928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 msgr2=0x559cf1e5c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1669800640 1 --2- 172.21.15.1:0/4171215928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 0x559cf1e5c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1669800640 1 -- 172.21.15.1:0/4171215928 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cf1d7f940 con 0x559cf1e5c6f0 2024-04-26T21:41:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f1668e00640 1 --2- 172.21.15.1:0/4171215928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 0x559cf1e693a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:41:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.063+0000 7f166a200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f164c0a3d90 0x7f164c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.064+0000 7f1662000640 1 -- 172.21.15.1:0/2296301071 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9730+0+0 (secure 0 0 0) 0x7f165c0d1320 con 0x7f1664108920 2024-04-26T21:41:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.064+0000 7f1669800640 1 --2- 172.21.15.1:0/4171215928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e66a70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1658009c70 tx=0x7f1658011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b54b4a8def2777 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.064+0000 7f1656a00640 1 -- 172.21.15.1:0/4171215928 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658004230 con 0x559cf1e5c6f0 2024-04-26T21:41:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.064+0000 7f1656a00640 1 -- 172.21.15.1:0/4171215928 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16580043d0 con 0x559cf1e5c6f0 2024-04-26T21:41:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.064+0000 7f1656a00640 1 -- 172.21.15.1:0/4171215928 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16580196b0 con 0x559cf1e5c6f0 2024-04-26T21:41:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166f1ba740 1 -- 172.21.15.1:0/4171215928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 msgr2=0x559cf1e66a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166f1ba740 1 --2- 172.21.15.1:0/4171215928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e66a70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1658009c70 tx=0x7f1658011810 comp rx=0 tx=0).stop 2024-04-26T21:41:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166a200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f164c0a3d90 0x7f164c0a6250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f165c099770 tx=0x7f165c00bef0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:12.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166f1ba740 1 -- 172.21.15.1:0/4171215928 shutdown_connections 2024-04-26T21:41:12.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166f1ba740 1 --2- 172.21.15.1:0/4171215928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e66a70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166f1ba740 1 --2- 172.21.15.1:0/4171215928 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 0x559cf1e693a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166f1ba740 1 --2- 172.21.15.1:0/4171215928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 0x559cf1e5c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166f1ba740 1 -- 172.21.15.1:0/4171215928 >> 172.21.15.1:0/4171215928 conn(0x559cf1e582e0 msgr2=0x559cf1e586c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:12.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166f1ba740 1 -- 172.21.15.1:0/4171215928 shutdown_connections 2024-04-26T21:41:12.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166f1ba740 1 -- 172.21.15.1:0/4171215928 wait complete. 2024-04-26T21:41:12.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.065+0000 7f166f1ba740 1 -- start start 2024-04-26T21:41:12.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.066+0000 7f166f1ba740 1 Processor -- start 2024-04-26T21:41:12.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.066+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 0x559cf1e63e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.066+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e643d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.066+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 0x559cf1e64910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.066+0000 7f166f1ba740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x559cf1e69be0 con 0x559cf1e5bdd0 2024-04-26T21:41:12.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.066+0000 7f166f1ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559cf1d48c60 con 0x559cf1e66fb0 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.066+0000 7f166f1ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559cf1e69d60 con 0x559cf1e5c6f0 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.066+0000 7f166f1ba740 10 client.0 _ll_register_callbacks cb 0x559cf1d74e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.067+0000 7f1668e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 0x559cf1e63e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.067+0000 7f1669800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e643d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.067+0000 7f1668e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 0x559cf1e63e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57296/0 (socket says 172.21.15.1:57296) 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.067+0000 7f1668e00640 1 -- 172.21.15.1:0/1028921285 learned_addr learned my addr 172.21.15.1:0/1028921285 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.067+0000 7f1668e00640 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 0x559cf1e64910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.067+0000 7f1669800640 1 -- 172.21.15.1:0/1028921285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 msgr2=0x559cf1e64910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.067+0000 7f1669800640 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 0x559cf1e64910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.067+0000 7f1669800640 1 -- 172.21.15.1:0/1028921285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 msgr2=0x559cf1e63e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.067+0000 7f1669800640 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 0x559cf1e63e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.067+0000 7f1669800640 1 -- 172.21.15.1:0/1028921285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cf1d57510 con 0x559cf1e5c6f0 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.068+0000 7f1669800640 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e643d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1658003e50 tx=0x7f1658005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=781a7e2d463b8518 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:12.070 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.068+0000 7f164b400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:41:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.068+0000 7f164b400640 1 -- 172.21.15.1:0/1028921285 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658027070 con 0x559cf1e5c6f0 2024-04-26T21:41:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.068+0000 7f164b400640 1 -- 172.21.15.1:0/1028921285 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9730+0+0 (secure 0 0 0) 0x7f1658003800 con 0x559cf1e5c6f0 2024-04-26T21:41:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.068+0000 7f166f1ba740 1 -- 172.21.15.1:0/1028921285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16580043d0 con 0x559cf1e5c6f0 2024-04-26T21:41:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.069+0000 7f164b400640 1 -- 172.21.15.1:0/1028921285 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1658031050 con 0x559cf1e5c6f0 2024-04-26T21:41:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.069+0000 7f166f1ba740 10 client.26365 fetch_fsmap learned FSMap version 36 2024-04-26T21:41:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.069+0000 7f166f1ba740 1 -- 172.21.15.1:0/1028921285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559cf1ad3210 con 0x559cf1e5c6f0 2024-04-26T21:41:12.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.069+0000 7f164b400640 1 -- 172.21.15.1:0/1028921285 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16580043d0 con 0x559cf1e5c6f0 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.069+0000 7f164b400640 1 -- 172.21.15.1:0/1028921285 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f1658018070 con 0x559cf1e5c6f0 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.070+0000 7f166f1ba740 10 client.26365 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.070+0000 7f166f1ba740 10 client.26365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.070+0000 7f166f1ba740 20 client.26365 populate_metadata read hostname 'smithi001' 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.070+0000 7f166f1ba740 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559cf1e76b00 0x559cf1e78ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.070+0000 7f166f1ba740 4 client.26365 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.070+0000 7f166f1ba740 1 -- 172.21.15.1:0/1028921285 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cf1dcef90 con 0x559cf1e76b00 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.070+0000 7f1668e00640 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559cf1e76b00 0x559cf1e78ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.071+0000 7f1668e00640 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559cf1e76b00 0x559cf1e78ef0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae18b05975638624 server_cookie=421ac6014f1109b1 in_seq=0 out_seq=0 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.071+0000 7f164b400640 10 client.26365 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:12.073 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.071+0000 7f164b400640 1 -- 172.21.15.1:0/1028921285 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559cf1dcef90 con 0x559cf1e76b00 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.071+0000 7f164b400640 10 client.26365 handle_command_reply: tid=0 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 client.26365 shutdown 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 -- 172.21.15.1:0/1028921285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 msgr2=0x559cf1e643d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e643d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1658003e50 tx=0x7f1658005d30 comp rx=0 tx=0).stop 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 -- 172.21.15.1:0/1028921285 shutdown_connections 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559cf1e76b00 0x559cf1e78ef0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5c6f0 0x559cf1e643d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e66fb0 0x559cf1e64910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f164b400640 0 client.26365 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 --2- 172.21.15.1:0/1028921285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5bdd0 0x559cf1e63e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 -- 172.21.15.1:0/1028921285 >> 172.21.15.1:0/1028921285 conn(0x559cf1e582e0 msgr2=0x559cf1e5e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 -- 172.21.15.1:0/1028921285 shutdown_connections 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 1 -- 172.21.15.1:0/1028921285 wait complete. 2024-04-26T21:41:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.072+0000 7f166f1ba740 20 client.26365 trim_cache size 0 max 16384 2024-04-26T21:41:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.095+0000 7f166f1ba740 1 Processor -- start 2024-04-26T21:41:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.095+0000 7f166f1ba740 1 -- start start 2024-04-26T21:41:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.095+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5bdd0 0x559cf1e72aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.095+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5c6f0 0x559cf1e62e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.095+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e66fb0 0x559cf1e7e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.095+0000 7f166f1ba740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f16641b4be0 con 0x559cf1e66fb0 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.095+0000 7f166f1ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16641b24b0 con 0x559cf1e5c6f0 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.095+0000 7f166f1ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f16641b0320 con 0x559cf1e5bdd0 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.095+0000 7f1668e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5bdd0 0x559cf1e72aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1669800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5c6f0 0x559cf1e62e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1668e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5bdd0 0x559cf1e72aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33650/0 (socket says 172.21.15.1:33650) 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1669800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5c6f0 0x559cf1e62e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40614/0 (socket says 172.21.15.1:40614) 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1668e00640 1 -- 172.21.15.1:0/521588182 learned_addr learned my addr 172.21.15.1:0/521588182 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1668e00640 1 --2- 172.21.15.1:0/521588182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e66fb0 0x559cf1e7e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1669800640 1 -- 172.21.15.1:0/521588182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5bdd0 msgr2=0x559cf1e72aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1669800640 1 --2- 172.21.15.1:0/521588182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5bdd0 0x559cf1e72aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1669800640 1 -- 172.21.15.1:0/521588182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e66fb0 msgr2=0x559cf1e7e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1669800640 1 --2- 172.21.15.1:0/521588182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e66fb0 0x559cf1e7e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1669800640 1 -- 172.21.15.1:0/521588182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cf1b9f620 con 0x559cf1e5c6f0 2024-04-26T21:41:12.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1668e00640 1 --2- 172.21.15.1:0/521588182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e66fb0 0x559cf1e7e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:41:12.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1669800640 1 --2- 172.21.15.1:0/521588182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5c6f0 0x559cf1e62e70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1658000c00 tx=0x7f1658003dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7393af9dfb905c78 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1656000640 1 -- 172.21.15.1:0/521588182 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658027070 con 0x559cf1e5c6f0 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1656000640 1 -- 172.21.15.1:0/521588182 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1658009850 con 0x559cf1e5c6f0 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.096+0000 7f1656000640 1 -- 172.21.15.1:0/521588182 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1658009310 con 0x559cf1e5c6f0 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.097+0000 7f166f1ba740 1 -- 172.21.15.1:0/521588182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5c6f0 msgr2=0x559cf1e62e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.097+0000 7f166f1ba740 1 --2- 172.21.15.1:0/521588182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5c6f0 0x559cf1e62e70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f1658000c00 tx=0x7f1658003dd0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.097+0000 7f166f1ba740 1 -- 172.21.15.1:0/521588182 shutdown_connections 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.097+0000 7f166f1ba740 1 --2- 172.21.15.1:0/521588182 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e66fb0 0x559cf1e7e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.097+0000 7f166f1ba740 1 --2- 172.21.15.1:0/521588182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5c6f0 0x559cf1e62e70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.097+0000 7f166f1ba740 1 --2- 172.21.15.1:0/521588182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e5bdd0 0x559cf1e72aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.097+0000 7f166f1ba740 1 -- 172.21.15.1:0/521588182 >> 172.21.15.1:0/521588182 conn(0x559cf1e582e0 msgr2=0x559cf1e62760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.097+0000 7f166f1ba740 1 -- 172.21.15.1:0/521588182 shutdown_connections 2024-04-26T21:41:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.098+0000 7f166f1ba740 1 -- 172.21.15.1:0/521588182 wait complete. 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.098+0000 7f166f1ba740 1 -- start start 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.098+0000 7f166f1ba740 1 Processor -- start 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5bdd0 0x559cf1e6a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5c6f0 0x559cf1e6a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f166f1ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e66fb0 0x559cf1edae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f166f1ba740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f16641b0970 con 0x559cf1e5c6f0 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f166f1ba740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f16641b1de0 con 0x559cf1e5bdd0 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f166f1ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f16641b1040 con 0x559cf1e66fb0 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f166f1ba740 10 client.0 _ll_register_callbacks cb 0x7f1658009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f1669800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5c6f0 0x559cf1e6a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f1668e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5bdd0 0x559cf1e6a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f1669800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5c6f0 0x559cf1e6a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57314/0 (socket says 172.21.15.1:57314) 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f1669800640 1 -- 172.21.15.1:0/3248678339 learned_addr learned my addr 172.21.15.1:0/3248678339 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f1668e00640 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e66fb0 0x559cf1edae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f1669800640 1 -- 172.21.15.1:0/3248678339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e66fb0 msgr2=0x559cf1edae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.099+0000 7f1669800640 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e66fb0 0x559cf1edae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.100+0000 7f1669800640 1 -- 172.21.15.1:0/3248678339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5bdd0 msgr2=0x559cf1e6a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.100+0000 7f1669800640 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5bdd0 0x559cf1e6a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.100+0000 7f1669800640 1 -- 172.21.15.1:0/3248678339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cf1d7bcb0 con 0x559cf1e5c6f0 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.100+0000 7f1669800640 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5c6f0 0x559cf1e6a6e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1658019bf0 tx=0x7f1658005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14016146fd2dc9b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.100+0000 7f164b400640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.100+0000 7f164b400640 1 -- 172.21.15.1:0/3248678339 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165803a070 con 0x559cf1e5c6f0 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.100+0000 7f164b400640 1 -- 172.21.15.1:0/3248678339 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(105..105 src has 1..105) ==== 9730+0+0 (secure 0 0 0) 0x7f1658003a00 con 0x559cf1e5c6f0 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.100+0000 7f166f1ba740 1 -- 172.21.15.1:0/3248678339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1658009850 con 0x559cf1e5c6f0 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f164b400640 1 -- 172.21.15.1:0/3248678339 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1658045050 con 0x559cf1e5c6f0 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f166f1ba740 10 client.16200 fetch_fsmap learned FSMap version 36 2024-04-26T21:41:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f166f1ba740 1 -- 172.21.15.1:0/3248678339 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559cf1edc580 con 0x559cf1e5c6f0 2024-04-26T21:41:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f164b400640 1 -- 172.21.15.1:0/3248678339 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1658009850 con 0x559cf1e5c6f0 2024-04-26T21:41:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f164b400640 1 -- 172.21.15.1:0/3248678339 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 36) ==== 8346+0+0 (secure 0 0 0) 0x7f1658018070 con 0x559cf1e5c6f0 2024-04-26T21:41:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f166f1ba740 10 client.16200 fetch_fsmap finished waiting for FSMap version 36 2024-04-26T21:41:12.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f166f1ba740 10 client.16200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:41:12.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f166f1ba740 20 client.16200 populate_metadata read hostname 'smithi001' 2024-04-26T21:41:12.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f166f1ba740 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559cf1edcc10 0x559cf1edf030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:12.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f166f1ba740 4 client.16200 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:41:12.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.101+0000 7f166f1ba740 1 -- 172.21.15.1:0/3248678339 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559cf1dc8e00 con 0x559cf1edcc10 2024-04-26T21:41:12.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.102+0000 7f1668e00640 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559cf1edcc10 0x559cf1edf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:12.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.102+0000 7f1668e00640 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559cf1edcc10 0x559cf1edf030 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34228472bbc1db05 server_cookie=23d3cc97bceb5257 in_seq=0 out_seq=0 2024-04-26T21:41:12.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.102+0000 7f164b400640 10 client.16200 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.181+0000 7f164b400640 1 -- 172.21.15.1:0/3248678339 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559cf1dc8e00 con 0x559cf1edcc10 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.181+0000 7f164b400640 10 client.16200 handle_command_reply: tid=0 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.181+0000 7f166f1ba740 1 client.16200 shutdown 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 1 -- 172.21.15.1:0/3248678339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5c6f0 msgr2=0x559cf1e6a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5c6f0 0x559cf1e6a6e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1658019bf0 tx=0x7f1658005d30 comp rx=0 tx=0).stop 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 1 -- 172.21.15.1:0/3248678339 shutdown_connections 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559cf1edcc10 0x559cf1edf030 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559cf1e66fb0 0x559cf1edae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559cf1e5c6f0 0x559cf1e6a6e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 1 --2- 172.21.15.1:0/3248678339 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559cf1e5bdd0 0x559cf1e6a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f164b400640 0 client.16200 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 1 -- 172.21.15.1:0/3248678339 >> 172.21.15.1:0/3248678339 conn(0x559cf1e582e0 msgr2=0x559cf1e56bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 1 -- 172.21.15.1:0/3248678339 shutdown_connections 2024-04-26T21:41:12.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 1 -- 172.21.15.1:0/3248678339 wait complete. 2024-04-26T21:41:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.182+0000 7f166f1ba740 20 client.16200 trim_cache size 0 max 16384 2024-04-26T21:41:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:41:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:41:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:41:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:41:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 -- 172.21.15.1:0/2296301071 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f164c0a3d90 msgr2=0x7f164c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f164c0a3d90 0x7f164c0a6250 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f165c099770 tx=0x7f165c00bef0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 -- 172.21.15.1:0/2296301071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1664108920 msgr2=0x7f166419fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1664108920 0x7f166419fbe0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f165c009fd0 tx=0x7f165c00ef10 comp rx=0 tx=0).stop 2024-04-26T21:41:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 -- 172.21.15.1:0/2296301071 shutdown_connections 2024-04-26T21:41:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f164c0a3d90 0x7f164c0a6250 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1664108920 0x7f166419fbe0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f16641032d0 0x7f166419b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 --2- 172.21.15.1:0/2296301071 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1664102910 0x7f166419b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 -- 172.21.15.1:0/2296301071 >> 172.21.15.1:0/2296301071 conn(0x7f16640fe3c0 msgr2=0x7f166410e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 -- 172.21.15.1:0/2296301071 shutdown_connections 2024-04-26T21:41:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:12.183+0000 7f166c200640 1 -- 172.21.15.1:0/2296301071 wait complete. 2024-04-26T21:41:12.256 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:11 smithi186 ceph-mon[32991]: pgmap v338: 129 pgs: 129 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-04-26T21:41:12.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:11 smithi001 ceph-mon[28054]: pgmap v338: 129 pgs: 129 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-04-26T21:41:12.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:11 smithi038 ceph-mon[32724]: pgmap v338: 129 pgs: 129 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-04-26T21:41:12.596 INFO:tasks.workunit.client.0.smithi001.stdout: 1 418824 4.43 MB/sec execute 430 sec latency 39.396 ms 2024-04-26T21:41:13.596 INFO:tasks.workunit.client.0.smithi001.stdout: 1 419045 4.42 MB/sec execute 431 sec latency 29.718 ms 2024-04-26T21:41:14.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:14 smithi001 ceph-mon[28054]: pgmap v339: 129 pgs: 129 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-04-26T21:41:14.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:14 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:14.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:14 smithi038 ceph-mon[32724]: pgmap v339: 129 pgs: 129 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-04-26T21:41:14.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:14 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:14 smithi186 ceph-mon[32991]: pgmap v339: 129 pgs: 129 active+clean; 277 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-04-26T21:41:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:14 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:14.596 INFO:tasks.workunit.client.0.smithi001.stdout: 1 419180 4.41 MB/sec execute 432 sec latency 765.952 ms 2024-04-26T21:41:15.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:15 smithi001 ceph-mon[28054]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T21:41:15.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:15 smithi038 ceph-mon[32724]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T21:41:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:15 smithi186 ceph-mon[32991]: osdmap e106: 12 total, 12 up, 12 in 2024-04-26T21:41:15.597 INFO:tasks.workunit.client.0.smithi001.stdout: 1 419180 4.40 MB/sec execute 433 sec latency 1766.107 ms 2024-04-26T21:41:16.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:16 smithi001 ceph-mon[28054]: pgmap v341: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 184 op/s 2024-04-26T21:41:16.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:16 smithi038 ceph-mon[32724]: pgmap v341: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 184 op/s 2024-04-26T21:41:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:16 smithi186 ceph-mon[32991]: pgmap v341: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 184 op/s 2024-04-26T21:41:16.597 INFO:tasks.workunit.client.0.smithi001.stdout: 1 419180 4.39 MB/sec execute 434 sec latency 2766.196 ms 2024-04-26T21:41:17.597 INFO:tasks.workunit.client.0.smithi001.stdout: 1 419496 4.38 MB/sec execute 435 sec latency 3317.314 ms 2024-04-26T21:41:18.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:18 smithi001 ceph-mon[28054]: pgmap v342: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.5 MiB/s wr, 148 op/s 2024-04-26T21:41:18.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:18 smithi038 ceph-mon[32724]: pgmap v342: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.5 MiB/s wr, 148 op/s 2024-04-26T21:41:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:18 smithi186 ceph-mon[32991]: pgmap v342: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.5 MiB/s wr, 148 op/s 2024-04-26T21:41:18.597 INFO:tasks.workunit.client.0.smithi001.stdout: 1 420276 4.38 MB/sec execute 436 sec latency 34.311 ms 2024-04-26T21:41:19.597 INFO:tasks.workunit.client.0.smithi001.stdout: 1 421178 4.38 MB/sec execute 437 sec latency 29.400 ms 2024-04-26T21:41:20.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:20 smithi001 ceph-mon[28054]: pgmap v343: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-04-26T21:41:20.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:20 smithi038 ceph-mon[32724]: pgmap v343: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-04-26T21:41:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:20 smithi186 ceph-mon[32991]: pgmap v343: 129 pgs: 129 active+clean; 279 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 1.5 MiB/s wr, 147 op/s 2024-04-26T21:41:20.597 INFO:tasks.workunit.client.0.smithi001.stdout: 1 422245 4.39 MB/sec execute 438 sec latency 26.428 ms 2024-04-26T21:41:21.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 422542 4.38 MB/sec execute 439 sec latency 27.593 ms 2024-04-26T21:41:22.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:22 smithi001 ceph-mon[28054]: pgmap v344: 129 pgs: 129 active+clean; 281 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-04-26T21:41:22.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:22 smithi038 ceph-mon[32724]: pgmap v344: 129 pgs: 129 active+clean; 281 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-04-26T21:41:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:22 smithi186 ceph-mon[32991]: pgmap v344: 129 pgs: 129 active+clean; 281 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-04-26T21:41:22.597 INFO:tasks.workunit.client.0.smithi001.stdout: 1 423192 4.38 MB/sec execute 440 sec latency 69.695 ms 2024-04-26T21:41:23.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 424382 4.37 MB/sec execute 441 sec latency 32.949 ms 2024-04-26T21:41:24.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:24 smithi001 ceph-mon[28054]: pgmap v345: 129 pgs: 129 active+clean; 281 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-04-26T21:41:24.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:24 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:24.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:24 smithi038 ceph-mon[32724]: pgmap v345: 129 pgs: 129 active+clean; 281 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-04-26T21:41:24.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:24 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:24 smithi186 ceph-mon[32991]: pgmap v345: 129 pgs: 129 active+clean; 281 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 1.3 MiB/s wr, 131 op/s 2024-04-26T21:41:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:24 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:24.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 425141 4.38 MB/sec execute 442 sec latency 38.773 ms 2024-04-26T21:41:25.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.344+0000 7fd6f5600640 1 -- 172.21.15.1:0/521620551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f0103110 msgr2=0x7fd6f010f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:25.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.344+0000 7fd6f5600640 1 --2- 172.21.15.1:0/521620551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f0103110 0x7fd6f010f620 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009ca0 tx=0x7fd6e802a8c0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.344+0000 7fd6f5600640 1 -- 172.21.15.1:0/521620551 shutdown_connections 2024-04-26T21:41:25.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.344+0000 7fd6f5600640 1 --2- 172.21.15.1:0/521620551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f0103110 0x7fd6f010f620 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.344+0000 7fd6f5600640 1 --2- 172.21.15.1:0/521620551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6f0102750 0x7fd6f0102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.344+0000 7fd6f5600640 1 --2- 172.21.15.1:0/521620551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0108760 0x7fd6f0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.344+0000 7fd6f5600640 1 -- 172.21.15.1:0/521620551 >> 172.21.15.1:0/521620551 conn(0x7fd6f00fe240 msgr2=0x7fd6f0100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:25.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.344+0000 7fd6f5600640 1 -- 172.21.15.1:0/521620551 shutdown_connections 2024-04-26T21:41:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.344+0000 7fd6f5600640 1 -- 172.21.15.1:0/521620551 wait complete. 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.345+0000 7fd6f5600640 1 Processor -- start 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.345+0000 7fd6f5600640 1 -- start start 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.346+0000 7fd6f5600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6f0102750 0x7fd6f019b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.346+0000 7fd6f5600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0103110 0x7fd6f019b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.346+0000 7fd6f5600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f0108760 0x7fd6f019f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.346+0000 7fd6f5600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f0112a70 con 0x7fd6f0102750 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.346+0000 7fd6f5600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f01128f0 con 0x7fd6f0108760 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.346+0000 7fd6f5600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f0112bf0 con 0x7fd6f0103110 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.346+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0103110 0x7fd6f019b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.346+0000 7fd6efe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6f0102750 0x7fd6f019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.346+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0103110 0x7fd6f019b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41008/0 (socket says 172.21.15.1:41008) 2024-04-26T21:41:25.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.346+0000 7fd6ef400640 1 -- 172.21.15.1:0/3653852942 learned_addr learned my addr 172.21.15.1:0/3653852942 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:25.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.347+0000 7fd6f4c00640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f0108760 0x7fd6f019f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.347+0000 7fd6ef400640 1 -- 172.21.15.1:0/3653852942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f0108760 msgr2=0x7fd6f019f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.347+0000 7fd6ef400640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f0108760 0x7fd6f019f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.347+0000 7fd6ef400640 1 -- 172.21.15.1:0/3653852942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6f0102750 msgr2=0x7fd6f019b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.347+0000 7fd6ef400640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6f0102750 0x7fd6f019b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.347+0000 7fd6ef400640 1 -- 172.21.15.1:0/3653852942 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f01a0100 con 0x7fd6f0103110 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.347+0000 7fd6efe00640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6f0102750 0x7fd6f019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.347+0000 7fd6ef400640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0103110 0x7fd6f019b650 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00e9f0 tx=0x7fd6dc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.348+0000 7fd6ecc00640 1 -- 172.21.15.1:0/3653852942 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc00cd80 con 0x7fd6f0103110 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.348+0000 7fd6f5600640 1 -- 172.21.15.1:0/3653852942 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f01a03f0 con 0x7fd6f0103110 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.348+0000 7fd6ecc00640 1 -- 172.21.15.1:0/3653852942 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6dc0108e0 con 0x7fd6f0103110 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.348+0000 7fd6ecc00640 1 -- 172.21.15.1:0/3653852942 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc018980 con 0x7fd6f0103110 2024-04-26T21:41:25.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.348+0000 7fd6f5600640 1 -- 172.21.15.1:0/3653852942 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f01a7c80 con 0x7fd6f0103110 2024-04-26T21:41:25.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.351+0000 7fd6f9979740 1 Processor -- start 2024-04-26T21:41:25.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.352+0000 7fd6f9979740 1 -- start start 2024-04-26T21:41:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.352+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da6809e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.352+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 0x55da680a8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.352+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da680a91c0 0x55da680ab5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.352+0000 7fd6f9979740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55da68003380 con 0x55da6809e900 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.352+0000 7fd6f9979740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55da67f9fc30 con 0x55da6809dfe0 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.352+0000 7fd6f9979740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55da67fff340 con 0x55da680a91c0 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6f4c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da6809e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6f4c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da6809e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47354/0 (socket says 172.21.15.1:47354) 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6f4c00640 1 -- 172.21.15.1:0/2836779955 learned_addr learned my addr 172.21.15.1:0/2836779955 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6efe00640 1 --2- 172.21.15.1:0/2836779955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 0x55da680a8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6f4c00640 1 -- 172.21.15.1:0/2836779955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da680a91c0 msgr2=0x55da680ab5b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6f4c00640 1 --2- 172.21.15.1:0/2836779955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da680a91c0 0x55da680ab5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6f4c00640 1 -- 172.21.15.1:0/2836779955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 msgr2=0x55da680a8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6f4c00640 1 --2- 172.21.15.1:0/2836779955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 0x55da680a8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6f4c00640 1 -- 172.21.15.1:0/2836779955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da67fc1940 con 0x55da6809dfe0 2024-04-26T21:41:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6efe00640 1 --2- 172.21.15.1:0/2836779955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 0x55da680a8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:41:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6ecc00640 1 -- 172.21.15.1:0/3653852942 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd6dc010430 con 0x7fd6f0103110 2024-04-26T21:41:25.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.353+0000 7fd6f4c00640 1 --2- 172.21.15.1:0/2836779955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da6809e3c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009ca0 tx=0x7fd6e80093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a6a72e29b91c88e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.354+0000 7fd6e5200640 1 -- 172.21.15.1:0/2836779955 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8031ce0 con 0x55da6809dfe0 2024-04-26T21:41:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.354+0000 7fd6e5200640 1 -- 172.21.15.1:0/2836779955 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6e8036030 con 0x55da6809dfe0 2024-04-26T21:41:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.354+0000 7fd6e5200640 1 -- 172.21.15.1:0/2836779955 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e80042a0 con 0x55da6809dfe0 2024-04-26T21:41:25.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.354+0000 7fd6f9979740 1 -- 172.21.15.1:0/2836779955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 msgr2=0x55da6809e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.354+0000 7fd6f9979740 1 --2- 172.21.15.1:0/2836779955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da6809e3c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009ca0 tx=0x7fd6e80093e0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.355+0000 7fd6f9979740 1 -- 172.21.15.1:0/2836779955 shutdown_connections 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.355+0000 7fd6f9979740 1 --2- 172.21.15.1:0/2836779955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da680a91c0 0x55da680ab5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.355+0000 7fd6f9979740 1 --2- 172.21.15.1:0/2836779955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 0x55da680a8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.355+0000 7fd6f9979740 1 --2- 172.21.15.1:0/2836779955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da6809e3c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.355+0000 7fd6f9979740 1 -- 172.21.15.1:0/2836779955 >> 172.21.15.1:0/2836779955 conn(0x55da6809a4f0 msgr2=0x55da6809a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.355+0000 7fd6f9979740 1 -- 172.21.15.1:0/2836779955 shutdown_connections 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.355+0000 7fd6f9979740 1 -- 172.21.15.1:0/2836779955 wait complete. 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.355+0000 7fd6ecc00640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd6d40a3c10 0x7fd6d40a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.356+0000 7fd6ecc00640 1 -- 172.21.15.1:0/3653852942 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7fd6dc0d0fa0 con 0x7fd6f0103110 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.356+0000 7fd6f9979740 1 -- start start 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.356+0000 7fd6f9979740 1 Processor -- start 2024-04-26T21:41:25.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.356+0000 7fd6efe00640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd6d40a3c10 0x7fd6d40a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 0x55da680a6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da680a91c0 0x55da680a6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6f9979740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55da680abc10 con 0x55da6809e900 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6f9979740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55da67f8ac60 con 0x55da6809dfe0 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6f9979740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55da680abd90 con 0x55da680a91c0 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6f9979740 10 client.0 _ll_register_callbacks cb 0x55da67fb6e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6efe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6efe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47368/0 (socket says 172.21.15.1:47368) 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6efe00640 1 -- 172.21.15.1:0/2238101145 learned_addr learned my addr 172.21.15.1:0/2238101145 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6efe00640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd6d40a3c10 0x7fd6d40a60d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55da6809f580 tx=0x7fd6e0005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6f4c00640 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da680a91c0 0x55da680a6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6ef400640 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 0x55da680a6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6efe00640 1 -- 172.21.15.1:0/2238101145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da680a91c0 msgr2=0x55da680a6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6efe00640 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da680a91c0 0x55da680a6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6efe00640 1 -- 172.21.15.1:0/2238101145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 msgr2=0x55da680a6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6efe00640 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 0x55da680a6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.357+0000 7fd6efe00640 1 -- 172.21.15.1:0/2238101145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da67f99510 con 0x55da6809dfe0 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.358+0000 7fd6efe00640 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a6110 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0011c60 tx=0x7fd6e00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f80aae0a2a297f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.358+0000 7fd6d9600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.358+0000 7fd6d9600640 1 -- 172.21.15.1:0/2238101145 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e001c660 con 0x55da6809dfe0 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.358+0000 7fd6f9979740 1 -- 172.21.15.1:0/2238101145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6e8036030 con 0x55da6809dfe0 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.358+0000 7fd6d9600640 1 -- 172.21.15.1:0/2238101145 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7fd6e001c800 con 0x55da6809dfe0 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.359+0000 7fd6d9600640 1 -- 172.21.15.1:0/2238101145 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd6e001ca40 con 0x55da6809dfe0 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.359+0000 7fd6f9979740 10 client.26405 fetch_fsmap learned FSMap version 38 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.359+0000 7fd6f9979740 1 -- 172.21.15.1:0/2238101145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55da67d15210 con 0x55da6809dfe0 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.359+0000 7fd6d9600640 1 -- 172.21.15.1:0/2238101145 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6e8036030 con 0x55da6809dfe0 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.359+0000 7fd6d9600640 1 -- 172.21.15.1:0/2238101145 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fd6e0025070 con 0x55da6809dfe0 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.360+0000 7fd6f9979740 10 client.26405 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.360+0000 7fd6f9979740 10 client.26405 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:41:25.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.360+0000 7fd6f9979740 20 client.26405 populate_metadata read hostname 'smithi001' 2024-04-26T21:41:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.360+0000 7fd6f9979740 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55da680b8d10 0x55da680bb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.360+0000 7fd6f9979740 4 client.26405 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:41:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.360+0000 7fd6f9979740 1 -- 172.21.15.1:0/2238101145 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da68010f90 con 0x55da680b8d10 2024-04-26T21:41:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.360+0000 7fd6ef400640 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55da680b8d10 0x55da680bb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.361+0000 7fd6ef400640 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55da680b8d10 0x55da680bb140 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79c42c9795882f70 server_cookie=3814a5e546dd21aa in_seq=0 out_seq=0 2024-04-26T21:41:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.361+0000 7fd6d9600640 10 client.26405 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.361+0000 7fd6d9600640 1 -- 172.21.15.1:0/2238101145 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55da68010f90 con 0x55da680b8d10 2024-04-26T21:41:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.361+0000 7fd6d9600640 10 client.26405 handle_command_reply: tid=0 2024-04-26T21:41:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.361+0000 7fd6f9979740 1 client.26405 shutdown 2024-04-26T21:41:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 1 -- 172.21.15.1:0/2238101145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 msgr2=0x55da680a6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a6110 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0011c60 tx=0x7fd6e00046c0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 1 -- 172.21.15.1:0/2238101145 shutdown_connections 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55da680b8d10 0x55da680bb140 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da680a91c0 0x55da680a6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da6809e900 0x55da680a6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 1 --2- 172.21.15.1:0/2238101145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a6110 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6d9600640 0 client.26405 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 1 -- 172.21.15.1:0/2238101145 >> 172.21.15.1:0/2238101145 conn(0x55da6809a4f0 msgr2=0x55da680a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 1 -- 172.21.15.1:0/2238101145 shutdown_connections 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 1 -- 172.21.15.1:0/2238101145 wait complete. 2024-04-26T21:41:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.362+0000 7fd6f9979740 20 client.26405 trim_cache size 0 max 16384 2024-04-26T21:41:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.380+0000 7fd6f9979740 1 Processor -- start 2024-04-26T21:41:25.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.380+0000 7fd6f9979740 1 -- start start 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.381+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.381+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 0x55da680a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.381+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 0x55da680a80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.381+0000 7fd6f9979740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f01b4810 con 0x55da680a91c0 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.381+0000 7fd6f9979740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f01b2080 con 0x55da6809dfe0 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.381+0000 7fd6f9979740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f01afef0 con 0x55da6809e900 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.381+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 0x55da680a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.381+0000 7fd6f4c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 0x55da680a80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.381+0000 7fd6efe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.382+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 0x55da680a09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41026/0 (socket says 172.21.15.1:41026) 2024-04-26T21:41:25.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.382+0000 7fd6ef400640 1 -- 172.21.15.1:0/465455395 learned_addr learned my addr 172.21.15.1:0/465455395 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.382+0000 7fd6efe00640 1 -- 172.21.15.1:0/465455395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 msgr2=0x55da680a09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.382+0000 7fd6efe00640 1 --2- 172.21.15.1:0/465455395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 0x55da680a09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.382+0000 7fd6efe00640 1 -- 172.21.15.1:0/465455395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 msgr2=0x55da680a80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.382+0000 7fd6efe00640 1 --2- 172.21.15.1:0/465455395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 0x55da680a80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.382+0000 7fd6efe00640 1 -- 172.21.15.1:0/465455395 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da67de1620 con 0x55da6809dfe0 2024-04-26T21:41:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.382+0000 7fd6efe00640 1 --2- 172.21.15.1:0/465455395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a5a10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0013cc0 tx=0x7fd6e001ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb413b02621119ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.383+0000 7fd6dbe00640 1 -- 172.21.15.1:0/465455395 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e001cde0 con 0x55da6809dfe0 2024-04-26T21:41:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.383+0000 7fd6dbe00640 1 -- 172.21.15.1:0/465455395 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6e0004020 con 0x55da6809dfe0 2024-04-26T21:41:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.383+0000 7fd6dbe00640 1 -- 172.21.15.1:0/465455395 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00053c0 con 0x55da6809dfe0 2024-04-26T21:41:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.383+0000 7fd6f9979740 1 -- 172.21.15.1:0/465455395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 msgr2=0x55da680a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.383+0000 7fd6f9979740 1 --2- 172.21.15.1:0/465455395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a5a10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0013cc0 tx=0x7fd6e001ec60 comp rx=0 tx=0).stop 2024-04-26T21:41:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.384+0000 7fd6f9979740 1 -- 172.21.15.1:0/465455395 shutdown_connections 2024-04-26T21:41:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.384+0000 7fd6f9979740 1 --2- 172.21.15.1:0/465455395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 0x55da680a09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.384+0000 7fd6f9979740 1 --2- 172.21.15.1:0/465455395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 0x55da680a80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.384+0000 7fd6f9979740 1 --2- 172.21.15.1:0/465455395 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680a5a10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.384+0000 7fd6f9979740 1 -- 172.21.15.1:0/465455395 >> 172.21.15.1:0/465455395 conn(0x55da6809a4f0 msgr2=0x55da68099590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.384+0000 7fd6f9979740 1 -- 172.21.15.1:0/465455395 shutdown_connections 2024-04-26T21:41:25.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.384+0000 7fd6f9979740 1 -- 172.21.15.1:0/465455395 wait complete. 2024-04-26T21:41:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.384+0000 7fd6f9979740 1 -- start start 2024-04-26T21:41:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.384+0000 7fd6f9979740 1 Processor -- start 2024-04-26T21:41:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 0x55da680b1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6f9979740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 0x55da680a2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6f9979740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f01b0540 con 0x55da680a91c0 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6f9979740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f01b19b0 con 0x55da6809dfe0 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6f9979740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f01b0c10 con 0x55da6809e900 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6f9979740 10 client.0 _ll_register_callbacks cb 0x7fd6e0013f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6efe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 0x55da680b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6efe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680b0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47384/0 (socket says 172.21.15.1:47384) 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6ef400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 0x55da680b1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41042/0 (socket says 172.21.15.1:41042) 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6efe00640 1 -- 172.21.15.1:0/3091464830 learned_addr learned my addr 172.21.15.1:0/3091464830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.385+0000 7fd6f4c00640 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 0x55da680a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6efe00640 1 -- 172.21.15.1:0/3091464830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 msgr2=0x55da680b1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6efe00640 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 0x55da680b1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6efe00640 1 -- 172.21.15.1:0/3091464830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 msgr2=0x55da680a2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6efe00640 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 0x55da680a2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6efe00640 1 -- 172.21.15.1:0/3091464830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da67fbdcb0 con 0x55da6809dfe0 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6f4c00640 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 0x55da680a2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6efe00640 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680b0fe0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd6e001d010 tx=0x7fd6e0004480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53a8a2a8f9fcdc43 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6d9600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6d9600640 1 -- 172.21.15.1:0/3091464830 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0044060 con 0x55da6809dfe0 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6d9600640 1 -- 172.21.15.1:0/3091464830 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7fd6e001c670 con 0x55da6809dfe0 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.386+0000 7fd6f9979740 1 -- 172.21.15.1:0/3091464830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6e0004020 con 0x55da6809dfe0 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6d9600640 1 -- 172.21.15.1:0/3091464830 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd6e004b050 con 0x55da6809dfe0 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6f9979740 10 client.26411 fetch_fsmap learned FSMap version 38 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6f9979740 1 -- 172.21.15.1:0/3091464830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55da68127200 con 0x55da6809dfe0 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6d9600640 1 -- 172.21.15.1:0/3091464830 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6e0004020 con 0x55da6809dfe0 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6d9600640 1 -- 172.21.15.1:0/3091464830 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7fd6e0025040 con 0x55da6809dfe0 2024-04-26T21:41:25.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6f9979740 10 client.26411 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T21:41:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6f9979740 10 client.26411 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:41:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6f9979740 20 client.26411 populate_metadata read hostname 'smithi001' 2024-04-26T21:41:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6f9979740 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55da68127810 0x55da68129c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6f9979740 4 client.26411 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:41:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.387+0000 7fd6f9979740 1 -- 172.21.15.1:0/3091464830 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55da6800ae00 con 0x55da68127810 2024-04-26T21:41:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.388+0000 7fd6ef400640 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55da68127810 0x55da68129c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.388+0000 7fd6ef400640 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55da68127810 0x55da68129c30 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c76d17a1f8a417e1 server_cookie=5b1cf40ca74296d1 in_seq=0 out_seq=0 2024-04-26T21:41:25.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.388+0000 7fd6d9600640 10 client.26411 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6d9600640 1 -- 172.21.15.1:0/3091464830 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55da6800ae00 con 0x55da68127810 2024-04-26T21:41:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6d9600640 10 client.26411 handle_command_reply: tid=0 2024-04-26T21:41:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6f9979740 1 client.26411 shutdown 2024-04-26T21:41:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6f9979740 1 -- 172.21.15.1:0/3091464830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 msgr2=0x55da680b0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6f9979740 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680b0fe0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd6e001d010 tx=0x7fd6e0004480 comp rx=0 tx=0).stop 2024-04-26T21:41:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6f9979740 1 -- 172.21.15.1:0/3091464830 shutdown_connections 2024-04-26T21:41:25.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6f9979740 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55da68127810 0x55da68129c30 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6f9979740 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55da6809e900 0x55da680b1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6f9979740 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55da680a91c0 0x55da680a2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6f9979740 1 --2- 172.21.15.1:0/3091464830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55da6809dfe0 0x55da680b0fe0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6f9979740 1 -- 172.21.15.1:0/3091464830 >> 172.21.15.1:0/3091464830 conn(0x55da6809a4f0 msgr2=0x55da680a6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6d9600640 0 client.26411 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.460+0000 7fd6f9979740 1 -- 172.21.15.1:0/3091464830 shutdown_connections 2024-04-26T21:41:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.461+0000 7fd6f9979740 1 -- 172.21.15.1:0/3091464830 wait complete. 2024-04-26T21:41:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.461+0000 7fd6f9979740 20 client.26411 trim_cache size 0 max 16384 2024-04-26T21:41:25.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:41:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:41:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:41:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:41:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.461+0000 7fd6f5600640 1 -- 172.21.15.1:0/3653852942 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd6d40a3c10 msgr2=0x7fd6d40a60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.461+0000 7fd6f5600640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd6d40a3c10 0x7fd6d40a60d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55da6809f580 tx=0x7fd6e0005f90 comp rx=0 tx=0).stop 2024-04-26T21:41:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.461+0000 7fd6f5600640 1 -- 172.21.15.1:0/3653852942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0103110 msgr2=0x7fd6f019b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.461+0000 7fd6f5600640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0103110 0x7fd6f019b650 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00e9f0 tx=0x7fd6dc00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.462+0000 7fd6f5600640 1 -- 172.21.15.1:0/3653852942 shutdown_connections 2024-04-26T21:41:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.462+0000 7fd6f5600640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd6d40a3c10 0x7fd6d40a60d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.462+0000 7fd6f5600640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f0103110 0x7fd6f019b650 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.462+0000 7fd6f5600640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6f0108760 0x7fd6f019f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.462+0000 7fd6f5600640 1 --2- 172.21.15.1:0/3653852942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd6f0102750 0x7fd6f019b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.462+0000 7fd6f5600640 1 -- 172.21.15.1:0/3653852942 >> 172.21.15.1:0/3653852942 conn(0x7fd6f00fe240 msgr2=0x7fd6f010df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.462+0000 7fd6f5600640 1 -- 172.21.15.1:0/3653852942 shutdown_connections 2024-04-26T21:41:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:25.462+0000 7fd6f5600640 1 -- 172.21.15.1:0/3653852942 wait complete. 2024-04-26T21:41:25.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 425997 4.38 MB/sec execute 443 sec latency 61.904 ms 2024-04-26T21:41:26.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:26 smithi038 ceph-mon[32724]: pgmap v346: 129 pgs: 129 active+clean; 286 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-04-26T21:41:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:26 smithi186 ceph-mon[32991]: pgmap v346: 129 pgs: 129 active+clean; 286 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-04-26T21:41:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:26 smithi001 ceph-mon[28054]: pgmap v346: 129 pgs: 129 active+clean; 286 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 187 op/s 2024-04-26T21:41:26.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 426293 4.37 MB/sec execute 444 sec latency 29.155 ms 2024-04-26T21:41:27.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 426924 4.37 MB/sec execute 445 sec latency 55.337 ms 2024-04-26T21:41:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:28 smithi038 ceph-mon[32724]: pgmap v347: 129 pgs: 129 active+clean; 286 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-26T21:41:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:28 smithi186 ceph-mon[32991]: pgmap v347: 129 pgs: 129 active+clean; 286 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-26T21:41:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:28 smithi001 ceph-mon[28054]: pgmap v347: 129 pgs: 129 active+clean; 286 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-26T21:41:28.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 428099 4.37 MB/sec execute 446 sec latency 39.062 ms 2024-04-26T21:41:29.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 429050 4.38 MB/sec execute 447 sec latency 22.110 ms 2024-04-26T21:41:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:30 smithi186 ceph-mon[32991]: pgmap v348: 129 pgs: 129 active+clean; 286 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-26T21:41:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:30 smithi001 ceph-mon[28054]: pgmap v348: 129 pgs: 129 active+clean; 286 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-26T21:41:30.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:30 smithi038 ceph-mon[32724]: pgmap v348: 129 pgs: 129 active+clean; 286 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 1.4 MiB/s wr, 102 op/s 2024-04-26T21:41:30.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 429586 4.38 MB/sec execute 448 sec latency 23.126 ms 2024-04-26T21:41:31.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 430096 4.37 MB/sec execute 449 sec latency 37.506 ms 2024-04-26T21:41:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:32 smithi186 ceph-mon[32991]: pgmap v349: 129 pgs: 129 active+clean; 290 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-04-26T21:41:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:32 smithi001 ceph-mon[28054]: pgmap v349: 129 pgs: 129 active+clean; 290 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-04-26T21:41:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:32 smithi038 ceph-mon[32724]: pgmap v349: 129 pgs: 129 active+clean; 290 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-04-26T21:41:32.598 INFO:tasks.workunit.client.0.smithi001.stdout: 1 430997 4.37 MB/sec execute 450 sec latency 53.417 ms 2024-04-26T21:41:33.599 INFO:tasks.workunit.client.0.smithi001.stdout: 1 431972 4.37 MB/sec execute 451 sec latency 78.086 ms 2024-04-26T21:41:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:34 smithi186 ceph-mon[32991]: pgmap v350: 129 pgs: 129 active+clean; 290 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-04-26T21:41:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:34 smithi001 ceph-mon[28054]: pgmap v350: 129 pgs: 129 active+clean; 290 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-04-26T21:41:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:34 smithi038 ceph-mon[32724]: pgmap v350: 129 pgs: 129 active+clean; 290 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 144 op/s 2024-04-26T21:41:34.599 INFO:tasks.workunit.client.0.smithi001.stdout: 1 432993 4.38 MB/sec execute 452 sec latency 26.038 ms 2024-04-26T21:41:35.599 INFO:tasks.workunit.client.0.smithi001.stdout: 1 433222 4.37 MB/sec execute 453 sec latency 25.499 ms 2024-04-26T21:41:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:36 smithi186 ceph-mon[32991]: pgmap v351: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T21:41:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:36 smithi001 ceph-mon[28054]: pgmap v351: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T21:41:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:36 smithi038 ceph-mon[32724]: pgmap v351: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 201 op/s 2024-04-26T21:41:36.599 INFO:tasks.workunit.client.0.smithi001.stdout: 1 433908 4.37 MB/sec execute 454 sec latency 31.852 ms 2024-04-26T21:41:37.599 INFO:tasks.workunit.client.0.smithi001.stdout: 1 435114 4.37 MB/sec execute 455 sec latency 42.562 ms 2024-04-26T21:41:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:38 smithi186 ceph-mon[32991]: pgmap v352: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-26T21:41:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:38 smithi001 ceph-mon[28054]: pgmap v352: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-26T21:41:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:38 smithi038 ceph-mon[32724]: pgmap v352: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-26T21:41:38.599 INFO:tasks.workunit.client.0.smithi001.stdout: 1 436140 4.38 MB/sec execute 456 sec latency 41.205 ms 2024-04-26T21:41:38.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.622+0000 7f311fe00640 1 -- 172.21.15.1:0/3608003351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f312010ee30 msgr2=0x7f312010f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:38.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.622+0000 7f311fe00640 1 --2- 172.21.15.1:0/3608003351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f312010ee30 0x7f312010f210 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3110009900 tx=0x7f311002a480 comp rx=0 tx=0).stop 2024-04-26T21:41:38.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.623+0000 7f311fe00640 1 -- 172.21.15.1:0/3608003351 shutdown_connections 2024-04-26T21:41:38.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.623+0000 7f311fe00640 1 --2- 172.21.15.1:0/3608003351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3120108d70 0x7f31201091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.623+0000 7f311fe00640 1 --2- 172.21.15.1:0/3608003351 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3120109710 0x7f3120115bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.623+0000 7f311fe00640 1 --2- 172.21.15.1:0/3608003351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f312010ee30 0x7f312010f210 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.623+0000 7f311fe00640 1 -- 172.21.15.1:0/3608003351 >> 172.21.15.1:0/3608003351 conn(0x7f3120078490 msgr2=0x7f312007a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.623+0000 7f311fe00640 1 -- 172.21.15.1:0/3608003351 shutdown_connections 2024-04-26T21:41:38.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.623+0000 7f311fe00640 1 -- 172.21.15.1:0/3608003351 wait complete. 2024-04-26T21:41:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.624+0000 7f311fe00640 1 Processor -- start 2024-04-26T21:41:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.624+0000 7f311fe00640 1 -- start start 2024-04-26T21:41:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.624+0000 7f311fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3120108d70 0x7f312019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.624+0000 7f311fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3120109710 0x7f312019f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.624+0000 7f311fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f312010ee30 0x7f31201a3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.624+0000 7f311fe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3120118ff0 con 0x7f3120108d70 2024-04-26T21:41:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.624+0000 7f311fe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3120118e70 con 0x7f312010ee30 2024-04-26T21:41:38.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.624+0000 7f311fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3120119170 con 0x7f3120109710 2024-04-26T21:41:38.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f311ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3120108d70 0x7f312019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f311f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f312010ee30 0x7f31201a3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f3117e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3120109710 0x7f312019f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f3117e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3120109710 0x7f312019f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:55394/0 (socket says 172.21.15.1:55394) 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f311ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3120108d70 0x7f312019f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49454/0 (socket says 172.21.15.1:49454) 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f311ea00640 1 -- 172.21.15.1:0/2103422933 learned_addr learned my addr 172.21.15.1:0/2103422933 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f311f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f312010ee30 0x7f31201a3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34162/0 (socket says 172.21.15.1:34162) 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f3117e00640 1 -- 172.21.15.1:0/2103422933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f312010ee30 msgr2=0x7f31201a3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f3117e00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f312010ee30 0x7f31201a3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f3117e00640 1 -- 172.21.15.1:0/2103422933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3120108d70 msgr2=0x7f312019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f3117e00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3120108d70 0x7f312019f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f3117e00640 1 -- 172.21.15.1:0/2103422933 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31201a44a0 con 0x7f3120109710 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.625+0000 7f311f400640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f312010ee30 0x7f31201a3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.626+0000 7f3117e00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3120109710 0x7f312019f9f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f310800e9f0 tx=0x7f310800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.626+0000 7f3124a00640 1 -- 172.21.15.1:0/2103422933 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310800cd80 con 0x7f3120109710 2024-04-26T21:41:38.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.626+0000 7f3124a00640 1 -- 172.21.15.1:0/2103422933 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31080035f0 con 0x7f3120109710 2024-04-26T21:41:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.626+0000 7f311fe00640 1 -- 172.21.15.1:0/2103422933 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31201a4790 con 0x7f3120109710 2024-04-26T21:41:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.626+0000 7f3124a00640 1 -- 172.21.15.1:0/2103422933 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3108010690 con 0x7f3120109710 2024-04-26T21:41:38.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.626+0000 7f311fe00640 1 -- 172.21.15.1:0/2103422933 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31201ac020 con 0x7f3120109710 2024-04-26T21:41:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.629+0000 7f3128b9f740 1 Processor -- start 2024-04-26T21:41:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f3124a00640 1 -- 172.21.15.1:0/2103422933 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f31080037a0 con 0x7f3120109710 2024-04-26T21:41:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f3128b9f740 1 -- start start 2024-04-26T21:41:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a2e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a38c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 0x560983a3b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f3128b9f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560983993380 con 0x560983a2e900 2024-04-26T21:41:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f3128b9f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56098392fc30 con 0x560983a2dfe0 2024-04-26T21:41:38.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f3128b9f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56098398f340 con 0x560983a391c0 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f311f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a2e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f311f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a2e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34178/0 (socket says 172.21.15.1:34178) 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f311f400640 1 -- 172.21.15.1:0/2399495252 learned_addr learned my addr 172.21.15.1:0/2399495252 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f311ea00640 1 --2- 172.21.15.1:0/2399495252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a38c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.630+0000 7f3117e00640 1 --2- 172.21.15.1:0/2399495252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 0x560983a3b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.631+0000 7f311f400640 1 -- 172.21.15.1:0/2399495252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 msgr2=0x560983a3b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.631+0000 7f311f400640 1 --2- 172.21.15.1:0/2399495252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 0x560983a3b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.631+0000 7f311f400640 1 -- 172.21.15.1:0/2399495252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 msgr2=0x560983a38c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.631+0000 7f311f400640 1 --2- 172.21.15.1:0/2399495252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a38c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.631+0000 7f311f400640 1 -- 172.21.15.1:0/2399495252 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560983951940 con 0x560983a2dfe0 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.631+0000 7f3117e00640 1 --2- 172.21.15.1:0/2399495252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 0x560983a3b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:41:38.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.631+0000 7f311f400640 1 --2- 172.21.15.1:0/2399495252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a2e3c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f31180117e0 tx=0x7f3118011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8bc515ee41680db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.631+0000 7f310fe00640 1 -- 172.21.15.1:0/2399495252 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118011310 con 0x560983a2dfe0 2024-04-26T21:41:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.631+0000 7f310fe00640 1 -- 172.21.15.1:0/2399495252 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3118003e20 con 0x560983a2dfe0 2024-04-26T21:41:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.631+0000 7f310fe00640 1 -- 172.21.15.1:0/2399495252 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31180044f0 con 0x560983a2dfe0 2024-04-26T21:41:38.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f3124a00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f31040a3d80 0x7f31040a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f3124a00640 1 -- 172.21.15.1:0/2103422933 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7f31080d0f00 con 0x7f3120109710 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f3128b9f740 1 -- 172.21.15.1:0/2399495252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 msgr2=0x560983a2e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f311ea00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f31040a3d80 0x7f31040a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f3128b9f740 1 --2- 172.21.15.1:0/2399495252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a2e3c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f31180117e0 tx=0x7f3118011ca0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f3128b9f740 1 -- 172.21.15.1:0/2399495252 shutdown_connections 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f3128b9f740 1 --2- 172.21.15.1:0/2399495252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 0x560983a3b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f3128b9f740 1 --2- 172.21.15.1:0/2399495252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a38c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f3128b9f740 1 --2- 172.21.15.1:0/2399495252 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a2e3c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f3128b9f740 1 -- 172.21.15.1:0/2399495252 >> 172.21.15.1:0/2399495252 conn(0x560983a2a4f0 msgr2=0x560983a2a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.632+0000 7f3128b9f740 1 -- 172.21.15.1:0/2399495252 shutdown_connections 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.633+0000 7f3128b9f740 1 -- 172.21.15.1:0/2399495252 wait complete. 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.633+0000 7f311ea00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f31040a3d80 0x7f31040a6240 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x560983a2f580 tx=0x7f3110005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.633+0000 7f3128b9f740 1 -- start start 2024-04-26T21:41:38.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.633+0000 7f3128b9f740 1 Processor -- start 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a36110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a36650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 0x560983a36b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f3128b9f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560983a3bdf0 con 0x560983a2e900 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f3128b9f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56098391ac60 con 0x560983a2dfe0 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f3128b9f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560983a3bf70 con 0x560983a391c0 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f3128b9f740 10 client.0 _ll_register_callbacks cb 0x560983946e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f311ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a36110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f311ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a36110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34184/0 (socket says 172.21.15.1:34184) 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f311ea00640 1 -- 172.21.15.1:0/2995854346 learned_addr learned my addr 172.21.15.1:0/2995854346 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f311f400640 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 0x560983a36b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.634+0000 7f3117e00640 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a36650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f311ea00640 1 -- 172.21.15.1:0/2995854346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 msgr2=0x560983a36b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f311ea00640 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 0x560983a36b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f311ea00640 1 -- 172.21.15.1:0/2995854346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 msgr2=0x560983a36650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f311ea00640 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a36650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f311ea00640 1 -- 172.21.15.1:0/2995854346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560983929510 con 0x560983a2dfe0 2024-04-26T21:41:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f3117e00640 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a36650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:41:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f311ea00640 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a36110 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3110006010 tx=0x7f3110009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c7862e6556af62c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f310cc00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:41:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f310cc00640 1 -- 172.21.15.1:0/2995854346 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110035040 con 0x560983a2dfe0 2024-04-26T21:41:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f310cc00640 1 -- 172.21.15.1:0/2995854346 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7f31100318b0 con 0x560983a2dfe0 2024-04-26T21:41:38.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f3128b9f740 1 -- 172.21.15.1:0/2995854346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3118003e20 con 0x560983a2dfe0 2024-04-26T21:41:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.635+0000 7f310cc00640 1 -- 172.21.15.1:0/2995854346 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f3110003e80 con 0x560983a2dfe0 2024-04-26T21:41:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.636+0000 7f3128b9f740 10 client.26423 fetch_fsmap learned FSMap version 38 2024-04-26T21:41:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.636+0000 7f3128b9f740 1 -- 172.21.15.1:0/2995854346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5609836a5210 con 0x560983a2dfe0 2024-04-26T21:41:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.636+0000 7f310cc00640 1 -- 172.21.15.1:0/2995854346 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3118003e20 con 0x560983a2dfe0 2024-04-26T21:41:38.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.636+0000 7f310cc00640 1 -- 172.21.15.1:0/2995854346 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f3110044030 con 0x560983a2dfe0 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.636+0000 7f3128b9f740 10 client.26423 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.636+0000 7f3128b9f740 10 client.26423 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.636+0000 7f3128b9f740 20 client.26423 populate_metadata read hostname 'smithi001' 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.636+0000 7f3128b9f740 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560983a48f60 0x560983a4b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.636+0000 7f3128b9f740 4 client.26423 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.636+0000 7f3128b9f740 1 -- 172.21.15.1:0/2995854346 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609839a0f90 con 0x560983a48f60 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.637+0000 7f3117e00640 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560983a48f60 0x560983a4b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.637+0000 7f3117e00640 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560983a48f60 0x560983a4b3b0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3deed6559eb8b9b0 server_cookie=4b166a2eacd1d08f in_seq=0 out_seq=0 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.637+0000 7f310cc00640 10 client.26423 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f310cc00640 1 -- 172.21.15.1:0/2995854346 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5609839a0f90 con 0x560983a48f60 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f310cc00640 10 client.26423 handle_command_reply: tid=0 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f3128b9f740 1 client.26423 shutdown 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f3128b9f740 1 -- 172.21.15.1:0/2995854346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 msgr2=0x560983a36110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f3128b9f740 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a36110 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3110006010 tx=0x7f3110009260 comp rx=0 tx=0).stop 2024-04-26T21:41:38.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f3128b9f740 1 -- 172.21.15.1:0/2995854346 shutdown_connections 2024-04-26T21:41:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f3128b9f740 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560983a48f60 0x560983a4b3b0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f3128b9f740 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a391c0 0x560983a36b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f3128b9f740 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a36650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f310cc00640 0 client.26423 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f3128b9f740 1 --2- 172.21.15.1:0/2995854346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2dfe0 0x560983a36110 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.638+0000 7f3128b9f740 1 -- 172.21.15.1:0/2995854346 >> 172.21.15.1:0/2995854346 conn(0x560983a2a4f0 msgr2=0x560983a30270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.639+0000 7f3128b9f740 1 -- 172.21.15.1:0/2995854346 shutdown_connections 2024-04-26T21:41:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.639+0000 7f3128b9f740 1 -- 172.21.15.1:0/2995854346 wait complete. 2024-04-26T21:41:38.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.639+0000 7f3128b9f740 20 client.26423 trim_cache size 0 max 16384 2024-04-26T21:41:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.656+0000 7f3128b9f740 1 Processor -- start 2024-04-26T21:41:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.656+0000 7f3128b9f740 1 -- start start 2024-04-26T21:41:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.656+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a44cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.656+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2e900 0x560983a309f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.656+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a391c0 0x560983a50cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.656+0000 7f3128b9f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f31201b8be0 con 0x560983a391c0 2024-04-26T21:41:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.656+0000 7f3128b9f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f31201b6450 con 0x560983a2e900 2024-04-26T21:41:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.656+0000 7f3128b9f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f31201b42c0 con 0x560983a2dfe0 2024-04-26T21:41:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f3117e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2e900 0x560983a309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f3117e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2e900 0x560983a309f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34188/0 (socket says 172.21.15.1:34188) 2024-04-26T21:41:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f3117e00640 1 -- 172.21.15.1:0/3871198618 learned_addr learned my addr 172.21.15.1:0/3871198618 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f311ea00640 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a44cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f311f400640 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a391c0 0x560983a50cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f311ea00640 1 -- 172.21.15.1:0/3871198618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2e900 msgr2=0x560983a309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f311ea00640 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2e900 0x560983a309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f311ea00640 1 -- 172.21.15.1:0/3871198618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a391c0 msgr2=0x560983a50cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f311ea00640 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a391c0 0x560983a50cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f311ea00640 1 -- 172.21.15.1:0/3871198618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560983771620 con 0x560983a2dfe0 2024-04-26T21:41:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f3117e00640 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2e900 0x560983a309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:41:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f311f400640 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a391c0 0x560983a50cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:41:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f311ea00640 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a44cb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3110051db0 tx=0x7f311002a480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9c5a10bd48d83cf server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.657+0000 7f310f400640 1 -- 172.21.15.1:0/3871198618 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110051470 con 0x560983a2dfe0 2024-04-26T21:41:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f310f400640 1 -- 172.21.15.1:0/3871198618 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f311004c070 con 0x560983a2dfe0 2024-04-26T21:41:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f310f400640 1 -- 172.21.15.1:0/3871198618 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311003d650 con 0x560983a2dfe0 2024-04-26T21:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f3128b9f740 1 -- 172.21.15.1:0/3871198618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 msgr2=0x560983a44cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f3128b9f740 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a44cb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3110051db0 tx=0x7f311002a480 comp rx=0 tx=0).stop 2024-04-26T21:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f3128b9f740 1 -- 172.21.15.1:0/3871198618 shutdown_connections 2024-04-26T21:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f3128b9f740 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a391c0 0x560983a50cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f3128b9f740 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a2e900 0x560983a309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f3128b9f740 1 --2- 172.21.15.1:0/3871198618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a44cb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f3128b9f740 1 -- 172.21.15.1:0/3871198618 >> 172.21.15.1:0/3871198618 conn(0x560983a2a4f0 msgr2=0x560983a29650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f3128b9f740 1 -- 172.21.15.1:0/3871198618 shutdown_connections 2024-04-26T21:41:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.658+0000 7f3128b9f740 1 -- 172.21.15.1:0/3871198618 wait complete. 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.659+0000 7f3128b9f740 1 -- start start 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.659+0000 7f3128b9f740 1 Processor -- start 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a41700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a3cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f3128b9f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a391c0 0x560983a3d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f3128b9f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f31201b4910 con 0x560983a2e900 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f3128b9f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f31201b5d80 con 0x560983a391c0 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f3128b9f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f31201b4fe0 con 0x560983a2dfe0 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f3128b9f740 10 client.0 _ll_register_callbacks cb 0x7f3110009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f311ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a41700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f311ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a41700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:55440/0 (socket says 172.21.15.1:55440) 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f311ea00640 1 -- 172.21.15.1:0/1649356325 learned_addr learned my addr 172.21.15.1:0/1649356325 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f3117e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a3cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f311f400640 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a391c0 0x560983a3d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f311ea00640 1 -- 172.21.15.1:0/1649356325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a391c0 msgr2=0x560983a3d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f311ea00640 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a391c0 0x560983a3d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f311ea00640 1 -- 172.21.15.1:0/1649356325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 msgr2=0x560983a3cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f311ea00640 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a3cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f311ea00640 1 -- 172.21.15.1:0/1649356325 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56098394dcb0 con 0x560983a2dfe0 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.660+0000 7f311ea00640 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a41700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3110042040 tx=0x7f3110009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94a858121dbd0f3b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.661+0000 7f310cc00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.661+0000 7f310cc00640 1 -- 172.21.15.1:0/1649356325 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110063060 con 0x560983a2dfe0 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.661+0000 7f3128b9f740 1 -- 172.21.15.1:0/1649356325 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f311004c070 con 0x560983a2dfe0 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.661+0000 7f310cc00640 1 -- 172.21.15.1:0/1649356325 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7f3110051470 con 0x560983a2dfe0 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.661+0000 7f310cc00640 1 -- 172.21.15.1:0/1649356325 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f311006a050 con 0x560983a2dfe0 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.661+0000 7f3128b9f740 10 client.26404 fetch_fsmap learned FSMap version 38 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.661+0000 7f3128b9f740 1 -- 172.21.15.1:0/1649356325 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x560983aaf510 con 0x560983a2dfe0 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.661+0000 7f310cc00640 1 -- 172.21.15.1:0/1649356325 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f311004c070 con 0x560983a2dfe0 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.662+0000 7f310cc00640 1 -- 172.21.15.1:0/1649356325 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 38) ==== 8354+0+0 (secure 0 0 0) 0x7f311005d070 con 0x560983a2dfe0 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.662+0000 7f3128b9f740 10 client.26404 fetch_fsmap finished waiting for FSMap version 38 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.662+0000 7f3128b9f740 10 client.26404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:41:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.662+0000 7f3128b9f740 20 client.26404 populate_metadata read hostname 'smithi001' 2024-04-26T21:41:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.662+0000 7f3128b9f740 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560983aafb10 0x560983ab1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.662+0000 7f3128b9f740 4 client.26404 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:41:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.662+0000 7f3128b9f740 1 -- 172.21.15.1:0/1649356325 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56098399ae00 con 0x560983aafb10 2024-04-26T21:41:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.662+0000 7f3117e00640 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560983aafb10 0x560983ab1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.662+0000 7f310cc00640 10 client.26404 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.662+0000 7f3117e00640 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560983aafb10 0x560983ab1f00 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f4d4893d87c819b server_cookie=86c0a82e09c27d10 in_seq=0 out_seq=0 2024-04-26T21:41:38.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.733+0000 7f310cc00640 1 -- 172.21.15.1:0/1649356325 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56098399ae00 con 0x560983aafb10 2024-04-26T21:41:38.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.733+0000 7f310cc00640 10 client.26404 handle_command_reply: tid=0 2024-04-26T21:41:38.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.733+0000 7f3128b9f740 1 client.26404 shutdown 2024-04-26T21:41:38.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 1 -- 172.21.15.1:0/1649356325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 msgr2=0x560983a41700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:38.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a41700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3110042040 tx=0x7f3110009680 comp rx=0 tx=0).stop 2024-04-26T21:41:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 1 -- 172.21.15.1:0/1649356325 shutdown_connections 2024-04-26T21:41:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560983aafb10 0x560983ab1f00 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560983a391c0 0x560983a3d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x560983a2e900 0x560983a3cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 1 --2- 172.21.15.1:0/1649356325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560983a2dfe0 0x560983a41700 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f310cc00640 0 client.26404 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 1 -- 172.21.15.1:0/1649356325 >> 172.21.15.1:0/1649356325 conn(0x560983a2a4f0 msgr2=0x560983a28dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 1 -- 172.21.15.1:0/1649356325 shutdown_connections 2024-04-26T21:41:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 1 -- 172.21.15.1:0/1649356325 wait complete. 2024-04-26T21:41:38.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.734+0000 7f3128b9f740 20 client.26404 trim_cache size 0 max 16384 2024-04-26T21:41:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:41:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:41:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:41:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:41:38.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 -- 172.21.15.1:0/2103422933 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f31040a3d80 msgr2=0x7f31040a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f31040a3d80 0x7f31040a6240 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x560983a2f580 tx=0x7f3110005a20 comp rx=0 tx=0).stop 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 -- 172.21.15.1:0/2103422933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3120109710 msgr2=0x7f312019f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3120109710 0x7f312019f9f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f310800e9f0 tx=0x7f310800eeb0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 -- 172.21.15.1:0/2103422933 shutdown_connections 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f31040a3d80 0x7f31040a6240 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3120109710 0x7f312019f9f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f312010ee30 0x7f31201a3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 --2- 172.21.15.1:0/2103422933 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3120108d70 0x7f312019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 -- 172.21.15.1:0/2103422933 >> 172.21.15.1:0/2103422933 conn(0x7f3120078490 msgr2=0x7f3120079ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.735+0000 7f311fe00640 1 -- 172.21.15.1:0/2103422933 shutdown_connections 2024-04-26T21:41:38.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:38.736+0000 7f311fe00640 1 -- 172.21.15.1:0/2103422933 wait complete. 2024-04-26T21:41:39.599 INFO:tasks.workunit.client.0.smithi001.stdout: 1 436697 4.38 MB/sec execute 457 sec latency 28.151 ms 2024-04-26T21:41:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:40 smithi186 ceph-mon[32991]: pgmap v353: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-26T21:41:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:40 smithi001 ceph-mon[28054]: pgmap v353: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-26T21:41:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:40 smithi038 ceph-mon[32724]: pgmap v353: 129 pgs: 129 active+clean; 295 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 140 op/s 2024-04-26T21:41:40.600 INFO:tasks.workunit.client.0.smithi001.stdout: 1 437120 4.37 MB/sec execute 458 sec latency 26.788 ms 2024-04-26T21:41:41.600 INFO:tasks.workunit.client.0.smithi001.stdout: 1 438139 4.37 MB/sec execute 459 sec latency 22.297 ms 2024-04-26T21:41:42.369 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:42 smithi001 ceph-mon[28054]: pgmap v354: 129 pgs: 129 active+clean; 296 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 227 op/s 2024-04-26T21:41:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:42 smithi186 ceph-mon[32991]: pgmap v354: 129 pgs: 129 active+clean; 296 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 227 op/s 2024-04-26T21:41:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:42 smithi038 ceph-mon[32724]: pgmap v354: 129 pgs: 129 active+clean; 296 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 227 op/s 2024-04-26T21:41:42.600 INFO:tasks.workunit.client.0.smithi001.stdout: 1 439233 4.37 MB/sec execute 460 sec latency 70.518 ms 2024-04-26T21:41:43.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:43 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:43.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:43 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:43.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:43 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:43.600 INFO:tasks.workunit.client.0.smithi001.stdout: 1 439662 4.37 MB/sec execute 461 sec latency 710.045 ms 2024-04-26T21:41:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:44 smithi186 ceph-mon[32991]: pgmap v355: 129 pgs: 129 active+clean; 296 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-04-26T21:41:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:44 smithi001 ceph-mon[28054]: pgmap v355: 129 pgs: 129 active+clean; 296 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-04-26T21:41:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:44 smithi038 ceph-mon[32724]: pgmap v355: 129 pgs: 129 active+clean; 296 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 143 op/s 2024-04-26T21:41:44.600 INFO:tasks.workunit.client.0.smithi001.stdout: 1 439662 4.36 MB/sec execute 462 sec latency 1710.144 ms 2024-04-26T21:41:45.600 INFO:tasks.workunit.client.0.smithi001.stdout: 1 439662 4.35 MB/sec execute 463 sec latency 2710.286 ms 2024-04-26T21:41:46.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:46 smithi186 ceph-mon[32991]: pgmap v356: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-04-26T21:41:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:46 smithi001 ceph-mon[28054]: pgmap v356: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-04-26T21:41:46.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:46 smithi038 ceph-mon[32724]: pgmap v356: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 193 op/s 2024-04-26T21:41:46.600 INFO:tasks.workunit.client.0.smithi001.stdout: 1 439675 4.34 MB/sec execute 464 sec latency 3412.272 ms 2024-04-26T21:41:47.600 INFO:tasks.workunit.client.0.smithi001.stdout: 1 440091 4.34 MB/sec execute 465 sec latency 769.464 ms 2024-04-26T21:41:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:48 smithi186 ceph-mon[32991]: pgmap v357: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-04-26T21:41:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:48 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:48 smithi001 ceph-mon[28054]: pgmap v357: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-04-26T21:41:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:48 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:48.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:48 smithi038 ceph-mon[32724]: pgmap v357: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-04-26T21:41:48.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:48 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:41:48.600 INFO:tasks.workunit.client.0.smithi001.stdout: 1 440320 4.33 MB/sec execute 466 sec latency 28.728 ms 2024-04-26T21:41:49.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:49 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:41:49.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:49 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:41:49.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:49 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:41:49.601 INFO:tasks.workunit.client.0.smithi001.stdout: 1 441076 4.33 MB/sec execute 467 sec latency 22.901 ms 2024-04-26T21:41:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:50 smithi186 ceph-mon[32991]: pgmap v358: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-04-26T21:41:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:50 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:41:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:50 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:41:50.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:50 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:41:50.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:50 smithi001 ceph-mon[28054]: pgmap v358: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-04-26T21:41:50.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:50 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:41:50.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:50 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:41:50.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:50 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:41:50.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:50 smithi038 ceph-mon[32724]: pgmap v358: 129 pgs: 129 active+clean; 299 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 136 op/s 2024-04-26T21:41:50.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:50 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:41:50.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:50 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:41:50.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:50 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:41:50.601 INFO:tasks.workunit.client.0.smithi001.stdout: 1 442285 4.33 MB/sec execute 468 sec latency 31.687 ms 2024-04-26T21:41:51.601 INFO:tasks.workunit.client.0.smithi001.stdout: 1 443367 4.34 MB/sec execute 469 sec latency 27.914 ms 2024-04-26T21:41:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.893+0000 7f1c2e600640 1 -- 172.21.15.1:0/995280779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c281014f0 msgr2=0x7f1c281018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.893+0000 7f1c2e600640 1 --2- 172.21.15.1:0/995280779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c281014f0 0x7f1c281018d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c009900 tx=0x7f1c1c02a410 comp rx=0 tx=0).stop 2024-04-26T21:41:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.893+0000 7f1c2e600640 1 -- 172.21.15.1:0/995280779 shutdown_connections 2024-04-26T21:41:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.893+0000 7f1c2e600640 1 --2- 172.21.15.1:0/995280779 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c2810f0a0 0x7f1c28111490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.893+0000 7f1c2e600640 1 --2- 172.21.15.1:0/995280779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c28101ea0 0x7f1c2810eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.893+0000 7f1c2e600640 1 --2- 172.21.15.1:0/995280779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c281014f0 0x7f1c281018d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.893+0000 7f1c2e600640 1 -- 172.21.15.1:0/995280779 >> 172.21.15.1:0/995280779 conn(0x7f1c280fd110 msgr2=0x7f1c280ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.893+0000 7f1c2e600640 1 -- 172.21.15.1:0/995280779 shutdown_connections 2024-04-26T21:41:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.893+0000 7f1c2e600640 1 -- 172.21.15.1:0/995280779 wait complete. 2024-04-26T21:41:51.897 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.894+0000 7f1c2e600640 1 Processor -- start 2024-04-26T21:41:51.897 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.894+0000 7f1c2e600640 1 -- start start 2024-04-26T21:41:51.897 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2e600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c281014f0 0x7f1c2819b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2e600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c28101ea0 0x7f1c2819b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2e600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c2810f0a0 0x7f1c2819faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2e600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1c28114c50 con 0x7f1c28101ea0 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2e600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1c28114ad0 con 0x7f1c281014f0 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2e600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1c28114dd0 con 0x7f1c2810f0a0 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c281014f0 0x7f1c2819b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2d200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c281014f0 0x7f1c2819b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:33364/0 (socket says 172.21.15.1:33364) 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2d200640 1 -- 172.21.15.1:0/2008355352 learned_addr learned my addr 172.21.15.1:0/2008355352 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c27e00640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c28101ea0 0x7f1c2819b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c2810f0a0 0x7f1c2819faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2d200640 1 -- 172.21.15.1:0/2008355352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c2810f0a0 msgr2=0x7f1c2819faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2d200640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c2810f0a0 0x7f1c2819faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2d200640 1 -- 172.21.15.1:0/2008355352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c28101ea0 msgr2=0x7f1c2819b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:51.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2d200640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c28101ea0 0x7f1c2819b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.895+0000 7f1c2d200640 1 -- 172.21.15.1:0/2008355352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c281a0220 con 0x7f1c281014f0 2024-04-26T21:41:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.896+0000 7f1c2d200640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c281014f0 0x7f1c2819b230 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c005ab0 tx=0x7f1c1c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.896+0000 7f1c25600640 1 -- 172.21.15.1:0/2008355352 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c03f070 con 0x7f1c281014f0 2024-04-26T21:41:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.896+0000 7f1c2e600640 1 -- 172.21.15.1:0/2008355352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c281a04b0 con 0x7f1c281014f0 2024-04-26T21:41:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.896+0000 7f1c2e600640 1 -- 172.21.15.1:0/2008355352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c281a7da0 con 0x7f1c281014f0 2024-04-26T21:41:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.896+0000 7f1c25600640 1 -- 172.21.15.1:0/2008355352 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c1c031b60 con 0x7f1c281014f0 2024-04-26T21:41:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.896+0000 7f1c25600640 1 -- 172.21.15.1:0/2008355352 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1c049060 con 0x7f1c281014f0 2024-04-26T21:41:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.901+0000 7f1c25600640 1 -- 172.21.15.1:0/2008355352 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f1c1c042070 con 0x7f1c281014f0 2024-04-26T21:41:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.901+0000 7f1c32a43740 1 Processor -- start 2024-04-26T21:41:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.901+0000 7f1c32a43740 1 -- start start 2024-04-26T21:41:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d003c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 0x55f0a8d0ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d0d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c32a43740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55f0a8c65380 con 0x55f0a8d00900 2024-04-26T21:41:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c32a43740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f0a8c01c30 con 0x55f0a8cfffe0 2024-04-26T21:41:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c32a43740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f0a8c61340 con 0x55f0a8d0b1c0 2024-04-26T21:41:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c2dc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d003c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c2dc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d003c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:33366/0 (socket says 172.21.15.1:33366) 2024-04-26T21:41:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c2dc00640 1 -- 172.21.15.1:0/898115960 learned_addr learned my addr 172.21.15.1:0/898115960 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c2d200640 1 --2- 172.21.15.1:0/898115960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 0x55f0a8d0ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c2dc00640 1 -- 172.21.15.1:0/898115960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 msgr2=0x55f0a8d0d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/898115960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d0d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c2dc00640 1 -- 172.21.15.1:0/898115960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 msgr2=0x55f0a8d0ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.902+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/898115960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 0x55f0a8d0ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.903+0000 7f1c2dc00640 1 -- 172.21.15.1:0/898115960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0a8c23940 con 0x55f0a8cfffe0 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.903+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/898115960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d003c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009f30 tx=0x7f1c200118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91111d0bb0d1a82b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.903+0000 7f1c1a000640 1 -- 172.21.15.1:0/898115960 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20004d20 con 0x55f0a8cfffe0 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.903+0000 7f1c1a000640 1 -- 172.21.15.1:0/898115960 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c20004050 con 0x55f0a8cfffe0 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.903+0000 7f1c1a000640 1 -- 172.21.15.1:0/898115960 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20005260 con 0x55f0a8cfffe0 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c25600640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1c100a3d00 0x7f1c100a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c32a43740 1 -- 172.21.15.1:0/898115960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8cfffe0 msgr2=0x55f0a8d003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c32a43740 1 --2- 172.21.15.1:0/898115960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d003c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009f30 tx=0x7f1c200118d0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c2d200640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1c100a3d00 0x7f1c100a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c32a43740 1 -- 172.21.15.1:0/898115960 shutdown_connections 2024-04-26T21:41:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c25600640 1 -- 172.21.15.1:0/2008355352 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7f1c1c0f0f90 con 0x7f1c281014f0 2024-04-26T21:41:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c32a43740 1 --2- 172.21.15.1:0/898115960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d0d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c32a43740 1 --2- 172.21.15.1:0/898115960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 0x55f0a8d0ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c32a43740 1 --2- 172.21.15.1:0/898115960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d003c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c32a43740 1 -- 172.21.15.1:0/898115960 >> 172.21.15.1:0/898115960 conn(0x55f0a8cfc4f0 msgr2=0x55f0a8cfc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c32a43740 1 -- 172.21.15.1:0/898115960 shutdown_connections 2024-04-26T21:41:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.904+0000 7f1c32a43740 1 -- 172.21.15.1:0/898115960 wait complete. 2024-04-26T21:41:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.905+0000 7f1c32a43740 1 -- start start 2024-04-26T21:41:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.905+0000 7f1c2d200640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1c100a3d00 0x7f1c100a61c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0ee330 tx=0x7f1c1c031400 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.905+0000 7f1c32a43740 1 Processor -- start 2024-04-26T21:41:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d08110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d08650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d08b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c32a43740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55f0a8d0ddf0 con 0x55f0a8cfffe0 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c32a43740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f0a8becc60 con 0x55f0a8d00900 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c32a43740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f0a8d0df70 con 0x55f0a8d0b1c0 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c32a43740 10 client.0 _ll_register_callbacks cb 0x55f0a8c18e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c27e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d08650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c27e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d08650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:33382/0 (socket says 172.21.15.1:33382) 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c27e00640 1 -- 172.21.15.1:0/4273968655 learned_addr learned my addr 172.21.15.1:0/4273968655 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d08b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.906+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d08110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.907+0000 7f1c27e00640 1 -- 172.21.15.1:0/4273968655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 msgr2=0x55f0a8d08b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.908+0000 7f1c27e00640 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d08b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.908+0000 7f1c27e00640 1 -- 172.21.15.1:0/4273968655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 msgr2=0x55f0a8d08110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.908+0000 7f1c27e00640 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d08110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.908+0000 7f1c27e00640 1 -- 172.21.15.1:0/4273968655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0a8bfb510 con 0x55f0a8d00900 2024-04-26T21:41:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.908+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d08110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.908+0000 7f1c27e00640 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d08650 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1c140116f0 tx=0x7f1c14011bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fed5fe7dab19e1f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.908+0000 7f1c0ea00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.908+0000 7f1c0ea00640 1 -- 172.21.15.1:0/4273968655 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400f8a0 con 0x55f0a8d00900 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.909+0000 7f1c0ea00640 1 -- 172.21.15.1:0/4273968655 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7f1c1400fa40 con 0x55f0a8d00900 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.909+0000 7f1c32a43740 1 -- 172.21.15.1:0/4273968655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c20004050 con 0x55f0a8d00900 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.909+0000 7f1c0ea00640 1 -- 172.21.15.1:0/4273968655 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1c1400fc40 con 0x55f0a8d00900 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.909+0000 7f1c32a43740 10 client.26441 fetch_fsmap learned FSMap version 40 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.909+0000 7f1c32a43740 1 -- 172.21.15.1:0/4273968655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f0a8977210 con 0x55f0a8d00900 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.910+0000 7f1c0ea00640 1 -- 172.21.15.1:0/4273968655 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c20004050 con 0x55f0a8d00900 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.910+0000 7f1c0ea00640 1 -- 172.21.15.1:0/4273968655 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f1c14018070 con 0x55f0a8d00900 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.910+0000 7f1c32a43740 10 client.26441 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.910+0000 7f1c32a43740 10 client.26441 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.910+0000 7f1c32a43740 20 client.26441 populate_metadata read hostname 'smithi001' 2024-04-26T21:41:51.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.910+0000 7f1c32a43740 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0a8d1afc0 0x55f0a8d1d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.910+0000 7f1c32a43740 4 client.26441 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:41:51.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.910+0000 7f1c32a43740 1 -- 172.21.15.1:0/4273968655 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0a8c72f90 con 0x55f0a8d1afc0 2024-04-26T21:41:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.911+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0a8d1afc0 0x55f0a8d1d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.911+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0a8d1afc0 0x55f0a8d1d410 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73e66f566b9731a2 server_cookie=1b842f821885e11a in_seq=0 out_seq=0 2024-04-26T21:41:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.911+0000 7f1c0ea00640 10 client.26441 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.912+0000 7f1c0ea00640 1 -- 172.21.15.1:0/4273968655 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f0a8c72f90 con 0x55f0a8d1afc0 2024-04-26T21:41:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.912+0000 7f1c0ea00640 10 client.26441 handle_command_reply: tid=0 2024-04-26T21:41:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.912+0000 7f1c32a43740 1 client.26441 shutdown 2024-04-26T21:41:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.912+0000 7f1c32a43740 1 -- 172.21.15.1:0/4273968655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 msgr2=0x55f0a8d08650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.912+0000 7f1c32a43740 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d08650 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1c140116f0 tx=0x7f1c14011bb0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.913+0000 7f1c32a43740 1 -- 172.21.15.1:0/4273968655 shutdown_connections 2024-04-26T21:41:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.913+0000 7f1c32a43740 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0a8d1afc0 0x55f0a8d1d410 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.913+0000 7f1c32a43740 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d08b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.913+0000 7f1c32a43740 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d08650 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.913+0000 7f1c32a43740 1 --2- 172.21.15.1:0/4273968655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d08110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.913+0000 7f1c32a43740 1 -- 172.21.15.1:0/4273968655 >> 172.21.15.1:0/4273968655 conn(0x55f0a8cfc4f0 msgr2=0x55f0a8d02270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.913+0000 7f1c0ea00640 0 client.26441 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.913+0000 7f1c32a43740 1 -- 172.21.15.1:0/4273968655 shutdown_connections 2024-04-26T21:41:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.913+0000 7f1c32a43740 1 -- 172.21.15.1:0/4273968655 wait complete. 2024-04-26T21:41:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.913+0000 7f1c32a43740 20 client.26441 trim_cache size 0 max 16384 2024-04-26T21:41:51.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.945+0000 7f1c32a43740 1 Processor -- start 2024-04-26T21:41:51.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c32a43740 1 -- start start 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d16cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d07270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d029a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c32a43740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1c281b48c0 con 0x55f0a8cfffe0 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c32a43740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1c281b2130 con 0x55f0a8d00900 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c32a43740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1c281affa0 con 0x55f0a8d0b1c0 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c27e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d07270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c2dc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d16cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c27e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d07270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:33396/0 (socket says 172.21.15.1:33396) 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.946+0000 7f1c27e00640 1 -- 172.21.15.1:0/1171417862 learned_addr learned my addr 172.21.15.1:0/1171417862 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:51.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.947+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/1171417862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.947+0000 7f1c27e00640 1 -- 172.21.15.1:0/1171417862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 msgr2=0x55f0a8d029a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:51.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.947+0000 7f1c27e00640 1 --2- 172.21.15.1:0/1171417862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d029a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.947+0000 7f1c27e00640 1 -- 172.21.15.1:0/1171417862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 msgr2=0x55f0a8d16cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:51.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.947+0000 7f1c27e00640 1 --2- 172.21.15.1:0/1171417862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d16cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.947+0000 7f1c27e00640 1 -- 172.21.15.1:0/1171417862 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0a8d00730 con 0x55f0a8d00900 2024-04-26T21:41:51.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.948+0000 7f1c27e00640 1 --2- 172.21.15.1:0/1171417862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d07270 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1c14023760 tx=0x7f1c14005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63e66bb5c9bab1ae server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:51.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.948+0000 7f1c19600640 1 -- 172.21.15.1:0/1171417862 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c1400faa0 con 0x55f0a8d00900 2024-04-26T21:41:51.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.949+0000 7f1c19600640 1 -- 172.21.15.1:0/1171417862 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c14028070 con 0x55f0a8d00900 2024-04-26T21:41:51.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.949+0000 7f1c19600640 1 -- 172.21.15.1:0/1171417862 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c140038e0 con 0x55f0a8d00900 2024-04-26T21:41:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.952+0000 7f1c32a43740 1 -- 172.21.15.1:0/1171417862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 msgr2=0x55f0a8d07270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.952+0000 7f1c32a43740 1 --2- 172.21.15.1:0/1171417862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d07270 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1c14023760 tx=0x7f1c14005f90 comp rx=0 tx=0).stop 2024-04-26T21:41:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.952+0000 7f1c32a43740 1 -- 172.21.15.1:0/1171417862 shutdown_connections 2024-04-26T21:41:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.952+0000 7f1c32a43740 1 --2- 172.21.15.1:0/1171417862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.952+0000 7f1c32a43740 1 --2- 172.21.15.1:0/1171417862 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d00900 0x55f0a8d07270 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.952+0000 7f1c32a43740 1 --2- 172.21.15.1:0/1171417862 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d16cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.952+0000 7f1c32a43740 1 -- 172.21.15.1:0/1171417862 >> 172.21.15.1:0/1171417862 conn(0x55f0a8cfc4f0 msgr2=0x55f0a8d09830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.952+0000 7f1c32a43740 1 -- 172.21.15.1:0/1171417862 shutdown_connections 2024-04-26T21:41:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.952+0000 7f1c32a43740 1 -- 172.21.15.1:0/1171417862 wait complete. 2024-04-26T21:41:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.953+0000 7f1c32a43740 1 -- start start 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.953+0000 7f1c32a43740 1 Processor -- start 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d131c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 0x55f0a8d13700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c32a43740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d04940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c32a43740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1c281b05f0 con 0x55f0a8d00900 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c32a43740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1c281b1a60 con 0x55f0a8d0b1c0 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c32a43740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1c281b0cc0 con 0x55f0a8cfffe0 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c32a43740 10 client.0 _ll_register_callbacks cb 0x7f1c14009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c27e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 0x55f0a8d13700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c2dc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d131c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c27e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 0x55f0a8d13700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38646/0 (socket says 172.21.15.1:38646) 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c27e00640 1 -- 172.21.15.1:0/994397589 learned_addr learned my addr 172.21.15.1:0/994397589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:41:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.954+0000 7f1c2dc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d131c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41004/0 (socket says 172.21.15.1:41004) 2024-04-26T21:41:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.955+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d04940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.955+0000 7f1c2dc00640 1 -- 172.21.15.1:0/994397589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d0b1c0 msgr2=0x55f0a8d04940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.955+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d04940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.955+0000 7f1c2dc00640 1 -- 172.21.15.1:0/994397589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 msgr2=0x55f0a8d13700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:41:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.955+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 0x55f0a8d13700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:51.958 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.955+0000 7f1c2dc00640 1 -- 172.21.15.1:0/994397589 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0a8a43620 con 0x55f0a8cfffe0 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.955+0000 7f1c27e00640 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 0x55f0a8d13700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.956+0000 7f1c2dc00640 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d131c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009f00 tx=0x7f1c20005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9f6805784f13f72 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.956+0000 7f1c0ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.956+0000 7f1c0ea00640 1 -- 172.21.15.1:0/994397589 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20003c40 con 0x55f0a8cfffe0 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.956+0000 7f1c0ea00640 1 -- 172.21.15.1:0/994397589 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(106..106 src has 1..106) ==== 9642+0+0 (secure 0 0 0) 0x7f1c20003de0 con 0x55f0a8cfffe0 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.956+0000 7f1c32a43740 1 -- 172.21.15.1:0/994397589 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c14028070 con 0x55f0a8cfffe0 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.956+0000 7f1c0ea00640 1 -- 172.21.15.1:0/994397589 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1c20004a00 con 0x55f0a8cfffe0 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.956+0000 7f1c32a43740 10 client.26419 fetch_fsmap learned FSMap version 40 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.956+0000 7f1c32a43740 1 -- 172.21.15.1:0/994397589 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55f0a8c1fcb0 con 0x55f0a8cfffe0 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.957+0000 7f1c0ea00640 1 -- 172.21.15.1:0/994397589 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c14028070 con 0x55f0a8cfffe0 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.957+0000 7f1c0ea00640 1 -- 172.21.15.1:0/994397589 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f1c20025040 con 0x55f0a8cfffe0 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.957+0000 7f1c32a43740 10 client.26419 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.957+0000 7f1c32a43740 10 client.26419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.957+0000 7f1c32a43740 20 client.26419 populate_metadata read hostname 'smithi001' 2024-04-26T21:41:51.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.957+0000 7f1c32a43740 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0a8d89b20 0x55f0a8d8bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:41:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.957+0000 7f1c32a43740 4 client.26419 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:41:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.957+0000 7f1c32a43740 1 -- 172.21.15.1:0/994397589 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f0a8c6ce00 con 0x55f0a8d89b20 2024-04-26T21:41:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.957+0000 7f1c27e00640 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0a8d89b20 0x55f0a8d8bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:41:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.958+0000 7f1c27e00640 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0a8d89b20 0x55f0a8d8bf40 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ab51a1544b39c4c server_cookie=e67e9d0d6afd9d8 in_seq=0 out_seq=0 2024-04-26T21:41:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:51.958+0000 7f1c0ea00640 10 client.26419 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c0ea00640 1 -- 172.21.15.1:0/994397589 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f0a8c6ce00 con 0x55f0a8d89b20 2024-04-26T21:41:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c0ea00640 10 client.26419 handle_command_reply: tid=0 2024-04-26T21:41:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 client.26419 shutdown 2024-04-26T21:41:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 -- 172.21.15.1:0/994397589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8cfffe0 msgr2=0x55f0a8d131c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d131c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009f00 tx=0x7f1c20005fd0 comp rx=0 tx=0).stop 2024-04-26T21:41:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 -- 172.21.15.1:0/994397589 shutdown_connections 2024-04-26T21:41:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f0a8d89b20 0x55f0a8d8bf40 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:52.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f0a8d0b1c0 0x55f0a8d04940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f0a8d00900 0x55f0a8d13700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 --2- 172.21.15.1:0/994397589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f0a8cfffe0 0x55f0a8d131c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 -- 172.21.15.1:0/994397589 >> 172.21.15.1:0/994397589 conn(0x55f0a8cfc4f0 msgr2=0x55f0a8cfb650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c0ea00640 0 client.26419 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 -- 172.21.15.1:0/994397589 shutdown_connections 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.008+0000 7f1c32a43740 1 -- 172.21.15.1:0/994397589 wait complete. 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c32a43740 20 client.26419 trim_cache size 0 max 16384 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:41:52.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:41:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c2e600640 1 -- 172.21.15.1:0/2008355352 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1c100a3d00 msgr2=0x7f1c100a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c2e600640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1c100a3d00 0x7f1c100a61c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0ee330 tx=0x7f1c1c031400 comp rx=0 tx=0).stop 2024-04-26T21:41:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c2e600640 1 -- 172.21.15.1:0/2008355352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c281014f0 msgr2=0x7f1c2819b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:41:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c2e600640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c281014f0 0x7f1c2819b230 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c005ab0 tx=0x7f1c1c003c10 comp rx=0 tx=0).stop 2024-04-26T21:41:52.012 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c2e600640 1 -- 172.21.15.1:0/2008355352 shutdown_connections 2024-04-26T21:41:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c2e600640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1c100a3d00 0x7f1c100a61c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c2e600640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c2810f0a0 0x7f1c2819faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c2e600640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1c28101ea0 0x7f1c2819b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c2e600640 1 --2- 172.21.15.1:0/2008355352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c281014f0 0x7f1c2819b230 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:41:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.009+0000 7f1c2e600640 1 -- 172.21.15.1:0/2008355352 >> 172.21.15.1:0/2008355352 conn(0x7f1c280fd110 msgr2=0x7f1c280ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:41:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.010+0000 7f1c2e600640 1 -- 172.21.15.1:0/2008355352 shutdown_connections 2024-04-26T21:41:52.013 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:41:52.010+0000 7f1c2e600640 1 -- 172.21.15.1:0/2008355352 wait complete. 2024-04-26T21:41:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:52 smithi186 ceph-mon[32991]: pgmap v359: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-04-26T21:41:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:52 smithi001 ceph-mon[28054]: pgmap v359: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-04-26T21:41:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:52 smithi038 ceph-mon[32724]: pgmap v359: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 166 op/s 2024-04-26T21:41:52.601 INFO:tasks.workunit.client.0.smithi001.stdout: 1 443827 4.34 MB/sec execute 470 sec latency 58.488 ms 2024-04-26T21:41:53.601 INFO:tasks.workunit.client.0.smithi001.stdout: 1 444477 4.33 MB/sec execute 471 sec latency 52.026 ms 2024-04-26T21:41:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:54 smithi186 ceph-mon[32991]: pgmap v360: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 876 KiB/s wr, 79 op/s 2024-04-26T21:41:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:54 smithi001 ceph-mon[28054]: pgmap v360: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 876 KiB/s wr, 79 op/s 2024-04-26T21:41:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:54 smithi038 ceph-mon[32724]: pgmap v360: 129 pgs: 129 active+clean; 301 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 876 KiB/s wr, 79 op/s 2024-04-26T21:41:54.601 INFO:tasks.workunit.client.0.smithi001.stdout: 1 445546 4.33 MB/sec execute 472 sec latency 29.798 ms 2024-04-26T21:41:55.602 INFO:tasks.workunit.client.0.smithi001.stdout: 1 446394 4.34 MB/sec execute 473 sec latency 26.480 ms 2024-04-26T21:41:56.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:56 smithi186 ceph-mon[32991]: pgmap v361: 129 pgs: 129 active+clean; 306 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 155 op/s 2024-04-26T21:41:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:56 smithi001 ceph-mon[28054]: pgmap v361: 129 pgs: 129 active+clean; 306 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 155 op/s 2024-04-26T21:41:56.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:56 smithi038 ceph-mon[32724]: pgmap v361: 129 pgs: 129 active+clean; 306 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.6 MiB/s wr, 155 op/s 2024-04-26T21:41:56.602 INFO:tasks.workunit.client.0.smithi001.stdout: 1 447258 4.34 MB/sec execute 474 sec latency 26.959 ms 2024-04-26T21:41:57.602 INFO:tasks.workunit.client.0.smithi001.stdout: 1 447614 4.33 MB/sec execute 475 sec latency 40.289 ms 2024-04-26T21:41:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:41:58 smithi186 ceph-mon[32991]: pgmap v362: 129 pgs: 129 active+clean; 306 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-04-26T21:41:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:41:58 smithi001 ceph-mon[28054]: pgmap v362: 129 pgs: 129 active+clean; 306 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-04-26T21:41:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:41:58 smithi038 ceph-mon[32724]: pgmap v362: 129 pgs: 129 active+clean; 306 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-04-26T21:41:58.602 INFO:tasks.workunit.client.0.smithi001.stdout: 1 448517 4.33 MB/sec execute 476 sec latency 21.788 ms 2024-04-26T21:41:59.602 INFO:tasks.workunit.client.0.smithi001.stdout: 1 449517 4.33 MB/sec execute 477 sec latency 32.614 ms 2024-04-26T21:42:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:00 smithi186 ceph-mon[32991]: pgmap v363: 129 pgs: 129 active+clean; 306 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-04-26T21:42:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:00 smithi001 ceph-mon[28054]: pgmap v363: 129 pgs: 129 active+clean; 306 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-04-26T21:42:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:00 smithi038 ceph-mon[32724]: pgmap v363: 129 pgs: 129 active+clean; 306 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 105 op/s 2024-04-26T21:42:00.602 INFO:tasks.workunit.client.0.smithi001.stdout: 1 450662 4.34 MB/sec execute 478 sec latency 70.956 ms 2024-04-26T21:42:01.602 INFO:tasks.workunit.client.0.smithi001.stdout: 1 450686 4.33 MB/sec execute 479 sec latency 126.204 ms 2024-04-26T21:42:02.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:02 smithi186 ceph-mon[32991]: pgmap v364: 129 pgs: 129 active+clean; 310 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-26T21:42:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:02 smithi001 ceph-mon[28054]: pgmap v364: 129 pgs: 129 active+clean; 310 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-26T21:42:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:02 smithi038 ceph-mon[32724]: pgmap v364: 129 pgs: 129 active+clean; 310 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 163 op/s 2024-04-26T21:42:02.602 INFO:tasks.workunit.client.0.smithi001.stdout: 1 450791 4.32 MB/sec execute 480 sec latency 111.431 ms 2024-04-26T21:42:03.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:03 smithi001 ceph-mon[28054]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T21:42:03.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:03 smithi038 ceph-mon[32724]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T21:42:03.602 INFO:tasks.workunit.client.0.smithi001.stdout: 1 451144 4.32 MB/sec execute 481 sec latency 78.746 ms 2024-04-26T21:42:03.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:03 smithi186 ceph-mon[32991]: osdmap e107: 12 total, 12 up, 12 in 2024-04-26T21:42:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:04 smithi001 ceph-mon[28054]: pgmap v366: 129 pgs: 129 active+clean; 310 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-04-26T21:42:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:04 smithi038 ceph-mon[32724]: pgmap v366: 129 pgs: 129 active+clean; 310 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-04-26T21:42:04.602 INFO:tasks.workunit.client.0.smithi001.stdout: 1 451884 4.32 MB/sec execute 482 sec latency 27.269 ms 2024-04-26T21:42:04.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:04 smithi186 ceph-mon[32991]: pgmap v366: 129 pgs: 129 active+clean; 310 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-04-26T21:42:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.178+0000 7f0ee6c00640 1 -- 172.21.15.1:0/2766817694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee0100c90 msgr2=0x7f0ee0101110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.178+0000 7f0ee6c00640 1 --2- 172.21.15.1:0/2766817694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee0100c90 0x7f0ee0101110 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009930 tx=0x7f0ecc02a420 comp rx=0 tx=0).stop 2024-04-26T21:42:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.178+0000 7f0ee6c00640 1 -- 172.21.15.1:0/2766817694 shutdown_connections 2024-04-26T21:42:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.178+0000 7f0ee6c00640 1 --2- 172.21.15.1:0/2766817694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee0100c90 0x7f0ee0101110 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.178+0000 7f0ee6c00640 1 --2- 172.21.15.1:0/2766817694 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ee0101650 0x7f0ee010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.178+0000 7f0ee6c00640 1 --2- 172.21.15.1:0/2766817694 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee0108880 0x7f0ee0108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.178+0000 7f0ee6c00640 1 -- 172.21.15.1:0/2766817694 >> 172.21.15.1:0/2766817694 conn(0x7f0ee0077bf0 msgr2=0x7f0ee00fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.178+0000 7f0ee6c00640 1 -- 172.21.15.1:0/2766817694 shutdown_connections 2024-04-26T21:42:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.178+0000 7f0ee6c00640 1 -- 172.21.15.1:0/2766817694 wait complete. 2024-04-26T21:42:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.179+0000 7f0ee6c00640 1 Processor -- start 2024-04-26T21:42:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.179+0000 7f0ee6c00640 1 -- start start 2024-04-26T21:42:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee6c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee0100c90 0x7f0ee019b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee6c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee0101650 0x7f0ee019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee6c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ee0108880 0x7f0ee019fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee6c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee0112b90 con 0x7f0ee0100c90 2024-04-26T21:42:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee6c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee0112a10 con 0x7f0ee0108880 2024-04-26T21:42:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee6c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee0112d10 con 0x7f0ee0101650 2024-04-26T21:42:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee5800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee0100c90 0x7f0ee019b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee4e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee0101650 0x7f0ee019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee6200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ee0108880 0x7f0ee019fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee5800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee0100c90 0x7f0ee019b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53024/0 (socket says 172.21.15.1:53024) 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.180+0000 7f0ee5800640 1 -- 172.21.15.1:0/1756921467 learned_addr learned my addr 172.21.15.1:0/1756921467 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.181+0000 7f0ee5800640 1 -- 172.21.15.1:0/1756921467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee0101650 msgr2=0x7f0ee019b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.181+0000 7f0ee5800640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee0101650 0x7f0ee019b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.181+0000 7f0ee5800640 1 -- 172.21.15.1:0/1756921467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ee0108880 msgr2=0x7f0ee019fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.181+0000 7f0ee5800640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ee0108880 0x7f0ee019fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.181+0000 7f0ee5800640 1 -- 172.21.15.1:0/1756921467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee01a02c0 con 0x7f0ee0100c90 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.181+0000 7f0ee6200640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ee0108880 0x7f0ee019fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.181+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee0101650 0x7f0ee019b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:05.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.181+0000 7f0ee5800640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee0100c90 0x7f0ee019b2d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400e9f0 tx=0x7f0ed400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.181+0000 7f0ede000640 1 -- 172.21.15.1:0/1756921467 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed400cd80 con 0x7f0ee0100c90 2024-04-26T21:42:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.181+0000 7f0ede000640 1 -- 172.21.15.1:0/1756921467 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ed40108e0 con 0x7f0ee0100c90 2024-04-26T21:42:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.182+0000 7f0ede000640 1 -- 172.21.15.1:0/1756921467 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4018980 con 0x7f0ee0100c90 2024-04-26T21:42:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.182+0000 7f0ee6c00640 1 -- 172.21.15.1:0/1756921467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee01a05b0 con 0x7f0ee0100c90 2024-04-26T21:42:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.182+0000 7f0ee6c00640 1 -- 172.21.15.1:0/1756921467 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee01a7e40 con 0x7f0ee0100c90 2024-04-26T21:42:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.185+0000 7f0eeafc6740 1 Processor -- start 2024-04-26T21:42:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0eeafc6740 1 -- start start 2024-04-26T21:42:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0ede000640 1 -- 172.21.15.1:0/1756921467 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0ed4010430 con 0x7f0ee0100c90 2024-04-26T21:42:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca32fe0 0x55e8eca333c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 0x55e8eca3dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca3e1c0 0x55e8eca405b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e8ec998380 con 0x55e8eca32fe0 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e8ec934c30 con 0x55e8eca3e1c0 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e8ec994340 con 0x55e8eca33900 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0ee4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca3e1c0 0x55e8eca405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0ee4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca3e1c0 0x55e8eca405b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59710/0 (socket says 172.21.15.1:59710) 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0ee4e00640 1 -- 172.21.15.1:0/2887633229 learned_addr learned my addr 172.21.15.1:0/2887633229 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.186+0000 7f0ee6200640 1 --2- 172.21.15.1:0/2887633229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca32fe0 0x55e8eca333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.187+0000 7f0ee5800640 1 --2- 172.21.15.1:0/2887633229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 0x55e8eca3dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.187+0000 7f0ee4e00640 1 -- 172.21.15.1:0/2887633229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 msgr2=0x55e8eca3dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.187+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/2887633229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 0x55e8eca3dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.187+0000 7f0ee4e00640 1 -- 172.21.15.1:0/2887633229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca32fe0 msgr2=0x55e8eca333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.187+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/2887633229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca32fe0 0x55e8eca333c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.187+0000 7f0ee4e00640 1 -- 172.21.15.1:0/2887633229 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8ec956940 con 0x55e8eca3e1c0 2024-04-26T21:42:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.187+0000 7f0ee6200640 1 --2- 172.21.15.1:0/2887633229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca32fe0 0x55e8eca333c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:42:05.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.187+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/2887633229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca3e1c0 0x55e8eca405b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc02aa10 tx=0x7f0ecc0043f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23edee6fa3caa2e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.187+0000 7f0ed2a00640 1 -- 172.21.15.1:0/2887633229 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc035040 con 0x55e8eca3e1c0 2024-04-26T21:42:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.188+0000 7f0ed2a00640 1 -- 172.21.15.1:0/2887633229 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ecc004770 con 0x55e8eca3e1c0 2024-04-26T21:42:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.188+0000 7f0ed2a00640 1 -- 172.21.15.1:0/2887633229 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc003920 con 0x55e8eca3e1c0 2024-04-26T21:42:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.188+0000 7f0ede000640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0ec80a3d80 0x7f0ec80a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.188+0000 7f0ede000640 1 -- 172.21.15.1:0/1756921467 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9730+0+0 (secure 0 0 0) 0x7f0ed40d0ef0 con 0x7f0ee0100c90 2024-04-26T21:42:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.188+0000 7f0ee6200640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0ec80a3d80 0x7f0ec80a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2887633229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca3e1c0 msgr2=0x55e8eca405b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/2887633229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca3e1c0 0x55e8eca405b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc02aa10 tx=0x7f0ecc0043f0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2887633229 shutdown_connections 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/2887633229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 0x55e8eca3dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/2887633229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca3e1c0 0x55e8eca405b0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/2887633229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca32fe0 0x55e8eca333c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2887633229 >> 172.21.15.1:0/2887633229 conn(0x55e8eca2f4f0 msgr2=0x55e8eca2f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0ee6200640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0ec80a3d80 0x7f0ec80a6240 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8005fd0 tx=0x7f0ed8009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2887633229 shutdown_connections 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2887633229 wait complete. 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.189+0000 7f0eeafc6740 1 -- start start 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.190+0000 7f0eeafc6740 1 Processor -- start 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.190+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 0x55e8eca3b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.190+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca3b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.190+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca3bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.190+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e8eca40d90 con 0x55e8eca3e1c0 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.190+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e8ec91fc60 con 0x55e8eca33900 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.190+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e8eca40f10 con 0x55e8eca32fe0 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.190+0000 7f0eeafc6740 10 client.0 _ll_register_callbacks cb 0x55e8ec94be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca3b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca3b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59726/0 (socket says 172.21.15.1:59726) 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee4e00640 1 -- 172.21.15.1:0/767098079 learned_addr learned my addr 172.21.15.1:0/767098079 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee5800640 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 0x55e8eca3b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee6200640 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca3bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee4e00640 1 -- 172.21.15.1:0/767098079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 msgr2=0x55e8eca3b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 0x55e8eca3b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee4e00640 1 -- 172.21.15.1:0/767098079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 msgr2=0x55e8eca3bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca3bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee4e00640 1 -- 172.21.15.1:0/767098079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8ec92e510 con 0x55e8eca33900 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee5800640 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 0x55e8eca3b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca3b5e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009900 tx=0x7f0ecc003890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db6f0d9642116294 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.191+0000 7f0ec7400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.192+0000 7f0ec7400640 1 -- 172.21.15.1:0/767098079 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc035040 con 0x55e8eca33900 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.192+0000 7f0ec7400640 1 -- 172.21.15.1:0/767098079 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9730+0+0 (secure 0 0 0) 0x7f0ecc0314a0 con 0x55e8eca33900 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.192+0000 7f0eeafc6740 1 -- 172.21.15.1:0/767098079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ecc004770 con 0x55e8eca33900 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.192+0000 7f0ec7400640 1 -- 172.21.15.1:0/767098079 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0ecc05a050 con 0x55e8eca33900 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.192+0000 7f0eeafc6740 10 client.26459 fetch_fsmap learned FSMap version 40 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.192+0000 7f0eeafc6740 1 -- 172.21.15.1:0/767098079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e8ec6aa210 con 0x55e8eca33900 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.192+0000 7f0ec7400640 1 -- 172.21.15.1:0/767098079 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ecc004770 con 0x55e8eca33900 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.193+0000 7f0ec7400640 1 -- 172.21.15.1:0/767098079 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f0ecc055030 con 0x55e8eca33900 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.193+0000 7f0eeafc6740 10 client.26459 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.193+0000 7f0eeafc6740 10 client.26459 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.193+0000 7f0eeafc6740 20 client.26459 populate_metadata read hostname 'smithi001' 2024-04-26T21:42:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.193+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e8eca4dcd0 0x55e8eca500c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.193+0000 7f0eeafc6740 4 client.26459 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:42:05.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.193+0000 7f0eeafc6740 1 -- 172.21.15.1:0/767098079 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8ec9a5f90 con 0x55e8eca4dcd0 2024-04-26T21:42:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.193+0000 7f0ee5800640 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e8eca4dcd0 0x55e8eca500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.194+0000 7f0ee5800640 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e8eca4dcd0 0x55e8eca500c0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73844b9796ee1f07 server_cookie=c6d30ff07a2e0974 in_seq=0 out_seq=0 2024-04-26T21:42:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.194+0000 7f0ec7400640 10 client.26459 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.194+0000 7f0ec7400640 1 -- 172.21.15.1:0/767098079 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e8ec9a5f90 con 0x55e8eca4dcd0 2024-04-26T21:42:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0ec7400640 10 client.26459 handle_command_reply: tid=0 2024-04-26T21:42:05.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 client.26459 shutdown 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 -- 172.21.15.1:0/767098079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 msgr2=0x55e8eca3b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca3b5e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009900 tx=0x7f0ecc003890 comp rx=0 tx=0).stop 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 -- 172.21.15.1:0/767098079 shutdown_connections 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e8eca4dcd0 0x55e8eca500c0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca3bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca3b5e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/767098079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 0x55e8eca3b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0ec7400640 0 client.26459 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 -- 172.21.15.1:0/767098079 >> 172.21.15.1:0/767098079 conn(0x55e8eca2f4f0 msgr2=0x55e8eca35270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 -- 172.21.15.1:0/767098079 shutdown_connections 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 1 -- 172.21.15.1:0/767098079 wait complete. 2024-04-26T21:42:05.199 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.195+0000 7f0eeafc6740 20 client.26459 trim_cache size 0 max 16384 2024-04-26T21:42:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0eeafc6740 1 Processor -- start 2024-04-26T21:42:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0eeafc6740 1 -- start start 2024-04-26T21:42:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca32fe0 0x55e8eca2edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 0x55e8eca3a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca359a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01b4a50 con 0x55e8eca3e1c0 2024-04-26T21:42:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01b22c0 con 0x55e8eca32fe0 2024-04-26T21:42:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01b0130 con 0x55e8eca33900 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0ee5800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca32fe0 0x55e8eca2edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0ee5800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca32fe0 0x55e8eca2edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59738/0 (socket says 172.21.15.1:59738) 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.214+0000 7f0ee5800640 1 -- 172.21.15.1:0/773007022 learned_addr learned my addr 172.21.15.1:0/773007022 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ee6200640 1 --2- 172.21.15.1:0/773007022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/773007022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 0x55e8eca3a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ee5800640 1 -- 172.21.15.1:0/773007022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 msgr2=0x55e8eca3a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ee5800640 1 --2- 172.21.15.1:0/773007022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 0x55e8eca3a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ee5800640 1 -- 172.21.15.1:0/773007022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 msgr2=0x55e8eca359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ee5800640 1 --2- 172.21.15.1:0/773007022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca359a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ee5800640 1 -- 172.21.15.1:0/773007022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8ec776620 con 0x55e8eca32fe0 2024-04-26T21:42:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/773007022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 0x55e8eca3a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:42:05.219 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ee5800640 1 --2- 172.21.15.1:0/773007022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca32fe0 0x55e8eca2edd0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400c6a0 tx=0x7f0ed40d3340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ea3547c6f738929 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ed2000640 1 -- 172.21.15.1:0/773007022 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed40a0d20 con 0x55e8eca32fe0 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ed2000640 1 -- 172.21.15.1:0/773007022 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ed40a55e0 con 0x55e8eca32fe0 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.215+0000 7f0ed2000640 1 -- 172.21.15.1:0/773007022 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed40d4280 con 0x55e8eca32fe0 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.216+0000 7f0eeafc6740 1 -- 172.21.15.1:0/773007022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca32fe0 msgr2=0x55e8eca2edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.216+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/773007022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca32fe0 0x55e8eca2edd0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400c6a0 tx=0x7f0ed40d3340 comp rx=0 tx=0).stop 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.216+0000 7f0eeafc6740 1 -- 172.21.15.1:0/773007022 shutdown_connections 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.216+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/773007022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca33900 0x55e8eca3a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.216+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/773007022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.216+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/773007022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca32fe0 0x55e8eca2edd0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.216+0000 7f0eeafc6740 1 -- 172.21.15.1:0/773007022 >> 172.21.15.1:0/773007022 conn(0x55e8eca2f4f0 msgr2=0x55e8eca3d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:05.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.216+0000 7f0eeafc6740 1 -- 172.21.15.1:0/773007022 shutdown_connections 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.216+0000 7f0eeafc6740 1 -- 172.21.15.1:0/773007022 wait complete. 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.217+0000 7f0eeafc6740 1 -- start start 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.217+0000 7f0eeafc6740 1 Processor -- start 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 0x55e8eca37c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca38160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0eeafc6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca386a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01b0780 con 0x55e8eca3e1c0 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01b1bf0 con 0x55e8eca33900 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0eeafc6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee01b0e50 con 0x55e8eca32fe0 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0eeafc6740 10 client.0 _ll_register_callbacks cb 0x7f0ecc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca38160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee5800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 0x55e8eca37c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee4e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca38160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59742/0 (socket says 172.21.15.1:59742) 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee4e00640 1 -- 172.21.15.1:0/2398492788 learned_addr learned my addr 172.21.15.1:0/2398492788 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:05.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee6200640 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca386a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee4e00640 1 -- 172.21.15.1:0/2398492788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 msgr2=0x55e8eca37c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 0x55e8eca37c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee4e00640 1 -- 172.21.15.1:0/2398492788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 msgr2=0x55e8eca386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca386a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee4e00640 1 -- 172.21.15.1:0/2398492788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8ec952cb0 con 0x55e8eca33900 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.218+0000 7f0ee5800640 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 0x55e8eca37c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.219+0000 7f0ee4e00640 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca38160 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc03ca40 tx=0x7f0ecc0092c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c40a17b4d50f9140 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.219+0000 7f0ec7400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.219+0000 7f0ec7400640 1 -- 172.21.15.1:0/2398492788 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc035030 con 0x55e8eca33900 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.219+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2398492788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ed40a55e0 con 0x55e8eca33900 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.219+0000 7f0ec7400640 1 -- 172.21.15.1:0/2398492788 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 9730+0+0 (secure 0 0 0) 0x7f0ecc0314a0 con 0x55e8eca33900 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.219+0000 7f0ec7400640 1 -- 172.21.15.1:0/2398492788 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f0ecc0044e0 con 0x55e8eca33900 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.219+0000 7f0eeafc6740 10 client.26465 fetch_fsmap learned FSMap version 40 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.219+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2398492788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55e8ecab4010 con 0x55e8eca33900 2024-04-26T21:42:05.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.220+0000 7f0ec7400640 1 -- 172.21.15.1:0/2398492788 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ed40a55e0 con 0x55e8eca33900 2024-04-26T21:42:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.220+0000 7f0ec7400640 1 -- 172.21.15.1:0/2398492788 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 40) ==== 8362+0+0 (secure 0 0 0) 0x7f0ecc055040 con 0x55e8eca33900 2024-04-26T21:42:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.220+0000 7f0eeafc6740 10 client.26465 fetch_fsmap finished waiting for FSMap version 40 2024-04-26T21:42:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.220+0000 7f0eeafc6740 10 client.26465 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:42:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.220+0000 7f0eeafc6740 20 client.26465 populate_metadata read hostname 'smithi001' 2024-04-26T21:42:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.220+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e8ecab4480 0x55e8ecab68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.220+0000 7f0eeafc6740 4 client.26465 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:42:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.220+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2398492788 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e8ec99fe00 con 0x55e8ecab4480 2024-04-26T21:42:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.220+0000 7f0ee5800640 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e8ecab4480 0x55e8ecab68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.221+0000 7f0ec7400640 10 client.26465 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.221+0000 7f0ee5800640 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e8ecab4480 0x55e8ecab68a0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b3d87b87c3d547c server_cookie=265b934ef2daac7b in_seq=0 out_seq=0 2024-04-26T21:42:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.289+0000 7f0ec7400640 1 -- 172.21.15.1:0/2398492788 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e8ec99fe00 con 0x55e8ecab4480 2024-04-26T21:42:05.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.289+0000 7f0ec7400640 10 client.26465 handle_command_reply: tid=0 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.289+0000 7f0eeafc6740 1 client.26465 shutdown 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2398492788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 msgr2=0x55e8eca38160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca38160 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc03ca40 tx=0x7f0ecc0092c0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2398492788 shutdown_connections 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e8ecab4480 0x55e8ecab68a0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e8eca3e1c0 0x55e8eca386a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e8eca33900 0x55e8eca38160 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0ec7400640 0 client.26465 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 1 --2- 172.21.15.1:0/2398492788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8eca32fe0 0x55e8eca37c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2398492788 >> 172.21.15.1:0/2398492788 conn(0x55e8eca2f4f0 msgr2=0x55e8eca3b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2398492788 shutdown_connections 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 1 -- 172.21.15.1:0/2398492788 wait complete. 2024-04-26T21:42:05.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.290+0000 7f0eeafc6740 20 client.26465 trim_cache size 0 max 16384 2024-04-26T21:42:05.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:42:05.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:42:05.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:42:05.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:42:05.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 -- 172.21.15.1:0/1756921467 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0ec80a3d80 msgr2=0x7f0ec80a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:05.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0ec80a3d80 0x7f0ec80a6240 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8005fd0 tx=0x7f0ed8009500 comp rx=0 tx=0).stop 2024-04-26T21:42:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 -- 172.21.15.1:0/1756921467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee0100c90 msgr2=0x7f0ee019b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee0100c90 0x7f0ee019b2d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0ed400e9f0 tx=0x7f0ed400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 -- 172.21.15.1:0/1756921467 shutdown_connections 2024-04-26T21:42:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0ec80a3d80 0x7f0ec80a6240 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ee0101650 0x7f0ee019b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ee0108880 0x7f0ee019fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 --2- 172.21.15.1:0/1756921467 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee0100c90 0x7f0ee019b2d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 -- 172.21.15.1:0/1756921467 >> 172.21.15.1:0/1756921467 conn(0x7f0ee0077bf0 msgr2=0x7f0ee0105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 -- 172.21.15.1:0/1756921467 shutdown_connections 2024-04-26T21:42:05.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:05.291+0000 7f0ee6c00640 1 -- 172.21.15.1:0/1756921467 wait complete. 2024-04-26T21:42:05.603 INFO:tasks.workunit.client.0.smithi001.stdout: 1 452885 4.31 MB/sec execute 483 sec latency 32.407 ms 2024-04-26T21:42:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:06 smithi001 ceph-mon[28054]: pgmap v367: 129 pgs: 129 active+clean; 312 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-26T21:42:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:06 smithi038 ceph-mon[32724]: pgmap v367: 129 pgs: 129 active+clean; 312 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-26T21:42:06.603 INFO:tasks.workunit.client.0.smithi001.stdout: 1 453787 4.32 MB/sec execute 484 sec latency 27.004 ms 2024-04-26T21:42:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:06 smithi186 ceph-mon[32991]: pgmap v367: 129 pgs: 129 active+clean; 312 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-26T21:42:07.603 INFO:tasks.workunit.client.0.smithi001.stdout: 1 454348 4.32 MB/sec execute 485 sec latency 28.939 ms 2024-04-26T21:42:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:08 smithi001 ceph-mon[28054]: pgmap v368: 129 pgs: 129 active+clean; 312 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-26T21:42:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:08 smithi038 ceph-mon[32724]: pgmap v368: 129 pgs: 129 active+clean; 312 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-26T21:42:08.603 INFO:tasks.workunit.client.0.smithi001.stdout: 1 454789 4.31 MB/sec execute 486 sec latency 53.789 ms 2024-04-26T21:42:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:08 smithi186 ceph-mon[32991]: pgmap v368: 129 pgs: 129 active+clean; 312 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-26T21:42:09.603 INFO:tasks.workunit.client.0.smithi001.stdout: 1 455646 4.31 MB/sec execute 487 sec latency 22.861 ms 2024-04-26T21:42:10.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:10 smithi001 ceph-mon[28054]: pgmap v369: 129 pgs: 129 active+clean; 312 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-26T21:42:10.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:10 smithi001 ceph-mon[28054]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T21:42:10.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:10 smithi038 ceph-mon[32724]: pgmap v369: 129 pgs: 129 active+clean; 312 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-26T21:42:10.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:10 smithi038 ceph-mon[32724]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T21:42:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:10 smithi186 ceph-mon[32991]: pgmap v369: 129 pgs: 129 active+clean; 312 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 170 op/s 2024-04-26T21:42:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:10 smithi186 ceph-mon[32991]: osdmap e108: 12 total, 12 up, 12 in 2024-04-26T21:42:10.603 INFO:tasks.workunit.client.0.smithi001.stdout: 1 456640 4.31 MB/sec execute 488 sec latency 25.899 ms 2024-04-26T21:42:11.603 INFO:tasks.workunit.client.0.smithi001.stdout: 1 457818 4.32 MB/sec execute 489 sec latency 31.579 ms 2024-04-26T21:42:12.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:12 smithi001 ceph-mon[28054]: pgmap v371: 129 pgs: 129 active+clean; 317 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 223 op/s 2024-04-26T21:42:12.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:12 smithi038 ceph-mon[32724]: pgmap v371: 129 pgs: 129 active+clean; 317 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 223 op/s 2024-04-26T21:42:12.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:12 smithi186 ceph-mon[32991]: pgmap v371: 129 pgs: 129 active+clean; 317 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.1 MiB/s wr, 223 op/s 2024-04-26T21:42:12.604 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458047 4.32 MB/sec execute 490 sec latency 24.274 ms 2024-04-26T21:42:13.604 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458375 4.31 MB/sec execute 491 sec latency 90.599 ms 2024-04-26T21:42:14.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:14 smithi001 ceph-mon[28054]: pgmap v372: 129 pgs: 129 active+clean; 317 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-26T21:42:14.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:14 smithi038 ceph-mon[32724]: pgmap v372: 129 pgs: 129 active+clean; 317 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-26T21:42:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:14 smithi186 ceph-mon[32991]: pgmap v372: 129 pgs: 129 active+clean; 317 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 194 op/s 2024-04-26T21:42:14.604 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458633 4.32 MB/sec execute 492 sec latency 558.302 ms 2024-04-26T21:42:15.604 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458633 4.31 MB/sec execute 493 sec latency 1558.543 ms 2024-04-26T21:42:16.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:16 smithi001 ceph-mon[28054]: pgmap v373: 129 pgs: 129 active+clean; 322 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-26T21:42:16.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:16 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:42:16.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:16 smithi038 ceph-mon[32724]: pgmap v373: 129 pgs: 129 active+clean; 322 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-26T21:42:16.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:16 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:42:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:16 smithi186 ceph-mon[32991]: pgmap v373: 129 pgs: 129 active+clean; 322 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-26T21:42:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:16 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:42:16.604 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458633 4.30 MB/sec execute 494 sec latency 2558.697 ms 2024-04-26T21:42:17.604 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458633 4.29 MB/sec execute 495 sec latency 3558.857 ms 2024-04-26T21:42:18.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:18 smithi001 ceph-mon[28054]: pgmap v374: 129 pgs: 129 active+clean; 322 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-26T21:42:18.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:18 smithi038 ceph-mon[32724]: pgmap v374: 129 pgs: 129 active+clean; 322 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-26T21:42:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.438+0000 7f82e2e00640 1 -- 172.21.15.1:0/2556712401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc103110 msgr2=0x7f82dc10f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.438+0000 7f82e2e00640 1 --2- 172.21.15.1:0/2556712401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc103110 0x7f82dc10f620 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f82d4009cd0 tx=0x7f82d402a8f0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.438+0000 7f82e2e00640 1 -- 172.21.15.1:0/2556712401 shutdown_connections 2024-04-26T21:42:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.438+0000 7f82e2e00640 1 --2- 172.21.15.1:0/2556712401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc103110 0x7f82dc10f620 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.438+0000 7f82e2e00640 1 --2- 172.21.15.1:0/2556712401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82dc102750 0x7f82dc102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.438+0000 7f82e2e00640 1 --2- 172.21.15.1:0/2556712401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82dc108760 0x7f82dc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.438+0000 7f82e2e00640 1 -- 172.21.15.1:0/2556712401 >> 172.21.15.1:0/2556712401 conn(0x7f82dc0fe240 msgr2=0x7f82dc100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.439+0000 7f82e2e00640 1 -- 172.21.15.1:0/2556712401 shutdown_connections 2024-04-26T21:42:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.439+0000 7f82e2e00640 1 -- 172.21.15.1:0/2556712401 wait complete. 2024-04-26T21:42:18.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.440+0000 7f82e2e00640 1 Processor -- start 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.440+0000 7f82e2e00640 1 -- start start 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.440+0000 7f82e2e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc102750 0x7f82dc19b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.440+0000 7f82e2e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82dc103110 0x7f82dc19b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.440+0000 7f82e2e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82dc108760 0x7f82dc19f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.440+0000 7f82e2e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc112a20 con 0x7f82dc102750 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.440+0000 7f82e2e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1128a0 con 0x7f82dc103110 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.440+0000 7f82e2e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc112ba0 con 0x7f82dc108760 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.440+0000 7f82e1a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc102750 0x7f82dc19b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e1000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82dc103110 0x7f82dc19b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e1a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc102750 0x7f82dc19b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50786/0 (socket says 172.21.15.1:50786) 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e1a00640 1 -- 172.21.15.1:0/394196745 learned_addr learned my addr 172.21.15.1:0/394196745 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e2400640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82dc108760 0x7f82dc19f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e1000640 1 -- 172.21.15.1:0/394196745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82dc108760 msgr2=0x7f82dc19f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e1000640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82dc108760 0x7f82dc19f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e1000640 1 -- 172.21.15.1:0/394196745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc102750 msgr2=0x7f82dc19b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e1000640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc102750 0x7f82dc19b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e1000640 1 -- 172.21.15.1:0/394196745 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82dc1a0100 con 0x7f82dc103110 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e2400640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82dc108760 0x7f82dc19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e1a00640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc102750 0x7f82dc19b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.441+0000 7f82e1000640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82dc103110 0x7f82dc19b650 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f82c800e9f0 tx=0x7f82c800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.442+0000 7f82da000640 1 -- 172.21.15.1:0/394196745 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c800ccc0 con 0x7f82dc103110 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.442+0000 7f82e2e00640 1 -- 172.21.15.1:0/394196745 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82dc1a03f0 con 0x7f82dc103110 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.442+0000 7f82da000640 1 -- 172.21.15.1:0/394196745 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82c80108e0 con 0x7f82dc103110 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.442+0000 7f82da000640 1 -- 172.21.15.1:0/394196745 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c8018880 con 0x7f82dc103110 2024-04-26T21:42:18.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.442+0000 7f82e2e00640 1 -- 172.21.15.1:0/394196745 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82dc1a7c80 con 0x7f82dc103110 2024-04-26T21:42:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.446+0000 7f82e7187740 1 Processor -- start 2024-04-26T21:42:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.446+0000 7f82e7187740 1 -- start start 2024-04-26T21:42:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.446+0000 7f82e7187740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0fea23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e7187740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 0x55de0feacc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e7187740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 0x55de0feaf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e7187740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55de0fe07380 con 0x55de0fead1c0 2024-04-26T21:42:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e7187740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55de0fda3c30 con 0x55de0fea1fe0 2024-04-26T21:42:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e7187740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55de0fe03340 con 0x55de0fea2900 2024-04-26T21:42:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82da000640 1 -- 172.21.15.1:0/394196745 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f82c8010430 con 0x7f82dc103110 2024-04-26T21:42:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e2400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0fea23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e1a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 0x55de0feacc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e2400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0fea23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50550/0 (socket says 172.21.15.1:50550) 2024-04-26T21:42:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e2400640 1 -- 172.21.15.1:0/1393170759 learned_addr learned my addr 172.21.15.1:0/1393170759 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e1000640 1 --2- 172.21.15.1:0/1393170759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 0x55de0feaf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e1a00640 1 -- 172.21.15.1:0/1393170759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 msgr2=0x55de0fea23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e1a00640 1 --2- 172.21.15.1:0/1393170759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0fea23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e1a00640 1 -- 172.21.15.1:0/1393170759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 msgr2=0x55de0feaf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e1a00640 1 --2- 172.21.15.1:0/1393170759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 0x55de0feaf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.447+0000 7f82e1a00640 1 -- 172.21.15.1:0/1393170759 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de0fdc5940 con 0x55de0fea2900 2024-04-26T21:42:18.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.448+0000 7f82e1a00640 1 --2- 172.21.15.1:0/1393170759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 0x55de0feacc80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82d00114f0 tx=0x7f82d00119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bd4ea90bf94c196 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:18.452 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.448+0000 7f82cea00640 1 -- 172.21.15.1:0/1393170759 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d001e070 con 0x55de0fea2900 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.448+0000 7f82cea00640 1 -- 172.21.15.1:0/1393170759 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d000fb30 con 0x55de0fea2900 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.448+0000 7f82cea00640 1 -- 172.21.15.1:0/1393170759 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0019bd0 con 0x55de0fea2900 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.448+0000 7f82da000640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82c40a3d00 0x7f82c40a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82da000640 1 -- 172.21.15.1:0/394196745 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7f82c80d1070 con 0x7f82dc103110 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82e2400640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82c40a3d00 0x7f82c40a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82e7187740 1 -- 172.21.15.1:0/1393170759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 msgr2=0x55de0feacc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82e7187740 1 --2- 172.21.15.1:0/1393170759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 0x55de0feacc80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f82d00114f0 tx=0x7f82d00119b0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82e7187740 1 -- 172.21.15.1:0/1393170759 shutdown_connections 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82e7187740 1 --2- 172.21.15.1:0/1393170759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 0x55de0feacc80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82e7187740 1 --2- 172.21.15.1:0/1393170759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 0x55de0feaf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82e7187740 1 --2- 172.21.15.1:0/1393170759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0fea23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82e7187740 1 -- 172.21.15.1:0/1393170759 >> 172.21.15.1:0/1393170759 conn(0x55de0fe9e4f0 msgr2=0x55de0fe9e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82e7187740 1 -- 172.21.15.1:0/1393170759 shutdown_connections 2024-04-26T21:42:18.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.449+0000 7f82e7187740 1 -- 172.21.15.1:0/1393170759 wait complete. 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.450+0000 7f82e2400640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82c40a3d00 0x7f82c40a61c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f82d4004af0 tx=0x7f82d4005c50 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.450+0000 7f82e7187740 1 -- start start 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.450+0000 7f82e7187740 1 Processor -- start 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e7187740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea1fe0 0x55de0feaa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e7187740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 0x55de0feaa640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e7187740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fead1c0 0x55de0feaab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e7187740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55de0feafdf0 con 0x55de0fea2900 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e7187740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55de0fd8ec60 con 0x55de0fead1c0 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e7187740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55de0feaff70 con 0x55de0fea1fe0 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e7187740 10 client.0 _ll_register_callbacks cb 0x55de0fdbae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e2400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fead1c0 0x55de0feaab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e1000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 0x55de0feaa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e1a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea1fe0 0x55de0feaa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e2400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fead1c0 0x55de0feaab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50564/0 (socket says 172.21.15.1:50564) 2024-04-26T21:42:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e2400640 1 -- 172.21.15.1:0/1440619579 learned_addr learned my addr 172.21.15.1:0/1440619579 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e2400640 1 -- 172.21.15.1:0/1440619579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea1fe0 msgr2=0x55de0feaa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e2400640 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea1fe0 0x55de0feaa100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e2400640 1 -- 172.21.15.1:0/1440619579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 msgr2=0x55de0feaa640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.451+0000 7f82e2400640 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 0x55de0feaa640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.452+0000 7f82e2400640 1 -- 172.21.15.1:0/1440619579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de0fd9d510 con 0x55de0fead1c0 2024-04-26T21:42:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.452+0000 7f82e1a00640 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea1fe0 0x55de0feaa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.452+0000 7f82e2400640 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fead1c0 0x55de0feaab80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f82d402ae10 tx=0x7f82d402a8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38dd7ea9c7d858c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:18.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.452+0000 7f82c3400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:42:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.452+0000 7f82c3400640 1 -- 172.21.15.1:0/1440619579 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4003970 con 0x55de0fead1c0 2024-04-26T21:42:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.452+0000 7f82e7187740 1 -- 172.21.15.1:0/1440619579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d000fb30 con 0x55de0fead1c0 2024-04-26T21:42:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.452+0000 7f82c3400640 1 -- 172.21.15.1:0/1440619579 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7f82d4003b10 con 0x55de0fead1c0 2024-04-26T21:42:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.452+0000 7f82c3400640 1 -- 172.21.15.1:0/1440619579 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f82d4003d10 con 0x55de0fead1c0 2024-04-26T21:42:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.452+0000 7f82e7187740 10 client.26477 fetch_fsmap learned FSMap version 41 2024-04-26T21:42:18.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.452+0000 7f82e7187740 1 -- 172.21.15.1:0/1440619579 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55de0fb19210 con 0x55de0fead1c0 2024-04-26T21:42:18.457 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:18 smithi186 ceph-mon[32991]: pgmap v374: 129 pgs: 129 active+clean; 322 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.453+0000 7f82c3400640 1 -- 172.21.15.1:0/1440619579 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d000fb30 con 0x55de0fead1c0 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.453+0000 7f82c3400640 1 -- 172.21.15.1:0/1440619579 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f82d4051070 con 0x55de0fead1c0 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.453+0000 7f82e7187740 10 client.26477 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.453+0000 7f82e7187740 10 client.26477 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.453+0000 7f82e7187740 20 client.26477 populate_metadata read hostname 'smithi001' 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.453+0000 7f82e7187740 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55de0febcf90 0x55de0febf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.453+0000 7f82e7187740 4 client.26477 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.453+0000 7f82e7187740 1 -- 172.21.15.1:0/1440619579 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de0fe14f90 con 0x55de0febcf90 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.453+0000 7f82e1a00640 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55de0febcf90 0x55de0febf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.454+0000 7f82e1a00640 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55de0febcf90 0x55de0febf3a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=929d5d9a68658018 server_cookie=35c558038715f5f7 in_seq=0 out_seq=0 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.454+0000 7f82c3400640 10 client.26477 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82c3400640 1 -- 172.21.15.1:0/1440619579 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55de0fe14f90 con 0x55de0febcf90 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82c3400640 10 client.26477 handle_command_reply: tid=0 2024-04-26T21:42:18.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 client.26477 shutdown 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 -- 172.21.15.1:0/1440619579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fead1c0 msgr2=0x55de0feaab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fead1c0 0x55de0feaab80 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f82d402ae10 tx=0x7f82d402a8f0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 -- 172.21.15.1:0/1440619579 shutdown_connections 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55de0febcf90 0x55de0febf3a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fead1c0 0x55de0feaab80 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 0x55de0feaa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 --2- 172.21.15.1:0/1440619579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea1fe0 0x55de0feaa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82c3400640 0 client.26477 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 -- 172.21.15.1:0/1440619579 >> 172.21.15.1:0/1440619579 conn(0x55de0fe9e4f0 msgr2=0x55de0fea4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 -- 172.21.15.1:0/1440619579 shutdown_connections 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 1 -- 172.21.15.1:0/1440619579 wait complete. 2024-04-26T21:42:18.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.455+0000 7f82e7187740 20 client.26477 trim_cache size 0 max 16384 2024-04-26T21:42:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.472+0000 7f82e7187740 1 Processor -- start 2024-04-26T21:42:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.472+0000 7f82e7187740 1 -- start start 2024-04-26T21:42:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.472+0000 7f82e7187740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0feb8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.472+0000 7f82e7187740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 0x55de0fe9df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.472+0000 7f82e7187740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fead1c0 0x55de0fea90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.472+0000 7f82e7187740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1b4920 con 0x55de0fea2900 2024-04-26T21:42:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.472+0000 7f82e7187740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1b2160 con 0x55de0fea1fe0 2024-04-26T21:42:18.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.472+0000 7f82e7187740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1affd0 con 0x55de0fead1c0 2024-04-26T21:42:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.472+0000 7f82e1a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0feb8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.472+0000 7f82e2400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fead1c0 0x55de0fea90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e2400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fead1c0 0x55de0fea90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50990/0 (socket says 172.21.15.1:50990) 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e1a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0feb8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50570/0 (socket says 172.21.15.1:50570) 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e2400640 1 -- 172.21.15.1:0/2295893777 learned_addr learned my addr 172.21.15.1:0/2295893777 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e1000640 1 --2- 172.21.15.1:0/2295893777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 0x55de0fe9df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e2400640 1 -- 172.21.15.1:0/2295893777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 msgr2=0x55de0feb8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e2400640 1 --2- 172.21.15.1:0/2295893777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0feb8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e2400640 1 -- 172.21.15.1:0/2295893777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 msgr2=0x55de0fe9df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e2400640 1 --2- 172.21.15.1:0/2295893777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 0x55de0fe9df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e2400640 1 -- 172.21.15.1:0/2295893777 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de0fea2730 con 0x55de0fead1c0 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e1000640 1 --2- 172.21.15.1:0/2295893777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 0x55de0fe9df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82e2400640 1 --2- 172.21.15.1:0/2295893777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fead1c0 0x55de0fea90e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f82d4004ac0 tx=0x7f82d4030ff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cc942774f31c2d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.473+0000 7f82ce000640 1 -- 172.21.15.1:0/2295893777 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d40308c0 con 0x55de0fead1c0 2024-04-26T21:42:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82ce000640 1 -- 172.21.15.1:0/2295893777 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d4036040 con 0x55de0fead1c0 2024-04-26T21:42:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82ce000640 1 -- 172.21.15.1:0/2295893777 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4004680 con 0x55de0fead1c0 2024-04-26T21:42:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82e7187740 1 -- 172.21.15.1:0/2295893777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fead1c0 msgr2=0x55de0fea90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82e7187740 1 --2- 172.21.15.1:0/2295893777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fead1c0 0x55de0fea90e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f82d4004ac0 tx=0x7f82d4030ff0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82e7187740 1 -- 172.21.15.1:0/2295893777 shutdown_connections 2024-04-26T21:42:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82e7187740 1 --2- 172.21.15.1:0/2295893777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fead1c0 0x55de0fea90e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82e7187740 1 --2- 172.21.15.1:0/2295893777 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fea2900 0x55de0fe9df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82e7187740 1 --2- 172.21.15.1:0/2295893777 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0feb8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82e7187740 1 -- 172.21.15.1:0/2295893777 >> 172.21.15.1:0/2295893777 conn(0x55de0fe9e4f0 msgr2=0x55de0fea89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82e7187740 1 -- 172.21.15.1:0/2295893777 shutdown_connections 2024-04-26T21:42:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.474+0000 7f82e7187740 1 -- 172.21.15.1:0/2295893777 wait complete. 2024-04-26T21:42:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.475+0000 7f82e7187740 1 -- start start 2024-04-26T21:42:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.475+0000 7f82e7187740 1 Processor -- start 2024-04-26T21:42:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.475+0000 7f82e7187740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0feb51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.475+0000 7f82e7187740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 0x55de0feb5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.475+0000 7f82e7187740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 0x55de0fea6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.475+0000 7f82e7187740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1b0620 con 0x55de0fead1c0 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.475+0000 7f82e7187740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1b1a90 con 0x55de0fea1fe0 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e7187740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1b0cf0 con 0x55de0fea2900 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e7187740 10 client.0 _ll_register_callbacks cb 0x7f82d0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e2400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 0x55de0fea6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e2400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 0x55de0fea6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50820/0 (socket says 172.21.15.1:50820) 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e2400640 1 -- 172.21.15.1:0/3306273438 learned_addr learned my addr 172.21.15.1:0/3306273438 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e1000640 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 0x55de0feb5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e1a00640 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0feb51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e2400640 1 -- 172.21.15.1:0/3306273438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 msgr2=0x55de0feb5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e2400640 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 0x55de0feb5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e2400640 1 -- 172.21.15.1:0/3306273438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 msgr2=0x55de0feb51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e2400640 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0feb51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e2400640 1 -- 172.21.15.1:0/3306273438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de0fbe5620 con 0x55de0fead1c0 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.476+0000 7f82e1000640 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 0x55de0feb5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.477+0000 7f82e2400640 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 0x55de0fea6510 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f82d402ae10 tx=0x7f82d402a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd016476f8d3522c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.477+0000 7f82c3400640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.477+0000 7f82c3400640 1 -- 172.21.15.1:0/3306273438 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4004130 con 0x55de0fead1c0 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.477+0000 7f82c3400640 1 -- 172.21.15.1:0/3306273438 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7f82d40042d0 con 0x55de0fead1c0 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.477+0000 7f82e7187740 1 -- 172.21.15.1:0/3306273438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82d4036040 con 0x55de0fead1c0 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.477+0000 7f82c3400640 1 -- 172.21.15.1:0/3306273438 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f82d40308c0 con 0x55de0fead1c0 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.477+0000 7f82e7187740 10 client.16287 fetch_fsmap learned FSMap version 41 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.477+0000 7f82e7187740 1 -- 172.21.15.1:0/3306273438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55de0fdc1cb0 con 0x55de0fead1c0 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.477+0000 7f82c3400640 1 -- 172.21.15.1:0/3306273438 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82d4036040 con 0x55de0fead1c0 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.478+0000 7f82c3400640 1 -- 172.21.15.1:0/3306273438 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f82d4051040 con 0x55de0fead1c0 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.478+0000 7f82e7187740 10 client.16287 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:42:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.478+0000 7f82e7187740 10 client.16287 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:42:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.478+0000 7f82e7187740 20 client.16287 populate_metadata read hostname 'smithi001' 2024-04-26T21:42:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.478+0000 7f82e7187740 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55de0ff231a0 0x55de0ff255c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.478+0000 7f82e7187740 4 client.16287 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:42:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.478+0000 7f82e7187740 1 -- 172.21.15.1:0/3306273438 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de0fe0ee00 con 0x55de0ff231a0 2024-04-26T21:42:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.478+0000 7f82e1a00640 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55de0ff231a0 0x55de0ff255c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.479+0000 7f82e1a00640 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55de0ff231a0 0x55de0ff255c0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed3857a667308a7 server_cookie=c7a42327b372bd30 in_seq=0 out_seq=0 2024-04-26T21:42:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.479+0000 7f82c3400640 10 client.16287 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:18.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.548+0000 7f82c3400640 1 -- 172.21.15.1:0/3306273438 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55de0fe0ee00 con 0x55de0ff231a0 2024-04-26T21:42:18.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.548+0000 7f82c3400640 10 client.16287 handle_command_reply: tid=0 2024-04-26T21:42:18.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.548+0000 7f82e7187740 1 client.16287 shutdown 2024-04-26T21:42:18.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.548+0000 7f82e7187740 1 -- 172.21.15.1:0/3306273438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 msgr2=0x55de0fea6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.548+0000 7f82e7187740 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 0x55de0fea6510 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f82d402ae10 tx=0x7f82d402a8f0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.548+0000 7f82e7187740 1 -- 172.21.15.1:0/3306273438 shutdown_connections 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.549+0000 7f82e7187740 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55de0ff231a0 0x55de0ff255c0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.549+0000 7f82e7187740 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de0fea2900 0x55de0feb5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.549+0000 7f82e7187740 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55de0fead1c0 0x55de0fea6510 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.549+0000 7f82e7187740 1 --2- 172.21.15.1:0/3306273438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55de0fea1fe0 0x55de0feb51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.549+0000 7f82c3400640 0 client.16287 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.549+0000 7f82e7187740 1 -- 172.21.15.1:0/3306273438 >> 172.21.15.1:0/3306273438 conn(0x55de0fe9e4f0 msgr2=0x55de0fea4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.549+0000 7f82e7187740 1 -- 172.21.15.1:0/3306273438 shutdown_connections 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.549+0000 7f82e7187740 1 -- 172.21.15.1:0/3306273438 wait complete. 2024-04-26T21:42:18.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.549+0000 7f82e7187740 20 client.16287 trim_cache size 0 max 16384 2024-04-26T21:42:18.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:42:18.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:42:18.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:42:18.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 -- 172.21.15.1:0/394196745 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82c40a3d00 msgr2=0x7f82c40a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82c40a3d00 0x7f82c40a61c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f82d4004af0 tx=0x7f82d4005c50 comp rx=0 tx=0).stop 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 -- 172.21.15.1:0/394196745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82dc103110 msgr2=0x7f82dc19b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82dc103110 0x7f82dc19b650 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f82c800e9f0 tx=0x7f82c800eeb0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 -- 172.21.15.1:0/394196745 shutdown_connections 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82c40a3d00 0x7f82c40a61c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82dc108760 0x7f82dc19f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82dc103110 0x7f82dc19b650 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 --2- 172.21.15.1:0/394196745 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82dc102750 0x7f82dc19b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 -- 172.21.15.1:0/394196745 >> 172.21.15.1:0/394196745 conn(0x7f82dc0fe240 msgr2=0x7f82dc10d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 -- 172.21.15.1:0/394196745 shutdown_connections 2024-04-26T21:42:18.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:18.550+0000 7f82e2e00640 1 -- 172.21.15.1:0/394196745 wait complete. 2024-04-26T21:42:18.604 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458693 4.29 MB/sec execute 496 sec latency 4445.014 ms 2024-04-26T21:42:19.605 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458857 4.29 MB/sec execute 497 sec latency 678.629 ms 2024-04-26T21:42:20.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:20 smithi001 ceph-mon[28054]: pgmap v375: 129 pgs: 129 active+clean; 322 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-26T21:42:20.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:20 smithi038 ceph-mon[32724]: pgmap v375: 129 pgs: 129 active+clean; 322 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-26T21:42:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:20 smithi186 ceph-mon[32991]: pgmap v375: 129 pgs: 129 active+clean; 322 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 161 op/s 2024-04-26T21:42:20.605 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458857 4.28 MB/sec execute 498 sec latency 1678.792 ms 2024-04-26T21:42:21.605 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458857 4.27 MB/sec execute 499 sec latency 2679.003 ms 2024-04-26T21:42:22.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:22 smithi001 ceph-mon[28054]: pgmap v376: 129 pgs: 129 active+clean; 325 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 169 op/s 2024-04-26T21:42:22.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:22 smithi038 ceph-mon[32724]: pgmap v376: 129 pgs: 129 active+clean; 325 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 169 op/s 2024-04-26T21:42:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:22 smithi186 ceph-mon[32991]: pgmap v376: 129 pgs: 129 active+clean; 325 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 169 op/s 2024-04-26T21:42:22.605 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458857 4.26 MB/sec execute 500 sec latency 3679.170 ms 2024-04-26T21:42:23.605 INFO:tasks.workunit.client.0.smithi001.stdout: 1 458857 4.26 MB/sec execute 501 sec latency 4679.329 ms 2024-04-26T21:42:24.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:24 smithi038 ceph-mon[32724]: pgmap v377: 129 pgs: 129 active+clean; 325 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-04-26T21:42:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:24 smithi186 ceph-mon[32991]: pgmap v377: 129 pgs: 129 active+clean; 325 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-04-26T21:42:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:24 smithi001 ceph-mon[28054]: pgmap v377: 129 pgs: 129 active+clean; 325 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-04-26T21:42:24.605 INFO:tasks.workunit.client.0.smithi001.stdout: 1 459133 4.26 MB/sec execute 502 sec latency 5367.317 ms 2024-04-26T21:42:25.606 INFO:tasks.workunit.client.0.smithi001.stdout: 1 460317 4.26 MB/sec execute 503 sec latency 22.908 ms 2024-04-26T21:42:26.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:26 smithi038 ceph-mon[32724]: pgmap v378: 129 pgs: 129 active+clean; 329 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.6 MiB/s wr, 104 op/s 2024-04-26T21:42:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:26 smithi186 ceph-mon[32991]: pgmap v378: 129 pgs: 129 active+clean; 329 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.6 MiB/s wr, 104 op/s 2024-04-26T21:42:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:26 smithi001 ceph-mon[28054]: pgmap v378: 129 pgs: 129 active+clean; 329 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 1.6 MiB/s wr, 104 op/s 2024-04-26T21:42:26.606 INFO:tasks.workunit.client.0.smithi001.stdout: 1 461328 4.26 MB/sec execute 504 sec latency 32.765 ms 2024-04-26T21:42:27.606 INFO:tasks.workunit.client.0.smithi001.stdout: 1 462141 4.27 MB/sec execute 505 sec latency 347.270 ms 2024-04-26T21:42:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:28 smithi186 ceph-mon[32991]: pgmap v379: 129 pgs: 129 active+clean; 329 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 667 KiB/s wr, 48 op/s 2024-04-26T21:42:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:28 smithi001 ceph-mon[28054]: pgmap v379: 129 pgs: 129 active+clean; 329 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 667 KiB/s wr, 48 op/s 2024-04-26T21:42:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:28 smithi038 ceph-mon[32724]: pgmap v379: 129 pgs: 129 active+clean; 329 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 667 KiB/s wr, 48 op/s 2024-04-26T21:42:28.606 INFO:tasks.workunit.client.0.smithi001.stdout: 1 462141 4.26 MB/sec execute 506 sec latency 1347.404 ms 2024-04-26T21:42:29.606 INFO:tasks.workunit.client.0.smithi001.stdout: 1 462283 4.25 MB/sec execute 507 sec latency 2030.868 ms 2024-04-26T21:42:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:30 smithi186 ceph-mon[32991]: pgmap v380: 129 pgs: 129 active+clean; 329 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 667 KiB/s wr, 48 op/s 2024-04-26T21:42:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:30 smithi001 ceph-mon[28054]: pgmap v380: 129 pgs: 129 active+clean; 329 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 667 KiB/s wr, 48 op/s 2024-04-26T21:42:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:30 smithi038 ceph-mon[32724]: pgmap v380: 129 pgs: 129 active+clean; 329 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 667 KiB/s wr, 48 op/s 2024-04-26T21:42:30.606 INFO:tasks.workunit.client.0.smithi001.stdout: 1 463011 4.25 MB/sec execute 508 sec latency 17.443 ms 2024-04-26T21:42:31.606 INFO:tasks.workunit.client.0.smithi001.stdout: 1 464272 4.25 MB/sec execute 509 sec latency 27.795 ms 2024-04-26T21:42:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.720+0000 7ff62d400640 1 -- 172.21.15.1:0/1273945233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff628103910 msgr2=0x7ff628103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.720+0000 7ff62d400640 1 --2- 172.21.15.1:0/1273945233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff628103910 0x7ff628103cf0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ff618009900 tx=0x7ff61802a480 comp rx=0 tx=0).stop 2024-04-26T21:42:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.721+0000 7ff62d400640 1 -- 172.21.15.1:0/1273945233 shutdown_connections 2024-04-26T21:42:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.721+0000 7ff62d400640 1 --2- 172.21.15.1:0/1273945233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff628104230 0x7ff62810c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.721 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.721+0000 7ff62d400640 1 --2- 172.21.15.1:0/1273945233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62810cd30 0x7ff62810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.721+0000 7ff62d400640 1 --2- 172.21.15.1:0/1273945233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff628103910 0x7ff628103cf0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.721+0000 7ff62d400640 1 -- 172.21.15.1:0/1273945233 >> 172.21.15.1:0/1273945233 conn(0x7ff6280fd110 msgr2=0x7ff6280ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:31.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.721+0000 7ff62d400640 1 -- 172.21.15.1:0/1273945233 shutdown_connections 2024-04-26T21:42:31.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.721+0000 7ff62d400640 1 -- 172.21.15.1:0/1273945233 wait complete. 2024-04-26T21:42:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.722+0000 7ff62d400640 1 Processor -- start 2024-04-26T21:42:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.722+0000 7ff62d400640 1 -- start start 2024-04-26T21:42:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.722+0000 7ff62d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff628103910 0x7ff62819b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.722+0000 7ff62d400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff628104230 0x7ff62819b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.722+0000 7ff62d400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62810cd30 0x7ff62819fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff62d400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff628112a30 con 0x7ff62810cd30 2024-04-26T21:42:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff62d400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff6281128b0 con 0x7ff628104230 2024-04-26T21:42:31.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff62d400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff628112bb0 con 0x7ff628103910 2024-04-26T21:42:31.724 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff627400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff628104230 0x7ff62819b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff627400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff628104230 0x7ff62819b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34398/0 (socket says 172.21.15.1:34398) 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff627400640 1 -- 172.21.15.1:0/943483576 learned_addr learned my addr 172.21.15.1:0/943483576 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff627e00640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff628103910 0x7ff62819b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff62ca00640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62810cd30 0x7ff62819fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff627400640 1 -- 172.21.15.1:0/943483576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff628103910 msgr2=0x7ff62819b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff627400640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff628103910 0x7ff62819b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff627400640 1 -- 172.21.15.1:0/943483576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62810cd30 msgr2=0x7ff62819fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff627400640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62810cd30 0x7ff62819fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.723+0000 7ff627400640 1 -- 172.21.15.1:0/943483576 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6281a01c0 con 0x7ff628104230 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.724+0000 7ff627400640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff628104230 0x7ff62819b710 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff61400d750 tx=0x7ff61400dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.724+0000 7ff624c00640 1 -- 172.21.15.1:0/943483576 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614017070 con 0x7ff628104230 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.724+0000 7ff624c00640 1 -- 172.21.15.1:0/943483576 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff614005020 con 0x7ff628104230 2024-04-26T21:42:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.724+0000 7ff62d400640 1 -- 172.21.15.1:0/943483576 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6281a04b0 con 0x7ff628104230 2024-04-26T21:42:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.724+0000 7ff624c00640 1 -- 172.21.15.1:0/943483576 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614004c10 con 0x7ff628104230 2024-04-26T21:42:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.724+0000 7ff62d400640 1 -- 172.21.15.1:0/943483576 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6281a7d40 con 0x7ff628104230 2024-04-26T21:42:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.728+0000 7ff6319a3740 1 Processor -- start 2024-04-26T21:42:31.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.728+0000 7ff6319a3740 1 -- start start 2024-04-26T21:42:31.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.728+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 0x55c7a38b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.728+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.728+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.728+0000 7ff6319a3740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c7a3819380 con 0x55c7a38bf1c0 2024-04-26T21:42:31.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.728+0000 7ff6319a3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c7a37b5c30 con 0x55c7a38b3fe0 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.728+0000 7ff6319a3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c7a3815340 con 0x55c7a38b4900 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.728+0000 7ff624c00640 1 -- 172.21.15.1:0/943483576 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7ff6140051d0 con 0x7ff628104230 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff627e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff62ca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 0x55c7a38b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff627e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:36306/0 (socket says 172.21.15.1:36306) 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff627e00640 1 -- 172.21.15.1:0/3432248831 learned_addr learned my addr 172.21.15.1:0/3432248831 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff627400640 1 --2- 172.21.15.1:0/3432248831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff62ca00640 1 -- 172.21.15.1:0/3432248831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 msgr2=0x55c7a38bec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff62ca00640 1 --2- 172.21.15.1:0/3432248831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff62ca00640 1 -- 172.21.15.1:0/3432248831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38bf1c0 msgr2=0x55c7a38c15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff62ca00640 1 --2- 172.21.15.1:0/3432248831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff62ca00640 1 -- 172.21.15.1:0/3432248831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7a37d7940 con 0x55c7a38b3fe0 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.729+0000 7ff627400640 1 --2- 172.21.15.1:0/3432248831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.730+0000 7ff62ca00640 1 --2- 172.21.15.1:0/3432248831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 0x55c7a38b43c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff620009eb0 tx=0x7ff6200107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c58047813baf095d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:31.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.730+0000 7ff61d200640 1 -- 172.21.15.1:0/3432248831 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff620003b70 con 0x55c7a38b3fe0 2024-04-26T21:42:31.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.730+0000 7ff61d200640 1 -- 172.21.15.1:0/3432248831 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff620015070 con 0x55c7a38b3fe0 2024-04-26T21:42:31.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.730+0000 7ff61d200640 1 -- 172.21.15.1:0/3432248831 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62001a8d0 con 0x55c7a38b3fe0 2024-04-26T21:42:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.730+0000 7ff624c00640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff60c0a3d80 0x7ff60c0a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff624c00640 1 -- 172.21.15.1:0/943483576 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7ff6140d04e0 con 0x7ff628104230 2024-04-26T21:42:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff627e00640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff60c0a3d80 0x7ff60c0a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff6319a3740 1 -- 172.21.15.1:0/3432248831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 msgr2=0x55c7a38b43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff6319a3740 1 --2- 172.21.15.1:0/3432248831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 0x55c7a38b43c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff620009eb0 tx=0x7ff6200107a0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff6319a3740 1 -- 172.21.15.1:0/3432248831 shutdown_connections 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff6319a3740 1 --2- 172.21.15.1:0/3432248831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff6319a3740 1 --2- 172.21.15.1:0/3432248831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff6319a3740 1 --2- 172.21.15.1:0/3432248831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 0x55c7a38b43c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff6319a3740 1 -- 172.21.15.1:0/3432248831 >> 172.21.15.1:0/3432248831 conn(0x55c7a38b04f0 msgr2=0x55c7a38b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff6319a3740 1 -- 172.21.15.1:0/3432248831 shutdown_connections 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.731+0000 7ff6319a3740 1 -- 172.21.15.1:0/3432248831 wait complete. 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.732+0000 7ff627e00640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff60c0a3d80 0x7ff60c0a6240 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff618005c40 tx=0x7ff618005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.732+0000 7ff6319a3740 1 -- start start 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.732+0000 7ff6319a3740 1 Processor -- start 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b3fe0 0x55c7a38bc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38bf1c0 0x55c7a38bcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff6319a3740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c7a38c1df0 con 0x55c7a38b3fe0 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff6319a3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c7a37a0c60 con 0x55c7a38bf1c0 2024-04-26T21:42:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff6319a3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c7a38c1f70 con 0x55c7a38b4900 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff6319a3740 10 client.0 _ll_register_callbacks cb 0x55c7a37cce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff627400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff62ca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38bf1c0 0x55c7a38bcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff627400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:36314/0 (socket says 172.21.15.1:36314) 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff627400640 1 -- 172.21.15.1:0/3295238887 learned_addr learned my addr 172.21.15.1:0/3295238887 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff62ca00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38bf1c0 0x55c7a38bcb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34408/0 (socket says 172.21.15.1:34408) 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.733+0000 7ff627e00640 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b3fe0 0x55c7a38bc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff62ca00640 1 -- 172.21.15.1:0/3295238887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 msgr2=0x55c7a38bc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff62ca00640 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff62ca00640 1 -- 172.21.15.1:0/3295238887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b3fe0 msgr2=0x55c7a38bc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff62ca00640 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b3fe0 0x55c7a38bc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff62ca00640 1 -- 172.21.15.1:0/3295238887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7a37af510 con 0x55c7a38bf1c0 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff627400640 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff62ca00640 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38bf1c0 0x55c7a38bcb00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff620000f80 tx=0x7ff620003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9829fe597980807 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:31.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff611600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:42:31.738 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff611600640 1 -- 172.21.15.1:0/3295238887 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff620027020 con 0x55c7a38bf1c0 2024-04-26T21:42:31.738 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff6319a3740 1 -- 172.21.15.1:0/3295238887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff620015070 con 0x55c7a38bf1c0 2024-04-26T21:42:31.738 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.734+0000 7ff611600640 1 -- 172.21.15.1:0/3295238887 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7ff620004250 con 0x55c7a38bf1c0 2024-04-26T21:42:31.738 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff611600640 1 -- 172.21.15.1:0/3295238887 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7ff620032050 con 0x55c7a38bf1c0 2024-04-26T21:42:31.738 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff6319a3740 10 client.26495 fetch_fsmap learned FSMap version 41 2024-04-26T21:42:31.738 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff6319a3740 1 -- 172.21.15.1:0/3295238887 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c7a352b210 con 0x55c7a38bf1c0 2024-04-26T21:42:31.738 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff611600640 1 -- 172.21.15.1:0/3295238887 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff620015070 con 0x55c7a38bf1c0 2024-04-26T21:42:31.738 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff611600640 1 -- 172.21.15.1:0/3295238887 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7ff620023470 con 0x55c7a38bf1c0 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff6319a3740 10 client.26495 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff6319a3740 10 client.26495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff6319a3740 20 client.26495 populate_metadata read hostname 'smithi001' 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff6319a3740 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c7a38ced40 0x55c7a38d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff6319a3740 4 client.26495 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.735+0000 7ff6319a3740 1 -- 172.21.15.1:0/3295238887 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7a3826f90 con 0x55c7a38ced40 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.736+0000 7ff627e00640 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c7a38ced40 0x55c7a38d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.736+0000 7ff627e00640 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c7a38ced40 0x55c7a38d1130 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e22543e970899674 server_cookie=ed6ccb54cd830b3f in_seq=0 out_seq=0 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.736+0000 7ff611600640 10 client.26495 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.737+0000 7ff611600640 1 -- 172.21.15.1:0/3295238887 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7a3826f90 con 0x55c7a38ced40 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.737+0000 7ff611600640 10 client.26495 handle_command_reply: tid=0 2024-04-26T21:42:31.740 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.737+0000 7ff6319a3740 1 client.26495 shutdown 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.737+0000 7ff6319a3740 1 -- 172.21.15.1:0/3295238887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38bf1c0 msgr2=0x55c7a38bcb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.737+0000 7ff6319a3740 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38bf1c0 0x55c7a38bcb00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff620000f80 tx=0x7ff620003c00 comp rx=0 tx=0).stop 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.737+0000 7ff6319a3740 1 -- 172.21.15.1:0/3295238887 shutdown_connections 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.737+0000 7ff6319a3740 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c7a38ced40 0x55c7a38d1130 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.737+0000 7ff6319a3740 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38b4900 0x55c7a38bc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.737+0000 7ff6319a3740 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38bf1c0 0x55c7a38bcb00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.737+0000 7ff6319a3740 1 --2- 172.21.15.1:0/3295238887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b3fe0 0x55c7a38bc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.738+0000 7ff611600640 0 client.26495 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.738+0000 7ff6319a3740 1 -- 172.21.15.1:0/3295238887 >> 172.21.15.1:0/3295238887 conn(0x55c7a38b04f0 msgr2=0x55c7a38b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.738+0000 7ff6319a3740 1 -- 172.21.15.1:0/3295238887 shutdown_connections 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.738+0000 7ff6319a3740 1 -- 172.21.15.1:0/3295238887 wait complete. 2024-04-26T21:42:31.741 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.738+0000 7ff6319a3740 20 client.26495 trim_cache size 0 max 16384 2024-04-26T21:42:31.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.757+0000 7ff6319a3740 1 Processor -- start 2024-04-26T21:42:31.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.757+0000 7ff6319a3740 1 -- start start 2024-04-26T21:42:31.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.757+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 0x55c7a38cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.757+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b4900 0x55c7a38b69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.757+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 0x55c7a38af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.757+0000 7ff6319a3740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff6281b4a30 con 0x55c7a38b4900 2024-04-26T21:42:31.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.757+0000 7ff6319a3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff6281b2270 con 0x55c7a38b3fe0 2024-04-26T21:42:31.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.757+0000 7ff6319a3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff6281b00e0 con 0x55c7a38bf1c0 2024-04-26T21:42:31.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.758+0000 7ff627400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b4900 0x55c7a38b69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.758+0000 7ff627400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b4900 0x55c7a38b69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52704/0 (socket says 172.21.15.1:52704) 2024-04-26T21:42:31.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.758+0000 7ff627400640 1 -- 172.21.15.1:0/1405359955 learned_addr learned my addr 172.21.15.1:0/1405359955 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:31.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.758+0000 7ff627e00640 1 --2- 172.21.15.1:0/1405359955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 0x55c7a38cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.760 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.758+0000 7ff62ca00640 1 --2- 172.21.15.1:0/1405359955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 0x55c7a38af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.760 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.758+0000 7ff627400640 1 -- 172.21.15.1:0/1405359955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 msgr2=0x55c7a38af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:31.760 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.758+0000 7ff627400640 1 --2- 172.21.15.1:0/1405359955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 0x55c7a38af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.760 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.758+0000 7ff627400640 1 -- 172.21.15.1:0/1405359955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 msgr2=0x55c7a38cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:31.760 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.758+0000 7ff627400640 1 --2- 172.21.15.1:0/1405359955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 0x55c7a38cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.760 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.758+0000 7ff627400640 1 -- 172.21.15.1:0/1405359955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7a35f7620 con 0x55c7a38b4900 2024-04-26T21:42:31.760 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.759+0000 7ff627400640 1 --2- 172.21.15.1:0/1405359955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b4900 0x55c7a38b69a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff6140a4760 tx=0x7ff61409c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a83c3a23747640 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:31.761 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.760+0000 7ff613e00640 1 -- 172.21.15.1:0/1405359955 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614020040 con 0x55c7a38b4900 2024-04-26T21:42:31.761 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.760+0000 7ff613e00640 1 -- 172.21.15.1:0/1405359955 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff614004db0 con 0x55c7a38b4900 2024-04-26T21:42:31.761 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.760+0000 7ff613e00640 1 -- 172.21.15.1:0/1405359955 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff614009a30 con 0x55c7a38b4900 2024-04-26T21:42:31.761 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.761+0000 7ff6319a3740 1 -- 172.21.15.1:0/1405359955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b4900 msgr2=0x55c7a38b69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:31.761 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.761+0000 7ff6319a3740 1 --2- 172.21.15.1:0/1405359955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b4900 0x55c7a38b69a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff6140a4760 tx=0x7ff61409c190 comp rx=0 tx=0).stop 2024-04-26T21:42:31.761 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.761+0000 7ff6319a3740 1 -- 172.21.15.1:0/1405359955 shutdown_connections 2024-04-26T21:42:31.761 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.761+0000 7ff6319a3740 1 --2- 172.21.15.1:0/1405359955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 0x55c7a38af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.761+0000 7ff6319a3740 1 --2- 172.21.15.1:0/1405359955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b4900 0x55c7a38b69a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.761+0000 7ff6319a3740 1 --2- 172.21.15.1:0/1405359955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b3fe0 0x55c7a38cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.761+0000 7ff6319a3740 1 -- 172.21.15.1:0/1405359955 >> 172.21.15.1:0/1405359955 conn(0x55c7a38b04f0 msgr2=0x55c7a38aedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:31.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.761+0000 7ff6319a3740 1 -- 172.21.15.1:0/1405359955 shutdown_connections 2024-04-26T21:42:31.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.761+0000 7ff6319a3740 1 -- 172.21.15.1:0/1405359955 wait complete. 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.761+0000 7ff6319a3740 1 -- start start 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff6319a3740 1 Processor -- start 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b3fe0 0x55c7a38c7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b4900 0x55c7a38c7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff6319a3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff6319a3740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff6281b0730 con 0x55c7a38b3fe0 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff6319a3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff6281b1ba0 con 0x55c7a38b4900 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff6319a3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff6281b0e00 con 0x55c7a38bf1c0 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff6319a3740 10 client.0 _ll_register_callbacks cb 0x7ff618005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff627e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b3fe0 0x55c7a38c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.764 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff627400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b4900 0x55c7a38c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff62ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff62ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:36338/0 (socket says 172.21.15.1:36338) 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff62ca00640 1 -- 172.21.15.1:0/1712241331 learned_addr learned my addr 172.21.15.1:0/1712241331 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.762+0000 7ff627400640 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b4900 0x55c7a38c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34422/0 (socket says 172.21.15.1:34422) 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff62ca00640 1 -- 172.21.15.1:0/1712241331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b4900 msgr2=0x55c7a38c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff62ca00640 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b4900 0x55c7a38c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff62ca00640 1 -- 172.21.15.1:0/1712241331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b3fe0 msgr2=0x55c7a38c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff62ca00640 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b3fe0 0x55c7a38c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff62ca00640 1 -- 172.21.15.1:0/1712241331 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7a37d3cb0 con 0x55c7a38bf1c0 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff627400640 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b4900 0x55c7a38c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff62ca00640 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c2920 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff620002230 tx=0x7ff620009c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=991afed392364c3a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff611600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff611600640 1 -- 172.21.15.1:0/1712241331 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62003b060 con 0x55c7a38bf1c0 2024-04-26T21:42:31.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff611600640 1 -- 172.21.15.1:0/1712241331 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7ff620023c60 con 0x55c7a38bf1c0 2024-04-26T21:42:31.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.763+0000 7ff6319a3740 1 -- 172.21.15.1:0/1712241331 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff614004db0 con 0x55c7a38bf1c0 2024-04-26T21:42:31.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff611600640 1 -- 172.21.15.1:0/1712241331 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7ff620042050 con 0x55c7a38bf1c0 2024-04-26T21:42:31.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff6319a3740 10 client.26473 fetch_fsmap learned FSMap version 41 2024-04-26T21:42:31.766 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff6319a3740 1 -- 172.21.15.1:0/1712241331 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c7a393d2c0 con 0x55c7a38bf1c0 2024-04-26T21:42:31.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff611600640 1 -- 172.21.15.1:0/1712241331 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff614004db0 con 0x55c7a38bf1c0 2024-04-26T21:42:31.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff611600640 1 -- 172.21.15.1:0/1712241331 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7ff6200049e0 con 0x55c7a38bf1c0 2024-04-26T21:42:31.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff6319a3740 10 client.26473 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:42:31.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff6319a3740 10 client.26473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:42:31.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff6319a3740 20 client.26473 populate_metadata read hostname 'smithi001' 2024-04-26T21:42:31.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff6319a3740 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c7a393da70 0x55c7a393fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:31.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff6319a3740 4 client.26473 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:42:31.767 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.764+0000 7ff6319a3740 1 -- 172.21.15.1:0/1712241331 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c7a3820e00 con 0x55c7a393da70 2024-04-26T21:42:31.768 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.765+0000 7ff627e00640 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c7a393da70 0x55c7a393fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:31.768 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.765+0000 7ff627e00640 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c7a393da70 0x55c7a393fe90 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb3bc0c832f0274c server_cookie=daf0f85c18c2914f in_seq=0 out_seq=0 2024-04-26T21:42:31.768 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.765+0000 7ff611600640 10 client.26473 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:31.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.802+0000 7ff611600640 1 -- 172.21.15.1:0/1712241331 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c7a3820e00 con 0x55c7a393da70 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.802+0000 7ff611600640 10 client.26473 handle_command_reply: tid=0 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.802+0000 7ff6319a3740 1 client.26473 shutdown 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 1 -- 172.21.15.1:0/1712241331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 msgr2=0x55c7a38c2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c2920 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff620002230 tx=0x7ff620009c30 comp rx=0 tx=0).stop 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 1 -- 172.21.15.1:0/1712241331 shutdown_connections 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c7a393da70 0x55c7a393fe90 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7a38bf1c0 0x55c7a38c2920 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c7a38b4900 0x55c7a38c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 1 --2- 172.21.15.1:0/1712241331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c7a38b3fe0 0x55c7a38c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff611600640 0 client.26473 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 1 -- 172.21.15.1:0/1712241331 >> 172.21.15.1:0/1712241331 conn(0x55c7a38b04f0 msgr2=0x55c7a38af650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 1 -- 172.21.15.1:0/1712241331 shutdown_connections 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 1 -- 172.21.15.1:0/1712241331 wait complete. 2024-04-26T21:42:31.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.803+0000 7ff6319a3740 20 client.26473 trim_cache size 0 max 16384 2024-04-26T21:42:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:42:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:42:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:42:31.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 -- 172.21.15.1:0/943483576 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff60c0a3d80 msgr2=0x7ff60c0a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff60c0a3d80 0x7ff60c0a6240 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff618005c40 tx=0x7ff618005a20 comp rx=0 tx=0).stop 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 -- 172.21.15.1:0/943483576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff628104230 msgr2=0x7ff62819b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff628104230 0x7ff62819b710 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff61400d750 tx=0x7ff61400dc10 comp rx=0 tx=0).stop 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 -- 172.21.15.1:0/943483576 shutdown_connections 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff60c0a3d80 0x7ff60c0a6240 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff62810cd30 0x7ff62819fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff628104230 0x7ff62819b710 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 --2- 172.21.15.1:0/943483576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff628103910 0x7ff62819b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 -- 172.21.15.1:0/943483576 >> 172.21.15.1:0/943483576 conn(0x7ff6280fd110 msgr2=0x7ff6281026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:31.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 -- 172.21.15.1:0/943483576 shutdown_connections 2024-04-26T21:42:31.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:31.804+0000 7ff62d400640 1 -- 172.21.15.1:0/943483576 wait complete. 2024-04-26T21:42:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:32 smithi186 ceph-mon[32991]: pgmap v381: 129 pgs: 129 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-04-26T21:42:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:32 smithi001 ceph-mon[28054]: pgmap v381: 129 pgs: 129 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-04-26T21:42:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:32 smithi038 ceph-mon[32724]: pgmap v381: 129 pgs: 129 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 1.2 MiB/s wr, 89 op/s 2024-04-26T21:42:32.606 INFO:tasks.workunit.client.0.smithi001.stdout: 1 465529 4.26 MB/sec execute 510 sec latency 21.712 ms 2024-04-26T21:42:33.606 INFO:tasks.workunit.client.0.smithi001.stdout: 1 465820 4.25 MB/sec execute 511 sec latency 267.953 ms 2024-04-26T21:42:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:34 smithi186 ceph-mon[32991]: pgmap v382: 129 pgs: 129 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 937 KiB/s wr, 56 op/s 2024-04-26T21:42:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:34 smithi001 ceph-mon[28054]: pgmap v382: 129 pgs: 129 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 937 KiB/s wr, 56 op/s 2024-04-26T21:42:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:34 smithi038 ceph-mon[32724]: pgmap v382: 129 pgs: 129 active+clean; 331 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 937 KiB/s wr, 56 op/s 2024-04-26T21:42:34.607 INFO:tasks.workunit.client.0.smithi001.stdout: 1 465894 4.25 MB/sec execute 512 sec latency 954.455 ms 2024-04-26T21:42:35.607 INFO:tasks.workunit.client.0.smithi001.stdout: 1 466601 4.24 MB/sec execute 513 sec latency 27.796 ms 2024-04-26T21:42:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:36 smithi186 ceph-mon[32991]: pgmap v383: 129 pgs: 129 active+clean; 332 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-26T21:42:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:36 smithi001 ceph-mon[28054]: pgmap v383: 129 pgs: 129 active+clean; 332 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-26T21:42:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:36 smithi038 ceph-mon[32724]: pgmap v383: 129 pgs: 129 active+clean; 332 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.4 MiB/s wr, 119 op/s 2024-04-26T21:42:36.607 INFO:tasks.workunit.client.0.smithi001.stdout: 1 467836 4.24 MB/sec execute 514 sec latency 30.067 ms 2024-04-26T21:42:37.607 INFO:tasks.workunit.client.0.smithi001.stdout: 1 468714 4.25 MB/sec execute 515 sec latency 28.515 ms 2024-04-26T21:42:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:38 smithi186 ceph-mon[32991]: pgmap v384: 129 pgs: 129 active+clean; 332 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-26T21:42:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:38 smithi001 ceph-mon[28054]: pgmap v384: 129 pgs: 129 active+clean; 332 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-26T21:42:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:38 smithi038 ceph-mon[32724]: pgmap v384: 129 pgs: 129 active+clean; 332 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-26T21:42:38.607 INFO:tasks.workunit.client.0.smithi001.stdout: 1 469302 4.25 MB/sec execute 516 sec latency 53.361 ms 2024-04-26T21:42:39.607 INFO:tasks.workunit.client.0.smithi001.stdout: 1 469734 4.24 MB/sec execute 517 sec latency 44.999 ms 2024-04-26T21:42:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:40 smithi186 ceph-mon[32991]: pgmap v385: 129 pgs: 129 active+clean; 332 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-26T21:42:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:40 smithi001 ceph-mon[28054]: pgmap v385: 129 pgs: 129 active+clean; 332 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-26T21:42:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:40 smithi038 ceph-mon[32724]: pgmap v385: 129 pgs: 129 active+clean; 332 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-04-26T21:42:40.607 INFO:tasks.workunit.client.0.smithi001.stdout: 1 470497 4.24 MB/sec execute 518 sec latency 67.180 ms 2024-04-26T21:42:41.607 INFO:tasks.workunit.client.0.smithi001.stdout: 1 471475 4.24 MB/sec execute 519 sec latency 37.461 ms 2024-04-26T21:42:42.370 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:42 smithi001 ceph-mon[28054]: pgmap v386: 129 pgs: 129 active+clean; 338 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-04-26T21:42:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:42 smithi186 ceph-mon[32991]: pgmap v386: 129 pgs: 129 active+clean; 338 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-04-26T21:42:42.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:42 smithi038 ceph-mon[32724]: pgmap v386: 129 pgs: 129 active+clean; 338 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-04-26T21:42:42.608 INFO:tasks.workunit.client.0.smithi001.stdout: 1 472630 4.25 MB/sec execute 520 sec latency 28.438 ms 2024-04-26T21:42:43.611 INFO:tasks.workunit.client.0.smithi001.stdout: 1 472958 4.25 MB/sec execute 521 sec latency 31.061 ms 2024-04-26T21:42:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:44 smithi186 ceph-mon[32991]: pgmap v387: 129 pgs: 129 active+clean; 338 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 1.2 MiB/s wr, 119 op/s 2024-04-26T21:42:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:44 smithi001 ceph-mon[28054]: pgmap v387: 129 pgs: 129 active+clean; 338 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 1.2 MiB/s wr, 119 op/s 2024-04-26T21:42:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:44 smithi038 ceph-mon[32724]: pgmap v387: 129 pgs: 129 active+clean; 338 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 1.2 MiB/s wr, 119 op/s 2024-04-26T21:42:44.611 INFO:tasks.workunit.client.0.smithi001.stdout: 1 473703 4.24 MB/sec execute 522 sec latency 62.520 ms 2024-04-26T21:42:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.962+0000 7f1f21a00640 1 -- 172.21.15.1:0/2358577361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f1c108760 msgr2=0x7f1f1c108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.962+0000 7f1f21a00640 1 --2- 172.21.15.1:0/2358577361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f1c108760 0x7f1f1c108b40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c009900 tx=0x7f1f0c02a3f0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.962+0000 7f1f21a00640 1 -- 172.21.15.1:0/2358577361 shutdown_connections 2024-04-26T21:42:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.962+0000 7f1f21a00640 1 --2- 172.21.15.1:0/2358577361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f1c102750 0x7f1f1c102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.962+0000 7f1f21a00640 1 --2- 172.21.15.1:0/2358577361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f1c103110 0x7f1f1c10f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.962+0000 7f1f21a00640 1 --2- 172.21.15.1:0/2358577361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f1c108760 0x7f1f1c108b40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.962+0000 7f1f21a00640 1 -- 172.21.15.1:0/2358577361 >> 172.21.15.1:0/2358577361 conn(0x7f1f1c0fe240 msgr2=0x7f1f1c100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.962+0000 7f1f21a00640 1 -- 172.21.15.1:0/2358577361 shutdown_connections 2024-04-26T21:42:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.962+0000 7f1f21a00640 1 -- 172.21.15.1:0/2358577361 wait complete. 2024-04-26T21:42:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.964+0000 7f1f21a00640 1 Processor -- start 2024-04-26T21:42:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.964+0000 7f1f21a00640 1 -- start start 2024-04-26T21:42:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.964+0000 7f1f21a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f1c102750 0x7f1f1c074c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.964+0000 7f1f21a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f1c103110 0x7f1f1c075170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.964+0000 7f1f21a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f1c108760 0x7f1f1c0756b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.964+0000 7f1f21a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c112a70 con 0x7f1f1c108760 2024-04-26T21:42:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.964+0000 7f1f21a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1128f0 con 0x7f1f1c103110 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.964+0000 7f1f21a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c112bf0 con 0x7f1f1c102750 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f1c102750 0x7f1f1c074c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f1c103110 0x7f1f1c075170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f1c102750 0x7f1f1c074c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50794/0 (socket says 172.21.15.1:50794) 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1be00640 1 -- 172.21.15.1:0/2629084727 learned_addr learned my addr 172.21.15.1:0/2629084727 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f1c103110 0x7f1f1c075170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51126/0 (socket says 172.21.15.1:51126) 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f21000640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f1c108760 0x7f1f1c0756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1be00640 1 -- 172.21.15.1:0/2629084727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f1c103110 msgr2=0x7f1f1c075170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1be00640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f1c103110 0x7f1f1c075170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1be00640 1 -- 172.21.15.1:0/2629084727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f1c108760 msgr2=0x7f1f1c0756b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1be00640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f1c108760 0x7f1f1c0756b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1be00640 1 -- 172.21.15.1:0/2629084727 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f1c1a7450 con 0x7f1f1c102750 2024-04-26T21:42:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.965+0000 7f1f1be00640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f1c102750 0x7f1f1c074c30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c02a9e0 tx=0x7f1f0c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.966+0000 7f1f18c00640 1 -- 172.21.15.1:0/2629084727 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c03f070 con 0x7f1f1c102750 2024-04-26T21:42:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.966+0000 7f1f21a00640 1 -- 172.21.15.1:0/2629084727 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f1c1a76e0 con 0x7f1f1c102750 2024-04-26T21:42:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.966+0000 7f1f18c00640 1 -- 172.21.15.1:0/2629084727 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f0c03bb20 con 0x7f1f1c102750 2024-04-26T21:42:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.966+0000 7f1f18c00640 1 -- 172.21.15.1:0/2629084727 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c035030 con 0x7f1f1c102750 2024-04-26T21:42:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.966+0000 7f1f21a00640 1 -- 172.21.15.1:0/2629084727 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f1c1a7c20 con 0x7f1f1c102750 2024-04-26T21:42:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.969+0000 7f1f25e3a740 1 Processor -- start 2024-04-26T21:42:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f25e3a740 1 -- start start 2024-04-26T21:42:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f18c00640 1 -- 172.21.15.1:0/2629084727 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f1f0c042070 con 0x7f1f1c102750 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e0fe0 0x562fc22e13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e1900 0x562fc22ebc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562fc2246380 con 0x562fc22e0fe0 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562fc21e2c30 con 0x562fc22ec1c0 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562fc2242340 con 0x562fc22e1900 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e1900 0x562fc22ebc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f21000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e0fe0 0x562fc22e13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e1900 0x562fc22ebc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50806/0 (socket says 172.21.15.1:50806) 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f1b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.970+0000 7f1f1be00640 1 -- 172.21.15.1:0/3349367927 learned_addr learned my addr 172.21.15.1:0/3349367927 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.971+0000 7f1f1be00640 1 -- 172.21.15.1:0/3349367927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 msgr2=0x562fc22ee5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.971+0000 7f1f1be00640 1 --2- 172.21.15.1:0/3349367927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22ee5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.971+0000 7f1f1be00640 1 -- 172.21.15.1:0/3349367927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e0fe0 msgr2=0x562fc22e13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.971+0000 7f1f1be00640 1 --2- 172.21.15.1:0/3349367927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e0fe0 0x562fc22e13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.971+0000 7f1f1be00640 1 -- 172.21.15.1:0/3349367927 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fc2204940 con 0x562fc22e1900 2024-04-26T21:42:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.971+0000 7f1f1be00640 1 --2- 172.21.15.1:0/3349367927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e1900 0x562fc22ebc80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0ee5e0 tx=0x7f1f0c0f9630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f002f9e3e6bfb03c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.971+0000 7f1f18c00640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1f000a3d30 0x7f1f000a61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.972+0000 7f1f18c00640 1 -- 172.21.15.1:0/2629084727 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7f1f0c0f1380 con 0x7f1f1c102750 2024-04-26T21:42:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.972+0000 7f1f11600640 1 -- 172.21.15.1:0/3349367927 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c0b9210 con 0x562fc22e1900 2024-04-26T21:42:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.972+0000 7f1f11600640 1 -- 172.21.15.1:0/3349367927 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f0c0fe030 con 0x562fc22e1900 2024-04-26T21:42:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.972+0000 7f1f21000640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1f000a3d30 0x7f1f000a61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.972+0000 7f1f11600640 1 -- 172.21.15.1:0/3349367927 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c0bc6f0 con 0x562fc22e1900 2024-04-26T21:42:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.973+0000 7f1f21000640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1f000a3d30 0x7f1f000a61f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1f14005fd0 tx=0x7f1f140096a0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.973+0000 7f1f25e3a740 1 -- 172.21.15.1:0/3349367927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e1900 msgr2=0x562fc22ebc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.973+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/3349367927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e1900 0x562fc22ebc80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0ee5e0 tx=0x7f1f0c0f9630 comp rx=0 tx=0).stop 2024-04-26T21:42:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.973+0000 7f1f25e3a740 1 -- 172.21.15.1:0/3349367927 shutdown_connections 2024-04-26T21:42:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.973+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/3349367927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e1900 0x562fc22ebc80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.973+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/3349367927 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.973+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/3349367927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e0fe0 0x562fc22e13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.973+0000 7f1f25e3a740 1 -- 172.21.15.1:0/3349367927 >> 172.21.15.1:0/3349367927 conn(0x562fc22dd4f0 msgr2=0x562fc22dd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.973+0000 7f1f25e3a740 1 -- 172.21.15.1:0/3349367927 shutdown_connections 2024-04-26T21:42:44.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.973+0000 7f1f25e3a740 1 -- 172.21.15.1:0/3349367927 wait complete. 2024-04-26T21:42:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.974+0000 7f1f25e3a740 1 -- start start 2024-04-26T21:42:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.974+0000 7f1f25e3a740 1 Processor -- start 2024-04-26T21:42:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.974+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 0x562fc22e90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.974+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22e9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.974+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.974+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562fc22eebb0 con 0x562fc22e1900 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.974+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562fc21cdc60 con 0x562fc22ec1c0 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.974+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562fc22eed30 con 0x562fc22e0fe0 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f25e3a740 10 client.0 _ll_register_callbacks cb 0x562fc21f9e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f21000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 0x562fc22e90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f21000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51154/0 (socket says 172.21.15.1:51154) 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f21000640 1 -- 172.21.15.1:0/43890856 learned_addr learned my addr 172.21.15.1:0/43890856 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 0x562fc22e90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50818/0 (socket says 172.21.15.1:50818) 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f1b400640 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22e9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f21000640 1 -- 172.21.15.1:0/43890856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 msgr2=0x562fc22e90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f21000640 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 0x562fc22e90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f21000640 1 -- 172.21.15.1:0/43890856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 msgr2=0x562fc22e9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f21000640 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22e9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.975+0000 7f1f21000640 1 -- 172.21.15.1:0/43890856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fc21dc510 con 0x562fc22ec1c0 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.976+0000 7f1f21000640 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e9b70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1f14010710 tx=0x7f1f14004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cb4d98457981a16 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.976+0000 7f1f0a000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.976+0000 7f1f0a000640 1 -- 172.21.15.1:0/43890856 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1402c060 con 0x562fc22ec1c0 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.976+0000 7f1f0a000640 1 -- 172.21.15.1:0/43890856 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7f1f14017030 con 0x562fc22ec1c0 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.976+0000 7f1f25e3a740 1 -- 172.21.15.1:0/43890856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f0c0fe030 con 0x562fc22ec1c0 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.976+0000 7f1f0a000640 1 -- 172.21.15.1:0/43890856 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f1f14033050 con 0x562fc22ec1c0 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.976+0000 7f1f25e3a740 10 client.26513 fetch_fsmap learned FSMap version 41 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.976+0000 7f1f25e3a740 1 -- 172.21.15.1:0/43890856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x562fc1f58210 con 0x562fc22ec1c0 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.977+0000 7f1f0a000640 1 -- 172.21.15.1:0/43890856 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f0c0fe030 con 0x562fc22ec1c0 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.977+0000 7f1f0a000640 1 -- 172.21.15.1:0/43890856 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f1f140279c0 con 0x562fc22ec1c0 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.977+0000 7f1f25e3a740 10 client.26513 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.977+0000 7f1f25e3a740 10 client.26513 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.977+0000 7f1f25e3a740 20 client.26513 populate_metadata read hostname 'smithi001' 2024-04-26T21:42:44.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.977+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562fc22fbd00 0x562fc22fe1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.977+0000 7f1f25e3a740 4 client.26513 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:42:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.977+0000 7f1f25e3a740 1 -- 172.21.15.1:0/43890856 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fc2253f90 con 0x562fc22fbd00 2024-04-26T21:42:44.980 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.977+0000 7f1f1be00640 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562fc22fbd00 0x562fc22fe1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.978+0000 7f1f1be00640 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562fc22fbd00 0x562fc22fe1a0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a52a9672094e8990 server_cookie=4573199df993cd in_seq=0 out_seq=0 2024-04-26T21:42:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.978+0000 7f1f0a000640 10 client.26513 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f0a000640 1 -- 172.21.15.1:0/43890856 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562fc2253f90 con 0x562fc22fbd00 2024-04-26T21:42:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f0a000640 10 client.26513 handle_command_reply: tid=0 2024-04-26T21:42:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 client.26513 shutdown 2024-04-26T21:42:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 -- 172.21.15.1:0/43890856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 msgr2=0x562fc22e9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e9b70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1f14010710 tx=0x7f1f14004ab0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 -- 172.21.15.1:0/43890856 shutdown_connections 2024-04-26T21:42:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562fc22fbd00 0x562fc22fe1a0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e9b70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22e9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/43890856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 0x562fc22e90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 -- 172.21.15.1:0/43890856 >> 172.21.15.1:0/43890856 conn(0x562fc22dd4f0 msgr2=0x562fc22e3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f0a000640 0 client.26513 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 -- 172.21.15.1:0/43890856 shutdown_connections 2024-04-26T21:42:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 1 -- 172.21.15.1:0/43890856 wait complete. 2024-04-26T21:42:44.982 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.979+0000 7f1f25e3a740 20 client.26513 trim_cache size 0 max 16384 2024-04-26T21:42:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.997+0000 7f1f25e3a740 1 Processor -- start 2024-04-26T21:42:44.998 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f25e3a740 1 -- start start 2024-04-26T21:42:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 0x562fc22f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22eb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:44.999 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1b48c0 con 0x562fc22e1900 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1b2100 con 0x562fc22ec1c0 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1aff70 con 0x562fc22e0fe0 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f21000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f1b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22eb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f21000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51158/0 (socket says 172.21.15.1:51158) 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 0x562fc22f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.998+0000 7f1f21000640 1 -- 172.21.15.1:0/3412024914 learned_addr learned my addr 172.21.15.1:0/3412024914 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.999+0000 7f1f21000640 1 -- 172.21.15.1:0/3412024914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 msgr2=0x562fc22f79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.999+0000 7f1f21000640 1 --2- 172.21.15.1:0/3412024914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 0x562fc22f79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.999+0000 7f1f21000640 1 -- 172.21.15.1:0/3412024914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 msgr2=0x562fc22eb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.999+0000 7f1f21000640 1 --2- 172.21.15.1:0/3412024914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22eb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.999+0000 7f1f21000640 1 -- 172.21.15.1:0/3412024914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fc2024620 con 0x562fc22ec1c0 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.999+0000 7f1f1be00640 1 --2- 172.21.15.1:0/3412024914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 0x562fc22f79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:45.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.999+0000 7f1f21000640 1 --2- 172.21.15.1:0/3412024914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e39f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1f14010fc0 tx=0x7f1f14011eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca07b6225871ad32 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.999+0000 7f1f10c00640 1 -- 172.21.15.1:0/3412024914 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1401f040 con 0x562fc22ec1c0 2024-04-26T21:42:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.999+0000 7f1f10c00640 1 -- 172.21.15.1:0/3412024914 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f14003e20 con 0x562fc22ec1c0 2024-04-26T21:42:45.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:44.999+0000 7f1f10c00640 1 -- 172.21.15.1:0/3412024914 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1401c790 con 0x562fc22ec1c0 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.000+0000 7f1f25e3a740 1 -- 172.21.15.1:0/3412024914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 msgr2=0x562fc22e39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.000+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/3412024914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e39f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1f14010fc0 tx=0x7f1f14011eb0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.000+0000 7f1f25e3a740 1 -- 172.21.15.1:0/3412024914 shutdown_connections 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.000+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/3412024914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22ec1c0 0x562fc22e39f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.000+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/3412024914 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22eb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.000+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/3412024914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22e0fe0 0x562fc22f79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.000+0000 7f1f25e3a740 1 -- 172.21.15.1:0/3412024914 >> 172.21.15.1:0/3412024914 conn(0x562fc22dd4f0 msgr2=0x562fc22e8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.000+0000 7f1f25e3a740 1 -- 172.21.15.1:0/3412024914 shutdown_connections 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.000+0000 7f1f25e3a740 1 -- 172.21.15.1:0/3412024914 wait complete. 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.001+0000 7f1f25e3a740 1 -- start start 2024-04-26T21:42:45.004 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.001+0000 7f1f25e3a740 1 Processor -- start 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.001+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22e0fe0 0x562fc22f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.001+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22ef490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.001+0000 7f1f25e3a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22ec1c0 0x562fc22ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.001+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1b05c0 con 0x562fc22e1900 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.001+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1b1a30 con 0x562fc22e0fe0 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.001+0000 7f1f25e3a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1f1c1b0c90 con 0x562fc22ec1c0 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.001+0000 7f1f25e3a740 10 client.0 _ll_register_callbacks cb 0x7f1f140105a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22e0fe0 0x562fc22f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f21000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22ec1c0 0x562fc22ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f1be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22e0fe0 0x562fc22f40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51170/0 (socket says 172.21.15.1:51170) 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f1b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22ef490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f1be00640 1 -- 172.21.15.1:0/2461812353 learned_addr learned my addr 172.21.15.1:0/2461812353 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f21000640 1 -- 172.21.15.1:0/2461812353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22e0fe0 msgr2=0x562fc22f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f21000640 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22e0fe0 0x562fc22f40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.005 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f21000640 1 -- 172.21.15.1:0/2461812353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 msgr2=0x562fc22ef490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f21000640 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22ef490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f21000640 1 -- 172.21.15.1:0/2461812353 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fc2200cb0 con 0x562fc22ec1c0 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f1be00640 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22e0fe0 0x562fc22f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f21000640 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22ec1c0 0x562fc22ef9d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1f14027e80 tx=0x7f1f14003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa4d735cdc6eabc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.002+0000 7f1f0a000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f0a000640 1 -- 172.21.15.1:0/2461812353 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1403b030 con 0x562fc22ec1c0 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f0a000640 1 -- 172.21.15.1:0/2461812353 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7f1f14004d10 con 0x562fc22ec1c0 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f25e3a740 1 -- 172.21.15.1:0/2461812353 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f14003e20 con 0x562fc22ec1c0 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f0a000640 1 -- 172.21.15.1:0/2461812353 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f1f14045050 con 0x562fc22ec1c0 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f25e3a740 10 client.26491 fetch_fsmap learned FSMap version 41 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f25e3a740 1 -- 172.21.15.1:0/2461812353 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x562fc2361fa0 con 0x562fc22ec1c0 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f0a000640 1 -- 172.21.15.1:0/2461812353 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f14003e20 con 0x562fc22ec1c0 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f0a000640 1 -- 172.21.15.1:0/2461812353 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f1f140273e0 con 0x562fc22ec1c0 2024-04-26T21:42:45.006 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f25e3a740 10 client.26491 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:42:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f25e3a740 10 client.26491 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:42:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f25e3a740 20 client.26491 populate_metadata read hostname 'smithi001' 2024-04-26T21:42:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.003+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562fc2362680 0x562fc2364aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.004+0000 7f1f25e3a740 4 client.26491 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:42:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.004+0000 7f1f25e3a740 1 -- 172.21.15.1:0/2461812353 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562fc224de00 con 0x562fc2362680 2024-04-26T21:42:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.004+0000 7f1f1be00640 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562fc2362680 0x562fc2364aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.004+0000 7f1f1be00640 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562fc2362680 0x562fc2364aa0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31f973ce985edbc0 server_cookie=6bfe8302749ddd3e in_seq=0 out_seq=0 2024-04-26T21:42:45.007 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.004+0000 7f1f0a000640 10 client.26491 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:45.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.057+0000 7f1f0a000640 1 -- 172.21.15.1:0/2461812353 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562fc224de00 con 0x562fc2362680 2024-04-26T21:42:45.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.057+0000 7f1f0a000640 10 client.26491 handle_command_reply: tid=0 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.057+0000 7f1f25e3a740 1 client.26491 shutdown 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 1 -- 172.21.15.1:0/2461812353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22ec1c0 msgr2=0x562fc22ef9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22ec1c0 0x562fc22ef9d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1f14027e80 tx=0x7f1f14003ba0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 1 -- 172.21.15.1:0/2461812353 shutdown_connections 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562fc2362680 0x562fc2364aa0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562fc22ec1c0 0x562fc22ef9d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562fc22e1900 0x562fc22ef490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 1 --2- 172.21.15.1:0/2461812353 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562fc22e0fe0 0x562fc22f40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f0a000640 0 client.26491 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 1 -- 172.21.15.1:0/2461812353 >> 172.21.15.1:0/2461812353 conn(0x562fc22dd4f0 msgr2=0x562fc22e7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 1 -- 172.21.15.1:0/2461812353 shutdown_connections 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 1 -- 172.21.15.1:0/2461812353 wait complete. 2024-04-26T21:42:45.059 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.058+0000 7f1f25e3a740 20 client.26491 trim_cache size 0 max 16384 2024-04-26T21:42:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:42:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:42:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:42:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:42:45.060 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 -- 172.21.15.1:0/2629084727 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1f000a3d30 msgr2=0x7f1f000a61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1f000a3d30 0x7f1f000a61f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1f14005fd0 tx=0x7f1f140096a0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 -- 172.21.15.1:0/2629084727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f1c102750 msgr2=0x7f1f1c074c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f1c102750 0x7f1f1c074c30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c02a9e0 tx=0x7f1f0c003c10 comp rx=0 tx=0).stop 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 -- 172.21.15.1:0/2629084727 shutdown_connections 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1f000a3d30 0x7f1f000a61f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1f1c108760 0x7f1f1c0756b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f1c103110 0x7f1f1c075170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 --2- 172.21.15.1:0/2629084727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f1c102750 0x7f1f1c074c30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 -- 172.21.15.1:0/2629084727 >> 172.21.15.1:0/2629084727 conn(0x7f1f1c0fe240 msgr2=0x7f1f1c0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 -- 172.21.15.1:0/2629084727 shutdown_connections 2024-04-26T21:42:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:45.059+0000 7f1f21a00640 1 -- 172.21.15.1:0/2629084727 wait complete. 2024-04-26T21:42:45.611 INFO:tasks.workunit.client.0.smithi001.stdout: 1 474941 4.24 MB/sec execute 523 sec latency 33.300 ms 2024-04-26T21:42:46.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:46 smithi186 ceph-mon[32991]: pgmap v388: 129 pgs: 129 active+clean; 343 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 178 op/s 2024-04-26T21:42:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:46 smithi001 ceph-mon[28054]: pgmap v388: 129 pgs: 129 active+clean; 343 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 178 op/s 2024-04-26T21:42:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:46 smithi038 ceph-mon[32724]: pgmap v388: 129 pgs: 129 active+clean; 343 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.1 MiB/s wr, 178 op/s 2024-04-26T21:42:46.611 INFO:tasks.workunit.client.0.smithi001.stdout: 1 476066 4.25 MB/sec execute 524 sec latency 22.173 ms 2024-04-26T21:42:47.611 INFO:tasks.workunit.client.0.smithi001.stdout: 1 476506 4.25 MB/sec execute 525 sec latency 34.280 ms 2024-04-26T21:42:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:48 smithi186 ceph-mon[32991]: pgmap v389: 129 pgs: 129 active+clean; 343 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-04-26T21:42:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:48 smithi001 ceph-mon[28054]: pgmap v389: 129 pgs: 129 active+clean; 343 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-04-26T21:42:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:48 smithi038 ceph-mon[32724]: pgmap v389: 129 pgs: 129 active+clean; 343 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-04-26T21:42:48.611 INFO:tasks.workunit.client.0.smithi001.stdout: 1 477133 4.25 MB/sec execute 526 sec latency 21.870 ms 2024-04-26T21:42:49.612 INFO:tasks.workunit.client.0.smithi001.stdout: 1 478429 4.25 MB/sec execute 527 sec latency 28.975 ms 2024-04-26T21:42:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:50 smithi186 ceph-mon[32991]: pgmap v390: 129 pgs: 129 active+clean; 343 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-04-26T21:42:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:50 smithi001 ceph-mon[28054]: pgmap v390: 129 pgs: 129 active+clean; 343 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-04-26T21:42:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:50 smithi038 ceph-mon[32724]: pgmap v390: 129 pgs: 129 active+clean; 343 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 1.6 MiB/s wr, 116 op/s 2024-04-26T21:42:50.612 INFO:tasks.workunit.client.0.smithi001.stdout: 1 479321 4.25 MB/sec execute 528 sec latency 40.740 ms 2024-04-26T21:42:51.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:51 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:42:51.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:51 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:42:51.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:51 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:42:51.612 INFO:tasks.workunit.client.0.smithi001.stdout: 1 480052 4.25 MB/sec execute 529 sec latency 21.873 ms 2024-04-26T21:42:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:52 smithi186 ceph-mon[32991]: pgmap v391: 129 pgs: 129 active+clean; 357 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 181 op/s 2024-04-26T21:42:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:52 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:42:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:52 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:42:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:52 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:42:52.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:52 smithi038 ceph-mon[32724]: pgmap v391: 129 pgs: 129 active+clean; 357 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 181 op/s 2024-04-26T21:42:52.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:52 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:42:52.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:52 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:42:52.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:52 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:42:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:52 smithi001 ceph-mon[28054]: pgmap v391: 129 pgs: 129 active+clean; 357 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 181 op/s 2024-04-26T21:42:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:52 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:42:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:52 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:42:52.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:52 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:42:52.612 INFO:tasks.workunit.client.0.smithi001.stdout: 1 480612 4.25 MB/sec execute 530 sec latency 23.663 ms 2024-04-26T21:42:53.612 INFO:tasks.workunit.client.0.smithi001.stdout: 1 481552 4.25 MB/sec execute 531 sec latency 42.040 ms 2024-04-26T21:42:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:54 smithi186 ceph-mon[32991]: pgmap v392: 129 pgs: 129 active+clean; 357 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.4 MiB/s wr, 125 op/s 2024-04-26T21:42:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:54 smithi001 ceph-mon[28054]: pgmap v392: 129 pgs: 129 active+clean; 357 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.4 MiB/s wr, 125 op/s 2024-04-26T21:42:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:54 smithi038 ceph-mon[32724]: pgmap v392: 129 pgs: 129 active+clean; 357 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.4 MiB/s wr, 125 op/s 2024-04-26T21:42:54.612 INFO:tasks.workunit.client.0.smithi001.stdout: 1 482558 4.25 MB/sec execute 532 sec latency 33.404 ms 2024-04-26T21:42:55.612 INFO:tasks.workunit.client.0.smithi001.stdout: 1 483456 4.26 MB/sec execute 533 sec latency 24.943 ms 2024-04-26T21:42:56.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:56 smithi186 ceph-mon[32991]: pgmap v393: 129 pgs: 129 active+clean; 364 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 173 op/s 2024-04-26T21:42:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:56 smithi001 ceph-mon[28054]: pgmap v393: 129 pgs: 129 active+clean; 364 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 173 op/s 2024-04-26T21:42:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:56 smithi038 ceph-mon[32724]: pgmap v393: 129 pgs: 129 active+clean; 364 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 173 op/s 2024-04-26T21:42:56.612 INFO:tasks.workunit.client.0.smithi001.stdout: 1 483761 4.25 MB/sec execute 534 sec latency 26.224 ms 2024-04-26T21:42:57.613 INFO:tasks.workunit.client.0.smithi001.stdout: 1 484425 4.25 MB/sec execute 535 sec latency 24.140 ms 2024-04-26T21:42:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.206+0000 7efc99600640 1 -- 172.21.15.1:0/2958457119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc94107650 msgr2=0x7efc94107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.206+0000 7efc99600640 1 --2- 172.21.15.1:0/2958457119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc94107650 0x7efc94107a30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7efc84009900 tx=0x7efc8402a3f0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.206+0000 7efc99600640 1 -- 172.21.15.1:0/2958457119 shutdown_connections 2024-04-26T21:42:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.206+0000 7efc99600640 1 --2- 172.21.15.1:0/2958457119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc94101640 0x7efc94101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.206+0000 7efc99600640 1 --2- 172.21.15.1:0/2958457119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc94102000 0x7efc9410e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.206+0000 7efc99600640 1 --2- 172.21.15.1:0/2958457119 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc94107650 0x7efc94107a30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.206+0000 7efc99600640 1 -- 172.21.15.1:0/2958457119 >> 172.21.15.1:0/2958457119 conn(0x7efc940fd110 msgr2=0x7efc940ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.206+0000 7efc99600640 1 -- 172.21.15.1:0/2958457119 shutdown_connections 2024-04-26T21:42:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.207+0000 7efc99600640 1 -- 172.21.15.1:0/2958457119 wait complete. 2024-04-26T21:42:58.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.208+0000 7efc99600640 1 Processor -- start 2024-04-26T21:42:58.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.208+0000 7efc99600640 1 -- start start 2024-04-26T21:42:58.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.208+0000 7efc99600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc94101640 0x7efc9410eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.208+0000 7efc99600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc94102000 0x7efc9410f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.208+0000 7efc99600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc94107650 0x7efc9410f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.208+0000 7efc99600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efc94112a80 con 0x7efc94101640 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.208+0000 7efc99600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efc94112900 con 0x7efc94107650 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.208+0000 7efc99600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efc94112c00 con 0x7efc94102000 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc98c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc94107650 0x7efc9410f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc93e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc94101640 0x7efc9410eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc98c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc94107650 0x7efc9410f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47492/0 (socket says 172.21.15.1:47492) 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc98c00640 1 -- 172.21.15.1:0/2246550375 learned_addr learned my addr 172.21.15.1:0/2246550375 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc8b400640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc94102000 0x7efc9410f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc98c00640 1 -- 172.21.15.1:0/2246550375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc94102000 msgr2=0x7efc9410f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc98c00640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc94102000 0x7efc9410f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc98c00640 1 -- 172.21.15.1:0/2246550375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc94101640 msgr2=0x7efc9410eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc98c00640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc94101640 0x7efc9410eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc98c00640 1 -- 172.21.15.1:0/2246550375 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc941a7460 con 0x7efc94107650 2024-04-26T21:42:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.209+0000 7efc98c00640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc94107650 0x7efc9410f720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00efd0 tx=0x7efc8c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.210+0000 7efc91600640 1 -- 172.21.15.1:0/2246550375 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c003b70 con 0x7efc94107650 2024-04-26T21:42:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.210+0000 7efc91600640 1 -- 172.21.15.1:0/2246550375 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc8c003d10 con 0x7efc94107650 2024-04-26T21:42:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.210+0000 7efc91600640 1 -- 172.21.15.1:0/2246550375 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0108a0 con 0x7efc94107650 2024-04-26T21:42:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.210+0000 7efc99600640 1 -- 172.21.15.1:0/2246550375 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc941a7750 con 0x7efc94107650 2024-04-26T21:42:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.210+0000 7efc99600640 1 -- 172.21.15.1:0/2246550375 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc941a7ca0 con 0x7efc94107650 2024-04-26T21:42:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.214+0000 7efc91600640 1 -- 172.21.15.1:0/2246550375 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7efc8c004020 con 0x7efc94107650 2024-04-26T21:42:58.215 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.214+0000 7efc9da25740 1 Processor -- start 2024-04-26T21:42:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.215+0000 7efc9da25740 1 -- start start 2024-04-26T21:42:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.215+0000 7efc9da25740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a3c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.215+0000 7efc9da25740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a4f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.215+0000 7efc9da25740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a51dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.215+0000 7efc9da25740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c0849a1380 con 0x55c084a3bf20 2024-04-26T21:42:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.215+0000 7efc9da25740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c08493dc30 con 0x55c084a4f9d0 2024-04-26T21:42:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.215+0000 7efc9da25740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c08499d340 con 0x55c084a3c840 2024-04-26T21:42:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.216+0000 7efc93e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a4f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.216+0000 7efc8b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a3c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.216+0000 7efc93e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a4f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:39904/0 (socket says 172.21.15.1:39904) 2024-04-26T21:42:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.216+0000 7efc93e00640 1 -- 172.21.15.1:0/3119529066 learned_addr learned my addr 172.21.15.1:0/3119529066 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.216+0000 7efc8b400640 1 --2- 172.21.15.1:0/3119529066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a51dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.216+0000 7efc93e00640 1 -- 172.21.15.1:0/3119529066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 msgr2=0x55c084a51dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.216+0000 7efc93e00640 1 --2- 172.21.15.1:0/3119529066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a51dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.216+0000 7efc93e00640 1 -- 172.21.15.1:0/3119529066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 msgr2=0x55c084a3c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.216+0000 7efc93e00640 1 --2- 172.21.15.1:0/3119529066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a3c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.216+0000 7efc93e00640 1 -- 172.21.15.1:0/3119529066 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c08495f940 con 0x55c084a3c840 2024-04-26T21:42:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.217+0000 7efc91600640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7efc780a3d80 0x7efc780a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.217+0000 7efc93e00640 1 --2- 172.21.15.1:0/3119529066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a4f490 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efc84009900 tx=0x7efc840040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3177fa7c31dc91e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.217+0000 7efc91600640 1 -- 172.21.15.1:0/2246550375 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7efc8c0d1070 con 0x7efc94107650 2024-04-26T21:42:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.218+0000 7efc89200640 1 -- 172.21.15.1:0/3119529066 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84035030 con 0x55c084a3c840 2024-04-26T21:42:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.218+0000 7efc89200640 1 -- 172.21.15.1:0/3119529066 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc840044b0 con 0x55c084a3c840 2024-04-26T21:42:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.218+0000 7efc89200640 1 -- 172.21.15.1:0/3119529066 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc840035f0 con 0x55c084a3c840 2024-04-26T21:42:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.218+0000 7efc8b400640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7efc780a3d80 0x7efc780a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.219+0000 7efc9da25740 1 -- 172.21.15.1:0/3119529066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 msgr2=0x55c084a4f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.219+0000 7efc9da25740 1 --2- 172.21.15.1:0/3119529066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a4f490 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efc84009900 tx=0x7efc840040c0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.219+0000 7efc9da25740 1 -- 172.21.15.1:0/3119529066 shutdown_connections 2024-04-26T21:42:58.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.219+0000 7efc9da25740 1 --2- 172.21.15.1:0/3119529066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a4f490 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.219+0000 7efc9da25740 1 --2- 172.21.15.1:0/3119529066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a51dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.219+0000 7efc9da25740 1 --2- 172.21.15.1:0/3119529066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a3c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.219+0000 7efc9da25740 1 -- 172.21.15.1:0/3119529066 >> 172.21.15.1:0/3119529066 conn(0x55c084a384f0 msgr2=0x55c084a388d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:58.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.219+0000 7efc8b400640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7efc780a3d80 0x7efc780a6240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009710 tx=0x7efc7c005850 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.219+0000 7efc9da25740 1 -- 172.21.15.1:0/3119529066 shutdown_connections 2024-04-26T21:42:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.220+0000 7efc9da25740 1 -- 172.21.15.1:0/3119529066 wait complete. 2024-04-26T21:42:58.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.220+0000 7efc9da25740 1 -- start start 2024-04-26T21:42:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.220+0000 7efc9da25740 1 Processor -- start 2024-04-26T21:42:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.221+0000 7efc9da25740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3bf20 0x55c084a4c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.221+0000 7efc9da25740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3c840 0x55c084a4ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.221+0000 7efc9da25740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.221+0000 7efc9da25740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c084a52590 con 0x55c084a3c840 2024-04-26T21:42:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.221+0000 7efc9da25740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c084928c60 con 0x55c084a4f9d0 2024-04-26T21:42:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.221+0000 7efc9da25740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c084a52710 con 0x55c084a3bf20 2024-04-26T21:42:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.221+0000 7efc9da25740 10 client.0 _ll_register_callbacks cb 0x55c084954e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:42:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.222+0000 7efc8b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3c840 0x55c084a4ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.222+0000 7efc8b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3c840 0x55c084a4ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47700/0 (socket says 172.21.15.1:47700) 2024-04-26T21:42:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.222+0000 7efc8b400640 1 -- 172.21.15.1:0/2076312823 learned_addr learned my addr 172.21.15.1:0/2076312823 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.222+0000 7efc93e00640 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3bf20 0x55c084a4c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.222+0000 7efc98c00640 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.222+0000 7efc8b400640 1 -- 172.21.15.1:0/2076312823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3bf20 msgr2=0x55c084a4c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.222+0000 7efc8b400640 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3bf20 0x55c084a4c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.222+0000 7efc8b400640 1 -- 172.21.15.1:0/2076312823 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 msgr2=0x55c084a4d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.223+0000 7efc8b400640 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.223+0000 7efc8b400640 1 -- 172.21.15.1:0/2076312823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c084937510 con 0x55c084a3c840 2024-04-26T21:42:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.223+0000 7efc98c00640 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:42:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.223+0000 7efc8b400640 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3c840 0x55c084a4ceb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efc7c011a80 tx=0x7efc7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4e74454a2decb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:58.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.223+0000 7efc81600640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:42:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.224+0000 7efc81600640 1 -- 172.21.15.1:0/2076312823 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c021930 con 0x55c084a3c840 2024-04-26T21:42:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.224+0000 7efc81600640 1 -- 172.21.15.1:0/2076312823 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7efc7c009040 con 0x55c084a3c840 2024-04-26T21:42:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.224+0000 7efc9da25740 1 -- 172.21.15.1:0/2076312823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc840044b0 con 0x55c084a3c840 2024-04-26T21:42:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.224+0000 7efc81600640 1 -- 172.21.15.1:0/2076312823 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7efc7c034050 con 0x55c084a3c840 2024-04-26T21:42:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.224+0000 7efc9da25740 10 client.16332 fetch_fsmap learned FSMap version 41 2024-04-26T21:42:58.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.224+0000 7efc9da25740 1 -- 172.21.15.1:0/2076312823 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c0846b3210 con 0x55c084a3c840 2024-04-26T21:42:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.224+0000 7efc81600640 1 -- 172.21.15.1:0/2076312823 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc840044b0 con 0x55c084a3c840 2024-04-26T21:42:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.225+0000 7efc81600640 1 -- 172.21.15.1:0/2076312823 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7efc7c01d070 con 0x55c084a3c840 2024-04-26T21:42:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.225+0000 7efc9da25740 10 client.16332 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:42:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.225+0000 7efc9da25740 10 client.16332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:42:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.225+0000 7efc9da25740 20 client.16332 populate_metadata read hostname 'smithi001' 2024-04-26T21:42:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.225+0000 7efc9da25740 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c084a5f7f0 0x55c084a61c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.225+0000 7efc9da25740 4 client.16332 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:42:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.225+0000 7efc9da25740 1 -- 172.21.15.1:0/2076312823 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0849aef90 con 0x55c084a5f7f0 2024-04-26T21:42:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.225+0000 7efc93e00640 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c084a5f7f0 0x55c084a61c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.226+0000 7efc93e00640 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c084a5f7f0 0x55c084a61c00 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be3f0dcf424dfcd1 server_cookie=a86543617ec3189b in_seq=0 out_seq=0 2024-04-26T21:42:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.226+0000 7efc81600640 10 client.16332 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.227+0000 7efc81600640 1 -- 172.21.15.1:0/2076312823 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0849aef90 con 0x55c084a5f7f0 2024-04-26T21:42:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.227+0000 7efc81600640 10 client.16332 handle_command_reply: tid=0 2024-04-26T21:42:58.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.227+0000 7efc9da25740 1 client.16332 shutdown 2024-04-26T21:42:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 1 -- 172.21.15.1:0/2076312823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3c840 msgr2=0x55c084a4ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3c840 0x55c084a4ceb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efc7c011a80 tx=0x7efc7c004730 comp rx=0 tx=0).stop 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 1 -- 172.21.15.1:0/2076312823 shutdown_connections 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c084a5f7f0 0x55c084a61c00 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3c840 0x55c084a4ceb0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 1 --2- 172.21.15.1:0/2076312823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3bf20 0x55c084a4c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 1 -- 172.21.15.1:0/2076312823 >> 172.21.15.1:0/2076312823 conn(0x55c084a384f0 msgr2=0x55c084a3e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc81600640 0 client.16332 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 1 -- 172.21.15.1:0/2076312823 shutdown_connections 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 1 -- 172.21.15.1:0/2076312823 wait complete. 2024-04-26T21:42:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.228+0000 7efc9da25740 20 client.16332 trim_cache size 0 max 16384 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.246+0000 7efc9da25740 1 Processor -- start 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.246+0000 7efc9da25740 1 -- start start 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.246+0000 7efc9da25740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a4f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.246+0000 7efc9da25740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a5b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.246+0000 7efc9da25740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.246+0000 7efc9da25740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efc941b04f0 con 0x55c084a3bf20 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.246+0000 7efc9da25740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efc941add30 con 0x55c084a4f9d0 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.246+0000 7efc9da25740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efc941abba0 con 0x55c084a3c840 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc98c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc93e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a4f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc98c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47518/0 (socket says 172.21.15.1:47518) 2024-04-26T21:42:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc98c00640 1 -- 172.21.15.1:0/572334810 learned_addr learned my addr 172.21.15.1:0/572334810 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc8b400640 1 --2- 172.21.15.1:0/572334810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a5b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc98c00640 1 -- 172.21.15.1:0/572334810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 msgr2=0x55c084a5b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc98c00640 1 --2- 172.21.15.1:0/572334810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a5b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc98c00640 1 -- 172.21.15.1:0/572334810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 msgr2=0x55c084a4f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc98c00640 1 --2- 172.21.15.1:0/572334810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a4f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc98c00640 1 -- 172.21.15.1:0/572334810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c084a3c670 con 0x55c084a4f9d0 2024-04-26T21:42:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc98c00640 1 --2- 172.21.15.1:0/572334810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d040 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0ce410 tx=0x7efc8c009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99e138d3bb3152ab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.247+0000 7efc83e00640 1 -- 172.21.15.1:0/572334810 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0a0f60 con 0x55c084a4f9d0 2024-04-26T21:42:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.248+0000 7efc83e00640 1 -- 172.21.15.1:0/572334810 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc8c0b6070 con 0x55c084a4f9d0 2024-04-26T21:42:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.248+0000 7efc83e00640 1 -- 172.21.15.1:0/572334810 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0ce5e0 con 0x55c084a4f9d0 2024-04-26T21:42:58.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.248+0000 7efc9da25740 1 -- 172.21.15.1:0/572334810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 msgr2=0x55c084a4d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.248+0000 7efc9da25740 1 --2- 172.21.15.1:0/572334810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d040 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0ce410 tx=0x7efc8c009820 comp rx=0 tx=0).stop 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.248+0000 7efc9da25740 1 -- 172.21.15.1:0/572334810 shutdown_connections 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.248+0000 7efc9da25740 1 --2- 172.21.15.1:0/572334810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a5b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.248+0000 7efc9da25740 1 --2- 172.21.15.1:0/572334810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a4d040 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.248+0000 7efc9da25740 1 --2- 172.21.15.1:0/572334810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a4f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.248+0000 7efc9da25740 1 -- 172.21.15.1:0/572334810 >> 172.21.15.1:0/572334810 conn(0x55c084a384f0 msgr2=0x55c084a4c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.248+0000 7efc9da25740 1 -- 172.21.15.1:0/572334810 shutdown_connections 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.249+0000 7efc9da25740 1 -- 172.21.15.1:0/572334810 wait complete. 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.249+0000 7efc9da25740 1 -- start start 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.249+0000 7efc9da25740 1 Processor -- start 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc9da25740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a57d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc9da25740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a53100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc9da25740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a53640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc9da25740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efc941ac1f0 con 0x55c084a3bf20 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc9da25740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7efc941ad660 con 0x55c084a4f9d0 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc9da25740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efc941ac8c0 con 0x55c084a3c840 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc9da25740 10 client.0 _ll_register_callbacks cb 0x7efc7c003f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc93e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a57d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc98c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a53640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc98c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a53640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47520/0 (socket says 172.21.15.1:47520) 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc98c00640 1 -- 172.21.15.1:0/4880117 learned_addr learned my addr 172.21.15.1:0/4880117 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc8b400640 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a53100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc98c00640 1 -- 172.21.15.1:0/4880117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 msgr2=0x55c084a53100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc98c00640 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a53100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc98c00640 1 -- 172.21.15.1:0/4880117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 msgr2=0x55c084a57d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc98c00640 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a57d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc98c00640 1 -- 172.21.15.1:0/4880117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c08477f620 con 0x55c084a4f9d0 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.250+0000 7efc8b400640 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a53100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.251+0000 7efc98c00640 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a53640 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efc8c09cb40 tx=0x7efc8c009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b404076218f40e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.251+0000 7efc81600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.251+0000 7efc81600640 1 -- 172.21.15.1:0/4880117 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c0a5820 con 0x55c084a4f9d0 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.251+0000 7efc9da25740 1 -- 172.21.15.1:0/4880117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc8c0b6070 con 0x55c084a4f9d0 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.251+0000 7efc81600640 1 -- 172.21.15.1:0/4880117 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7efc8c099220 con 0x55c084a4f9d0 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.251+0000 7efc81600640 1 -- 172.21.15.1:0/4880117 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7efc8c004560 con 0x55c084a4f9d0 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.251+0000 7efc9da25740 10 client.26537 fetch_fsmap learned FSMap version 41 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.251+0000 7efc9da25740 1 -- 172.21.15.1:0/4880117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55c08495bcb0 con 0x55c084a4f9d0 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.251+0000 7efc81600640 1 -- 172.21.15.1:0/4880117 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc8c0b6070 con 0x55c084a4f9d0 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.251+0000 7efc81600640 1 -- 172.21.15.1:0/4880117 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7efc8c01f020 con 0x55c084a4f9d0 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.252+0000 7efc9da25740 10 client.26537 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:42:58.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.252+0000 7efc9da25740 10 client.26537 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:42:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.252+0000 7efc9da25740 20 client.26537 populate_metadata read hostname 'smithi001' 2024-04-26T21:42:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.252+0000 7efc9da25740 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c084ac6170 0x55c084ac8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:42:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.252+0000 7efc9da25740 4 client.26537 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:42:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.252+0000 7efc9da25740 1 -- 172.21.15.1:0/4880117 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c0849a8e00 con 0x55c084ac6170 2024-04-26T21:42:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.252+0000 7efc93e00640 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c084ac6170 0x55c084ac8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:42:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.252+0000 7efc93e00640 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c084ac6170 0x55c084ac8590 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c08c213f5dab0f7 server_cookie=5cbe7c064888931c in_seq=0 out_seq=0 2024-04-26T21:42:58.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.252+0000 7efc81600640 10 client.26537 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.310+0000 7efc81600640 1 -- 172.21.15.1:0/4880117 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c0849a8e00 con 0x55c084ac6170 2024-04-26T21:42:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.310+0000 7efc81600640 10 client.26537 handle_command_reply: tid=0 2024-04-26T21:42:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.310+0000 7efc9da25740 1 client.26537 shutdown 2024-04-26T21:42:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 1 -- 172.21.15.1:0/4880117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 msgr2=0x55c084a53640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a53640 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efc8c09cb40 tx=0x7efc8c009bf0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 1 -- 172.21.15.1:0/4880117 shutdown_connections 2024-04-26T21:42:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c084ac6170 0x55c084ac8590 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c084a3c840 0x55c084a53100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c084a4f9d0 0x55c084a53640 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 1 --2- 172.21.15.1:0/4880117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c084a3bf20 0x55c084a57d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc81600640 0 client.26537 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:42:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 1 -- 172.21.15.1:0/4880117 >> 172.21.15.1:0/4880117 conn(0x55c084a384f0 msgr2=0x55c084a4d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 1 -- 172.21.15.1:0/4880117 shutdown_connections 2024-04-26T21:42:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 1 -- 172.21.15.1:0/4880117 wait complete. 2024-04-26T21:42:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.311+0000 7efc9da25740 20 client.26537 trim_cache size 0 max 16384 2024-04-26T21:42:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:42:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:42:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:42:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:42:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.312+0000 7efc99600640 1 -- 172.21.15.1:0/2246550375 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7efc780a3d80 msgr2=0x7efc780a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.312+0000 7efc99600640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7efc780a3d80 0x7efc780a6240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009710 tx=0x7efc7c005850 comp rx=0 tx=0).stop 2024-04-26T21:42:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.312+0000 7efc99600640 1 -- 172.21.15.1:0/2246550375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc94107650 msgr2=0x7efc9410f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:42:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.312+0000 7efc99600640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc94107650 0x7efc9410f720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00efd0 tx=0x7efc8c00c6a0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.313+0000 7efc99600640 1 -- 172.21.15.1:0/2246550375 shutdown_connections 2024-04-26T21:42:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.313+0000 7efc99600640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7efc780a3d80 0x7efc780a6240 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.313+0000 7efc99600640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc94102000 0x7efc9410f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.313+0000 7efc99600640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc94107650 0x7efc9410f720 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.313+0000 7efc99600640 1 --2- 172.21.15.1:0/2246550375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc94101640 0x7efc9410eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:42:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.313+0000 7efc99600640 1 -- 172.21.15.1:0/2246550375 >> 172.21.15.1:0/2246550375 conn(0x7efc940fd110 msgr2=0x7efc940fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:42:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.313+0000 7efc99600640 1 -- 172.21.15.1:0/2246550375 shutdown_connections 2024-04-26T21:42:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:42:58.313+0000 7efc99600640 1 -- 172.21.15.1:0/2246550375 wait complete. 2024-04-26T21:42:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:42:58 smithi186 ceph-mon[32991]: pgmap v394: 129 pgs: 129 active+clean; 364 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 2.2 MiB/s wr, 113 op/s 2024-04-26T21:42:58.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:42:58 smithi001 ceph-mon[28054]: pgmap v394: 129 pgs: 129 active+clean; 364 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 2.2 MiB/s wr, 113 op/s 2024-04-26T21:42:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:42:58 smithi038 ceph-mon[32724]: pgmap v394: 129 pgs: 129 active+clean; 364 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 2.2 MiB/s wr, 113 op/s 2024-04-26T21:42:58.613 INFO:tasks.workunit.client.0.smithi001.stdout: 1 485649 4.25 MB/sec execute 536 sec latency 52.633 ms 2024-04-26T21:42:59.613 INFO:tasks.workunit.client.0.smithi001.stdout: 1 486833 4.26 MB/sec execute 537 sec latency 46.389 ms 2024-04-26T21:43:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:00 smithi186 ceph-mon[32991]: pgmap v395: 129 pgs: 129 active+clean; 364 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 2.2 MiB/s wr, 113 op/s 2024-04-26T21:43:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:00 smithi001 ceph-mon[28054]: pgmap v395: 129 pgs: 129 active+clean; 364 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 2.2 MiB/s wr, 113 op/s 2024-04-26T21:43:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:00 smithi038 ceph-mon[32724]: pgmap v395: 129 pgs: 129 active+clean; 364 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 2.2 MiB/s wr, 113 op/s 2024-04-26T21:43:00.613 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487169 4.25 MB/sec execute 538 sec latency 107.262 ms 2024-04-26T21:43:01.613 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487169 4.25 MB/sec execute 539 sec latency 1107.404 ms 2024-04-26T21:43:02.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:02 smithi186 ceph-mon[32991]: pgmap v396: 129 pgs: 129 active+clean; 375 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-04-26T21:43:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:02 smithi001 ceph-mon[28054]: pgmap v396: 129 pgs: 129 active+clean; 375 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-04-26T21:43:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:02 smithi038 ceph-mon[32724]: pgmap v396: 129 pgs: 129 active+clean; 375 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-04-26T21:43:02.613 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487169 4.24 MB/sec execute 540 sec latency 2107.544 ms 2024-04-26T21:43:03.613 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487169 4.23 MB/sec execute 541 sec latency 3107.664 ms 2024-04-26T21:43:04.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:04 smithi186 ceph-mon[32991]: pgmap v397: 129 pgs: 129 active+clean; 375 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 1.7 MiB/s wr, 114 op/s 2024-04-26T21:43:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:04 smithi001 ceph-mon[28054]: pgmap v397: 129 pgs: 129 active+clean; 375 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 1.7 MiB/s wr, 114 op/s 2024-04-26T21:43:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:04 smithi038 ceph-mon[32724]: pgmap v397: 129 pgs: 129 active+clean; 375 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 1.7 MiB/s wr, 114 op/s 2024-04-26T21:43:04.613 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487224 4.22 MB/sec execute 542 sec latency 3792.375 ms 2024-04-26T21:43:05.614 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487224 4.21 MB/sec execute 543 sec latency 1128.685 ms 2024-04-26T21:43:06.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:06 smithi186 ceph-mon[32991]: pgmap v398: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-26T21:43:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:06 smithi001 ceph-mon[28054]: pgmap v398: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-26T21:43:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:06 smithi038 ceph-mon[32724]: pgmap v398: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.1 MiB/s wr, 154 op/s 2024-04-26T21:43:06.614 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487224 4.21 MB/sec execute 544 sec latency 2128.837 ms 2024-04-26T21:43:07.615 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487224 4.20 MB/sec execute 545 sec latency 3128.942 ms 2024-04-26T21:43:08.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:08 smithi186 ceph-mon[32991]: pgmap v399: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.4 MiB/s wr, 105 op/s 2024-04-26T21:43:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:08 smithi001 ceph-mon[28054]: pgmap v399: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.4 MiB/s wr, 105 op/s 2024-04-26T21:43:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:08 smithi038 ceph-mon[32724]: pgmap v399: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.4 MiB/s wr, 105 op/s 2024-04-26T21:43:08.614 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487224 4.19 MB/sec execute 546 sec latency 4129.036 ms 2024-04-26T21:43:09.614 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487498 4.18 MB/sec execute 547 sec latency 4802.917 ms 2024-04-26T21:43:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:10 smithi186 ceph-mon[32991]: pgmap v400: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.4 MiB/s wr, 105 op/s 2024-04-26T21:43:10.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:10 smithi001 ceph-mon[28054]: pgmap v400: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.4 MiB/s wr, 105 op/s 2024-04-26T21:43:10.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:10 smithi038 ceph-mon[32724]: pgmap v400: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 1.4 MiB/s wr, 105 op/s 2024-04-26T21:43:10.614 INFO:tasks.workunit.client.0.smithi001.stdout: 1 487791 4.18 MB/sec execute 548 sec latency 116.199 ms 2024-04-26T21:43:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.466+0000 7f70a1400640 1 -- 172.21.15.1:0/4277171516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c10cd20 msgr2=0x7f709c10f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.466+0000 7f70a1400640 1 --2- 172.21.15.1:0/4277171516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c10cd20 0x7f709c10f1c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7094009ca0 tx=0x7f709402a950 comp rx=0 tx=0).stop 2024-04-26T21:43:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.466+0000 7f70a1400640 1 -- 172.21.15.1:0/4277171516 shutdown_connections 2024-04-26T21:43:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.466+0000 7f70a1400640 1 --2- 172.21.15.1:0/4277171516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c10cd20 0x7f709c10f1c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.466+0000 7f70a1400640 1 --2- 172.21.15.1:0/4277171516 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c104220 0x7f709c10c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.466+0000 7f70a1400640 1 --2- 172.21.15.1:0/4277171516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c103900 0x7f709c103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.466+0000 7f70a1400640 1 -- 172.21.15.1:0/4277171516 >> 172.21.15.1:0/4277171516 conn(0x7f709c0fd140 msgr2=0x7f709c0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.467+0000 7f70a1400640 1 -- 172.21.15.1:0/4277171516 shutdown_connections 2024-04-26T21:43:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.467+0000 7f70a1400640 1 -- 172.21.15.1:0/4277171516 wait complete. 2024-04-26T21:43:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.468+0000 7f70a1400640 1 Processor -- start 2024-04-26T21:43:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.468+0000 7f70a1400640 1 -- start start 2024-04-26T21:43:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a1400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c103900 0x7f709c19b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a1400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c104220 0x7f709c19b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a1400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c10cd20 0x7f709c19fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a1400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f709c102380 con 0x7f709c103900 2024-04-26T21:43:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a1400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f709c102200 con 0x7f709c10cd20 2024-04-26T21:43:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a1400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f709c102500 con 0x7f709c104220 2024-04-26T21:43:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c10cd20 0x7f709c19fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f709be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c103900 0x7f709c19b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c10cd20 0x7f709c19fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40356/0 (socket says 172.21.15.1:40356) 2024-04-26T21:43:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a0a00640 1 -- 172.21.15.1:0/3558165938 learned_addr learned my addr 172.21.15.1:0/3558165938 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f709be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c103900 0x7f709c19b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56716/0 (socket says 172.21.15.1:56716) 2024-04-26T21:43:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f709b400640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c104220 0x7f709c19b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a0a00640 1 -- 172.21.15.1:0/3558165938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c104220 msgr2=0x7f709c19b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a0a00640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c104220 0x7f709c19b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a0a00640 1 -- 172.21.15.1:0/3558165938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c103900 msgr2=0x7f709c19b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.469+0000 7f70a0a00640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c103900 0x7f709c19b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.470+0000 7f70a0a00640 1 -- 172.21.15.1:0/3558165938 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709c1a0130 con 0x7f709c10cd20 2024-04-26T21:43:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.470+0000 7f70a0a00640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c10cd20 0x7f709c19fa00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7094004af0 tx=0x7f7094031f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.470+0000 7f7098c00640 1 -- 172.21.15.1:0/3558165938 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094034040 con 0x7f709c10cd20 2024-04-26T21:43:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.470+0000 7f7098c00640 1 -- 172.21.15.1:0/3558165938 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70940308f0 con 0x7f709c10cd20 2024-04-26T21:43:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.470+0000 7f70a1400640 1 -- 172.21.15.1:0/3558165938 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709c1a03c0 con 0x7f709c10cd20 2024-04-26T21:43:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.470+0000 7f7098c00640 1 -- 172.21.15.1:0/3558165938 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7094004600 con 0x7f709c10cd20 2024-04-26T21:43:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.470+0000 7f70a1400640 1 -- 172.21.15.1:0/3558165938 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709c1a7cb0 con 0x7f709c10cd20 2024-04-26T21:43:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.474+0000 7f7098c00640 1 -- 172.21.15.1:0/3558165938 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f7094037020 con 0x7f709c10cd20 2024-04-26T21:43:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.474+0000 7f70a5808740 1 Processor -- start 2024-04-26T21:43:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.474+0000 7f70a5808740 1 -- start start 2024-04-26T21:43:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f70a5808740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 0x5631ba7f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f70a5808740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba7fdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f70a5808740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba8005b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f70a5808740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5631ba758380 con 0x5631ba7f2fe0 2024-04-26T21:43:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f70a5808740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5631ba6f4c30 con 0x5631ba7f3900 2024-04-26T21:43:11.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f70a5808740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5631ba754340 con 0x5631ba7fe1c0 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f709b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 0x5631ba7f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f709be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba7fdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f709be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba7fdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40364/0 (socket says 172.21.15.1:40364) 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f709be00640 1 -- 172.21.15.1:0/4104294054 learned_addr learned my addr 172.21.15.1:0/4104294054 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f709b400640 1 --2- 172.21.15.1:0/4104294054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba8005b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.475+0000 7f7098c00640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f70800a3d80 0x7f70800a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f709be00640 1 -- 172.21.15.1:0/4104294054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 msgr2=0x5631ba8005b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f709be00640 1 --2- 172.21.15.1:0/4104294054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba8005b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f709be00640 1 -- 172.21.15.1:0/4104294054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 msgr2=0x5631ba7f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f709be00640 1 --2- 172.21.15.1:0/4104294054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 0x5631ba7f33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f709be00640 1 -- 172.21.15.1:0/4104294054 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631ba716940 con 0x5631ba7f3900 2024-04-26T21:43:11.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f70a0a00640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f70800a3d80 0x7f70800a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f7098c00640 1 -- 172.21.15.1:0/3558165938 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7f70940f17c0 con 0x7f709c10cd20 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f709be00640 1 --2- 172.21.15.1:0/4104294054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba7fdc80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f708c009d20 tx=0x7f708c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe014245914ad0eb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f7091200640 1 -- 172.21.15.1:0/4104294054 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c01e070 con 0x5631ba7f3900 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f7091200640 1 -- 172.21.15.1:0/4104294054 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f708c019ae0 con 0x5631ba7f3900 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.476+0000 7f7091200640 1 -- 172.21.15.1:0/4104294054 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c00eac0 con 0x5631ba7f3900 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.477+0000 7f70a5808740 1 -- 172.21.15.1:0/4104294054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 msgr2=0x5631ba7fdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.477+0000 7f70a5808740 1 --2- 172.21.15.1:0/4104294054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba7fdc80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f708c009d20 tx=0x7f708c011810 comp rx=0 tx=0).stop 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.477+0000 7f70a0a00640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f70800a3d80 0x7f70800a6240 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f70940bd0b0 tx=0x7f7094031fc0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.477+0000 7f70a5808740 1 -- 172.21.15.1:0/4104294054 shutdown_connections 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.477+0000 7f70a5808740 1 --2- 172.21.15.1:0/4104294054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba8005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.477+0000 7f70a5808740 1 --2- 172.21.15.1:0/4104294054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba7fdc80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.477+0000 7f70a5808740 1 --2- 172.21.15.1:0/4104294054 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 0x5631ba7f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.477+0000 7f70a5808740 1 -- 172.21.15.1:0/4104294054 >> 172.21.15.1:0/4104294054 conn(0x5631ba7ef4f0 msgr2=0x5631ba7ef8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.478+0000 7f70a5808740 1 -- 172.21.15.1:0/4104294054 shutdown_connections 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.478+0000 7f70a5808740 1 -- 172.21.15.1:0/4104294054 wait complete. 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.478+0000 7f70a5808740 1 -- start start 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.478+0000 7f70a5808740 1 Processor -- start 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f70a5808740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f70a5808740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 0x5631ba7fb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f70a5808740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7fe1c0 0x5631ba7fbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f70a5808740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5631ba800df0 con 0x5631ba7f3900 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f70a5808740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5631ba6dfc60 con 0x5631ba7fe1c0 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f70a5808740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5631ba800f70 con 0x5631ba7f2fe0 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f70a5808740 10 client.0 _ll_register_callbacks cb 0x5631ba70be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f709b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f709b400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:45426/0 (socket says 172.21.15.1:45426) 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f709b400640 1 -- 172.21.15.1:0/1694392809 learned_addr learned my addr 172.21.15.1:0/1694392809 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f709be00640 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 0x5631ba7fb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.479+0000 7f709b400640 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7fe1c0 0x5631ba7fbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f709b400640 1 -- 172.21.15.1:0/1694392809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7fe1c0 msgr2=0x5631ba7fbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f709b400640 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7fe1c0 0x5631ba7fbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f709b400640 1 -- 172.21.15.1:0/1694392809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 msgr2=0x5631ba7fb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f709b400640 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 0x5631ba7fb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f709b400640 1 -- 172.21.15.1:0/1694392809 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631ba6ee510 con 0x5631ba7f2fe0 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f709b400640 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fb090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f709c19c1c0 tx=0x7f7088015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c65945a70475f338 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f7085600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f7085600640 1 -- 172.21.15.1:0/1694392809 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7088005020 con 0x5631ba7f2fe0 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f7085600640 1 -- 172.21.15.1:0/1694392809 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7f70880051c0 con 0x5631ba7f2fe0 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f70a5808740 1 -- 172.21.15.1:0/1694392809 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f708c019ae0 con 0x5631ba7f2fe0 2024-04-26T21:43:11.489 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f7085600640 1 -- 172.21.15.1:0/1694392809 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f70880053c0 con 0x5631ba7f2fe0 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f70a5808740 10 client.26521 fetch_fsmap learned FSMap version 41 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.480+0000 7f70a5808740 1 -- 172.21.15.1:0/1694392809 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5631ba46a210 con 0x5631ba7f2fe0 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.481+0000 7f7085600640 1 -- 172.21.15.1:0/1694392809 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f708c019ae0 con 0x5631ba7f2fe0 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.481+0000 7f7085600640 1 -- 172.21.15.1:0/1694392809 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f70880047a0 con 0x5631ba7f2fe0 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.481+0000 7f70a5808740 10 client.26521 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.481+0000 7f70a5808740 10 client.26521 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.481+0000 7f70a5808740 20 client.26521 populate_metadata read hostname 'smithi001' 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.481+0000 7f70a5808740 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5631ba80df60 0x5631ba810370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.481+0000 7f70a5808740 4 client.26521 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.481+0000 7f70a5808740 1 -- 172.21.15.1:0/1694392809 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631ba765f90 con 0x5631ba80df60 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.481+0000 7f709be00640 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5631ba80df60 0x5631ba810370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.482+0000 7f709be00640 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5631ba80df60 0x5631ba810370 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cad2f6ac9538f3e6 server_cookie=f46ec79058286e22 in_seq=0 out_seq=0 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.482+0000 7f7085600640 10 client.26521 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f7085600640 1 -- 172.21.15.1:0/1694392809 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5631ba765f90 con 0x5631ba80df60 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f7085600640 10 client.26521 handle_command_reply: tid=0 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 client.26521 shutdown 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 -- 172.21.15.1:0/1694392809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7f2fe0 msgr2=0x5631ba7fb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fb090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f709c19c1c0 tx=0x7f7088015040 comp rx=0 tx=0).stop 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 -- 172.21.15.1:0/1694392809 shutdown_connections 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5631ba80df60 0x5631ba810370 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7fe1c0 0x5631ba7fbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 0x5631ba7fb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 --2- 172.21.15.1:0/1694392809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fb090 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f7085600640 0 client.26521 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 -- 172.21.15.1:0/1694392809 >> 172.21.15.1:0/1694392809 conn(0x5631ba7ef4f0 msgr2=0x5631ba7f5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 -- 172.21.15.1:0/1694392809 shutdown_connections 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.483+0000 7f70a5808740 1 -- 172.21.15.1:0/1694392809 wait complete. 2024-04-26T21:43:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.484+0000 7f70a5808740 20 client.26521 trim_cache size 0 max 16384 2024-04-26T21:43:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.503+0000 7f70a5808740 1 Processor -- start 2024-04-26T21:43:11.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.503+0000 7f70a5808740 1 -- start start 2024-04-26T21:43:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.503+0000 7f70a5808740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.503+0000 7f70a5808740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba809ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.503+0000 7f70a5808740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba7ee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.503+0000 7f70a5808740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f709c1b4910 con 0x5631ba7f2fe0 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.503+0000 7f70a5808740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f709c1b2150 con 0x5631ba7f3900 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.503+0000 7f70a5808740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f709c1affc0 con 0x5631ba7fe1c0 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba809ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba809ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40376/0 (socket says 172.21.15.1:40376) 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56760/0 (socket says 172.21.15.1:56760) 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709be00640 1 -- 172.21.15.1:0/3990414940 learned_addr learned my addr 172.21.15.1:0/3990414940 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709b400640 1 --2- 172.21.15.1:0/3990414940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba7ee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709be00640 1 -- 172.21.15.1:0/3990414940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 msgr2=0x5631ba7ee4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:11.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709be00640 1 --2- 172.21.15.1:0/3990414940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba7ee4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709be00640 1 -- 172.21.15.1:0/3990414940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 msgr2=0x5631ba7fd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709be00640 1 --2- 172.21.15.1:0/3990414940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709be00640 1 -- 172.21.15.1:0/3990414940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631ba536620 con 0x5631ba7f3900 2024-04-26T21:43:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709b400640 1 --2- 172.21.15.1:0/3990414940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:43:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.504+0000 7f709be00640 1 --2- 172.21.15.1:0/3990414940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba809ce0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f708c009250 tx=0x7f708c00e3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7975e42f6fda0910 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.505+0000 7f7087e00640 1 -- 172.21.15.1:0/3990414940 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c01e040 con 0x5631ba7f3900 2024-04-26T21:43:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.505+0000 7f7087e00640 1 -- 172.21.15.1:0/3990414940 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f708c019430 con 0x5631ba7f3900 2024-04-26T21:43:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.505+0000 7f7087e00640 1 -- 172.21.15.1:0/3990414940 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c00e9b0 con 0x5631ba7f3900 2024-04-26T21:43:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.506+0000 7f70a5808740 1 -- 172.21.15.1:0/3990414940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 msgr2=0x5631ba809ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.506+0000 7f70a5808740 1 --2- 172.21.15.1:0/3990414940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba809ce0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f708c009250 tx=0x7f708c00e3d0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.506+0000 7f70a5808740 1 -- 172.21.15.1:0/3990414940 shutdown_connections 2024-04-26T21:43:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.506+0000 7f70a5808740 1 --2- 172.21.15.1:0/3990414940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba7ee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.506+0000 7f70a5808740 1 --2- 172.21.15.1:0/3990414940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f3900 0x5631ba809ce0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.506+0000 7f70a5808740 1 --2- 172.21.15.1:0/3990414940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f2fe0 0x5631ba7fd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.506+0000 7f70a5808740 1 -- 172.21.15.1:0/3990414940 >> 172.21.15.1:0/3990414940 conn(0x5631ba7ef4f0 msgr2=0x5631ba7eddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.506+0000 7f70a5808740 1 -- 172.21.15.1:0/3990414940 shutdown_connections 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.506+0000 7f70a5808740 1 -- 172.21.15.1:0/3990414940 wait complete. 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.506+0000 7f70a5808740 1 -- start start 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f70a5808740 1 Processor -- start 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f70a5808740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f2fe0 0x5631ba7f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f70a5808740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 0x5631ba7f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f70a5808740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba7f8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f70a5808740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f709c1b0610 con 0x5631ba7f3900 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f70a5808740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f709c1b1a80 con 0x5631ba7f2fe0 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f70a5808740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f709c1b0ce0 con 0x5631ba7fe1c0 2024-04-26T21:43:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f70a5808740 10 client.0 _ll_register_callbacks cb 0x7f7088009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f709b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f2fe0 0x5631ba7f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f709b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f2fe0 0x5631ba7f7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40384/0 (socket says 172.21.15.1:40384) 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f709b400640 1 -- 172.21.15.1:0/403235589 learned_addr learned my addr 172.21.15.1:0/403235589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.507+0000 7f709b400640 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba7f8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f709be00640 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 0x5631ba7f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f709b400640 1 -- 172.21.15.1:0/403235589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 msgr2=0x5631ba7f8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f709b400640 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba7f8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f709b400640 1 -- 172.21.15.1:0/403235589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 msgr2=0x5631ba7f81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f709b400640 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 0x5631ba7f81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f709b400640 1 -- 172.21.15.1:0/403235589 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631ba712cb0 con 0x5631ba7f2fe0 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f709be00640 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 0x5631ba7f81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f709b400640 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f2fe0 0x5631ba7f7cb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7088009380 tx=0x7f7088001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d5200c950513ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f7085600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:43:11.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f7085600640 1 -- 172.21.15.1:0/403235589 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70880045d0 con 0x5631ba7f2fe0 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f7085600640 1 -- 172.21.15.1:0/403235589 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(108..108 src has 1..108) ==== 9642+0+0 (secure 0 0 0) 0x7f7088004770 con 0x5631ba7f2fe0 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.508+0000 7f70a5808740 1 -- 172.21.15.1:0/403235589 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f708c019430 con 0x5631ba7f2fe0 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f7085600640 1 -- 172.21.15.1:0/403235589 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f7088004970 con 0x5631ba7f2fe0 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f70a5808740 10 client.26555 fetch_fsmap learned FSMap version 41 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f70a5808740 1 -- 172.21.15.1:0/403235589 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5631ba8744f0 con 0x5631ba7f2fe0 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f7085600640 1 -- 172.21.15.1:0/403235589 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f708c019430 con 0x5631ba7f2fe0 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f7085600640 1 -- 172.21.15.1:0/403235589 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f7088003920 con 0x5631ba7f2fe0 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f70a5808740 10 client.26555 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f70a5808740 10 client.26555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f70a5808740 20 client.26555 populate_metadata read hostname 'smithi001' 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f70a5808740 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5631ba874bb0 0x5631ba876fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:11.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f70a5808740 4 client.26555 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:43:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.509+0000 7f70a5808740 1 -- 172.21.15.1:0/403235589 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5631ba75fe00 con 0x5631ba874bb0 2024-04-26T21:43:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.510+0000 7f709be00640 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5631ba874bb0 0x5631ba876fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.510+0000 7f709be00640 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5631ba874bb0 0x5631ba876fd0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a456ddf3f4c94b6 server_cookie=44ebcc4a15b438af in_seq=0 out_seq=0 2024-04-26T21:43:11.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.510+0000 7f7085600640 10 client.26555 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:11.616 INFO:tasks.workunit.client.0.smithi001.stdout: 1 488420 4.18 MB/sec execute 549 sec latency 66.369 ms 2024-04-26T21:43:11.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f7085600640 1 -- 172.21.15.1:0/403235589 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5631ba75fe00 con 0x5631ba874bb0 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f7085600640 10 client.26555 handle_command_reply: tid=0 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 client.26555 shutdown 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 -- 172.21.15.1:0/403235589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f2fe0 msgr2=0x5631ba7f7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f2fe0 0x5631ba7f7cb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7088009380 tx=0x7f7088001e50 comp rx=0 tx=0).stop 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 -- 172.21.15.1:0/403235589 shutdown_connections 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5631ba874bb0 0x5631ba876fd0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5631ba7fe1c0 0x5631ba7f8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5631ba7f3900 0x5631ba7f81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f7085600640 0 client.26555 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 --2- 172.21.15.1:0/403235589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631ba7f2fe0 0x5631ba7f7cb0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 -- 172.21.15.1:0/403235589 >> 172.21.15.1:0/403235589 conn(0x5631ba7ef4f0 msgr2=0x5631ba7ee650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 -- 172.21.15.1:0/403235589 shutdown_connections 2024-04-26T21:43:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.634+0000 7f70a5808740 1 -- 172.21.15.1:0/403235589 wait complete. 2024-04-26T21:43:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.635+0000 7f70a5808740 20 client.26555 trim_cache size 0 max 16384 2024-04-26T21:43:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:43:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:43:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:43:11.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:43:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.635+0000 7f70a1400640 1 -- 172.21.15.1:0/3558165938 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f70800a3d80 msgr2=0x7f70800a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.635+0000 7f70a1400640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f70800a3d80 0x7f70800a6240 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f70940bd0b0 tx=0x7f7094031fc0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.635+0000 7f70a1400640 1 -- 172.21.15.1:0/3558165938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c10cd20 msgr2=0x7f709c19fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:11.638 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.635+0000 7f70a1400640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c10cd20 0x7f709c19fa00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7094004af0 tx=0x7f7094031f50 comp rx=0 tx=0).stop 2024-04-26T21:43:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.636+0000 7f70a1400640 1 -- 172.21.15.1:0/3558165938 shutdown_connections 2024-04-26T21:43:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.636+0000 7f70a1400640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f70800a3d80 0x7f70800a6240 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.636+0000 7f70a1400640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f709c104220 0x7f709c19b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.636+0000 7f70a1400640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f709c10cd20 0x7f709c19fa00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.636+0000 7f70a1400640 1 --2- 172.21.15.1:0/3558165938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f709c103900 0x7f709c19b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.636+0000 7f70a1400640 1 -- 172.21.15.1:0/3558165938 >> 172.21.15.1:0/3558165938 conn(0x7f709c0fd140 msgr2=0x7f709c10e260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.636+0000 7f70a1400640 1 -- 172.21.15.1:0/3558165938 shutdown_connections 2024-04-26T21:43:11.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:11.636+0000 7f70a1400640 1 -- 172.21.15.1:0/3558165938 wait complete. 2024-04-26T21:43:12.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:12 smithi186 ceph-mon[32991]: pgmap v401: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.7 MiB/s wr, 128 op/s 2024-04-26T21:43:12.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:12 smithi001 ceph-mon[28054]: pgmap v401: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.7 MiB/s wr, 128 op/s 2024-04-26T21:43:12.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:12 smithi038 ceph-mon[32724]: pgmap v401: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.7 MiB/s wr, 128 op/s 2024-04-26T21:43:12.616 INFO:tasks.workunit.client.0.smithi001.stdout: 1 489259 4.18 MB/sec execute 550 sec latency 54.949 ms 2024-04-26T21:43:13.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:13 smithi001 ceph-mon[28054]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T21:43:13.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:13 smithi038 ceph-mon[32724]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T21:43:13.616 INFO:tasks.workunit.client.0.smithi001.stdout: 1 490422 4.19 MB/sec execute 551 sec latency 26.825 ms 2024-04-26T21:43:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:13 smithi186 ceph-mon[32991]: osdmap e109: 12 total, 12 up, 12 in 2024-04-26T21:43:14.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:14 smithi001 ceph-mon[28054]: pgmap v403: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 917 KiB/s wr, 74 op/s 2024-04-26T21:43:14.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:14 smithi038 ceph-mon[32724]: pgmap v403: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 917 KiB/s wr, 74 op/s 2024-04-26T21:43:14.616 INFO:tasks.workunit.client.0.smithi001.stdout: 1 490733 4.18 MB/sec execute 552 sec latency 48.745 ms 2024-04-26T21:43:14.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:14 smithi186 ceph-mon[32991]: pgmap v403: 129 pgs: 129 active+clean; 377 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 917 KiB/s wr, 74 op/s 2024-04-26T21:43:15.616 INFO:tasks.workunit.client.0.smithi001.stdout: 1 491354 4.18 MB/sec execute 553 sec latency 28.428 ms 2024-04-26T21:43:16.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:16 smithi001 ceph-mon[28054]: pgmap v404: 129 pgs: 129 active+clean; 380 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-26T21:43:16.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:16 smithi038 ceph-mon[32724]: pgmap v404: 129 pgs: 129 active+clean; 380 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-26T21:43:16.616 INFO:tasks.workunit.client.0.smithi001.stdout: 1 492287 4.18 MB/sec execute 554 sec latency 26.604 ms 2024-04-26T21:43:16.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:16 smithi186 ceph-mon[32991]: pgmap v404: 129 pgs: 129 active+clean; 380 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-26T21:43:17.616 INFO:tasks.workunit.client.0.smithi001.stdout: 1 493208 4.18 MB/sec execute 555 sec latency 32.833 ms 2024-04-26T21:43:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:18 smithi001 ceph-mon[28054]: pgmap v405: 129 pgs: 129 active+clean; 380 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-26T21:43:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:18 smithi038 ceph-mon[32724]: pgmap v405: 129 pgs: 129 active+clean; 380 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-26T21:43:18.616 INFO:tasks.workunit.client.0.smithi001.stdout: 1 494096 4.18 MB/sec execute 556 sec latency 30.130 ms 2024-04-26T21:43:18.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:18 smithi186 ceph-mon[32991]: pgmap v405: 129 pgs: 129 active+clean; 380 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-26T21:43:19.617 INFO:tasks.workunit.client.0.smithi001.stdout: 1 494427 4.18 MB/sec execute 557 sec latency 69.990 ms 2024-04-26T21:43:20.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:20 smithi001 ceph-mon[28054]: pgmap v406: 129 pgs: 129 active+clean; 380 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-26T21:43:20.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:20 smithi001 ceph-mon[28054]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T21:43:20.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:20 smithi038 ceph-mon[32724]: pgmap v406: 129 pgs: 129 active+clean; 380 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-26T21:43:20.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:20 smithi038 ceph-mon[32724]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T21:43:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:20 smithi186 ceph-mon[32991]: pgmap v406: 129 pgs: 129 active+clean; 380 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-04-26T21:43:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:20 smithi186 ceph-mon[32991]: osdmap e110: 12 total, 12 up, 12 in 2024-04-26T21:43:20.617 INFO:tasks.workunit.client.0.smithi001.stdout: 1 495340 4.18 MB/sec execute 558 sec latency 24.703 ms 2024-04-26T21:43:21.617 INFO:tasks.workunit.client.0.smithi001.stdout: 1 496447 4.18 MB/sec execute 559 sec latency 25.962 ms 2024-04-26T21:43:22.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:22 smithi001 ceph-mon[28054]: pgmap v408: 129 pgs: 129 active+clean; 385 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 179 op/s 2024-04-26T21:43:22.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:22 smithi038 ceph-mon[32724]: pgmap v408: 129 pgs: 129 active+clean; 385 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 179 op/s 2024-04-26T21:43:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:22 smithi186 ceph-mon[32991]: pgmap v408: 129 pgs: 129 active+clean; 385 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 179 op/s 2024-04-26T21:43:22.617 INFO:tasks.workunit.client.0.smithi001.stdout: 1 497631 4.19 MB/sec execute 560 sec latency 21.728 ms 2024-04-26T21:43:23.618 INFO:tasks.workunit.client.0.smithi001.stdout: 1 497873 4.18 MB/sec execute 561 sec latency 28.842 ms 2024-04-26T21:43:24.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:24 smithi001 ceph-mon[28054]: pgmap v409: 129 pgs: 129 active+clean; 385 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 1.8 MiB/s wr, 157 op/s 2024-04-26T21:43:24.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:24 smithi038 ceph-mon[32724]: pgmap v409: 129 pgs: 129 active+clean; 385 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 1.8 MiB/s wr, 157 op/s 2024-04-26T21:43:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:24 smithi186 ceph-mon[32991]: pgmap v409: 129 pgs: 129 active+clean; 385 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 1.8 MiB/s wr, 157 op/s 2024-04-26T21:43:24.617 INFO:tasks.workunit.client.0.smithi001.stdout: 1 498472 4.18 MB/sec execute 562 sec latency 60.295 ms 2024-04-26T21:43:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.786+0000 7f17c1400640 1 -- 172.21.15.1:0/1253791069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103910 msgr2=0x7f17bc103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.786+0000 7f17c1400640 1 --2- 172.21.15.1:0/1253791069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103910 0x7f17bc103cf0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f17ac009900 tx=0x7f17ac02a480 comp rx=0 tx=0).stop 2024-04-26T21:43:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.787+0000 7f17c1400640 1 -- 172.21.15.1:0/1253791069 shutdown_connections 2024-04-26T21:43:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.787+0000 7f17c1400640 1 --2- 172.21.15.1:0/1253791069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc104230 0x7f17bc10c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.787+0000 7f17c1400640 1 --2- 172.21.15.1:0/1253791069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc10cd30 0x7f17bc10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.787+0000 7f17c1400640 1 --2- 172.21.15.1:0/1253791069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc103910 0x7f17bc103cf0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.787+0000 7f17c1400640 1 -- 172.21.15.1:0/1253791069 >> 172.21.15.1:0/1253791069 conn(0x7f17bc0fd110 msgr2=0x7f17bc0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.787+0000 7f17c1400640 1 -- 172.21.15.1:0/1253791069 shutdown_connections 2024-04-26T21:43:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.787+0000 7f17c1400640 1 -- 172.21.15.1:0/1253791069 wait complete. 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.788+0000 7f17c1400640 1 Processor -- start 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.788+0000 7f17c1400640 1 -- start start 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.788+0000 7f17c1400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc103910 0x7f17bc10c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.788+0000 7f17c1400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc104230 0x7f17bc108360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.788+0000 7f17c1400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc10cd30 0x7f17bc1088a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.788+0000 7f17c1400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc06c920 con 0x7f17bc104230 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.788+0000 7f17c1400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc06c7a0 con 0x7f17bc10cd30 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.788+0000 7f17c1400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc06caa0 con 0x7f17bc103910 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17c0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc10cd30 0x7f17bc1088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17bbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc103910 0x7f17bc10c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17c0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc10cd30 0x7f17bc1088a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51844/0 (socket says 172.21.15.1:51844) 2024-04-26T21:43:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17bbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc103910 0x7f17bc10c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37598/0 (socket says 172.21.15.1:37598) 2024-04-26T21:43:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17c0a00640 1 -- 172.21.15.1:0/313080555 learned_addr learned my addr 172.21.15.1:0/313080555 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:24.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17bb400640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc104230 0x7f17bc108360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17c0a00640 1 -- 172.21.15.1:0/313080555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc103910 msgr2=0x7f17bc10c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17c0a00640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc103910 0x7f17bc10c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17c0a00640 1 -- 172.21.15.1:0/313080555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc104230 msgr2=0x7f17bc108360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17c0a00640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc104230 0x7f17bc108360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17c0a00640 1 -- 172.21.15.1:0/313080555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17bc108e60 con 0x7f17bc10cd30 2024-04-26T21:43:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.789+0000 7f17c0a00640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc10cd30 0x7f17bc1088a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f17b400efd0 tx=0x7f17b400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.790+0000 7f17b8c00640 1 -- 172.21.15.1:0/313080555 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4003b70 con 0x7f17bc10cd30 2024-04-26T21:43:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.790+0000 7f17b8c00640 1 -- 172.21.15.1:0/313080555 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17b4003d10 con 0x7f17bc10cd30 2024-04-26T21:43:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.790+0000 7f17b8c00640 1 -- 172.21.15.1:0/313080555 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4010990 con 0x7f17bc10cd30 2024-04-26T21:43:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.790+0000 7f17c1400640 1 -- 172.21.15.1:0/313080555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17bc109150 con 0x7f17bc10cd30 2024-04-26T21:43:24.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.790+0000 7f17c1400640 1 -- 172.21.15.1:0/313080555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17bc1b0520 con 0x7f17bc10cd30 2024-04-26T21:43:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.794+0000 7f17c577e740 1 Processor -- start 2024-04-26T21:43:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.794+0000 7f17b8c00640 1 -- 172.21.15.1:0/313080555 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f17b4003ec0 con 0x7f17bc10cd30 2024-04-26T21:43:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.794+0000 7f17c577e740 1 -- start start 2024-04-26T21:43:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17c577e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 0x562e4427d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17c577e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44287c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17c577e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e442881c0 0x562e4428a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17c577e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562e441e2380 con 0x562e4427cfe0 2024-04-26T21:43:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17c577e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562e4417ec30 con 0x562e4427d900 2024-04-26T21:43:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17c577e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562e441de340 con 0x562e442881c0 2024-04-26T21:43:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44287c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17bbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44287c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51846/0 (socket says 172.21.15.1:51846) 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17bbe00640 1 -- 172.21.15.1:0/2273637508 learned_addr learned my addr 172.21.15.1:0/2273637508 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17bb400640 1 --2- 172.21.15.1:0/2273637508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 0x562e4427d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17bb400640 1 --2- 172.21.15.1:0/2273637508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e442881c0 0x562e4428a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17bbe00640 1 -- 172.21.15.1:0/2273637508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e442881c0 msgr2=0x562e4428a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17bbe00640 1 --2- 172.21.15.1:0/2273637508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e442881c0 0x562e4428a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17bbe00640 1 -- 172.21.15.1:0/2273637508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 msgr2=0x562e4427d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17bbe00640 1 --2- 172.21.15.1:0/2273637508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 0x562e4427d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.795+0000 7f17bbe00640 1 -- 172.21.15.1:0/2273637508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e441a0940 con 0x562e4427d900 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.796+0000 7f17bbe00640 1 --2- 172.21.15.1:0/2273637508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44287c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f17ac000c00 tx=0x7f17ac004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6505d7ed315cbe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.796+0000 7f17bb400640 1 --2- 172.21.15.1:0/2273637508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e442881c0 0x562e4428a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:43:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.796+0000 7f17b8c00640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f17a00a3db0 0x7f17a00a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.796+0000 7f17b1200640 1 -- 172.21.15.1:0/2273637508 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac035040 con 0x562e4427d900 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.796+0000 7f17b1200640 1 -- 172.21.15.1:0/2273637508 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17ac0054f0 con 0x562e4427d900 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.796+0000 7f17b1200640 1 -- 172.21.15.1:0/2273637508 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ac0035f0 con 0x562e4427d900 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.796+0000 7f17b8c00640 1 -- 172.21.15.1:0/313080555 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9642+0+0 (secure 0 0 0) 0x7f17b40d0f30 con 0x7f17bc10cd30 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.797+0000 7f17c577e740 1 -- 172.21.15.1:0/2273637508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 msgr2=0x562e44287c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.797+0000 7f17c577e740 1 --2- 172.21.15.1:0/2273637508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44287c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f17ac000c00 tx=0x7f17ac004040 comp rx=0 tx=0).stop 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.797+0000 7f17bb400640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f17a00a3db0 0x7f17a00a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.797+0000 7f17c577e740 1 -- 172.21.15.1:0/2273637508 shutdown_connections 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.797+0000 7f17c577e740 1 --2- 172.21.15.1:0/2273637508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e442881c0 0x562e4428a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.797+0000 7f17c577e740 1 --2- 172.21.15.1:0/2273637508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44287c80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.797+0000 7f17c577e740 1 --2- 172.21.15.1:0/2273637508 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 0x562e4427d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.797+0000 7f17c577e740 1 -- 172.21.15.1:0/2273637508 >> 172.21.15.1:0/2273637508 conn(0x562e442794f0 msgr2=0x562e442798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.798+0000 7f17c577e740 1 -- 172.21.15.1:0/2273637508 shutdown_connections 2024-04-26T21:43:24.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.798+0000 7f17c577e740 1 -- 172.21.15.1:0/2273637508 wait complete. 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.798+0000 7f17c577e740 1 -- start start 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.798+0000 7f17bb400640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f17a00a3db0 0x7f17a00a6270 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f17a80095f0 tx=0x7f17a80096c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17c577e740 1 Processor -- start 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17c577e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427cfe0 0x562e442850d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17c577e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44285610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17c577e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e442881c0 0x562e44285b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17c577e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562e4428ad90 con 0x562e442881c0 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17c577e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562e44169c60 con 0x562e4427d900 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17c577e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562e4428af10 con 0x562e4427cfe0 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17c577e740 10 client.0 _ll_register_callbacks cb 0x562e44195e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17bb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44285610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17bb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44285610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51860/0 (socket says 172.21.15.1:51860) 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17bb400640 1 -- 172.21.15.1:0/3120234908 learned_addr learned my addr 172.21.15.1:0/3120234908 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.799+0000 7f17bbe00640 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427cfe0 0x562e442850d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17c0a00640 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e442881c0 0x562e44285b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17bb400640 1 -- 172.21.15.1:0/3120234908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427cfe0 msgr2=0x562e442850d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17bb400640 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427cfe0 0x562e442850d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17bb400640 1 -- 172.21.15.1:0/3120234908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e442881c0 msgr2=0x562e44285b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17bb400640 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e442881c0 0x562e44285b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17bb400640 1 -- 172.21.15.1:0/3120234908 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e44178510 con 0x562e4427d900 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17c0a00640 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e442881c0 0x562e44285b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17bb400640 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44285610 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f17a8004590 tx=0x7f17a8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf52dd51d53854db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17a5600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17a5600640 1 -- 172.21.15.1:0/3120234908 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a802a470 con 0x562e4427d900 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.800+0000 7f17a5600640 1 -- 172.21.15.1:0/3120234908 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9642+0+0 (secure 0 0 0) 0x7f17a8021430 con 0x562e4427d900 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17c577e740 1 -- 172.21.15.1:0/3120234908 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17ac0054f0 con 0x562e4427d900 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17a5600640 1 -- 172.21.15.1:0/3120234908 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f17a8021630 con 0x562e4427d900 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17c577e740 10 client.26567 fetch_fsmap learned FSMap version 41 2024-04-26T21:43:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17c577e740 1 -- 172.21.15.1:0/3120234908 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x562e43ef4210 con 0x562e4427d900 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17a5600640 1 -- 172.21.15.1:0/3120234908 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17ac0054f0 con 0x562e4427d900 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17a5600640 1 -- 172.21.15.1:0/3120234908 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f17a802ab60 con 0x562e4427d900 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17c577e740 10 client.26567 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17c577e740 10 client.26567 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17c577e740 20 client.26567 populate_metadata read hostname 'smithi001' 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17c577e740 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562e44297ef0 0x562e4429a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17c577e740 4 client.26567 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.801+0000 7f17c577e740 1 -- 172.21.15.1:0/3120234908 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e441eff90 con 0x562e44297ef0 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.802+0000 7f17bbe00640 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562e44297ef0 0x562e4429a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.802+0000 7f17bbe00640 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562e44297ef0 0x562e4429a340 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b67f42a7e00ae53 server_cookie=978afbd06d0ae2c2 in_seq=0 out_seq=0 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.802+0000 7f17a5600640 10 client.26567 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17a5600640 1 -- 172.21.15.1:0/3120234908 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e441eff90 con 0x562e44297ef0 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17a5600640 10 client.26567 handle_command_reply: tid=0 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17c577e740 1 client.26567 shutdown 2024-04-26T21:43:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17c577e740 1 -- 172.21.15.1:0/3120234908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 msgr2=0x562e44285610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17c577e740 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44285610 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f17a8004590 tx=0x7f17a8004730 comp rx=0 tx=0).stop 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17c577e740 1 -- 172.21.15.1:0/3120234908 shutdown_connections 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17c577e740 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562e44297ef0 0x562e4429a340 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17c577e740 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e442881c0 0x562e44285b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17c577e740 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e4427d900 0x562e44285610 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17a5600640 0 client.26567 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17c577e740 1 --2- 172.21.15.1:0/3120234908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427cfe0 0x562e442850d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.803+0000 7f17c577e740 1 -- 172.21.15.1:0/3120234908 >> 172.21.15.1:0/3120234908 conn(0x562e442794f0 msgr2=0x562e4427f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.804+0000 7f17c577e740 1 -- 172.21.15.1:0/3120234908 shutdown_connections 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.804+0000 7f17c577e740 1 -- 172.21.15.1:0/3120234908 wait complete. 2024-04-26T21:43:24.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.804+0000 7f17c577e740 20 client.26567 trim_cache size 0 max 16384 2024-04-26T21:43:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.824+0000 7f17c577e740 1 Processor -- start 2024-04-26T21:43:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.824+0000 7f17c577e740 1 -- start start 2024-04-26T21:43:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.824+0000 7f17c577e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 0x562e44293c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.824+0000 7f17c577e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427d900 0x562e442840b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.824+0000 7f17c577e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e4429fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.824+0000 7f17c577e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc1bd1d0 con 0x562e4427cfe0 2024-04-26T21:43:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.824+0000 7f17c577e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc1baa10 con 0x562e442881c0 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.824+0000 7f17c577e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc1b8880 con 0x562e4427d900 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427d900 0x562e442840b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17c0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e4429fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427d900 0x562e442840b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37620/0 (socket says 172.21.15.1:37620) 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bb400640 1 -- 172.21.15.1:0/3161781450 learned_addr learned my addr 172.21.15.1:0/3161781450 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17c0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e4429fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51862/0 (socket says 172.21.15.1:51862) 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bbe00640 1 --2- 172.21.15.1:0/3161781450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 0x562e44293c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bb400640 1 -- 172.21.15.1:0/3161781450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 msgr2=0x562e4429fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bb400640 1 --2- 172.21.15.1:0/3161781450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e4429fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bb400640 1 -- 172.21.15.1:0/3161781450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 msgr2=0x562e44293c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bb400640 1 --2- 172.21.15.1:0/3161781450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 0x562e44293c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bb400640 1 -- 172.21.15.1:0/3161781450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e4427d730 con 0x562e4427d900 2024-04-26T21:43:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bbe00640 1 --2- 172.21.15.1:0/3161781450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 0x562e44293c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:43:24.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17c0a00640 1 --2- 172.21.15.1:0/3161781450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e4429fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.825+0000 7f17bb400640 1 --2- 172.21.15.1:0/3161781450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427d900 0x562e442840b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f17a801a040 tx=0x7f17a8009b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaa83c9b6ba8da49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.826+0000 7f17a7e00640 1 -- 172.21.15.1:0/3161781450 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a8005040 con 0x562e4427d900 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.826+0000 7f17a7e00640 1 -- 172.21.15.1:0/3161781450 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17a803c070 con 0x562e4427d900 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.826+0000 7f17a7e00640 1 -- 172.21.15.1:0/3161781450 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a80047a0 con 0x562e4427d900 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.826+0000 7f17c577e740 1 -- 172.21.15.1:0/3161781450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427d900 msgr2=0x562e442840b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.826+0000 7f17c577e740 1 --2- 172.21.15.1:0/3161781450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427d900 0x562e442840b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f17a801a040 tx=0x7f17a8009b10 comp rx=0 tx=0).stop 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.827+0000 7f17c577e740 1 -- 172.21.15.1:0/3161781450 shutdown_connections 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.827+0000 7f17c577e740 1 --2- 172.21.15.1:0/3161781450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427d900 0x562e442840b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.827+0000 7f17c577e740 1 --2- 172.21.15.1:0/3161781450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e4429fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.827+0000 7f17c577e740 1 --2- 172.21.15.1:0/3161781450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427cfe0 0x562e44293c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.827+0000 7f17c577e740 1 -- 172.21.15.1:0/3161781450 >> 172.21.15.1:0/3161781450 conn(0x562e442794f0 msgr2=0x562e442839a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.827+0000 7f17c577e740 1 -- 172.21.15.1:0/3161781450 shutdown_connections 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.827+0000 7f17c577e740 1 -- 172.21.15.1:0/3161781450 wait complete. 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.827+0000 7f17c577e740 1 -- start start 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.827+0000 7f17c577e740 1 Processor -- start 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17c577e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427cfe0 0x562e44290160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17c577e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427d900 0x562e442906a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17c577e740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e44281900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17c577e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc1b8ed0 con 0x562e4427d900 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17c577e740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc1ba340 con 0x562e442881c0 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17c577e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17bc1b95a0 con 0x562e4427cfe0 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17c577e740 10 client.0 _ll_register_callbacks cb 0x7f17a8009960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:43:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17c0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e44281900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17bb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427d900 0x562e442906a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17c0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e44281900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51872/0 (socket says 172.21.15.1:51872) 2024-04-26T21:43:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17c0a00640 1 -- 172.21.15.1:0/365642331 learned_addr learned my addr 172.21.15.1:0/365642331 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.828+0000 7f17bbe00640 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427cfe0 0x562e44290160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.829+0000 7f17c0a00640 1 -- 172.21.15.1:0/365642331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427cfe0 msgr2=0x562e44290160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.829+0000 7f17c0a00640 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427cfe0 0x562e44290160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.829+0000 7f17c0a00640 1 -- 172.21.15.1:0/365642331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427d900 msgr2=0x562e442906a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.829+0000 7f17c0a00640 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427d900 0x562e442906a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.829+0000 7f17c0a00640 1 -- 172.21.15.1:0/365642331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e43fc0620 con 0x562e442881c0 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.829+0000 7f17c0a00640 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e44281900 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f17b4098e70 tx=0x7f17b4009960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56203ce3e47ba4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.829+0000 7f17a5600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.829+0000 7f17a5600640 1 -- 172.21.15.1:0/365642331 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b40a0dc0 con 0x562e442881c0 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.829+0000 7f17c577e740 1 -- 172.21.15.1:0/365642331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17a803c070 con 0x562e442881c0 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.829+0000 7f17a5600640 1 -- 172.21.15.1:0/365642331 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9642+0+0 (secure 0 0 0) 0x7f17b40041b0 con 0x562e442881c0 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17a5600640 1 -- 172.21.15.1:0/365642331 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f17b40a5680 con 0x562e442881c0 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17c577e740 10 client.26573 fetch_fsmap learned FSMap version 41 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17c577e740 1 -- 172.21.15.1:0/365642331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x562e4419ccb0 con 0x562e442881c0 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17a5600640 1 -- 172.21.15.1:0/365642331 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17a803c070 con 0x562e442881c0 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17a5600640 1 -- 172.21.15.1:0/365642331 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f17b401f020 con 0x562e442881c0 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17c577e740 10 client.26573 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17c577e740 10 client.26573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:43:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17c577e740 20 client.26573 populate_metadata read hostname 'smithi001' 2024-04-26T21:43:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17c577e740 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562e442fe340 0x562e44300760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17c577e740 4 client.26573 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:43:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.830+0000 7f17c577e740 1 -- 172.21.15.1:0/365642331 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e441e9e00 con 0x562e442fe340 2024-04-26T21:43:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.831+0000 7f17bbe00640 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562e442fe340 0x562e44300760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.831+0000 7f17bbe00640 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562e442fe340 0x562e44300760 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ec0247e16385dcc server_cookie=96c91f7901406a30 in_seq=0 out_seq=0 2024-04-26T21:43:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.831+0000 7f17a5600640 10 client.26573 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.918+0000 7f17a5600640 1 -- 172.21.15.1:0/365642331 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e441e9e00 con 0x562e442fe340 2024-04-26T21:43:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.918+0000 7f17a5600640 10 client.26573 handle_command_reply: tid=0 2024-04-26T21:43:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.918+0000 7f17c577e740 1 client.26573 shutdown 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.918+0000 7f17c577e740 1 -- 172.21.15.1:0/365642331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 msgr2=0x562e44281900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.918+0000 7f17c577e740 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e44281900 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f17b4098e70 tx=0x7f17b4009960 comp rx=0 tx=0).stop 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.919+0000 7f17c577e740 1 -- 172.21.15.1:0/365642331 shutdown_connections 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.919+0000 7f17c577e740 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562e442fe340 0x562e44300760 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.919+0000 7f17c577e740 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e442881c0 0x562e44281900 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.919+0000 7f17c577e740 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562e4427d900 0x562e442906a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.919+0000 7f17a5600640 0 client.26573 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.919+0000 7f17c577e740 1 --2- 172.21.15.1:0/365642331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e4427cfe0 0x562e44290160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.919+0000 7f17c577e740 1 -- 172.21.15.1:0/365642331 >> 172.21.15.1:0/365642331 conn(0x562e442794f0 msgr2=0x562e4427f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.919+0000 7f17c577e740 1 -- 172.21.15.1:0/365642331 shutdown_connections 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.919+0000 7f17c577e740 1 -- 172.21.15.1:0/365642331 wait complete. 2024-04-26T21:43:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.919+0000 7f17c577e740 20 client.26573 trim_cache size 0 max 16384 2024-04-26T21:43:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:43:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:43:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:43:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:43:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 -- 172.21.15.1:0/313080555 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f17a00a3db0 msgr2=0x7f17a00a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f17a00a3db0 0x7f17a00a6270 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f17a80095f0 tx=0x7f17a80096c0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 -- 172.21.15.1:0/313080555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc10cd30 msgr2=0x7f17bc1088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc10cd30 0x7f17bc1088a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f17b400efd0 tx=0x7f17b400c6a0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 -- 172.21.15.1:0/313080555 shutdown_connections 2024-04-26T21:43:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f17a00a3db0 0x7f17a00a6270 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17bc10cd30 0x7f17bc1088a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17bc104230 0x7f17bc108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 --2- 172.21.15.1:0/313080555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17bc103910 0x7f17bc10c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 -- 172.21.15.1:0/313080555 >> 172.21.15.1:0/313080555 conn(0x7f17bc0fd110 msgr2=0x7f17bc1015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 -- 172.21.15.1:0/313080555 shutdown_connections 2024-04-26T21:43:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:24.920+0000 7f17c1400640 1 -- 172.21.15.1:0/313080555 wait complete. 2024-04-26T21:43:25.617 INFO:tasks.workunit.client.0.smithi001.stdout: 1 499730 4.18 MB/sec execute 563 sec latency 80.102 ms 2024-04-26T21:43:26.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:26 smithi038 ceph-mon[32724]: pgmap v410: 129 pgs: 129 active+clean; 394 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 148 op/s 2024-04-26T21:43:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:26 smithi186 ceph-mon[32991]: pgmap v410: 129 pgs: 129 active+clean; 394 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 148 op/s 2024-04-26T21:43:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:26 smithi001 ceph-mon[28054]: pgmap v410: 129 pgs: 129 active+clean; 394 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 148 op/s 2024-04-26T21:43:26.617 INFO:tasks.workunit.client.0.smithi001.stdout: 1 500382 4.18 MB/sec execute 564 sec latency 45.034 ms 2024-04-26T21:43:27.618 INFO:tasks.workunit.client.0.smithi001.stdout: 1 501194 4.19 MB/sec execute 565 sec latency 30.853 ms 2024-04-26T21:43:28.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:28 smithi038 ceph-mon[32724]: pgmap v411: 129 pgs: 129 active+clean; 394 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 148 op/s 2024-04-26T21:43:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:28 smithi186 ceph-mon[32991]: pgmap v411: 129 pgs: 129 active+clean; 394 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 148 op/s 2024-04-26T21:43:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:28 smithi001 ceph-mon[28054]: pgmap v411: 129 pgs: 129 active+clean; 394 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 148 op/s 2024-04-26T21:43:28.618 INFO:tasks.workunit.client.0.smithi001.stdout: 1 501478 4.18 MB/sec execute 566 sec latency 27.917 ms 2024-04-26T21:43:29.618 INFO:tasks.workunit.client.0.smithi001.stdout: 1 502121 4.17 MB/sec execute 567 sec latency 26.133 ms 2024-04-26T21:43:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:30 smithi186 ceph-mon[32991]: pgmap v412: 129 pgs: 129 active+clean; 394 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 148 op/s 2024-04-26T21:43:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:30 smithi001 ceph-mon[28054]: pgmap v412: 129 pgs: 129 active+clean; 394 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 148 op/s 2024-04-26T21:43:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:30 smithi038 ceph-mon[32724]: pgmap v412: 129 pgs: 129 active+clean; 394 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 148 op/s 2024-04-26T21:43:30.618 INFO:tasks.workunit.client.0.smithi001.stdout: 1 503311 4.18 MB/sec execute 568 sec latency 49.371 ms 2024-04-26T21:43:31.618 INFO:tasks.workunit.client.0.smithi001.stdout: 1 504136 4.18 MB/sec execute 569 sec latency 26.638 ms 2024-04-26T21:43:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:32 smithi186 ceph-mon[32991]: pgmap v413: 129 pgs: 129 active+clean; 395 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 199 op/s 2024-04-26T21:43:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:32 smithi001 ceph-mon[28054]: pgmap v413: 129 pgs: 129 active+clean; 395 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 199 op/s 2024-04-26T21:43:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:32 smithi038 ceph-mon[32724]: pgmap v413: 129 pgs: 129 active+clean; 395 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 199 op/s 2024-04-26T21:43:32.618 INFO:tasks.workunit.client.0.smithi001.stdout: 1 504796 4.18 MB/sec execute 570 sec latency 34.905 ms 2024-04-26T21:43:33.618 INFO:tasks.workunit.client.0.smithi001.stdout: 1 505173 4.18 MB/sec execute 571 sec latency 27.970 ms 2024-04-26T21:43:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:34 smithi186 ceph-mon[32991]: pgmap v414: 129 pgs: 129 active+clean; 395 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-04-26T21:43:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:34 smithi001 ceph-mon[28054]: pgmap v414: 129 pgs: 129 active+clean; 395 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-04-26T21:43:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:34 smithi038 ceph-mon[32724]: pgmap v414: 129 pgs: 129 active+clean; 395 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 1.6 MiB/s wr, 142 op/s 2024-04-26T21:43:34.618 INFO:tasks.workunit.client.0.smithi001.stdout: 1 505926 4.18 MB/sec execute 572 sec latency 21.688 ms 2024-04-26T21:43:35.619 INFO:tasks.workunit.client.0.smithi001.stdout: 1 506969 4.17 MB/sec execute 573 sec latency 28.559 ms 2024-04-26T21:43:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:36 smithi186 ceph-mon[32991]: pgmap v415: 129 pgs: 129 active+clean; 400 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 191 op/s 2024-04-26T21:43:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:36 smithi001 ceph-mon[28054]: pgmap v415: 129 pgs: 129 active+clean; 400 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 191 op/s 2024-04-26T21:43:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:36 smithi038 ceph-mon[32724]: pgmap v415: 129 pgs: 129 active+clean; 400 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 191 op/s 2024-04-26T21:43:36.619 INFO:tasks.workunit.client.0.smithi001.stdout: 1 508111 4.18 MB/sec execute 574 sec latency 31.554 ms 2024-04-26T21:43:37.619 INFO:tasks.workunit.client.0.smithi001.stdout: 1 508489 4.18 MB/sec execute 575 sec latency 29.567 ms 2024-04-26T21:43:38.075 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.075+0000 7f5750000640 1 -- 172.21.15.1:0/4007632298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5748104b50 msgr2=0x7f5748104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.075+0000 7f5750000640 1 --2- 172.21.15.1:0/4007632298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5748104b50 0x7f5748104fd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5734009890 tx=0x7f573402a360 comp rx=0 tx=0).stop 2024-04-26T21:43:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.075+0000 7f5750000640 1 -- 172.21.15.1:0/4007632298 shutdown_connections 2024-04-26T21:43:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.075+0000 7f5750000640 1 --2- 172.21.15.1:0/4007632298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5748104b50 0x7f5748104fd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.075+0000 7f5750000640 1 --2- 172.21.15.1:0/4007632298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748105510 0x7f57481119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.075+0000 7f5750000640 1 --2- 172.21.15.1:0/4007632298 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f574810ac10 0x7f574810aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.075+0000 7f5750000640 1 -- 172.21.15.1:0/4007632298 >> 172.21.15.1:0/4007632298 conn(0x7f5748100620 msgr2=0x7f5748102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.075+0000 7f5750000640 1 -- 172.21.15.1:0/4007632298 shutdown_connections 2024-04-26T21:43:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.075+0000 7f5750000640 1 -- 172.21.15.1:0/4007632298 wait complete. 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.076+0000 7f5750000640 1 Processor -- start 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f5750000640 1 -- start start 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f5750000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748104b50 0x7f574819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f5750000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5748105510 0x7f574819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f5750000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f574810ac10 0x7f574819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f5750000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5748114e00 con 0x7f574810ac10 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f5750000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5748114c80 con 0x7f5748104b50 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f5750000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5748114f80 con 0x7f5748105510 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f574d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748104b50 0x7f574819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f574cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5748105510 0x7f574819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f574d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748104b50 0x7f574819b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54638/0 (socket says 172.21.15.1:54638) 2024-04-26T21:43:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f574cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5748105510 0x7f574819b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51158/0 (socket says 172.21.15.1:51158) 2024-04-26T21:43:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.077+0000 7f574d600640 1 -- 172.21.15.1:0/458397983 learned_addr learned my addr 172.21.15.1:0/458397983 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.078+0000 7f574e000640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f574810ac10 0x7f574819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.078+0000 7f574d600640 1 -- 172.21.15.1:0/458397983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5748105510 msgr2=0x7f574819b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.078+0000 7f574d600640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5748105510 0x7f574819b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.078+0000 7f574d600640 1 -- 172.21.15.1:0/458397983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f574810ac10 msgr2=0x7f574819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.078+0000 7f574d600640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f574810ac10 0x7f574819fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.078+0000 7f574d600640 1 -- 172.21.15.1:0/458397983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57481a02b0 con 0x7f5748104b50 2024-04-26T21:43:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.079+0000 7f574d600640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748104b50 0x7f574819b2c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e9f0 tx=0x7f573c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.079+0000 7f5746000640 1 -- 172.21.15.1:0/458397983 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c00cd80 con 0x7f5748104b50 2024-04-26T21:43:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.079+0000 7f5746000640 1 -- 172.21.15.1:0/458397983 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f573c0108e0 con 0x7f5748104b50 2024-04-26T21:43:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.079+0000 7f5750000640 1 -- 172.21.15.1:0/458397983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57481a05a0 con 0x7f5748104b50 2024-04-26T21:43:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.079+0000 7f5746000640 1 -- 172.21.15.1:0/458397983 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c018980 con 0x7f5748104b50 2024-04-26T21:43:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.079+0000 7f5750000640 1 -- 172.21.15.1:0/458397983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57481a7e30 con 0x7f5748104b50 2024-04-26T21:43:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.084+0000 7f5752ddc740 1 Processor -- start 2024-04-26T21:43:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.084+0000 7f5752ddc740 1 -- start start 2024-04-26T21:43:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.084+0000 7f5746000640 1 -- 172.21.15.1:0/458397983 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f573c010430 con 0x7f5748104b50 2024-04-26T21:43:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b02dd0 0x559d70b031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 0x559d70b0da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 0x559d70b103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f5752ddc740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x559d70a68380 con 0x559d70b02dd0 2024-04-26T21:43:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f5752ddc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559d70a04c30 con 0x559d70b0dfb0 2024-04-26T21:43:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f5752ddc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559d70a64340 con 0x559d70b036f0 2024-04-26T21:43:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f574cc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 0x559d70b103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f574cc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 0x559d70b103a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54650/0 (socket says 172.21.15.1:54650) 2024-04-26T21:43:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f574cc00640 1 -- 172.21.15.1:0/3096248813 learned_addr learned my addr 172.21.15.1:0/3096248813 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f574d600640 1 --2- 172.21.15.1:0/3096248813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 0x559d70b0da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f574e000640 1 --2- 172.21.15.1:0/3096248813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b02dd0 0x559d70b031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.085+0000 7f574cc00640 1 -- 172.21.15.1:0/3096248813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 msgr2=0x559d70b0da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.086+0000 7f574cc00640 1 --2- 172.21.15.1:0/3096248813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 0x559d70b0da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.086+0000 7f574cc00640 1 -- 172.21.15.1:0/3096248813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b02dd0 msgr2=0x559d70b031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.086+0000 7f574cc00640 1 --2- 172.21.15.1:0/3096248813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b02dd0 0x559d70b031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.086+0000 7f574cc00640 1 -- 172.21.15.1:0/3096248813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d70a26940 con 0x559d70b0dfb0 2024-04-26T21:43:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.086+0000 7f574cc00640 1 --2- 172.21.15.1:0/3096248813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 0x559d70b103a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f573402a880 tx=0x7f5734005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f864b8c3d7ad8595 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.087+0000 7f573a600640 1 -- 172.21.15.1:0/3096248813 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5734041020 con 0x559d70b0dfb0 2024-04-26T21:43:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.087+0000 7f573a600640 1 -- 172.21.15.1:0/3096248813 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5734004040 con 0x559d70b0dfb0 2024-04-26T21:43:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.087+0000 7f573a600640 1 -- 172.21.15.1:0/3096248813 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5734037040 con 0x559d70b0dfb0 2024-04-26T21:43:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.087+0000 7f5746000640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f57300a3d90 0x7f57300a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.087+0000 7f5746000640 1 -- 172.21.15.1:0/458397983 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9642+0+0 (secure 0 0 0) 0x7f573c0d0ed0 con 0x7f5748104b50 2024-04-26T21:43:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.087+0000 7f574e000640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f57300a3d90 0x7f57300a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.088+0000 7f5752ddc740 1 -- 172.21.15.1:0/3096248813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 msgr2=0x559d70b103a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.088+0000 7f5752ddc740 1 --2- 172.21.15.1:0/3096248813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 0x559d70b103a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f573402a880 tx=0x7f5734005b20 comp rx=0 tx=0).stop 2024-04-26T21:43:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.088+0000 7f5752ddc740 1 -- 172.21.15.1:0/3096248813 shutdown_connections 2024-04-26T21:43:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.088+0000 7f5752ddc740 1 --2- 172.21.15.1:0/3096248813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 0x559d70b0da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.088+0000 7f5752ddc740 1 --2- 172.21.15.1:0/3096248813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 0x559d70b103a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.088+0000 7f5752ddc740 1 --2- 172.21.15.1:0/3096248813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b02dd0 0x559d70b031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.088+0000 7f5752ddc740 1 -- 172.21.15.1:0/3096248813 >> 172.21.15.1:0/3096248813 conn(0x559d70aff2e0 msgr2=0x559d70aff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.088+0000 7f5752ddc740 1 -- 172.21.15.1:0/3096248813 shutdown_connections 2024-04-26T21:43:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.089+0000 7f574e000640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f57300a3d90 0x7f57300a6250 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5740005fd0 tx=0x7f5740010040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.089+0000 7f5752ddc740 1 -- 172.21.15.1:0/3096248813 wait complete. 2024-04-26T21:43:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.089+0000 7f5752ddc740 1 -- start start 2024-04-26T21:43:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.090+0000 7f5752ddc740 1 Processor -- start 2024-04-26T21:43:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.090+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 0x559d70b0aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.090+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 0x559d70b0b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.090+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b0dfb0 0x559d70b0b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.090+0000 7f5752ddc740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x559d70b19390 con 0x559d70b036f0 2024-04-26T21:43:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.090+0000 7f5752ddc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x559d709efc60 con 0x559d70b02dd0 2024-04-26T21:43:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.090+0000 7f5752ddc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559d70b19510 con 0x559d70b0dfb0 2024-04-26T21:43:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.090+0000 7f5752ddc740 10 client.0 _ll_register_callbacks cb 0x559d70a1be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.091+0000 7f574e000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b0dfb0 0x559d70b0b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.091+0000 7f574cc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 0x559d70b0b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.091+0000 7f574e000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b0dfb0 0x559d70b0b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51180/0 (socket says 172.21.15.1:51180) 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.091+0000 7f574e000640 1 -- 172.21.15.1:0/3599264672 learned_addr learned my addr 172.21.15.1:0/3599264672 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.091+0000 7f574d600640 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 0x559d70b0aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.091+0000 7f574cc00640 1 -- 172.21.15.1:0/3599264672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b0dfb0 msgr2=0x559d70b0b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.092+0000 7f574cc00640 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b0dfb0 0x559d70b0b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.092+0000 7f574cc00640 1 -- 172.21.15.1:0/3599264672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 msgr2=0x559d70b0aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.092+0000 7f574cc00640 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 0x559d70b0aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.092+0000 7f574cc00640 1 -- 172.21.15.1:0/3599264672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d709fe510 con 0x559d70b036f0 2024-04-26T21:43:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.092+0000 7f574e000640 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b0dfb0 0x559d70b0b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:43:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.092+0000 7f574cc00640 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 0x559d70b0b3f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f573402a950 tx=0x7f57340303c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29284a1f8394391f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.092+0000 7f572ea00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.093+0000 7f572ea00640 1 -- 172.21.15.1:0/3599264672 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5734037050 con 0x559d70b036f0 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.093+0000 7f5752ddc740 1 -- 172.21.15.1:0/3599264672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5734004040 con 0x559d70b036f0 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.093+0000 7f572ea00640 1 -- 172.21.15.1:0/3599264672 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9642+0+0 (secure 0 0 0) 0x7f5734030b20 con 0x559d70b036f0 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.093+0000 7f572ea00640 1 -- 172.21.15.1:0/3599264672 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f5734059050 con 0x559d70b036f0 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.093+0000 7f5752ddc740 10 client.16380 fetch_fsmap learned FSMap version 41 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.093+0000 7f5752ddc740 1 -- 172.21.15.1:0/3599264672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x559d7077a210 con 0x559d70b036f0 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.094+0000 7f572ea00640 1 -- 172.21.15.1:0/3599264672 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5734004040 con 0x559d70b036f0 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.094+0000 7f572ea00640 1 -- 172.21.15.1:0/3599264672 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f573403d480 con 0x559d70b036f0 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.094+0000 7f5752ddc740 10 client.16380 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.094+0000 7f5752ddc740 10 client.16380 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:43:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.094+0000 7f5752ddc740 20 client.16380 populate_metadata read hostname 'smithi001' 2024-04-26T21:43:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.094+0000 7f5752ddc740 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559d70b262b0 0x559d70b286a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.094+0000 7f5752ddc740 4 client.16380 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:43:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.094+0000 7f5752ddc740 1 -- 172.21.15.1:0/3599264672 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d70a75f90 con 0x559d70b262b0 2024-04-26T21:43:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.095+0000 7f574d600640 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559d70b262b0 0x559d70b286a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.096+0000 7f574d600640 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559d70b262b0 0x559d70b286a0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94e8a55197dae258 server_cookie=d5920d48f9079da9 in_seq=0 out_seq=0 2024-04-26T21:43:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.096+0000 7f572ea00640 10 client.16380 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f572ea00640 1 -- 172.21.15.1:0/3599264672 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559d70a75f90 con 0x559d70b262b0 2024-04-26T21:43:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f572ea00640 10 client.16380 handle_command_reply: tid=0 2024-04-26T21:43:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f5752ddc740 1 client.16380 shutdown 2024-04-26T21:43:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f5752ddc740 1 -- 172.21.15.1:0/3599264672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 msgr2=0x559d70b0b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f5752ddc740 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 0x559d70b0b3f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f573402a950 tx=0x7f57340303c0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f5752ddc740 1 -- 172.21.15.1:0/3599264672 shutdown_connections 2024-04-26T21:43:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f5752ddc740 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559d70b262b0 0x559d70b286a0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f5752ddc740 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b0dfb0 0x559d70b0b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f5752ddc740 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 0x559d70b0b3f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f5752ddc740 1 --2- 172.21.15.1:0/3599264672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 0x559d70b0aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f5752ddc740 1 -- 172.21.15.1:0/3599264672 >> 172.21.15.1:0/3599264672 conn(0x559d70aff2e0 msgr2=0x559d70b05060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.097+0000 7f572ea00640 0 client.16380 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.098+0000 7f5752ddc740 1 -- 172.21.15.1:0/3599264672 shutdown_connections 2024-04-26T21:43:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.098+0000 7f5752ddc740 1 -- 172.21.15.1:0/3599264672 wait complete. 2024-04-26T21:43:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.098+0000 7f5752ddc740 20 client.16380 trim_cache size 0 max 16384 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.116+0000 7f5752ddc740 1 Processor -- start 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.116+0000 7f5752ddc740 1 -- start start 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.116+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 0x559d70b22250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.116+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 0x559d70b05790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.116+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b0dfb0 0x559d70b0a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.116+0000 7f5752ddc740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f57481b06b0 con 0x559d70b0dfb0 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.116+0000 7f5752ddc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f57481adef0 con 0x559d70b02dd0 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.116+0000 7f5752ddc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57481abd60 con 0x559d70b036f0 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 0x559d70b22250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574d600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 0x559d70b22250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54666/0 (socket says 172.21.15.1:54666) 2024-04-26T21:43:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574d600640 1 -- 172.21.15.1:0/229604685 learned_addr learned my addr 172.21.15.1:0/229604685 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574e000640 1 --2- 172.21.15.1:0/229604685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b0dfb0 0x559d70b0a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574cc00640 1 --2- 172.21.15.1:0/229604685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 0x559d70b05790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574d600640 1 -- 172.21.15.1:0/229604685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 msgr2=0x559d70b05790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574d600640 1 --2- 172.21.15.1:0/229604685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 0x559d70b05790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574d600640 1 -- 172.21.15.1:0/229604685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b0dfb0 msgr2=0x559d70b0a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574d600640 1 --2- 172.21.15.1:0/229604685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b0dfb0 0x559d70b0a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574d600640 1 -- 172.21.15.1:0/229604685 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d70846620 con 0x559d70b02dd0 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574cc00640 1 --2- 172.21.15.1:0/229604685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 0x559d70b05790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.117+0000 7f574d600640 1 --2- 172.21.15.1:0/229604685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 0x559d70b22250 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f573c09ca50 tx=0x7f573c0bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4c3ca3d111906f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.118+0000 7f5739c00640 1 -- 172.21.15.1:0/229604685 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c09cac0 con 0x559d70b02dd0 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.118+0000 7f5739c00640 1 -- 172.21.15.1:0/229604685 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f573c0a0dc0 con 0x559d70b02dd0 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.118+0000 7f5739c00640 1 -- 172.21.15.1:0/229604685 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c099130 con 0x559d70b02dd0 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.118+0000 7f5752ddc740 1 -- 172.21.15.1:0/229604685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 msgr2=0x559d70b22250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.118+0000 7f5752ddc740 1 --2- 172.21.15.1:0/229604685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 0x559d70b22250 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f573c09ca50 tx=0x7f573c0bd040 comp rx=0 tx=0).stop 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.119+0000 7f5752ddc740 1 -- 172.21.15.1:0/229604685 shutdown_connections 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.119+0000 7f5752ddc740 1 --2- 172.21.15.1:0/229604685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b036f0 0x559d70b05790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.119+0000 7f5752ddc740 1 --2- 172.21.15.1:0/229604685 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b0dfb0 0x559d70b0a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.119+0000 7f5752ddc740 1 --2- 172.21.15.1:0/229604685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b02dd0 0x559d70b22250 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.119+0000 7f5752ddc740 1 -- 172.21.15.1:0/229604685 >> 172.21.15.1:0/229604685 conn(0x559d70aff2e0 msgr2=0x559d70b0c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.119+0000 7f5752ddc740 1 -- 172.21.15.1:0/229604685 shutdown_connections 2024-04-26T21:43:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.119+0000 7f5752ddc740 1 -- 172.21.15.1:0/229604685 wait complete. 2024-04-26T21:43:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.119+0000 7f5752ddc740 1 -- start start 2024-04-26T21:43:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.119+0000 7f5752ddc740 1 Processor -- start 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b02dd0 0x559d70b1eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 0x559d70b07780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f5752ddc740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 0x559d70b07cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f5752ddc740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f57481ac3b0 con 0x559d70b036f0 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f5752ddc740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f57481ad820 con 0x559d70b0dfb0 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f5752ddc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57481aca80 con 0x559d70b02dd0 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f5752ddc740 10 client.0 _ll_register_callbacks cb 0x7f5740009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f574d600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b02dd0 0x559d70b1eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f574d600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b02dd0 0x559d70b1eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51190/0 (socket says 172.21.15.1:51190) 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f574d600640 1 -- 172.21.15.1:0/2277417461 learned_addr learned my addr 172.21.15.1:0/2277417461 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f574cc00640 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 0x559d70b07780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f574e000640 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 0x559d70b07cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f574d600640 1 -- 172.21.15.1:0/2277417461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 msgr2=0x559d70b07cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f574d600640 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 0x559d70b07cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f574d600640 1 -- 172.21.15.1:0/2277417461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 msgr2=0x559d70b07780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f574d600640 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 0x559d70b07780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.120+0000 7f574d600640 1 -- 172.21.15.1:0/2277417461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d70a22cb0 con 0x559d70b02dd0 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.121+0000 7f574d600640 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b02dd0 0x559d70b1eaa0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f573c00c6a0 tx=0x7f573c0ce1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13d3954c808fa39a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.121+0000 7f572ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.121+0000 7f572ea00640 1 -- 172.21.15.1:0/2277417461 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c09cac0 con 0x559d70b02dd0 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.121+0000 7f5752ddc740 1 -- 172.21.15.1:0/2277417461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f573c0a0dc0 con 0x559d70b02dd0 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.121+0000 7f572ea00640 1 -- 172.21.15.1:0/2277417461 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9642+0+0 (secure 0 0 0) 0x7f573c0bb8d0 con 0x559d70b02dd0 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.121+0000 7f572ea00640 1 -- 172.21.15.1:0/2277417461 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f573c0aa050 con 0x559d70b02dd0 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.121+0000 7f5752ddc740 10 client.26560 fetch_fsmap learned FSMap version 41 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.121+0000 7f5752ddc740 1 -- 172.21.15.1:0/2277417461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x559d70b8c380 con 0x559d70b02dd0 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f572ea00640 1 -- 172.21.15.1:0/2277417461 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f573c0a0dc0 con 0x559d70b02dd0 2024-04-26T21:43:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f572ea00640 1 -- 172.21.15.1:0/2277417461 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 41) ==== 8366+0+0 (secure 0 0 0) 0x7f573c0a51f0 con 0x559d70b02dd0 2024-04-26T21:43:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f5752ddc740 10 client.26560 fetch_fsmap finished waiting for FSMap version 41 2024-04-26T21:43:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f5752ddc740 10 client.26560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:43:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f5752ddc740 20 client.26560 populate_metadata read hostname 'smithi001' 2024-04-26T21:43:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f5752ddc740 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559d70b8c9f0 0x559d70b8ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f5752ddc740 4 client.26560 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:43:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f5752ddc740 1 -- 172.21.15.1:0/2277417461 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559d70a6fe00 con 0x559d70b8c9f0 2024-04-26T21:43:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f574cc00640 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559d70b8c9f0 0x559d70b8ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f574cc00640 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559d70b8c9f0 0x559d70b8ee10 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ef6217bc6497970 server_cookie=ddfb04e79e17213 in_seq=0 out_seq=0 2024-04-26T21:43:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.122+0000 7f572ea00640 10 client.26560 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.188+0000 7f572ea00640 1 -- 172.21.15.1:0/2277417461 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559d70a6fe00 con 0x559d70b8c9f0 2024-04-26T21:43:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f572ea00640 10 client.26560 handle_command_reply: tid=0 2024-04-26T21:43:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 client.26560 shutdown 2024-04-26T21:43:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 -- 172.21.15.1:0/2277417461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b02dd0 msgr2=0x559d70b1eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b02dd0 0x559d70b1eaa0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f573c00c6a0 tx=0x7f573c0ce1b0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 -- 172.21.15.1:0/2277417461 shutdown_connections 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x559d70b8c9f0 0x559d70b8ee10 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559d70b0dfb0 0x559d70b07cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x559d70b036f0 0x559d70b07780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 --2- 172.21.15.1:0/2277417461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559d70b02dd0 0x559d70b1eaa0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f572ea00640 0 client.26560 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 -- 172.21.15.1:0/2277417461 >> 172.21.15.1:0/2277417461 conn(0x559d70aff2e0 msgr2=0x559d70b0b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 -- 172.21.15.1:0/2277417461 shutdown_connections 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 1 -- 172.21.15.1:0/2277417461 wait complete. 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.189+0000 7f5752ddc740 20 client.26560 trim_cache size 0 max 16384 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:43:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:43:38.192 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:43:38.192 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:43:38.192 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 -- 172.21.15.1:0/458397983 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f57300a3d90 msgr2=0x7f57300a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f57300a3d90 0x7f57300a6250 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5740005fd0 tx=0x7f5740010040 comp rx=0 tx=0).stop 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 -- 172.21.15.1:0/458397983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748104b50 msgr2=0x7f574819b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748104b50 0x7f574819b2c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e9f0 tx=0x7f573c00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 -- 172.21.15.1:0/458397983 shutdown_connections 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f57300a3d90 0x7f57300a6250 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5748105510 0x7f574819b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f574810ac10 0x7f574819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 --2- 172.21.15.1:0/458397983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748104b50 0x7f574819b2c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 -- 172.21.15.1:0/458397983 >> 172.21.15.1:0/458397983 conn(0x7f5748100620 msgr2=0x7f5748102030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 -- 172.21.15.1:0/458397983 shutdown_connections 2024-04-26T21:43:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:38.190+0000 7f5750000640 1 -- 172.21.15.1:0/458397983 wait complete. 2024-04-26T21:43:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:38 smithi186 ceph-mon[32991]: pgmap v416: 129 pgs: 129 active+clean; 400 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-04-26T21:43:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:38 smithi001 ceph-mon[28054]: pgmap v416: 129 pgs: 129 active+clean; 400 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-04-26T21:43:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:38 smithi038 ceph-mon[32724]: pgmap v416: 129 pgs: 129 active+clean; 400 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-04-26T21:43:38.619 INFO:tasks.workunit.client.0.smithi001.stdout: 1 509197 4.18 MB/sec execute 576 sec latency 50.778 ms 2024-04-26T21:43:39.619 INFO:tasks.workunit.client.0.smithi001.stdout: 1 510370 4.18 MB/sec execute 577 sec latency 26.629 ms 2024-04-26T21:43:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:40 smithi186 ceph-mon[32991]: pgmap v417: 129 pgs: 129 active+clean; 400 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-04-26T21:43:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:40 smithi001 ceph-mon[28054]: pgmap v417: 129 pgs: 129 active+clean; 400 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-04-26T21:43:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:40 smithi038 ceph-mon[32724]: pgmap v417: 129 pgs: 129 active+clean; 400 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 1.2 MiB/s wr, 123 op/s 2024-04-26T21:43:40.619 INFO:tasks.workunit.client.0.smithi001.stdout: 1 511397 4.18 MB/sec execute 578 sec latency 23.529 ms 2024-04-26T21:43:41.619 INFO:tasks.workunit.client.0.smithi001.stdout: 1 511966 4.18 MB/sec execute 579 sec latency 27.997 ms 2024-04-26T21:43:42.370 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:42 smithi001 ceph-mon[28054]: pgmap v418: 129 pgs: 129 active+clean; 406 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-26T21:43:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:42 smithi186 ceph-mon[32991]: pgmap v418: 129 pgs: 129 active+clean; 406 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-26T21:43:42.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:42 smithi038 ceph-mon[32724]: pgmap v418: 129 pgs: 129 active+clean; 406 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 MiB/s wr, 189 op/s 2024-04-26T21:43:42.619 INFO:tasks.workunit.client.0.smithi001.stdout: 1 512630 4.18 MB/sec execute 580 sec latency 43.325 ms 2024-04-26T21:43:43.619 INFO:tasks.workunit.client.0.smithi001.stdout: 1 513643 4.18 MB/sec execute 581 sec latency 27.761 ms 2024-04-26T21:43:44.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:44 smithi186 ceph-mon[32991]: pgmap v419: 129 pgs: 129 active+clean; 406 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-04-26T21:43:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:44 smithi001 ceph-mon[28054]: pgmap v419: 129 pgs: 129 active+clean; 406 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-04-26T21:43:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:44 smithi038 ceph-mon[32724]: pgmap v419: 129 pgs: 129 active+clean; 406 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 1.2 MiB/s wr, 114 op/s 2024-04-26T21:43:44.620 INFO:tasks.workunit.client.0.smithi001.stdout: 1 514554 4.18 MB/sec execute 582 sec latency 29.164 ms 2024-04-26T21:43:45.620 INFO:tasks.workunit.client.0.smithi001.stdout: 1 515404 4.19 MB/sec execute 583 sec latency 35.127 ms 2024-04-26T21:43:46.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:46 smithi186 ceph-mon[32991]: pgmap v420: 129 pgs: 129 active+clean; 412 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-04-26T21:43:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:46 smithi001 ceph-mon[28054]: pgmap v420: 129 pgs: 129 active+clean; 412 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-04-26T21:43:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:46 smithi038 ceph-mon[32724]: pgmap v420: 129 pgs: 129 active+clean; 412 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-04-26T21:43:46.620 INFO:tasks.workunit.client.0.smithi001.stdout: 1 515859 4.18 MB/sec execute 584 sec latency 42.917 ms 2024-04-26T21:43:47.620 INFO:tasks.workunit.client.0.smithi001.stdout: 1 516682 4.18 MB/sec execute 585 sec latency 26.014 ms 2024-04-26T21:43:48.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:48 smithi186 ceph-mon[32991]: pgmap v421: 129 pgs: 129 active+clean; 412 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-04-26T21:43:48.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:48 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:43:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:48 smithi001 ceph-mon[28054]: pgmap v421: 129 pgs: 129 active+clean; 412 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-04-26T21:43:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:48 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:43:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:48 smithi038 ceph-mon[32724]: pgmap v421: 129 pgs: 129 active+clean; 412 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-04-26T21:43:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:48 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:43:48.620 INFO:tasks.workunit.client.0.smithi001.stdout: 1 517614 4.18 MB/sec execute 586 sec latency 28.221 ms 2024-04-26T21:43:49.620 INFO:tasks.workunit.client.0.smithi001.stdout: 1 518785 4.19 MB/sec execute 587 sec latency 30.473 ms 2024-04-26T21:43:50.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:50 smithi186 ceph-mon[32991]: pgmap v422: 129 pgs: 129 active+clean; 412 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-04-26T21:43:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:50 smithi001 ceph-mon[28054]: pgmap v422: 129 pgs: 129 active+clean; 412 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-04-26T21:43:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:50 smithi038 ceph-mon[32724]: pgmap v422: 129 pgs: 129 active+clean; 412 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.1 MiB/s wr, 117 op/s 2024-04-26T21:43:50.620 INFO:tasks.workunit.client.0.smithi001.stdout: 1 519083 4.18 MB/sec execute 588 sec latency 121.687 ms 2024-04-26T21:43:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.336+0000 7f51f0a00640 1 -- 172.21.15.1:0/3515789266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec103110 msgr2=0x7f51ec10f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.336+0000 7f51f0a00640 1 --2- 172.21.15.1:0/3515789266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec103110 0x7f51ec10f620 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f51e4009ca0 tx=0x7f51e402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.337+0000 7f51f0a00640 1 -- 172.21.15.1:0/3515789266 shutdown_connections 2024-04-26T21:43:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.337+0000 7f51f0a00640 1 --2- 172.21.15.1:0/3515789266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec103110 0x7f51ec10f620 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.337 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.337+0000 7f51f0a00640 1 --2- 172.21.15.1:0/3515789266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51ec102750 0x7f51ec102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.337+0000 7f51f0a00640 1 --2- 172.21.15.1:0/3515789266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec108760 0x7f51ec108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.337+0000 7f51f0a00640 1 -- 172.21.15.1:0/3515789266 >> 172.21.15.1:0/3515789266 conn(0x7f51ec0fe240 msgr2=0x7f51ec100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.337+0000 7f51f0a00640 1 -- 172.21.15.1:0/3515789266 shutdown_connections 2024-04-26T21:43:51.338 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.337+0000 7f51f0a00640 1 -- 172.21.15.1:0/3515789266 wait complete. 2024-04-26T21:43:51.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51f0a00640 1 Processor -- start 2024-04-26T21:43:51.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51f0a00640 1 -- start start 2024-04-26T21:43:51.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51f0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec102750 0x7f51ec19b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51f0a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51ec103110 0x7f51ec19b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.339 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51f0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec108760 0x7f51ec19f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51f0a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec112a70 con 0x7f51ec103110 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51f0a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec1128f0 con 0x7f51ec102750 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51f0a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec112bf0 con 0x7f51ec108760 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51eb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec102750 0x7f51ec19b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51eb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec102750 0x7f51ec19b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39872/0 (socket says 172.21.15.1:39872) 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.339+0000 7f51eb400640 1 -- 172.21.15.1:0/1524151937 learned_addr learned my addr 172.21.15.1:0/1524151937 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.338+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51ec103110 0x7f51ec19b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.339+0000 7f51ebe00640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec108760 0x7f51ec19f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.339+0000 7f51eb400640 1 -- 172.21.15.1:0/1524151937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec108760 msgr2=0x7f51ec19f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.339+0000 7f51eb400640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec108760 0x7f51ec19f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.340 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.339+0000 7f51eb400640 1 -- 172.21.15.1:0/1524151937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51ec103110 msgr2=0x7f51ec19b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.339+0000 7f51eb400640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51ec103110 0x7f51ec19b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.339+0000 7f51eb400640 1 -- 172.21.15.1:0/1524151937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec1a00a0 con 0x7f51ec102750 2024-04-26T21:43:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.339+0000 7f51eb400640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec102750 0x7f51ec19b0b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00d950 tx=0x7f51dc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.339+0000 7f51e3e00640 1 -- 172.21.15.1:0/1524151937 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc017030 con 0x7f51ec102750 2024-04-26T21:43:51.341 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.340+0000 7f51e3e00640 1 -- 172.21.15.1:0/1524151937 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51dc004020 con 0x7f51ec102750 2024-04-26T21:43:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.340+0000 7f51e3e00640 1 -- 172.21.15.1:0/1524151937 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0047a0 con 0x7f51ec102750 2024-04-26T21:43:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.340+0000 7f51f0a00640 1 -- 172.21.15.1:0/1524151937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ec1a0390 con 0x7f51ec102750 2024-04-26T21:43:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.340+0000 7f51f0a00640 1 -- 172.21.15.1:0/1524151937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ec1a7c20 con 0x7f51ec102750 2024-04-26T21:43:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.344+0000 7f51f4e40740 1 Processor -- start 2024-04-26T21:43:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.344+0000 7f51f4e40740 1 -- start start 2024-04-26T21:43:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.344+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.344+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 0x558bb50b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.344+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 0x558bb50b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.344+0000 7f51f4e40740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558bb500b380 con 0x558bb50a5fe0 2024-04-26T21:43:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.344+0000 7f51f4e40740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558bb4fa7c30 con 0x558bb50a6900 2024-04-26T21:43:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.344+0000 7f51f4e40740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558bb5007340 con 0x558bb50b11c0 2024-04-26T21:43:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 0x558bb50b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50a63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38056/0 (socket says 172.21.15.1:38056) 2024-04-26T21:43:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51ebe00640 1 -- 172.21.15.1:0/2070293760 learned_addr learned my addr 172.21.15.1:0/2070293760 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 0x558bb50b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42620/0 (socket says 172.21.15.1:42620) 2024-04-26T21:43:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51e3e00640 1 -- 172.21.15.1:0/1524151937 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f51dc009a30 con 0x7f51ec102750 2024-04-26T21:43:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51eaa00640 1 -- 172.21.15.1:0/2070293760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 msgr2=0x558bb50b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51eaa00640 1 --2- 172.21.15.1:0/2070293760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 0x558bb50b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51eaa00640 1 -- 172.21.15.1:0/2070293760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 msgr2=0x558bb50a63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:51.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51eaa00640 1 --2- 172.21.15.1:0/2070293760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50a63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51eaa00640 1 -- 172.21.15.1:0/2070293760 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bb4fc9940 con 0x558bb50b11c0 2024-04-26T21:43:51.347 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.345+0000 7f51eaa00640 1 --2- 172.21.15.1:0/2070293760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 0x558bb50b35b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f51d800bbc0 tx=0x7f51d80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=409642308f4f3c5e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:51.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.346+0000 7f51d7e00640 1 -- 172.21.15.1:0/2070293760 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8004030 con 0x558bb50b11c0 2024-04-26T21:43:51.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.346+0000 7f51d7e00640 1 -- 172.21.15.1:0/2070293760 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d80035f0 con 0x558bb50b11c0 2024-04-26T21:43:51.348 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.346+0000 7f51d7e00640 1 -- 172.21.15.1:0/2070293760 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d80116a0 con 0x558bb50b11c0 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.346+0000 7f51e3e00640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51d00a3d00 0x7f51d00a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51e3e00640 1 -- 172.21.15.1:0/1524151937 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 9642+0+0 (secure 0 0 0) 0x7f51dc0cfc40 con 0x7f51ec102750 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51ebe00640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51d00a3d00 0x7f51d00a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51f4e40740 1 -- 172.21.15.1:0/2070293760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 msgr2=0x558bb50b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51f4e40740 1 --2- 172.21.15.1:0/2070293760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 0x558bb50b35b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f51d800bbc0 tx=0x7f51d80107a0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51f4e40740 1 -- 172.21.15.1:0/2070293760 shutdown_connections 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51f4e40740 1 --2- 172.21.15.1:0/2070293760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 0x558bb50b35b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51f4e40740 1 --2- 172.21.15.1:0/2070293760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 0x558bb50b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51f4e40740 1 --2- 172.21.15.1:0/2070293760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50a63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51f4e40740 1 -- 172.21.15.1:0/2070293760 >> 172.21.15.1:0/2070293760 conn(0x558bb50a24f0 msgr2=0x558bb50a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51f4e40740 1 -- 172.21.15.1:0/2070293760 shutdown_connections 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.347+0000 7f51f4e40740 1 -- 172.21.15.1:0/2070293760 wait complete. 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.348+0000 7f51ebe00640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51d00a3d00 0x7f51d00a61c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f51e4009430 tx=0x7f51e4005d80 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.348+0000 7f51f4e40740 1 -- start start 2024-04-26T21:43:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.348+0000 7f51f4e40740 1 Processor -- start 2024-04-26T21:43:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50ae160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 0x558bb50ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 0x558bb50aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51f4e40740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558bb50b3df0 con 0x558bb50a5fe0 2024-04-26T21:43:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51f4e40740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558bb4f92c60 con 0x558bb50a6900 2024-04-26T21:43:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51f4e40740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558bb50b3f70 con 0x558bb50b11c0 2024-04-26T21:43:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51f4e40740 10 client.0 _ll_register_callbacks cb 0x558bb4fbee50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:43:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 0x558bb50ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 0x558bb50aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51eb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 0x558bb50ae6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39892/0 (socket says 172.21.15.1:39892) 2024-04-26T21:43:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51eaa00640 1 -- 172.21.15.1:0/2114610589 learned_addr learned my addr 172.21.15.1:0/2114610589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51eaa00640 1 -- 172.21.15.1:0/2114610589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 msgr2=0x558bb50aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51eaa00640 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 0x558bb50aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51eaa00640 1 -- 172.21.15.1:0/2114610589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 msgr2=0x558bb50ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51eaa00640 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.349+0000 7f51eaa00640 1 -- 172.21.15.1:0/2114610589 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bb4fa1510 con 0x558bb50a6900 2024-04-26T21:43:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.350+0000 7f51eaa00640 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 0x558bb50ae6a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f51d8003f00 tx=0x7f51d800bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb160c587cb43be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.350+0000 7f51d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:43:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.350+0000 7f51d4c00640 1 -- 172.21.15.1:0/2114610589 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d802a060 con 0x558bb50a6900 2024-04-26T21:43:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.350+0000 7f51f4e40740 1 -- 172.21.15.1:0/2114610589 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d80035f0 con 0x558bb50a6900 2024-04-26T21:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.350+0000 7f51d4c00640 1 -- 172.21.15.1:0/2114610589 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9642+0+0 (secure 0 0 0) 0x7f51d8003a30 con 0x558bb50a6900 2024-04-26T21:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.350+0000 7f51d4c00640 1 -- 172.21.15.1:0/2114610589 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f51d8031050 con 0x558bb50a6900 2024-04-26T21:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.350+0000 7f51f4e40740 10 client.26600 fetch_fsmap learned FSMap version 42 2024-04-26T21:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.350+0000 7f51f4e40740 1 -- 172.21.15.1:0/2114610589 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558bb4d1d210 con 0x558bb50a6900 2024-04-26T21:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.351+0000 7f51d4c00640 1 -- 172.21.15.1:0/2114610589 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d80035f0 con 0x558bb50a6900 2024-04-26T21:43:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.351+0000 7f51d4c00640 1 -- 172.21.15.1:0/2114610589 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f51d80046c0 con 0x558bb50a6900 2024-04-26T21:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.351+0000 7f51f4e40740 10 client.26600 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T21:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.351+0000 7f51f4e40740 10 client.26600 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.351+0000 7f51f4e40740 20 client.26600 populate_metadata read hostname 'smithi001' 2024-04-26T21:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.351+0000 7f51f4e40740 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558bb50c0e00 0x558bb50c31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.351+0000 7f51f4e40740 4 client.26600 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.351+0000 7f51f4e40740 1 -- 172.21.15.1:0/2114610589 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bb5018f90 con 0x558bb50c0e00 2024-04-26T21:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.351+0000 7f51eb400640 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558bb50c0e00 0x558bb50c31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.352+0000 7f51eb400640 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558bb50c0e00 0x558bb50c31f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90dff4c6fb00f6ac server_cookie=400633ba2e446a09 in_seq=0 out_seq=0 2024-04-26T21:43:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.352+0000 7f51d4c00640 10 client.26600 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.352+0000 7f51d4c00640 1 -- 172.21.15.1:0/2114610589 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558bb5018f90 con 0x558bb50c0e00 2024-04-26T21:43:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.352+0000 7f51d4c00640 10 client.26600 handle_command_reply: tid=0 2024-04-26T21:43:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 client.26600 shutdown 2024-04-26T21:43:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 -- 172.21.15.1:0/2114610589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 msgr2=0x558bb50ae6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 0x558bb50ae6a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f51d8003f00 tx=0x7f51d800bfa0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 -- 172.21.15.1:0/2114610589 shutdown_connections 2024-04-26T21:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558bb50c0e00 0x558bb50c31f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50b11c0 0x558bb50aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50a6900 0x558bb50ae6a0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 --2- 172.21.15.1:0/2114610589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 -- 172.21.15.1:0/2114610589 >> 172.21.15.1:0/2114610589 conn(0x558bb50a24f0 msgr2=0x558bb50a8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51d4c00640 0 client.26600 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 -- 172.21.15.1:0/2114610589 shutdown_connections 2024-04-26T21:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 1 -- 172.21.15.1:0/2114610589 wait complete. 2024-04-26T21:43:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.353+0000 7f51f4e40740 20 client.26600 trim_cache size 0 max 16384 2024-04-26T21:43:51.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.371+0000 7f51f4e40740 1 Processor -- start 2024-04-26T21:43:51.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.371+0000 7f51f4e40740 1 -- start start 2024-04-26T21:43:51.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.371+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a5fe0 0x558bb50b0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.371+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a6900 0x558bb50a89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.371+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 0x558bb50bc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.371+0000 7f51f4e40740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec1b49c0 con 0x558bb50a6900 2024-04-26T21:43:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.371+0000 7f51f4e40740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec1b2200 con 0x558bb50b11c0 2024-04-26T21:43:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.371+0000 7f51f4e40740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec1b0070 con 0x558bb50a5fe0 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a6900 0x558bb50a89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 0x558bb50bc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51eb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a5fe0 0x558bb50b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51ebe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 0x558bb50bc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39904/0 (socket says 172.21.15.1:39904) 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51ebe00640 1 -- 172.21.15.1:0/181020327 learned_addr learned my addr 172.21.15.1:0/181020327 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a6900 0x558bb50a89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38064/0 (socket says 172.21.15.1:38064) 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51eaa00640 1 -- 172.21.15.1:0/181020327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a5fe0 msgr2=0x558bb50b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51eaa00640 1 --2- 172.21.15.1:0/181020327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a5fe0 0x558bb50b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51eaa00640 1 -- 172.21.15.1:0/181020327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 msgr2=0x558bb50bc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51eaa00640 1 --2- 172.21.15.1:0/181020327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 0x558bb50bc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51eaa00640 1 -- 172.21.15.1:0/181020327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bb50a6730 con 0x558bb50a6900 2024-04-26T21:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51eb400640 1 --2- 172.21.15.1:0/181020327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a5fe0 0x558bb50b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.372+0000 7f51ebe00640 1 --2- 172.21.15.1:0/181020327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 0x558bb50bc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:43:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.373+0000 7f51eaa00640 1 --2- 172.21.15.1:0/181020327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a6900 0x558bb50a89a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f51d800b620 tx=0x7f51d800bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d143df223a0cc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.373+0000 7f51d7400640 1 -- 172.21.15.1:0/181020327 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d802a070 con 0x558bb50a6900 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.373+0000 7f51d7400640 1 -- 172.21.15.1:0/181020327 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d80035f0 con 0x558bb50a6900 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.373+0000 7f51d7400640 1 -- 172.21.15.1:0/181020327 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d80041c0 con 0x558bb50a6900 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.374+0000 7f51f4e40740 1 -- 172.21.15.1:0/181020327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a6900 msgr2=0x558bb50a89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.374+0000 7f51f4e40740 1 --2- 172.21.15.1:0/181020327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a6900 0x558bb50a89a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f51d800b620 tx=0x7f51d800bac0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.374+0000 7f51f4e40740 1 -- 172.21.15.1:0/181020327 shutdown_connections 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.374+0000 7f51f4e40740 1 --2- 172.21.15.1:0/181020327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 0x558bb50bc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.374+0000 7f51f4e40740 1 --2- 172.21.15.1:0/181020327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a6900 0x558bb50a89a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.374+0000 7f51f4e40740 1 --2- 172.21.15.1:0/181020327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a5fe0 0x558bb50b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.374+0000 7f51f4e40740 1 -- 172.21.15.1:0/181020327 >> 172.21.15.1:0/181020327 conn(0x558bb50a24f0 msgr2=0x558bb50bc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.375+0000 7f51f4e40740 1 -- 172.21.15.1:0/181020327 shutdown_connections 2024-04-26T21:43:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.375+0000 7f51f4e40740 1 -- 172.21.15.1:0/181020327 wait complete. 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.375+0000 7f51f4e40740 1 -- start start 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.376+0000 7f51f4e40740 1 Processor -- start 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.376+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50c6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.376+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a6900 0x558bb50c3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.376+0000 7f51f4e40740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 0x558bb50c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.376+0000 7f51f4e40740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec1b06c0 con 0x558bb50a5fe0 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.376+0000 7f51f4e40740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec1b1b30 con 0x558bb50b11c0 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.376+0000 7f51f4e40740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec1b0d90 con 0x558bb50a6900 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.376+0000 7f51f4e40740 10 client.0 _ll_register_callbacks cb 0x7f51e4009360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.376+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a6900 0x558bb50c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51eaa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a6900 0x558bb50c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42652/0 (socket says 172.21.15.1:42652) 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51eaa00640 1 -- 172.21.15.1:0/3615967520 learned_addr learned my addr 172.21.15.1:0/3615967520 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:43:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51ebe00640 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 0x558bb50c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51eb400640 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51eaa00640 1 -- 172.21.15.1:0/3615967520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 msgr2=0x558bb50c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51eaa00640 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 0x558bb50c3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51eaa00640 1 -- 172.21.15.1:0/3615967520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 msgr2=0x558bb50c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51eaa00640 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50c6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51eaa00640 1 -- 172.21.15.1:0/3615967520 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bb4de9620 con 0x558bb50a6900 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51eaa00640 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a6900 0x558bb50c3900 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x558bb50a0d60 tx=0x7f51d8005950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ab827e775a969aa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.377+0000 7f51d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.378+0000 7f51d4c00640 1 -- 172.21.15.1:0/3615967520 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d802a070 con 0x558bb50a6900 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.378+0000 7f51d4c00640 1 -- 172.21.15.1:0/3615967520 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 9642+0+0 (secure 0 0 0) 0x7f51d8003a30 con 0x558bb50a6900 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.378+0000 7f51f4e40740 1 -- 172.21.15.1:0/3615967520 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d80035f0 con 0x558bb50a6900 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.378+0000 7f51d4c00640 1 -- 172.21.15.1:0/3615967520 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f51d8045050 con 0x558bb50a6900 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.378+0000 7f51f4e40740 10 client.26578 fetch_fsmap learned FSMap version 42 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.378+0000 7f51f4e40740 1 -- 172.21.15.1:0/3615967520 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x558bb4fc5cb0 con 0x558bb50a6900 2024-04-26T21:43:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.378+0000 7f51d4c00640 1 -- 172.21.15.1:0/3615967520 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d80035f0 con 0x558bb50a6900 2024-04-26T21:43:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.378+0000 7f51d4c00640 1 -- 172.21.15.1:0/3615967520 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7f51d803c070 con 0x558bb50a6900 2024-04-26T21:43:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.379+0000 7f51f4e40740 10 client.26578 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T21:43:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.379+0000 7f51f4e40740 10 client.26578 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:43:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.379+0000 7f51f4e40740 20 client.26578 populate_metadata read hostname 'smithi001' 2024-04-26T21:43:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.379+0000 7f51f4e40740 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558bb5127300 0x558bb5129720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:43:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.379+0000 7f51f4e40740 4 client.26578 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:43:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.379+0000 7f51f4e40740 1 -- 172.21.15.1:0/3615967520 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558bb5012e00 con 0x558bb5127300 2024-04-26T21:43:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.379+0000 7f51eb400640 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558bb5127300 0x558bb5129720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:43:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.379+0000 7f51eb400640 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558bb5127300 0x558bb5129720 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8a36a24fe916edf server_cookie=6b39d31a95bf4b64 in_seq=0 out_seq=0 2024-04-26T21:43:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.379+0000 7f51d4c00640 10 client.26578 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51d4c00640 1 -- 172.21.15.1:0/3615967520 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558bb5012e00 con 0x558bb5127300 2024-04-26T21:43:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51d4c00640 10 client.26578 handle_command_reply: tid=0 2024-04-26T21:43:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 client.26578 shutdown 2024-04-26T21:43:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 -- 172.21.15.1:0/3615967520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a6900 msgr2=0x558bb50c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a6900 0x558bb50c3900 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x558bb50a0d60 tx=0x7f51d8005950 comp rx=0 tx=0).stop 2024-04-26T21:43:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 -- 172.21.15.1:0/3615967520 shutdown_connections 2024-04-26T21:43:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558bb5127300 0x558bb5129720 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558bb50a6900 0x558bb50c3900 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bb50b11c0 0x558bb50c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51d4c00640 0 client.26578 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 --2- 172.21.15.1:0/3615967520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558bb50a5fe0 0x558bb50c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 -- 172.21.15.1:0/3615967520 >> 172.21.15.1:0/3615967520 conn(0x558bb50a24f0 msgr2=0x558bb50bca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 -- 172.21.15.1:0/3615967520 shutdown_connections 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.446+0000 7f51f4e40740 1 -- 172.21.15.1:0/3615967520 wait complete. 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.447+0000 7f51f4e40740 20 client.26578 trim_cache size 0 max 16384 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:43:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:43:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:43:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.447+0000 7f51f0a00640 1 -- 172.21.15.1:0/1524151937 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51d00a3d00 msgr2=0x7f51d00a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.447+0000 7f51f0a00640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51d00a3d00 0x7f51d00a61c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f51e4009430 tx=0x7f51e4005d80 comp rx=0 tx=0).stop 2024-04-26T21:43:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.447+0000 7f51f0a00640 1 -- 172.21.15.1:0/1524151937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec102750 msgr2=0x7f51ec19b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:43:51.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.447+0000 7f51f0a00640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec102750 0x7f51ec19b0b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00d950 tx=0x7f51dc00de10 comp rx=0 tx=0).stop 2024-04-26T21:43:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.448+0000 7f51f0a00640 1 -- 172.21.15.1:0/1524151937 shutdown_connections 2024-04-26T21:43:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.448+0000 7f51f0a00640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f51d00a3d00 0x7f51d00a61c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.448+0000 7f51f0a00640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51ec108760 0x7f51ec19f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.448+0000 7f51f0a00640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51ec103110 0x7f51ec19b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.448+0000 7f51f0a00640 1 --2- 172.21.15.1:0/1524151937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f51ec102750 0x7f51ec19b0b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:43:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.448+0000 7f51f0a00640 1 -- 172.21.15.1:0/1524151937 >> 172.21.15.1:0/1524151937 conn(0x7f51ec0fe240 msgr2=0x7f51ec0fe820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:43:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.448+0000 7f51f0a00640 1 -- 172.21.15.1:0/1524151937 shutdown_connections 2024-04-26T21:43:51.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:43:51.448+0000 7f51f0a00640 1 -- 172.21.15.1:0/1524151937 wait complete. 2024-04-26T21:43:51.620 INFO:tasks.workunit.client.0.smithi001.stdout: 1 519745 4.18 MB/sec execute 589 sec latency 54.040 ms 2024-04-26T21:43:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:52 smithi186 ceph-mon[32991]: pgmap v423: 129 pgs: 129 active+clean; 417 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-04-26T21:43:52.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:52 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:43:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:52 smithi001 ceph-mon[28054]: pgmap v423: 129 pgs: 129 active+clean; 417 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-04-26T21:43:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:52 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:43:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:52 smithi038 ceph-mon[32724]: pgmap v423: 129 pgs: 129 active+clean; 417 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 174 op/s 2024-04-26T21:43:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:52 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:43:52.620 INFO:tasks.workunit.client.0.smithi001.stdout: 1 521009 4.18 MB/sec execute 590 sec latency 24.016 ms 2024-04-26T21:43:53.621 INFO:tasks.workunit.client.0.smithi001.stdout: 1 522221 4.19 MB/sec execute 591 sec latency 24.187 ms 2024-04-26T21:43:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:54 smithi186 ceph-mon[32991]: pgmap v424: 129 pgs: 129 active+clean; 417 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-26T21:43:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:54 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:43:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:54 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:43:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:54 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:43:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:54 smithi001 ceph-mon[28054]: pgmap v424: 129 pgs: 129 active+clean; 417 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-26T21:43:54.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:54 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:43:54.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:54 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:43:54.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:54 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:43:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:54 smithi038 ceph-mon[32724]: pgmap v424: 129 pgs: 129 active+clean; 417 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 1.4 MiB/s wr, 108 op/s 2024-04-26T21:43:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:54 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:43:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:54 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:43:54.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:54 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:43:54.621 INFO:tasks.workunit.client.0.smithi001.stdout: 1 522656 4.19 MB/sec execute 592 sec latency 37.653 ms 2024-04-26T21:43:55.621 INFO:tasks.workunit.client.0.smithi001.stdout: 1 523194 4.18 MB/sec execute 593 sec latency 59.401 ms 2024-04-26T21:43:56.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:56 smithi186 ceph-mon[32991]: pgmap v425: 129 pgs: 129 active+clean; 418 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 187 op/s 2024-04-26T21:43:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:56 smithi001 ceph-mon[28054]: pgmap v425: 129 pgs: 129 active+clean; 418 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 187 op/s 2024-04-26T21:43:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:56 smithi038 ceph-mon[32724]: pgmap v425: 129 pgs: 129 active+clean; 418 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.2 MiB/s wr, 187 op/s 2024-04-26T21:43:56.621 INFO:tasks.workunit.client.0.smithi001.stdout: 1 524040 4.18 MB/sec execute 594 sec latency 35.509 ms 2024-04-26T21:43:57.621 INFO:tasks.workunit.client.0.smithi001.stdout: 1 524927 4.19 MB/sec execute 595 sec latency 37.847 ms 2024-04-26T21:43:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:43:58 smithi186 ceph-mon[32991]: pgmap v426: 129 pgs: 129 active+clean; 418 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-26T21:43:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:43:58 smithi001 ceph-mon[28054]: pgmap v426: 129 pgs: 129 active+clean; 418 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-26T21:43:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:43:58 smithi038 ceph-mon[32724]: pgmap v426: 129 pgs: 129 active+clean; 418 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-26T21:43:58.621 INFO:tasks.workunit.client.0.smithi001.stdout: 1 525990 4.19 MB/sec execute 596 sec latency 29.192 ms 2024-04-26T21:43:59.621 INFO:tasks.workunit.client.0.smithi001.stdout: 1 526295 4.19 MB/sec execute 597 sec latency 31.906 ms 2024-04-26T21:44:00.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:00 smithi186 ceph-mon[32991]: pgmap v427: 129 pgs: 129 active+clean; 418 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-26T21:44:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:00 smithi001 ceph-mon[28054]: pgmap v427: 129 pgs: 129 active+clean; 418 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-26T21:44:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:00 smithi038 ceph-mon[32724]: pgmap v427: 129 pgs: 129 active+clean; 418 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 1.6 MiB/s wr, 135 op/s 2024-04-26T21:44:00.621 INFO:tasks.workunit.client.0.smithi001.stdout: 1 526985 4.18 MB/sec execute 598 sec latency 51.133 ms 2024-04-26T21:44:01.622 INFO:tasks.workunit.client.0.smithi001.stdout: 1 528035 4.18 MB/sec execute 599 sec latency 92.163 ms 2024-04-26T21:44:02.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:02 smithi186 ceph-mon[32991]: pgmap v428: 129 pgs: 129 active+clean; 423 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 190 op/s 2024-04-26T21:44:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:02 smithi001 ceph-mon[28054]: pgmap v428: 129 pgs: 129 active+clean; 423 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 190 op/s 2024-04-26T21:44:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:02 smithi038 ceph-mon[32724]: pgmap v428: 129 pgs: 129 active+clean; 423 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 190 op/s 2024-04-26T21:44:02.622 INFO:tasks.workunit.client.0.smithi001.stdout: 1 cleanup 600 sec 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: 0 cleanup 600 sec 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: Operation Count AvgLat MaxLat 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: ---------------------------------------- 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: NTCreateX 79814 3.287 5367.299 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: Close 58625 0.010 1.900 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: Rename 3377 11.377 94.569 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: Unlink 16117 4.291 4802.898 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: Deltree 2 155.965 221.350 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: Mkdir 1 0.045 0.045 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: Qpathinfo 72333 1.886 3317.282 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: Qfileinfo 12671 0.005 0.250 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: Qfsinfo 13260 0.554 31.641 2024-04-26T21:44:03.008 INFO:tasks.workunit.client.0.smithi001.stdout: Sfileinfo 6490 0.080 0.681 2024-04-26T21:44:03.009 INFO:tasks.workunit.client.0.smithi001.stdout: Find 27957 0.693 7044.277 2024-04-26T21:44:03.009 INFO:tasks.workunit.client.0.smithi001.stdout: WriteX 39772 0.371 178.059 2024-04-26T21:44:03.009 INFO:tasks.workunit.client.0.smithi001.stdout: ReadX 125096 0.031 42.542 2024-04-26T21:44:03.009 INFO:tasks.workunit.client.0.smithi001.stdout: LockX 260 1.009 3.593 2024-04-26T21:44:03.009 INFO:tasks.workunit.client.0.smithi001.stdout: UnlockX 260 0.875 3.771 2024-04-26T21:44:03.009 INFO:tasks.workunit.client.0.smithi001.stdout: Flush 5592 7.558 283.361 2024-04-26T21:44:03.009 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-04-26T21:44:03.009 INFO:tasks.workunit.client.0.smithi001.stdout:Throughput 4.18177 MB/sec 1 clients 1 procs max_latency=7044.292 ms 2024-04-26T21:44:03.009 INFO:tasks.workunit.client.0.smithi001.stderr:+ dbench 10 2024-04-26T21:44:03.010 INFO:tasks.workunit.client.0.smithi001.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-26T21:44:03.010 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-04-26T21:44:03.011 INFO:tasks.workunit.client.0.smithi001.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-26T21:44:03.014 INFO:tasks.workunit.client.0.smithi001.stdout:8 of 10 processes prepared for launch 0 sec 2024-04-26T21:44:03.115 INFO:tasks.workunit.client.0.smithi001.stdout:10 of 10 processes prepared for launch 0 sec 2024-04-26T21:44:03.115 INFO:tasks.workunit.client.0.smithi001.stdout:releasing clients 2024-04-26T21:44:03.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:03 smithi001 ceph-mon[28054]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T21:44:03.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:03 smithi038 ceph-mon[32724]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T21:44:03.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:03 smithi186 ceph-mon[32991]: osdmap e111: 12 total, 12 up, 12 in 2024-04-26T21:44:04.115 INFO:tasks.workunit.client.0.smithi001.stdout: 10 343 123.89 MB/sec warmup 1 sec latency 633.730 ms 2024-04-26T21:44:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:04 smithi001 ceph-mon[28054]: pgmap v430: 129 pgs: 129 active+clean; 423 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-04-26T21:44:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:04 smithi038 ceph-mon[32724]: pgmap v430: 129 pgs: 129 active+clean; 423 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-04-26T21:44:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.612+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2018750975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8ac10a970 msgr2=0x7fa8ac10ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.612+0000 7fa8b1e00640 1 --2- 172.21.15.1:0/2018750975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8ac10a970 0x7fa8ac10ad50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa89c009ca0 tx=0x7fa89c02a8c0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.612+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2018750975 shutdown_connections 2024-04-26T21:44:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.612+0000 7fa8b1e00640 1 --2- 172.21.15.1:0/2018750975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8ac105320 0x7fa8ac111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.612+0000 7fa8b1e00640 1 --2- 172.21.15.1:0/2018750975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ac104960 0x7fa8ac104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.612+0000 7fa8b1e00640 1 --2- 172.21.15.1:0/2018750975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8ac10a970 0x7fa8ac10ad50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.612+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2018750975 >> 172.21.15.1:0/2018750975 conn(0x7fa8ac100430 msgr2=0x7fa8ac102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.612+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2018750975 shutdown_connections 2024-04-26T21:44:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.612+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2018750975 wait complete. 2024-04-26T21:44:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.613+0000 7fa8b1e00640 1 Processor -- start 2024-04-26T21:44:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.614+0000 7fa8b1e00640 1 -- start start 2024-04-26T21:44:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.614+0000 7fa8b1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8ac104960 0x7fa8ac19b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.614+0000 7fa8b1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8ac105320 0x7fa8ac19b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.614+0000 7fa8b1e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ac10a970 0x7fa8ac19fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.614+0000 7fa8b1e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac114c80 con 0x7fa8ac10a970 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.614+0000 7fa8b1e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac114b00 con 0x7fa8ac104960 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.614+0000 7fa8b1e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac114e00 con 0x7fa8ac105320 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.614+0000 7fa8b0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8ac104960 0x7fa8ac19b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.614+0000 7fa8b0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8ac104960 0x7fa8ac19b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54668/0 (socket says 172.21.15.1:54668) 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.614+0000 7fa8b0a00640 1 -- 172.21.15.1:0/2974698267 learned_addr learned my addr 172.21.15.1:0/2974698267 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.615+0000 7fa8a3e00640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8ac105320 0x7fa8ac19b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.615+0000 7fa8b1400640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ac10a970 0x7fa8ac19fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.615+0000 7fa8a3e00640 1 -- 172.21.15.1:0/2974698267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8ac104960 msgr2=0x7fa8ac19b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.615+0000 7fa8a3e00640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8ac104960 0x7fa8ac19b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.615+0000 7fa8a3e00640 1 -- 172.21.15.1:0/2974698267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ac10a970 msgr2=0x7fa8ac19fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.615+0000 7fa8a3e00640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ac10a970 0x7fa8ac19fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.615+0000 7fa8a3e00640 1 -- 172.21.15.1:0/2974698267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8ac1a0150 con 0x7fa8ac105320 2024-04-26T21:44:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.616+0000 7fa8a3e00640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8ac105320 0x7fa8ac19b6a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa89800ea60 tx=0x7fa89800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.616+0000 7fa8aa000640 1 -- 172.21.15.1:0/2974698267 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89800ce40 con 0x7fa8ac105320 2024-04-26T21:44:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.616+0000 7fa8aa000640 1 -- 172.21.15.1:0/2974698267 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8980108e0 con 0x7fa8ac105320 2024-04-26T21:44:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.616+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2974698267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8ac1a0440 con 0x7fa8ac105320 2024-04-26T21:44:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.616+0000 7fa8aa000640 1 -- 172.21.15.1:0/2974698267 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa898018a80 con 0x7fa8ac105320 2024-04-26T21:44:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.616+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2974698267 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8ac1a7cd0 con 0x7fa8ac105320 2024-04-26T21:44:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.620+0000 7fa8b6228740 1 Processor -- start 2024-04-26T21:44:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.620+0000 7fa8b6228740 1 -- start start 2024-04-26T21:44:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.620+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.620+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332b0900 0x5604332bac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.620+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332bb1c0 0x5604332bd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.620+0000 7fa8b6228740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x560433215380 con 0x5604332affe0 2024-04-26T21:44:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.620+0000 7fa8b6228740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5604331b1c30 con 0x5604332b0900 2024-04-26T21:44:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.620+0000 7fa8b6228740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560433211340 con 0x5604332bb1c0 2024-04-26T21:44:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8aa000640 1 -- 172.21.15.1:0/2974698267 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa898018c20 con 0x7fa8ac105320 2024-04-26T21:44:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8b0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332b0900 0x5604332bac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8a3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332bb1c0 0x5604332bd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8b1400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8b0a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332b0900 0x5604332bac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54670/0 (socket says 172.21.15.1:54670) 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8b0a00640 1 -- 172.21.15.1:0/2654462666 learned_addr learned my addr 172.21.15.1:0/2654462666 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8b0a00640 1 -- 172.21.15.1:0/2654462666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332bb1c0 msgr2=0x5604332bd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8b0a00640 1 --2- 172.21.15.1:0/2654462666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332bb1c0 0x5604332bd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8b0a00640 1 -- 172.21.15.1:0/2654462666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 msgr2=0x5604332b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8b0a00640 1 --2- 172.21.15.1:0/2654462666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8b0a00640 1 -- 172.21.15.1:0/2654462666 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604331d3940 con 0x5604332b0900 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.621+0000 7fa8b0a00640 1 --2- 172.21.15.1:0/2654462666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332b0900 0x5604332bac80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa89c0097b0 tx=0x7fa89c02f310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e82c3d93c075782 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.622+0000 7fa8a1c00640 1 -- 172.21.15.1:0/2654462666 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89c031ce0 con 0x5604332b0900 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.622+0000 7fa8a1c00640 1 -- 172.21.15.1:0/2654462666 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa89c036030 con 0x5604332b0900 2024-04-26T21:44:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.622+0000 7fa8a1c00640 1 -- 172.21.15.1:0/2654462666 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89c0042a0 con 0x5604332b0900 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.622+0000 7fa8aa000640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa8900a3e00 0x7fa8900a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8aa000640 1 -- 172.21.15.1:0/2974698267 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 9730+0+0 (secure 0 0 0) 0x7fa8980d10e0 con 0x7fa8ac105320 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8b6228740 1 -- 172.21.15.1:0/2654462666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332b0900 msgr2=0x5604332bac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8b6228740 1 --2- 172.21.15.1:0/2654462666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332b0900 0x5604332bac80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa89c0097b0 tx=0x7fa89c02f310 comp rx=0 tx=0).stop 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8b1400640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa8900a3e00 0x7fa8900a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8b6228740 1 -- 172.21.15.1:0/2654462666 shutdown_connections 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8b6228740 1 --2- 172.21.15.1:0/2654462666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332bb1c0 0x5604332bd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8b6228740 1 --2- 172.21.15.1:0/2654462666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332b0900 0x5604332bac80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8b6228740 1 --2- 172.21.15.1:0/2654462666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8b6228740 1 -- 172.21.15.1:0/2654462666 >> 172.21.15.1:0/2654462666 conn(0x5604332ac4f0 msgr2=0x5604332ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8b6228740 1 -- 172.21.15.1:0/2654462666 shutdown_connections 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.623+0000 7fa8b6228740 1 -- 172.21.15.1:0/2654462666 wait complete. 2024-04-26T21:44:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.624+0000 7fa8b6228740 1 -- start start 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.624+0000 7fa8b1400640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa8900a3e00 0x7fa8900a62c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4009c00 tx=0x7fa8a4005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.624+0000 7fa8b6228740 1 Processor -- start 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.624+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332b8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.624+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 0x5604332b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.625+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.625+0000 7fa8b6228740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5604332bdc10 con 0x5604332affe0 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.625+0000 7fa8b6228740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56043319cc60 con 0x5604332bb1c0 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.625+0000 7fa8b6228740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604332bdd90 con 0x5604332b0900 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.625+0000 7fa8b6228740 10 client.0 _ll_register_callbacks cb 0x5604331c8e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.625+0000 7fa8b0a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332b8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.625+0000 7fa8b0a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332b8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40442/0 (socket says 172.21.15.1:40442) 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.625+0000 7fa8b0a00640 1 -- 172.21.15.1:0/1112658619 learned_addr learned my addr 172.21.15.1:0/1112658619 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.625+0000 7fa8a3e00640 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 0x5604332b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.625+0000 7fa8b1400640 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.626+0000 7fa8a3e00640 1 -- 172.21.15.1:0/1112658619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 msgr2=0x5604332b8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.626+0000 7fa8a3e00640 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.626+0000 7fa8a3e00640 1 -- 172.21.15.1:0/1112658619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 msgr2=0x5604332b8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.626+0000 7fa8a3e00640 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332b8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.626+0000 7fa8a3e00640 1 -- 172.21.15.1:0/1112658619 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604331ab510 con 0x5604332b0900 2024-04-26T21:44:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.626+0000 7fa8b1400640 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:44:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.627+0000 7fa8a3e00640 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 0x5604332b85b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa8980ce310 tx=0x7fa8980057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f45f936e0c15b499 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.627+0000 7fa896000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:44:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.627+0000 7fa896000640 1 -- 172.21.15.1:0/1112658619 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89809c570 con 0x5604332b0900 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.627+0000 7fa896000640 1 -- 172.21.15.1:0/1112658619 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9730+0+0 (secure 0 0 0) 0x7fa898099280 con 0x5604332b0900 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.627+0000 7fa8b6228740 1 -- 172.21.15.1:0/1112658619 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa89c036030 con 0x5604332b0900 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.627+0000 7fa896000640 1 -- 172.21.15.1:0/1112658619 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fa89809cc10 con 0x5604332b0900 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.627+0000 7fa8b6228740 10 client.26590 fetch_fsmap learned FSMap version 42 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.627+0000 7fa8b6228740 1 -- 172.21.15.1:0/1112658619 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x560432f27210 con 0x5604332b0900 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.628+0000 7fa896000640 1 -- 172.21.15.1:0/1112658619 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa89c036030 con 0x5604332b0900 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.628+0000 7fa896000640 1 -- 172.21.15.1:0/1112658619 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fa8980d3110 con 0x5604332b0900 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.628+0000 7fa8b6228740 10 client.26590 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.628+0000 7fa8b6228740 10 client.26590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.628+0000 7fa8b6228740 20 client.26590 populate_metadata read hostname 'smithi001' 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.628+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5604332d3470 0x5604332d58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.628+0000 7fa8b6228740 4 client.26590 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.628+0000 7fa8b6228740 1 -- 172.21.15.1:0/1112658619 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560433222f90 con 0x5604332d3470 2024-04-26T21:44:04.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.629+0000 7fa8b0a00640 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5604332d3470 0x5604332d58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.629+0000 7fa8b0a00640 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5604332d3470 0x5604332d58a0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41210c6dbceae98e server_cookie=989bab5dc5b30b41 in_seq=0 out_seq=0 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.629+0000 7fa896000640 10 client.26590 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.630+0000 7fa896000640 1 -- 172.21.15.1:0/1112658619 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560433222f90 con 0x5604332d3470 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.630+0000 7fa896000640 10 client.26590 handle_command_reply: tid=0 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.630+0000 7fa8b6228740 1 client.26590 shutdown 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 1 -- 172.21.15.1:0/1112658619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 msgr2=0x5604332b85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 0x5604332b85b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa8980ce310 tx=0x7fa8980057f0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 1 -- 172.21.15.1:0/1112658619 shutdown_connections 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5604332d3470 0x5604332d58a0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 0x5604332b85b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1112658619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332b8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa896000640 0 client.26590 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 1 -- 172.21.15.1:0/1112658619 >> 172.21.15.1:0/1112658619 conn(0x5604332ac4f0 msgr2=0x5604332b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 1 -- 172.21.15.1:0/1112658619 shutdown_connections 2024-04-26T21:44:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 1 -- 172.21.15.1:0/1112658619 wait complete. 2024-04-26T21:44:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.631+0000 7fa8b6228740 20 client.26590 trim_cache size 0 max 16384 2024-04-26T21:44:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.651+0000 7fa8b6228740 1 Processor -- start 2024-04-26T21:44:04.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.651+0000 7fa8b6228740 1 -- start start 2024-04-26T21:44:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332ba7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 0x5604332b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b6228740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac1b05c0 con 0x5604332affe0 2024-04-26T21:44:04.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b6228740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac1ade00 con 0x5604332bb1c0 2024-04-26T21:44:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b6228740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac1abc70 con 0x5604332b0900 2024-04-26T21:44:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b1400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b0a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332ba7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b1400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54696/0 (socket says 172.21.15.1:54696) 2024-04-26T21:44:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b1400640 1 -- 172.21.15.1:0/3702479161 learned_addr learned my addr 172.21.15.1:0/3702479161 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:04.653 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8a3e00640 1 --2- 172.21.15.1:0/3702479161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 0x5604332b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b1400640 1 -- 172.21.15.1:0/3702479161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 msgr2=0x5604332b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b1400640 1 --2- 172.21.15.1:0/3702479161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 0x5604332b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b1400640 1 -- 172.21.15.1:0/3702479161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 msgr2=0x5604332ba7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b1400640 1 --2- 172.21.15.1:0/3702479161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332ba7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.652+0000 7fa8b1400640 1 -- 172.21.15.1:0/3702479161 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560432ff3620 con 0x5604332bb1c0 2024-04-26T21:44:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.653+0000 7fa8b1400640 1 --2- 172.21.15.1:0/3702479161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b71d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4004480 tx=0x7fa8a401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=951b9b853a92a995 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.653+0000 7fa8a1200640 1 -- 172.21.15.1:0/3702479161 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a402e070 con 0x5604332bb1c0 2024-04-26T21:44:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.653+0000 7fa8a1200640 1 -- 172.21.15.1:0/3702479161 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8a401e920 con 0x5604332bb1c0 2024-04-26T21:44:04.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.653+0000 7fa8a1200640 1 -- 172.21.15.1:0/3702479161 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4029930 con 0x5604332bb1c0 2024-04-26T21:44:04.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.654+0000 7fa8b6228740 1 -- 172.21.15.1:0/3702479161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 msgr2=0x5604332b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.654+0000 7fa8b6228740 1 --2- 172.21.15.1:0/3702479161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b71d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4004480 tx=0x7fa8a401b000 comp rx=0 tx=0).stop 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.654+0000 7fa8b6228740 1 -- 172.21.15.1:0/3702479161 shutdown_connections 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.654+0000 7fa8b6228740 1 --2- 172.21.15.1:0/3702479161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332b0900 0x5604332b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.654+0000 7fa8b6228740 1 --2- 172.21.15.1:0/3702479161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x5604332b71d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.654+0000 7fa8b6228740 1 --2- 172.21.15.1:0/3702479161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332affe0 0x5604332ba7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.654+0000 7fa8b6228740 1 -- 172.21.15.1:0/3702479161 >> 172.21.15.1:0/3702479161 conn(0x5604332ac4f0 msgr2=0x5604332aadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.654+0000 7fa8b6228740 1 -- 172.21.15.1:0/3702479161 shutdown_connections 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.654+0000 7fa8b6228740 1 -- 172.21.15.1:0/3702479161 wait complete. 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.655+0000 7fa8b6228740 1 -- start start 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.655+0000 7fa8b6228740 1 Processor -- start 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.655+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332affe0 0x5604332c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.655+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332b0900 0x5604332c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.655+0000 7fa8b6228740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x560433337a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.655+0000 7fa8b6228740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac1ac2c0 con 0x5604332b0900 2024-04-26T21:44:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.655+0000 7fa8b6228740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac1ad730 con 0x5604332bb1c0 2024-04-26T21:44:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b6228740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa8ac1ac990 con 0x5604332affe0 2024-04-26T21:44:04.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b6228740 10 client.0 _ll_register_callbacks cb 0x7fa898018420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:44:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b1400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x560433337a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332affe0 0x5604332c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332affe0 0x5604332c69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43644/0 (socket says 172.21.15.1:43644) 2024-04-26T21:44:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b1400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x560433337a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54708/0 (socket says 172.21.15.1:54708) 2024-04-26T21:44:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b0a00640 1 -- 172.21.15.1:0/1743215740 learned_addr learned my addr 172.21.15.1:0/1743215740 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8a3e00640 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332b0900 0x5604332c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b1400640 1 -- 172.21.15.1:0/1743215740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332affe0 msgr2=0x5604332c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b1400640 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332affe0 0x5604332c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b1400640 1 -- 172.21.15.1:0/1743215740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332b0900 msgr2=0x5604332c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b1400640 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332b0900 0x5604332c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b1400640 1 -- 172.21.15.1:0/1743215740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604331cfcb0 con 0x5604332bb1c0 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.656+0000 7fa8b0a00640 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332affe0 0x5604332c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.657+0000 7fa8b1400640 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x560433337a10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4000c00 tx=0x7fa8a4004160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1e0332f45032ff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.657+0000 7fa896000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.657+0000 7fa896000640 1 -- 172.21.15.1:0/1743215740 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a4037060 con 0x5604332bb1c0 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.657+0000 7fa896000640 1 -- 172.21.15.1:0/1743215740 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 9730+0+0 (secure 0 0 0) 0x7fa8a401e530 con 0x5604332bb1c0 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.657+0000 7fa8b6228740 1 -- 172.21.15.1:0/1743215740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8a401e920 con 0x5604332bb1c0 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.657+0000 7fa896000640 1 -- 172.21.15.1:0/1743215740 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fa8a403e050 con 0x5604332bb1c0 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.657+0000 7fa8b6228740 10 client.26624 fetch_fsmap learned FSMap version 42 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.657+0000 7fa8b6228740 1 -- 172.21.15.1:0/1743215740 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x560433339070 con 0x5604332bb1c0 2024-04-26T21:44:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.658+0000 7fa896000640 1 -- 172.21.15.1:0/1743215740 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8a401e920 con 0x5604332bb1c0 2024-04-26T21:44:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.658+0000 7fa896000640 1 -- 172.21.15.1:0/1743215740 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 42) ==== 8362+0+0 (secure 0 0 0) 0x7fa8a40046c0 con 0x5604332bb1c0 2024-04-26T21:44:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.658+0000 7fa8b6228740 10 client.26624 fetch_fsmap finished waiting for FSMap version 42 2024-04-26T21:44:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.658+0000 7fa8b6228740 10 client.26624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:44:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.658+0000 7fa8b6228740 20 client.26624 populate_metadata read hostname 'smithi001' 2024-04-26T21:44:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.658+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560433339680 0x56043333baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.658+0000 7fa8b6228740 4 client.26624 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:44:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.658+0000 7fa8b6228740 1 -- 172.21.15.1:0/1743215740 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56043321ce00 con 0x560433339680 2024-04-26T21:44:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.658+0000 7fa8b0a00640 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560433339680 0x56043333baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.659+0000 7fa8b0a00640 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560433339680 0x56043333baa0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c032ad317d4a7f7 server_cookie=b91c04ccd8911553 in_seq=0 out_seq=0 2024-04-26T21:44:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:04.659+0000 7fa896000640 10 client.26624 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:04.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:04 smithi186 ceph-mon[32991]: pgmap v430: 129 pgs: 129 active+clean; 423 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 160 op/s 2024-04-26T21:44:05.115 INFO:tasks.workunit.client.0.smithi001.stdout: 10 699 98.69 MB/sec warmup 2 sec latency 1633.840 ms 2024-04-26T21:44:06.116 INFO:tasks.workunit.client.0.smithi001.stdout: 10 1053 70.74 MB/sec warmup 3 sec latency 2634.589 ms 2024-04-26T21:44:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:06 smithi001 ceph-mon[28054]: pgmap v431: 129 pgs: 129 active+clean; 425 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.9 MiB/s wr, 177 op/s 2024-04-26T21:44:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:06 smithi038 ceph-mon[32724]: pgmap v431: 129 pgs: 129 active+clean; 425 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.9 MiB/s wr, 177 op/s 2024-04-26T21:44:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:06 smithi186 ceph-mon[32991]: pgmap v431: 129 pgs: 129 active+clean; 425 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.9 MiB/s wr, 177 op/s 2024-04-26T21:44:07.116 INFO:tasks.workunit.client.0.smithi001.stdout: 10 1582 60.39 MB/sec warmup 4 sec latency 3634.698 ms 2024-04-26T21:44:08.116 INFO:tasks.workunit.client.0.smithi001.stdout: 10 2227 52.10 MB/sec warmup 5 sec latency 4634.782 ms 2024-04-26T21:44:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:08 smithi001 ceph-mon[28054]: pgmap v432: 129 pgs: 129 active+clean; 425 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-26T21:44:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:08 smithi038 ceph-mon[32724]: pgmap v432: 129 pgs: 129 active+clean; 425 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-26T21:44:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:08 smithi186 ceph-mon[32991]: pgmap v432: 129 pgs: 129 active+clean; 425 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-26T21:44:09.116 INFO:tasks.workunit.client.0.smithi001.stdout: 10 2769 51.48 MB/sec warmup 6 sec latency 5634.937 ms 2024-04-26T21:44:10.116 INFO:tasks.workunit.client.0.smithi001.stdout: 10 3298 50.44 MB/sec warmup 7 sec latency 6635.011 ms 2024-04-26T21:44:10.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:10 smithi038 ceph-mon[32724]: pgmap v433: 129 pgs: 129 active+clean; 425 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-26T21:44:10.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:10 smithi038 ceph-mon[32724]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T21:44:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:10 smithi186 ceph-mon[32991]: pgmap v433: 129 pgs: 129 active+clean; 425 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-26T21:44:10.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:10 smithi186 ceph-mon[32991]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T21:44:10.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:10 smithi001 ceph-mon[28054]: pgmap v433: 129 pgs: 129 active+clean; 425 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-04-26T21:44:10.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:10 smithi001 ceph-mon[28054]: osdmap e112: 12 total, 12 up, 12 in 2024-04-26T21:44:11.117 INFO:tasks.workunit.client.0.smithi001.stdout: 10 3453 45.38 MB/sec warmup 8 sec latency 7635.862 ms 2024-04-26T21:44:12.117 INFO:tasks.workunit.client.0.smithi001.stdout: 10 3601 40.64 MB/sec warmup 9 sec latency 8635.951 ms 2024-04-26T21:44:12.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:12 smithi038 ceph-mon[32724]: pgmap v435: 129 pgs: 129 active+clean; 444 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-04-26T21:44:12.369 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:12 smithi001 ceph-mon[28054]: pgmap v435: 129 pgs: 129 active+clean; 444 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-04-26T21:44:12.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:12 smithi186 ceph-mon[32991]: pgmap v435: 129 pgs: 129 active+clean; 444 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.7 MiB/s wr, 273 op/s 2024-04-26T21:44:13.118 INFO:tasks.workunit.client.0.smithi001.stdout: 10 3753 36.61 MB/sec warmup 10 sec latency 9636.068 ms 2024-04-26T21:44:14.118 INFO:tasks.workunit.client.0.smithi001.stdout: 10 3950 33.69 MB/sec warmup 11 sec latency 10636.171 ms 2024-04-26T21:44:14.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:14 smithi186 ceph-mon[32991]: pgmap v436: 129 pgs: 129 active+clean; 444 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 241 op/s 2024-04-26T21:44:14.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:14 smithi001 ceph-mon[28054]: pgmap v436: 129 pgs: 129 active+clean; 444 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 241 op/s 2024-04-26T21:44:14.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:14 smithi038 ceph-mon[32724]: pgmap v436: 129 pgs: 129 active+clean; 444 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 241 op/s 2024-04-26T21:44:15.118 INFO:tasks.workunit.client.0.smithi001.stdout: 10 4347 32.49 MB/sec warmup 12 sec latency 10845.153 ms 2024-04-26T21:44:16.118 INFO:tasks.workunit.client.0.smithi001.stdout: 10 4874 32.50 MB/sec warmup 13 sec latency 227.095 ms 2024-04-26T21:44:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:16 smithi186 ceph-mon[32991]: pgmap v437: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-04-26T21:44:16.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:16 smithi001 ceph-mon[28054]: pgmap v437: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-04-26T21:44:16.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:16 smithi038 ceph-mon[32724]: pgmap v437: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-04-26T21:44:17.118 INFO:tasks.workunit.client.0.smithi001.stdout: 10 5443 31.40 MB/sec warmup 14 sec latency 1227.183 ms 2024-04-26T21:44:18.118 INFO:tasks.workunit.client.0.smithi001.stdout: 10 5972 32.57 MB/sec warmup 15 sec latency 2227.269 ms 2024-04-26T21:44:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:18 smithi186 ceph-mon[32991]: pgmap v438: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-04-26T21:44:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:18 smithi001 ceph-mon[28054]: pgmap v438: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-04-26T21:44:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:18 smithi038 ceph-mon[32724]: pgmap v438: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-04-26T21:44:19.119 INFO:tasks.workunit.client.0.smithi001.stdout: 10 6503 33.32 MB/sec warmup 16 sec latency 3227.400 ms 2024-04-26T21:44:20.118 INFO:tasks.workunit.client.0.smithi001.stdout: 10 6692 31.97 MB/sec warmup 17 sec latency 4227.513 ms 2024-04-26T21:44:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:20 smithi186 ceph-mon[32991]: pgmap v439: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-04-26T21:44:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:20 smithi001 ceph-mon[28054]: pgmap v439: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-04-26T21:44:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:20 smithi038 ceph-mon[32724]: pgmap v439: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.4 MiB/s wr, 235 op/s 2024-04-26T21:44:21.118 INFO:tasks.workunit.client.0.smithi001.stdout: 10 6826 30.34 MB/sec warmup 18 sec latency 5227.597 ms 2024-04-26T21:44:21.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.363+0000 7fa896000640 1 -- 172.21.15.1:0/1743215740 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56043321ce00 con 0x560433339680 2024-04-26T21:44:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.363+0000 7fa896000640 10 client.26624 handle_command_reply: tid=0 2024-04-26T21:44:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.363+0000 7fa8b6228740 1 client.26624 shutdown 2024-04-26T21:44:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.364+0000 7fa8b6228740 1 -- 172.21.15.1:0/1743215740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 msgr2=0x560433337a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:21.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.364+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x560433337a10 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4000c00 tx=0x7fa8a4004160 comp rx=0 tx=0).stop 2024-04-26T21:44:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.365+0000 7fa8b6228740 1 -- 172.21.15.1:0/1743215740 shutdown_connections 2024-04-26T21:44:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.365+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x560433339680 0x56043333baa0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.365+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5604332bb1c0 0x560433337a10 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.365+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5604332b0900 0x5604332c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.365+0000 7fa8b6228740 1 --2- 172.21.15.1:0/1743215740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604332affe0 0x5604332c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.365+0000 7fa8b6228740 1 -- 172.21.15.1:0/1743215740 >> 172.21.15.1:0/1743215740 conn(0x5604332ac4f0 msgr2=0x5604332ab650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.365+0000 7fa896000640 0 client.26624 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.365+0000 7fa8b6228740 1 -- 172.21.15.1:0/1743215740 shutdown_connections 2024-04-26T21:44:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.366+0000 7fa8b6228740 1 -- 172.21.15.1:0/1743215740 wait complete. 2024-04-26T21:44:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.366+0000 7fa8b6228740 20 client.26624 trim_cache size 0 max 16384 2024-04-26T21:44:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:44:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:44:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:44:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:44:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2974698267 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa8900a3e00 msgr2=0x7fa8900a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa8900a3e00 0x7fa8900a62c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4009c00 tx=0x7fa8a4005f90 comp rx=0 tx=0).stop 2024-04-26T21:44:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2974698267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8ac105320 msgr2=0x7fa8ac19b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8ac105320 0x7fa8ac19b6a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa89800ea60 tx=0x7fa89800ef20 comp rx=0 tx=0).stop 2024-04-26T21:44:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2974698267 shutdown_connections 2024-04-26T21:44:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa8900a3e00 0x7fa8900a62c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8ac105320 0x7fa8ac19b6a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa8ac10a970 0x7fa8ac19fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 --2- 172.21.15.1:0/2974698267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8ac104960 0x7fa8ac19b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2974698267 >> 172.21.15.1:0/2974698267 conn(0x7fa8ac100430 msgr2=0x7fa8ac100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2974698267 shutdown_connections 2024-04-26T21:44:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:21.368+0000 7fa8b1e00640 1 -- 172.21.15.1:0/2974698267 wait complete. 2024-04-26T21:44:21.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:21 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:44:21.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:21 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:44:21.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:21 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:44:22.118 INFO:tasks.workunit.client.0.smithi001.stdout: 10 6954 28.82 MB/sec warmup 19 sec latency 5602.409 ms 2024-04-26T21:44:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:22 smithi186 ceph-mon[32991]: pgmap v440: 129 pgs: 129 active+clean; 467 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 275 op/s 2024-04-26T21:44:22.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:22 smithi038 ceph-mon[32724]: pgmap v440: 129 pgs: 129 active+clean; 467 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 275 op/s 2024-04-26T21:44:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:22 smithi001 ceph-mon[28054]: pgmap v440: 129 pgs: 129 active+clean; 467 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.0 MiB/s wr, 275 op/s 2024-04-26T21:44:23.119 INFO:tasks.workunit.client.0.smithi001.stdout: 10 7136 27.61 MB/sec warmup 20 sec latency 338.544 ms 2024-04-26T21:44:24.119 INFO:tasks.workunit.client.0.smithi001.stdout: 10 7525 27.26 MB/sec warmup 21 sec latency 134.322 ms 2024-04-26T21:44:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:24 smithi186 ceph-mon[32991]: pgmap v441: 129 pgs: 129 active+clean; 467 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 165 op/s 2024-04-26T21:44:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:24 smithi001 ceph-mon[28054]: pgmap v441: 129 pgs: 129 active+clean; 467 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 165 op/s 2024-04-26T21:44:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:24 smithi038 ceph-mon[32724]: pgmap v441: 129 pgs: 129 active+clean; 467 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 165 op/s 2024-04-26T21:44:25.119 INFO:tasks.workunit.client.0.smithi001.stdout: 10 8005 27.40 MB/sec warmup 22 sec latency 79.667 ms 2024-04-26T21:44:26.119 INFO:tasks.workunit.client.0.smithi001.stdout: 10 8693 27.05 MB/sec warmup 23 sec latency 85.185 ms 2024-04-26T21:44:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:26 smithi186 ceph-mon[32991]: pgmap v442: 129 pgs: 129 active+clean; 506 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.2 MiB/s wr, 273 op/s 2024-04-26T21:44:26.571 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:26 smithi001 ceph-mon[28054]: pgmap v442: 129 pgs: 129 active+clean; 506 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.2 MiB/s wr, 273 op/s 2024-04-26T21:44:26.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:26 smithi038 ceph-mon[32724]: pgmap v442: 129 pgs: 129 active+clean; 506 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 7.2 MiB/s wr, 273 op/s 2024-04-26T21:44:27.119 INFO:tasks.workunit.client.0.smithi001.stdout: 10 8801 26.21 MB/sec warmup 24 sec latency 514.398 ms 2024-04-26T21:44:28.119 INFO:tasks.workunit.client.0.smithi001.stdout: 10 8834 25.20 MB/sec warmup 25 sec latency 596.571 ms 2024-04-26T21:44:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:28 smithi186 ceph-mon[32991]: pgmap v443: 129 pgs: 129 active+clean; 506 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.4 MiB/s wr, 185 op/s 2024-04-26T21:44:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:28 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:44:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:28 smithi001 ceph-mon[28054]: pgmap v443: 129 pgs: 129 active+clean; 506 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.4 MiB/s wr, 185 op/s 2024-04-26T21:44:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:28 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:44:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:28 smithi038 ceph-mon[32724]: pgmap v443: 129 pgs: 129 active+clean; 506 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.4 MiB/s wr, 185 op/s 2024-04-26T21:44:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:28 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:44:29.119 INFO:tasks.workunit.client.0.smithi001.stdout: 10 9165 25.60 MB/sec warmup 26 sec latency 657.137 ms 2024-04-26T21:44:30.119 INFO:tasks.workunit.client.0.smithi001.stdout: 10 9505 25.31 MB/sec warmup 27 sec latency 134.007 ms 2024-04-26T21:44:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:30 smithi186 ceph-mon[32991]: pgmap v444: 129 pgs: 129 active+clean; 506 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.4 MiB/s wr, 185 op/s 2024-04-26T21:44:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:30 smithi001 ceph-mon[28054]: pgmap v444: 129 pgs: 129 active+clean; 506 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.4 MiB/s wr, 185 op/s 2024-04-26T21:44:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:30 smithi038 ceph-mon[32724]: pgmap v444: 129 pgs: 129 active+clean; 506 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.4 MiB/s wr, 185 op/s 2024-04-26T21:44:31.120 INFO:tasks.workunit.client.0.smithi001.stdout: 10 9892 25.67 MB/sec warmup 28 sec latency 116.249 ms 2024-04-26T21:44:32.120 INFO:tasks.workunit.client.0.smithi001.stdout: 10 10119 25.22 MB/sec warmup 29 sec latency 184.000 ms 2024-04-26T21:44:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:32 smithi186 ceph-mon[32991]: pgmap v445: 129 pgs: 129 active+clean; 694 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 25 MiB/s wr, 414 op/s 2024-04-26T21:44:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:32 smithi001 ceph-mon[28054]: pgmap v445: 129 pgs: 129 active+clean; 694 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 25 MiB/s wr, 414 op/s 2024-04-26T21:44:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:32 smithi038 ceph-mon[32724]: pgmap v445: 129 pgs: 129 active+clean; 694 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 25 MiB/s wr, 414 op/s 2024-04-26T21:44:33.120 INFO:tasks.workunit.client.0.smithi001.stdout: 10 10462 24.65 MB/sec warmup 30 sec latency 64.510 ms 2024-04-26T21:44:34.120 INFO:tasks.workunit.client.0.smithi001.stdout: 10 10999 24.65 MB/sec warmup 31 sec latency 151.134 ms 2024-04-26T21:44:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:34 smithi001 ceph-mon[28054]: pgmap v446: 129 pgs: 129 active+clean; 694 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 24 MiB/s wr, 336 op/s 2024-04-26T21:44:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.568+0000 7f8c5be00640 1 -- 172.21.15.1:0/2530654517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c54074d00 msgr2=0x7f8c540750e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.568+0000 7f8c5be00640 1 --2- 172.21.15.1:0/2530654517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c54074d00 0x7f8c540750e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009900 tx=0x7f8c4802a480 comp rx=0 tx=0).stop 2024-04-26T21:44:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.568+0000 7f8c5be00640 1 -- 172.21.15.1:0/2530654517 shutdown_connections 2024-04-26T21:44:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.568+0000 7f8c5be00640 1 --2- 172.21.15.1:0/2530654517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c5410f1c0 0x7f8c541115d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.568+0000 7f8c5be00640 1 --2- 172.21.15.1:0/2530654517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c54075620 0x7f8c5410ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.568+0000 7f8c5be00640 1 --2- 172.21.15.1:0/2530654517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c54074d00 0x7f8c540750e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.568+0000 7f8c5be00640 1 -- 172.21.15.1:0/2530654517 >> 172.21.15.1:0/2530654517 conn(0x7f8c540fe3f0 msgr2=0x7f8c54100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:34.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.568+0000 7f8c5be00640 1 -- 172.21.15.1:0/2530654517 shutdown_connections 2024-04-26T21:44:34.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.569+0000 7f8c5be00640 1 -- 172.21.15.1:0/2530654517 wait complete. 2024-04-26T21:44:34.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.569+0000 7f8c5be00640 1 Processor -- start 2024-04-26T21:44:34.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.570+0000 7f8c5be00640 1 -- start start 2024-04-26T21:44:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.571+0000 7f8c5be00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c54074d00 0x7f8c54074180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.571+0000 7f8c5be00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c54075620 0x7f8c5406d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.571+0000 7f8c5be00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c5410f1c0 0x7f8c5406d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.571+0000 7f8c5be00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8c54114e20 con 0x7f8c54075620 2024-04-26T21:44:34.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.571+0000 7f8c5be00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8c54114ca0 con 0x7f8c54074d00 2024-04-26T21:44:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.571+0000 7f8c5be00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c54114fa0 con 0x7f8c5410f1c0 2024-04-26T21:44:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.571+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c54075620 0x7f8c5406d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.572+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c54075620 0x7f8c5406d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40218/0 (socket says 172.21.15.1:40218) 2024-04-26T21:44:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.572+0000 7f8c58a00640 1 -- 172.21.15.1:0/318835313 learned_addr learned my addr 172.21.15.1:0/318835313 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.572+0000 7f8c59400640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c54074d00 0x7f8c54074180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.572 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.572+0000 7f8c59e00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c5410f1c0 0x7f8c5406d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.573+0000 7f8c59e00640 1 -- 172.21.15.1:0/318835313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c54074d00 msgr2=0x7f8c54074180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.573+0000 7f8c59e00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c54074d00 0x7f8c54074180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.573+0000 7f8c59e00640 1 -- 172.21.15.1:0/318835313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c54075620 msgr2=0x7f8c5406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.573+0000 7f8c59e00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c54075620 0x7f8c5406d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.573+0000 7f8c59e00640 1 -- 172.21.15.1:0/318835313 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5406dd80 con 0x7f8c5410f1c0 2024-04-26T21:44:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.573+0000 7f8c58a00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c54075620 0x7f8c5406d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:44:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.573+0000 7f8c59e00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c5410f1c0 0x7f8c5406d740 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8c54075c50 tx=0x7f8c4c00cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.573+0000 7f8c52000640 1 -- 172.21.15.1:0/318835313 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c00ada0 con 0x7f8c5410f1c0 2024-04-26T21:44:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.573+0000 7f8c52000640 1 -- 172.21.15.1:0/318835313 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c4c00f900 con 0x7f8c5410f1c0 2024-04-26T21:44:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.573+0000 7f8c52000640 1 -- 172.21.15.1:0/318835313 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c0189b0 con 0x7f8c5410f1c0 2024-04-26T21:44:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.574+0000 7f8c5be00640 1 -- 172.21.15.1:0/318835313 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c54071ca0 con 0x7f8c5410f1c0 2024-04-26T21:44:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.574+0000 7f8c5be00640 1 -- 172.21.15.1:0/318835313 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c540721a0 con 0x7f8c5410f1c0 2024-04-26T21:44:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:34 smithi038 ceph-mon[32724]: pgmap v446: 129 pgs: 129 active+clean; 694 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 24 MiB/s wr, 336 op/s 2024-04-26T21:44:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.578+0000 7f8c52000640 1 -- 172.21.15.1:0/318835313 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f8c4c00fab0 con 0x7f8c5410f1c0 2024-04-26T21:44:34.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.581+0000 7f8c5ec10740 1 Processor -- start 2024-04-26T21:44:34.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.582+0000 7f8c5ec10740 1 -- start start 2024-04-26T21:44:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.582+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0955e0 0x5601ab0959c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.582+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 0x5601ab0a0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.583+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0955e0 0x5601ab0959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.583+0000 7f8c59400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 0x5601ab0a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.583+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0955e0 0x5601ab0959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40230/0 (socket says 172.21.15.1:40230) 2024-04-26T21:44:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.583+0000 7f8c59400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 0x5601ab0a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:45328/0 (socket says 172.21.15.1:45328) 2024-04-26T21:44:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.585+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0a07c0 0x5601ab0a2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.585+0000 7f8c5ec10740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5601aaff2380 con 0x5601ab0955e0 2024-04-26T21:44:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.585+0000 7f8c5ec10740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5601aaf8ec30 con 0x5601ab095f00 2024-04-26T21:44:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.585+0000 7f8c5ec10740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5601aafee340 con 0x5601ab0a07c0 2024-04-26T21:44:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.585+0000 7f8c58a00640 1 -- 172.21.15.1:0/536259716 learned_addr learned my addr 172.21.15.1:0/536259716 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.585+0000 7f8c52000640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8c3c0a3de0 0x7f8c3c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.586+0000 7f8c52000640 1 -- 172.21.15.1:0/318835313 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9642+0+0 (secure 0 0 0) 0x7f8c4c0d0e00 con 0x7f8c5410f1c0 2024-04-26T21:44:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.586+0000 7f8c58a00640 1 --2- 172.21.15.1:0/536259716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0a07c0 0x5601ab0a2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.586+0000 7f8c59e00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8c3c0a3de0 0x7f8c3c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.587+0000 7f8c58a00640 1 -- 172.21.15.1:0/536259716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0a07c0 msgr2=0x5601ab0a2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.587+0000 7f8c58a00640 1 --2- 172.21.15.1:0/536259716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0a07c0 0x5601ab0a2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.587+0000 7f8c58a00640 1 -- 172.21.15.1:0/536259716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 msgr2=0x5601ab0a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.587+0000 7f8c58a00640 1 --2- 172.21.15.1:0/536259716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 0x5601ab0a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.587+0000 7f8c58a00640 1 -- 172.21.15.1:0/536259716 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601aafb0940 con 0x5601ab0955e0 2024-04-26T21:44:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.587+0000 7f8c59400640 1 --2- 172.21.15.1:0/536259716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 0x5601ab0a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:44:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.588+0000 7f8c59e00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8c3c0a3de0 0x7f8c3c0a62a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c099310 tx=0x7f8c4c00af40 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.588+0000 7f8c58a00640 1 --2- 172.21.15.1:0/536259716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0955e0 0x5601ab0959c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000af90 tx=0x7f8c4000aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b10b0b0bd78acb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.588+0000 7f8c46600640 1 -- 172.21.15.1:0/536259716 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4000dcd0 con 0x5601ab0955e0 2024-04-26T21:44:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.589+0000 7f8c46600640 1 -- 172.21.15.1:0/536259716 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c4000f8d0 con 0x5601ab0955e0 2024-04-26T21:44:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.589+0000 7f8c5ec10740 1 -- 172.21.15.1:0/536259716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0955e0 msgr2=0x5601ab0959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.589+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/536259716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0955e0 0x5601ab0959c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000af90 tx=0x7f8c4000aac0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.590+0000 7f8c5ec10740 1 -- 172.21.15.1:0/536259716 shutdown_connections 2024-04-26T21:44:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.590+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/536259716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0a07c0 0x5601ab0a2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.590+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/536259716 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 0x5601ab0a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.590+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/536259716 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0955e0 0x5601ab0959c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.590+0000 7f8c5ec10740 1 -- 172.21.15.1:0/536259716 >> 172.21.15.1:0/536259716 conn(0x5601ab091af0 msgr2=0x5601ab091ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.590+0000 7f8c5ec10740 1 -- 172.21.15.1:0/536259716 shutdown_connections 2024-04-26T21:44:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.590+0000 7f8c5ec10740 1 -- 172.21.15.1:0/536259716 wait complete. 2024-04-26T21:44:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.591+0000 7f8c5ec10740 1 -- start start 2024-04-26T21:44:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.591+0000 7f8c5ec10740 1 Processor -- start 2024-04-26T21:44:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.592+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0955e0 0x5601ab0a0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.592+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0955e0 0x5601ab0a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.592+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0955e0 0x5601ab0a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:45334/0 (socket says 172.21.15.1:45334) 2024-04-26T21:44:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.592+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab095f00 0x5601ab09d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c58a00640 1 -- 172.21.15.1:0/2083087314 learned_addr learned my addr 172.21.15.1:0/2083087314 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 0x5601ab09dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5601ab0a33f0 con 0x5601ab0a07c0 2024-04-26T21:44:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5601aaf79c60 con 0x5601ab0955e0 2024-04-26T21:44:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5601ab0a3570 con 0x5601ab095f00 2024-04-26T21:44:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c5ec10740 10 client.0 _ll_register_callbacks cb 0x5601aafa5e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:44:34.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c59400640 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab095f00 0x5601ab09d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c58a00640 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 0x5601ab09dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c59400640 1 -- 172.21.15.1:0/2083087314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0955e0 msgr2=0x5601ab0a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c59400640 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0955e0 0x5601ab0a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c59400640 1 -- 172.21.15.1:0/2083087314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 msgr2=0x5601ab09dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c59400640 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 0x5601ab09dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.593+0000 7f8c59400640 1 -- 172.21.15.1:0/2083087314 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601aaf88510 con 0x5601ab095f00 2024-04-26T21:44:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.594+0000 7f8c58a00640 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 0x5601ab09dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:44:34.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.594+0000 7f8c59400640 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab095f00 0x5601ab09d610 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8c4802aa70 tx=0x7f8c48003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=353701cb87d26135 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.594+0000 7f8c3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:44:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.594+0000 7f8c3aa00640 1 -- 172.21.15.1:0/2083087314 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c48035040 con 0x5601ab095f00 2024-04-26T21:44:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.594+0000 7f8c3aa00640 1 -- 172.21.15.1:0/2083087314 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9642+0+0 (secure 0 0 0) 0x7f8c48003da0 con 0x5601ab095f00 2024-04-26T21:44:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.594+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c4000f8d0 con 0x5601ab095f00 2024-04-26T21:44:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.595+0000 7f8c3aa00640 1 -- 172.21.15.1:0/2083087314 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8c48005020 con 0x5601ab095f00 2024-04-26T21:44:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.595+0000 7f8c5ec10740 10 client.26605 fetch_fsmap learned FSMap version 44 2024-04-26T21:44:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.595+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5601aad04210 con 0x5601ab095f00 2024-04-26T21:44:34.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.595+0000 7f8c3aa00640 1 -- 172.21.15.1:0/2083087314 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c4000f8d0 con 0x5601ab095f00 2024-04-26T21:44:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.595+0000 7f8c3aa00640 1 -- 172.21.15.1:0/2083087314 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7f8c480314a0 con 0x5601ab095f00 2024-04-26T21:44:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.596+0000 7f8c5ec10740 10 client.26605 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T21:44:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.596+0000 7f8c5ec10740 10 client.26605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:44:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.596+0000 7f8c5ec10740 20 client.26605 populate_metadata read hostname 'smithi001' 2024-04-26T21:44:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.596+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5601ab0b0480 0x5601ab0b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.596+0000 7f8c5ec10740 4 client.26605 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:44:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.596+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601aaffff90 con 0x5601ab0b0480 2024-04-26T21:44:34.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.597+0000 7f8c58a00640 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5601ab0b0480 0x5601ab0b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.598+0000 7f8c3aa00640 10 client.26605 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:34.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.598+0000 7f8c58a00640 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5601ab0b0480 0x5601ab0b28b0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f5274a49f88439b server_cookie=194224cc25f3a42d in_seq=0 out_seq=0 2024-04-26T21:44:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.601+0000 7f8c3aa00640 1 -- 172.21.15.1:0/2083087314 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5601aaffff90 con 0x5601ab0b0480 2024-04-26T21:44:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.601+0000 7f8c3aa00640 10 client.26605 handle_command_reply: tid=0 2024-04-26T21:44:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.601+0000 7f8c5ec10740 1 client.26605 shutdown 2024-04-26T21:44:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.601+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab095f00 msgr2=0x5601ab09d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.601+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab095f00 0x5601ab09d610 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8c4802aa70 tx=0x7f8c48003820 comp rx=0 tx=0).stop 2024-04-26T21:44:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.601+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 shutdown_connections 2024-04-26T21:44:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.601+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5601ab0b0480 0x5601ab0b28b0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.602+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab095f00 0x5601ab09d610 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.602+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 0x5601ab09dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.602+0000 7f8c3aa00640 0 client.26605 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.602+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/2083087314 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0955e0 0x5601ab0a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.602+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 >> 172.21.15.1:0/2083087314 conn(0x5601ab091af0 msgr2=0x5601ab097870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.602+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 shutdown_connections 2024-04-26T21:44:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.602+0000 7f8c5ec10740 1 -- 172.21.15.1:0/2083087314 wait complete. 2024-04-26T21:44:34.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.602+0000 7f8c5ec10740 20 client.26605 trim_cache size 0 max 16384 2024-04-26T21:44:34.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.623+0000 7f8c5ec10740 1 Processor -- start 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.623+0000 7f8c5ec10740 1 -- start start 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.623+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 0x5601ab097fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.623+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab095f00 0x5601ab0b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.623+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0a07c0 0x5601ab0b8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.623+0000 7f8c5ec10740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8c541b8e70 con 0x5601ab095f00 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.623+0000 7f8c5ec10740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8c541b6680 con 0x5601ab0a07c0 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.623+0000 7f8c5ec10740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c541b44f0 con 0x5601ab0955e0 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.624+0000 7f8c59400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab095f00 0x5601ab0b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.624+0000 7f8c59400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab095f00 0x5601ab0b81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40250/0 (socket says 172.21.15.1:40250) 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.624+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 0x5601ab097fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.624+0000 7f8c59400640 1 -- 172.21.15.1:0/1645228232 learned_addr learned my addr 172.21.15.1:0/1645228232 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.624+0000 7f8c58a00640 1 --2- 172.21.15.1:0/1645228232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0a07c0 0x5601ab0b8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.624+0000 7f8c59400640 1 -- 172.21.15.1:0/1645228232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 msgr2=0x5601ab097fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.624+0000 7f8c59400640 1 --2- 172.21.15.1:0/1645228232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 0x5601ab097fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.624+0000 7f8c59400640 1 -- 172.21.15.1:0/1645228232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0a07c0 msgr2=0x5601ab0b8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.624+0000 7f8c59400640 1 --2- 172.21.15.1:0/1645228232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0a07c0 0x5601ab0b8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.624+0000 7f8c59400640 1 -- 172.21.15.1:0/1645228232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601aadd0620 con 0x5601ab095f00 2024-04-26T21:44:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.625+0000 7f8c59400640 1 --2- 172.21.15.1:0/1645228232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab095f00 0x5601ab0b81c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009850 tx=0x7f8c48004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ef36ead0402f398 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.625+0000 7f8c45c00640 1 -- 172.21.15.1:0/1645228232 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c48035030 con 0x5601ab095f00 2024-04-26T21:44:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.625+0000 7f8c45c00640 1 -- 172.21.15.1:0/1645228232 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c480314a0 con 0x5601ab095f00 2024-04-26T21:44:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.625+0000 7f8c45c00640 1 -- 172.21.15.1:0/1645228232 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c48005400 con 0x5601ab095f00 2024-04-26T21:44:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.626+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1645228232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab095f00 msgr2=0x5601ab0b81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:34.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.626+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/1645228232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab095f00 0x5601ab0b81c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009850 tx=0x7f8c48004610 comp rx=0 tx=0).stop 2024-04-26T21:44:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.627+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1645228232 shutdown_connections 2024-04-26T21:44:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.627+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/1645228232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab0a07c0 0x5601ab0b8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.627+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/1645228232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab095f00 0x5601ab0b81c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.627+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/1645228232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 0x5601ab097fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.627+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1645228232 >> 172.21.15.1:0/1645228232 conn(0x5601ab091af0 msgr2=0x5601ab098a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.627+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1645228232 shutdown_connections 2024-04-26T21:44:34.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.627+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1645228232 wait complete. 2024-04-26T21:44:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.627+0000 7f8c5ec10740 1 -- start start 2024-04-26T21:44:34.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.628+0000 7f8c5ec10740 1 Processor -- start 2024-04-26T21:44:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.629+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 0x5601ab1124a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 0x5601ab0a8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c5ec10740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 0x5601ab10f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c5ec10740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8c541b4b40 con 0x5601ab0a07c0 2024-04-26T21:44:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c5ec10740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f8c541b5fb0 con 0x5601ab095f00 2024-04-26T21:44:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c5ec10740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c541b5210 con 0x5601ab0955e0 2024-04-26T21:44:34.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c5ec10740 10 client.0 _ll_register_callbacks cb 0x7f8c48004100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 0x5601ab1124a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c58a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 0x5601ab1124a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:47770/0 (socket says 172.21.15.1:47770) 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c58a00640 1 -- 172.21.15.1:0/1698859159 learned_addr learned my addr 172.21.15.1:0/1698859159 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c59400640 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 0x5601ab0a8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c58a00640 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 0x5601ab10f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c58a00640 1 -- 172.21.15.1:0/1698859159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 msgr2=0x5601ab0a8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c58a00640 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 0x5601ab0a8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c58a00640 1 -- 172.21.15.1:0/1698859159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 msgr2=0x5601ab10f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c58a00640 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 0x5601ab10f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.630+0000 7f8c58a00640 1 -- 172.21.15.1:0/1698859159 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601aafaccb0 con 0x5601ab0955e0 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.631+0000 7f8c58a00640 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 0x5601ab1124a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000a610 tx=0x7f8c4001d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5616395241fc57e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.631+0000 7f8c3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.631+0000 7f8c3aa00640 1 -- 172.21.15.1:0/1698859159 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4000dea0 con 0x5601ab0955e0 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.631+0000 7f8c3aa00640 1 -- 172.21.15.1:0/1698859159 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9642+0+0 (secure 0 0 0) 0x7f8c40013020 con 0x5601ab0955e0 2024-04-26T21:44:34.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.631+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1698859159 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c480314a0 con 0x5601ab0955e0 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c3aa00640 1 -- 172.21.15.1:0/1698859159 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8c4003b050 con 0x5601ab0955e0 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c5ec10740 10 client.26611 fetch_fsmap learned FSMap version 44 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1698859159 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5601ab08d240 con 0x5601ab0955e0 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c3aa00640 1 -- 172.21.15.1:0/1698859159 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c480314a0 con 0x5601ab0955e0 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c3aa00640 1 -- 172.21.15.1:0/1698859159 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7f8c40028530 con 0x5601ab0955e0 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c5ec10740 10 client.26611 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c5ec10740 10 client.26611 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c5ec10740 20 client.26611 populate_metadata read hostname 'smithi001' 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5601ab08d6c0 0x5601ab08fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c5ec10740 4 client.26611 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:44:34.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.632+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1698859159 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5601aaff9e00 con 0x5601ab08d6c0 2024-04-26T21:44:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.633+0000 7f8c59400640 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5601ab08d6c0 0x5601ab08fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.633+0000 7f8c59400640 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5601ab08d6c0 0x5601ab08fae0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2fd9347bfadd3ba server_cookie=9b6bc80614859059 in_seq=0 out_seq=0 2024-04-26T21:44:34.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.633+0000 7f8c3aa00640 10 client.26611 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:34 smithi186 ceph-mon[32991]: pgmap v446: 129 pgs: 129 active+clean; 694 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 24 MiB/s wr, 336 op/s 2024-04-26T21:44:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c3aa00640 1 -- 172.21.15.1:0/1698859159 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5601aaff9e00 con 0x5601ab08d6c0 2024-04-26T21:44:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c3aa00640 10 client.26611 handle_command_reply: tid=0 2024-04-26T21:44:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 client.26611 shutdown 2024-04-26T21:44:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1698859159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 msgr2=0x5601ab1124a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 0x5601ab1124a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000a610 tx=0x7f8c4001d040 comp rx=0 tx=0).stop 2024-04-26T21:44:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1698859159 shutdown_connections 2024-04-26T21:44:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5601ab08d6c0 0x5601ab08fae0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5601ab0a07c0 0x5601ab10f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601ab095f00 0x5601ab0a8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c3aa00640 0 client.26611 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 --2- 172.21.15.1:0/1698859159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5601ab0955e0 0x5601ab1124a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1698859159 >> 172.21.15.1:0/1698859159 conn(0x5601ab091af0 msgr2=0x5601ab087bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1698859159 shutdown_connections 2024-04-26T21:44:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 1 -- 172.21.15.1:0/1698859159 wait complete. 2024-04-26T21:44:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.936+0000 7f8c5ec10740 20 client.26611 trim_cache size 0 max 16384 2024-04-26T21:44:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:44:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:44:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:44:34.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 -- 172.21.15.1:0/318835313 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8c3c0a3de0 msgr2=0x7f8c3c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8c3c0a3de0 0x7f8c3c0a62a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c099310 tx=0x7f8c4c00af40 comp rx=0 tx=0).stop 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 -- 172.21.15.1:0/318835313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c5410f1c0 msgr2=0x7f8c5406d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c5410f1c0 0x7f8c5406d740 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8c54075c50 tx=0x7f8c4c00cf10 comp rx=0 tx=0).stop 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 -- 172.21.15.1:0/318835313 shutdown_connections 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f8c3c0a3de0 0x7f8c3c0a62a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c5410f1c0 0x7f8c5406d740 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c54075620 0x7f8c5406d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 --2- 172.21.15.1:0/318835313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8c54074d00 0x7f8c54074180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 -- 172.21.15.1:0/318835313 >> 172.21.15.1:0/318835313 conn(0x7f8c540fe3f0 msgr2=0x7f8c540ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:34.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 -- 172.21.15.1:0/318835313 shutdown_connections 2024-04-26T21:44:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:34.937+0000 7f8c5be00640 1 -- 172.21.15.1:0/318835313 wait complete. 2024-04-26T21:44:35.120 INFO:tasks.workunit.client.0.smithi001.stdout: 10 11695 24.96 MB/sec warmup 32 sec latency 183.694 ms 2024-04-26T21:44:36.120 INFO:tasks.workunit.client.0.smithi001.stdout: 10 12499 25.35 MB/sec warmup 33 sec latency 194.418 ms 2024-04-26T21:44:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:36 smithi001 ceph-mon[28054]: pgmap v447: 129 pgs: 129 active+clean; 693 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 25 MiB/s wr, 508 op/s 2024-04-26T21:44:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:36 smithi038 ceph-mon[32724]: pgmap v447: 129 pgs: 129 active+clean; 693 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 25 MiB/s wr, 508 op/s 2024-04-26T21:44:36.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:36 smithi186 ceph-mon[32991]: pgmap v447: 129 pgs: 129 active+clean; 693 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 25 MiB/s wr, 508 op/s 2024-04-26T21:44:37.120 INFO:tasks.workunit.client.0.smithi001.stdout: 10 13201 26.30 MB/sec warmup 34 sec latency 53.822 ms 2024-04-26T21:44:38.120 INFO:tasks.workunit.client.0.smithi001.stdout: 10 13632 26.21 MB/sec warmup 35 sec latency 62.294 ms 2024-04-26T21:44:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:38 smithi001 ceph-mon[28054]: pgmap v448: 129 pgs: 129 active+clean; 693 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 21 MiB/s wr, 400 op/s 2024-04-26T21:44:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:38 smithi038 ceph-mon[32724]: pgmap v448: 129 pgs: 129 active+clean; 693 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 21 MiB/s wr, 400 op/s 2024-04-26T21:44:38.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:38 smithi186 ceph-mon[32991]: pgmap v448: 129 pgs: 129 active+clean; 693 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 21 MiB/s wr, 400 op/s 2024-04-26T21:44:39.120 INFO:tasks.workunit.client.0.smithi001.stdout: 10 14035 25.90 MB/sec warmup 36 sec latency 64.127 ms 2024-04-26T21:44:40.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 14732 25.98 MB/sec warmup 37 sec latency 102.953 ms 2024-04-26T21:44:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:40 smithi001 ceph-mon[28054]: pgmap v449: 129 pgs: 129 active+clean; 693 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 21 MiB/s wr, 400 op/s 2024-04-26T21:44:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:40 smithi038 ceph-mon[32724]: pgmap v449: 129 pgs: 129 active+clean; 693 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 21 MiB/s wr, 400 op/s 2024-04-26T21:44:40.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:40 smithi186 ceph-mon[32991]: pgmap v449: 129 pgs: 129 active+clean; 693 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 21 MiB/s wr, 400 op/s 2024-04-26T21:44:41.120 INFO:tasks.workunit.client.0.smithi001.stdout: 10 15499 26.34 MB/sec warmup 38 sec latency 71.469 ms 2024-04-26T21:44:42.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 16260 27.05 MB/sec warmup 39 sec latency 67.553 ms 2024-04-26T21:44:42.505 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:42 smithi186 ceph-mon[32991]: pgmap v450: 129 pgs: 129 active+clean; 709 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 23 MiB/s wr, 636 op/s 2024-04-26T21:44:42.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:42 smithi001 ceph-mon[28054]: pgmap v450: 129 pgs: 129 active+clean; 709 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 23 MiB/s wr, 636 op/s 2024-04-26T21:44:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:42 smithi038 ceph-mon[32724]: pgmap v450: 129 pgs: 129 active+clean; 709 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 23 MiB/s wr, 636 op/s 2024-04-26T21:44:43.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 16791 27.05 MB/sec warmup 40 sec latency 65.356 ms 2024-04-26T21:44:44.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 17361 27.22 MB/sec warmup 41 sec latency 55.942 ms 2024-04-26T21:44:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:44 smithi001 ceph-mon[28054]: pgmap v451: 129 pgs: 129 active+clean; 709 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.6 MiB/s wr, 407 op/s 2024-04-26T21:44:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:44 smithi038 ceph-mon[32724]: pgmap v451: 129 pgs: 129 active+clean; 709 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.6 MiB/s wr, 407 op/s 2024-04-26T21:44:44.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:44 smithi186 ceph-mon[32991]: pgmap v451: 129 pgs: 129 active+clean; 709 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.6 MiB/s wr, 407 op/s 2024-04-26T21:44:45.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 18059 27.27 MB/sec warmup 42 sec latency 66.272 ms 2024-04-26T21:44:46.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 18802 27.55 MB/sec warmup 43 sec latency 44.723 ms 2024-04-26T21:44:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:46 smithi001 ceph-mon[28054]: pgmap v452: 129 pgs: 129 active+clean; 730 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 581 op/s 2024-04-26T21:44:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:46 smithi038 ceph-mon[32724]: pgmap v452: 129 pgs: 129 active+clean; 730 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 581 op/s 2024-04-26T21:44:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:46 smithi186 ceph-mon[32991]: pgmap v452: 129 pgs: 129 active+clean; 730 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 581 op/s 2024-04-26T21:44:47.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 19426 27.91 MB/sec warmup 44 sec latency 91.075 ms 2024-04-26T21:44:48.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 19904 27.70 MB/sec warmup 45 sec latency 117.848 ms 2024-04-26T21:44:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.149+0000 7f1eef000640 1 -- 172.21.15.1:0/1808487143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee0098b10 msgr2=0x7f1ee009cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.149+0000 7f1eef000640 1 --2- 172.21.15.1:0/1808487143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee0098b10 0x7f1ee009cb50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1ee80680a0 tx=0x7f1ee8097360 comp rx=0 tx=0).stop 2024-04-26T21:44:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.150+0000 7f1eef000640 1 -- 172.21.15.1:0/1808487143 shutdown_connections 2024-04-26T21:44:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.150+0000 7f1eef000640 1 --2- 172.21.15.1:0/1808487143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ee009d230 0x7f1ee00a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.150+0000 7f1eef000640 1 --2- 172.21.15.1:0/1808487143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee0098b10 0x7f1ee009cb50 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.150+0000 7f1eef000640 1 --2- 172.21.15.1:0/1808487143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ee0098160 0x7f1ee0098540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.150+0000 7f1eef000640 1 -- 172.21.15.1:0/1808487143 >> 172.21.15.1:0/1808487143 conn(0x7f1ee00939c0 msgr2=0x7f1ee0095de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.151+0000 7f1eef000640 1 -- 172.21.15.1:0/1808487143 shutdown_connections 2024-04-26T21:44:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.151+0000 7f1eef000640 1 -- 172.21.15.1:0/1808487143 wait complete. 2024-04-26T21:44:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.153+0000 7f1eef000640 1 Processor -- start 2024-04-26T21:44:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.153+0000 7f1eef000640 1 -- start start 2024-04-26T21:44:48.153 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.153+0000 7f1eef000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee0098160 0x7f1ee00a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.153+0000 7f1eef000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ee009d230 0x7f1ee00a24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.154+0000 7f1eef000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ee00a2a00 0x7f1ee013eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.154+0000 7f1eef000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee00a81a0 con 0x7f1ee009d230 2024-04-26T21:44:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.154+0000 7f1eef000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee00a8020 con 0x7f1ee0098160 2024-04-26T21:44:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.154+0000 7f1eef000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee00a8320 con 0x7f1ee00a2a00 2024-04-26T21:44:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.154+0000 7f1eedc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee0098160 0x7f1ee00a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.155 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.154+0000 7f1eedc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee0098160 0x7f1ee00a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39446/0 (socket says 172.21.15.1:39446) 2024-04-26T21:44:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.154+0000 7f1eedc00640 1 -- 172.21.15.1:0/734795825 learned_addr learned my addr 172.21.15.1:0/734795825 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.154+0000 7f1eee600640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ee00a2a00 0x7f1ee013eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.155+0000 7f1eedc00640 1 -- 172.21.15.1:0/734795825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ee00a2a00 msgr2=0x7f1ee013eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.155+0000 7f1eedc00640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ee00a2a00 0x7f1ee013eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.155+0000 7f1eedc00640 1 -- 172.21.15.1:0/734795825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ee009d230 msgr2=0x7f1ee00a24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.155+0000 7f1eedc00640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ee009d230 0x7f1ee00a24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.155+0000 7f1eedc00640 1 -- 172.21.15.1:0/734795825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee013f420 con 0x7f1ee0098160 2024-04-26T21:44:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.155+0000 7f1eedc00640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee0098160 0x7f1ee00a1f80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00b450 tx=0x7f1edc00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:48.156 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.156+0000 7f1ee6000640 1 -- 172.21.15.1:0/734795825 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc015020 con 0x7f1ee0098160 2024-04-26T21:44:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.156+0000 7f1ee6000640 1 -- 172.21.15.1:0/734795825 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1edc011d40 con 0x7f1ee0098160 2024-04-26T21:44:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.156+0000 7f1ee6000640 1 -- 172.21.15.1:0/734795825 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc00f340 con 0x7f1ee0098160 2024-04-26T21:44:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.156+0000 7f1eef000640 1 -- 172.21.15.1:0/734795825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ee013f650 con 0x7f1ee0098160 2024-04-26T21:44:48.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.156+0000 7f1eef000640 1 -- 172.21.15.1:0/734795825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ee013fac0 con 0x7f1ee0098160 2024-04-26T21:44:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.163+0000 7f1ee6000640 1 -- 172.21.15.1:0/734795825 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f1edc018020 con 0x7f1ee0098160 2024-04-26T21:44:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.166+0000 7f1ee6000640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1ed00a3da0 0x7f1ed00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.166+0000 7f1ee6000640 1 -- 172.21.15.1:0/734795825 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9642+0+0 (secure 0 0 0) 0x7f1edc0d06a0 con 0x7f1ee0098160 2024-04-26T21:44:48.168 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.168+0000 7f1ef3419740 1 Processor -- start 2024-04-26T21:44:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.172+0000 7f1ef3419740 1 -- start start 2024-04-26T21:44:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.172+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c8158fc0 0x5563c81593a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.172+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81598e0 0x5563c8163c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.172+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81641a0 0x5563c8166590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.172+0000 7f1ef3419740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5563c80be380 con 0x5563c8158fc0 2024-04-26T21:44:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.172+0000 7f1ef3419740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5563c805ac30 con 0x5563c81641a0 2024-04-26T21:44:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.172+0000 7f1ef3419740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5563c80ba340 con 0x5563c81598e0 2024-04-26T21:44:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.173+0000 7f1eee600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81641a0 0x5563c8166590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.173+0000 7f1eed200640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1ed00a3da0 0x7f1ed00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.175+0000 7f1eedc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c8158fc0 0x5563c81593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.175+0000 7f1eed200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81598e0 0x5563c8163c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.176+0000 7f1eed200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81598e0 0x5563c8163c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:48066/0 (socket says 172.21.15.1:48066) 2024-04-26T21:44:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.176+0000 7f1eed200640 1 -- 172.21.15.1:0/301002112 learned_addr learned my addr 172.21.15.1:0/301002112 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.176+0000 7f1eed200640 1 -- 172.21.15.1:0/301002112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81641a0 msgr2=0x5563c8166590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.176+0000 7f1eed200640 1 --2- 172.21.15.1:0/301002112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81641a0 0x5563c8166590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.177+0000 7f1eed200640 1 -- 172.21.15.1:0/301002112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c8158fc0 msgr2=0x5563c81593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.177+0000 7f1eed200640 1 --2- 172.21.15.1:0/301002112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c8158fc0 0x5563c81593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.178+0000 7f1eed200640 1 -- 172.21.15.1:0/301002112 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563c807c940 con 0x5563c81598e0 2024-04-26T21:44:48.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.178+0000 7f1eed200640 1 --2- 172.21.15.1:0/301002112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81598e0 0x5563c8163c60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8051930 tx=0x7f1ee80af000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86d985214a02b467 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:48.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.179+0000 7f1edaa00640 1 -- 172.21.15.1:0/301002112 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee80a0470 con 0x5563c81598e0 2024-04-26T21:44:48.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.179+0000 7f1eed200640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1ed00a3da0 0x7f1ed00a6260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8052070 tx=0x7f1ee8068bf0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:48.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.179+0000 7f1edaa00640 1 -- 172.21.15.1:0/301002112 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ee809b040 con 0x5563c81598e0 2024-04-26T21:44:48.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.179+0000 7f1edaa00640 1 -- 172.21.15.1:0/301002112 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee80a0470 con 0x5563c81598e0 2024-04-26T21:44:48.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.180+0000 7f1ef3419740 1 -- 172.21.15.1:0/301002112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81598e0 msgr2=0x5563c8163c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:48.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.180+0000 7f1ef3419740 1 --2- 172.21.15.1:0/301002112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81598e0 0x5563c8163c60 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8051930 tx=0x7f1ee80af000 comp rx=0 tx=0).stop 2024-04-26T21:44:48.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.182+0000 7f1ef3419740 1 -- 172.21.15.1:0/301002112 shutdown_connections 2024-04-26T21:44:48.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.182+0000 7f1ef3419740 1 --2- 172.21.15.1:0/301002112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81598e0 0x5563c8163c60 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.182+0000 7f1ef3419740 1 --2- 172.21.15.1:0/301002112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81641a0 0x5563c8166590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.182+0000 7f1ef3419740 1 --2- 172.21.15.1:0/301002112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c8158fc0 0x5563c81593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.182+0000 7f1ef3419740 1 -- 172.21.15.1:0/301002112 >> 172.21.15.1:0/301002112 conn(0x5563c81554d0 msgr2=0x5563c81558b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:48.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.183+0000 7f1ef3419740 1 -- 172.21.15.1:0/301002112 shutdown_connections 2024-04-26T21:44:48.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.183+0000 7f1ef3419740 1 -- 172.21.15.1:0/301002112 wait complete. 2024-04-26T21:44:48.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.184+0000 7f1ef3419740 1 -- start start 2024-04-26T21:44:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.185+0000 7f1ef3419740 1 Processor -- start 2024-04-26T21:44:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.186+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c8158fc0 0x5563c8161140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.186+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81598e0 0x5563c8161680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.187+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c8161bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.187+0000 7f1ef3419740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5563c8166d70 con 0x5563c8158fc0 2024-04-26T21:44:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.187+0000 7f1ef3419740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5563c8045c60 con 0x5563c81598e0 2024-04-26T21:44:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.187+0000 7f1ef3419740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5563c8166ef0 con 0x5563c81641a0 2024-04-26T21:44:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.187+0000 7f1ef3419740 10 client.0 _ll_register_callbacks cb 0x5563c8071e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:44:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.188+0000 7f1eed200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81598e0 0x5563c8161680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.188+0000 7f1eed200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81598e0 0x5563c8161680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39468/0 (socket says 172.21.15.1:39468) 2024-04-26T21:44:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.188+0000 7f1eed200640 1 -- 172.21.15.1:0/40152752 learned_addr learned my addr 172.21.15.1:0/40152752 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.188+0000 7f1eee600640 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c8161bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.189+0000 7f1eedc00640 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c8158fc0 0x5563c8161140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.191+0000 7f1eed200640 1 -- 172.21.15.1:0/40152752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 msgr2=0x5563c8161bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.191+0000 7f1eed200640 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c8161bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.191+0000 7f1eed200640 1 -- 172.21.15.1:0/40152752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c8158fc0 msgr2=0x5563c8161140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.191+0000 7f1eed200640 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c8158fc0 0x5563c8161140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.191+0000 7f1eed200640 1 -- 172.21.15.1:0/40152752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563c8054510 con 0x5563c81598e0 2024-04-26T21:44:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.192+0000 7f1eee600640 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c8161bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:44:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.193+0000 7f1eed200640 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81598e0 0x5563c8161680 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8051c10 tx=0x7f1ee806b190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfb2e74d1616a8cb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.193+0000 7f1ecf400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:44:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.193+0000 7f1ecf400640 1 -- 172.21.15.1:0/40152752 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8069ab0 con 0x5563c81598e0 2024-04-26T21:44:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.194+0000 7f1ecf400640 1 -- 172.21.15.1:0/40152752 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9642+0+0 (secure 0 0 0) 0x7f1ee8069c50 con 0x5563c81598e0 2024-04-26T21:44:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.195+0000 7f1ef3419740 1 -- 172.21.15.1:0/40152752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ee809b040 con 0x5563c81598e0 2024-04-26T21:44:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.195+0000 7f1ecf400640 1 -- 172.21.15.1:0/40152752 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1ee8069e50 con 0x5563c81598e0 2024-04-26T21:44:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.195+0000 7f1ef3419740 10 client.26651 fetch_fsmap learned FSMap version 44 2024-04-26T21:44:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.196+0000 7f1ef3419740 1 -- 172.21.15.1:0/40152752 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5563c7dd0210 con 0x5563c81598e0 2024-04-26T21:44:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.196+0000 7f1ecf400640 1 -- 172.21.15.1:0/40152752 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ee809b040 con 0x5563c81598e0 2024-04-26T21:44:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.196+0000 7f1ecf400640 1 -- 172.21.15.1:0/40152752 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7f1ee80c9030 con 0x5563c81598e0 2024-04-26T21:44:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.197+0000 7f1ef3419740 10 client.26651 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T21:44:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.197+0000 7f1ef3419740 10 client.26651 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:44:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.197+0000 7f1ef3419740 20 client.26651 populate_metadata read hostname 'smithi001' 2024-04-26T21:44:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.197+0000 7f1ef3419740 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5563c8173b80 0x5563c8175f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.197+0000 7f1ef3419740 4 client.26651 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:44:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.197+0000 7f1ef3419740 1 -- 172.21.15.1:0/40152752 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563c80cbf90 con 0x5563c8173b80 2024-04-26T21:44:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.198+0000 7f1eedc00640 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5563c8173b80 0x5563c8175f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.198+0000 7f1ecf400640 10 client.26651 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.202+0000 7f1eedc00640 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5563c8173b80 0x5563c8175f70 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92e9018c332778b4 server_cookie=dd592473996f5912 in_seq=0 out_seq=0 2024-04-26T21:44:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.202+0000 7f1ecf400640 1 -- 172.21.15.1:0/40152752 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563c80cbf90 con 0x5563c8173b80 2024-04-26T21:44:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.202+0000 7f1ecf400640 10 client.26651 handle_command_reply: tid=0 2024-04-26T21:44:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.203+0000 7f1ef3419740 1 client.26651 shutdown 2024-04-26T21:44:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.204+0000 7f1ef3419740 1 -- 172.21.15.1:0/40152752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81598e0 msgr2=0x5563c8161680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.204+0000 7f1ef3419740 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81598e0 0x5563c8161680 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8051c10 tx=0x7f1ee806b190 comp rx=0 tx=0).stop 2024-04-26T21:44:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.204+0000 7f1ef3419740 1 -- 172.21.15.1:0/40152752 shutdown_connections 2024-04-26T21:44:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.204+0000 7f1ef3419740 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5563c8173b80 0x5563c8175f70 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.204+0000 7f1ef3419740 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c8161bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.204+0000 7f1ef3419740 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81598e0 0x5563c8161680 secure :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8051c10 tx=0x7f1ee806b190 comp rx=0 tx=0).stop 2024-04-26T21:44:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.204+0000 7f1ef3419740 1 --2- 172.21.15.1:0/40152752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c8158fc0 0x5563c8161140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.204+0000 7f1ef3419740 1 -- 172.21.15.1:0/40152752 >> 172.21.15.1:0/40152752 conn(0x5563c81554d0 msgr2=0x5563c815b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.204+0000 7f1ecf400640 0 client.26651 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.206+0000 7f1ef3419740 1 -- 172.21.15.1:0/40152752 shutdown_connections 2024-04-26T21:44:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.206+0000 7f1ef3419740 1 -- 172.21.15.1:0/40152752 wait complete. 2024-04-26T21:44:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.206+0000 7f1ef3419740 20 client.26651 trim_cache size 0 max 16384 2024-04-26T21:44:48.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.242+0000 7f1ef3419740 1 Processor -- start 2024-04-26T21:44:48.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.242+0000 7f1ef3419740 1 -- start start 2024-04-26T21:44:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.243+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c8158fc0 0x5563c816fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.243+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c81641a0 0x5563c8162f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.243+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81634b0 0x5563c817b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.243+0000 7f1ef3419740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee014c370 con 0x5563c81641a0 2024-04-26T21:44:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.243+0000 7f1ef3419740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee0149b80 con 0x5563c81634b0 2024-04-26T21:44:48.243 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.243+0000 7f1ef3419740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee01479f0 con 0x5563c8158fc0 2024-04-26T21:44:48.245 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.244+0000 7f1eee600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81634b0 0x5563c817b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.245 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.244+0000 7f1eee600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81634b0 0x5563c817b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39478/0 (socket says 172.21.15.1:39478) 2024-04-26T21:44:48.245 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.245+0000 7f1eee600640 1 -- 172.21.15.1:0/1193837792 learned_addr learned my addr 172.21.15.1:0/1193837792 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:48.245 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.245+0000 7f1eedc00640 1 --2- 172.21.15.1:0/1193837792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c8158fc0 0x5563c816fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.245+0000 7f1eedc00640 1 -- 172.21.15.1:0/1193837792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81634b0 msgr2=0x5563c817b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.246+0000 7f1eedc00640 1 --2- 172.21.15.1:0/1193837792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81634b0 0x5563c817b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.246+0000 7f1eedc00640 1 -- 172.21.15.1:0/1193837792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c81641a0 msgr2=0x5563c8162f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.246+0000 7f1eed200640 1 --2- 172.21.15.1:0/1193837792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c81641a0 0x5563c8162f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.246+0000 7f1eedc00640 1 --2- 172.21.15.1:0/1193837792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c81641a0 0x5563c8162f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.246+0000 7f1eedc00640 1 -- 172.21.15.1:0/1193837792 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563c81600f0 con 0x5563c8158fc0 2024-04-26T21:44:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.247+0000 7f1eed200640 1 --2- 172.21.15.1:0/1193837792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c81641a0 0x5563c8162f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:44:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.247+0000 7f1eedc00640 1 --2- 172.21.15.1:0/1193837792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c8158fc0 0x5563c816fc60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0cdaa0 tx=0x7f1edc010970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ce541758ac66888 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.248+0000 7f1eda000640 1 -- 172.21.15.1:0/1193837792 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc096e60 con 0x5563c8158fc0 2024-04-26T21:44:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.249+0000 7f1eda000640 1 -- 172.21.15.1:0/1193837792 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1edc0a4f60 con 0x5563c8158fc0 2024-04-26T21:44:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.249+0000 7f1eda000640 1 -- 172.21.15.1:0/1193837792 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc098960 con 0x5563c8158fc0 2024-04-26T21:44:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.250+0000 7f1ef3419740 1 -- 172.21.15.1:0/1193837792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c8158fc0 msgr2=0x5563c816fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.250+0000 7f1ef3419740 1 --2- 172.21.15.1:0/1193837792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c8158fc0 0x5563c816fc60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0cdaa0 tx=0x7f1edc010970 comp rx=0 tx=0).stop 2024-04-26T21:44:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.250+0000 7f1ef3419740 1 -- 172.21.15.1:0/1193837792 shutdown_connections 2024-04-26T21:44:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.250+0000 7f1ef3419740 1 --2- 172.21.15.1:0/1193837792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c81634b0 0x5563c817b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.250+0000 7f1ef3419740 1 --2- 172.21.15.1:0/1193837792 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c81641a0 0x5563c8162f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.250+0000 7f1ef3419740 1 --2- 172.21.15.1:0/1193837792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c8158fc0 0x5563c816fc60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.250+0000 7f1ef3419740 1 -- 172.21.15.1:0/1193837792 >> 172.21.15.1:0/1193837792 conn(0x5563c81554d0 msgr2=0x5563c8162860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.252+0000 7f1ef3419740 1 -- 172.21.15.1:0/1193837792 shutdown_connections 2024-04-26T21:44:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.252+0000 7f1ef3419740 1 -- 172.21.15.1:0/1193837792 wait complete. 2024-04-26T21:44:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.253+0000 7f1ef3419740 1 -- start start 2024-04-26T21:44:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.253+0000 7f1ef3419740 1 Processor -- start 2024-04-26T21:44:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.254+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c8158fc0 0x5563c815df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.255+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c81634b0 0x5563c815e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.255+0000 7f1ef3419740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c815e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.257+0000 7f1eee600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c815e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.258+0000 7f1eee600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c815e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:48090/0 (socket says 172.21.15.1:48090) 2024-04-26T21:44:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.258+0000 7f1eee600640 1 -- 172.21.15.1:0/1540093114 learned_addr learned my addr 172.21.15.1:0/1540093114 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:44:48.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.257+0000 7f1eedc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c8158fc0 0x5563c815df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.258+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee0148040 con 0x5563c81634b0 2024-04-26T21:44:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.258+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee01494b0 con 0x5563c8158fc0 2024-04-26T21:44:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.258+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1ee0148710 con 0x5563c81641a0 2024-04-26T21:44:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.259+0000 7f1ef3419740 10 client.0 _ll_register_callbacks cb 0x7f1ed400b160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:44:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.259+0000 7f1eee600640 1 -- 172.21.15.1:0/1540093114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c8158fc0 msgr2=0x5563c815df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.259+0000 7f1eee600640 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c8158fc0 0x5563c815df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.259+0000 7f1eee600640 1 -- 172.21.15.1:0/1540093114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c81634b0 msgr2=0x5563c815e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:44:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.259+0000 7f1eee600640 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c81634b0 0x5563c815e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:48.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.259+0000 7f1eee600640 1 -- 172.21.15.1:0/1540093114 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563c7e9c620 con 0x5563c81641a0 2024-04-26T21:44:48.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.260+0000 7f1eee600640 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c815e9a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4016b60 tx=0x7f1ed4014780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8cbe0961edf6e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:44:48.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.260+0000 7f1ecf400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:44:48.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.260+0000 7f1ecf400640 1 -- 172.21.15.1:0/1540093114 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4023060 con 0x5563c81641a0 2024-04-26T21:44:48.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.261+0000 7f1ecf400640 1 -- 172.21.15.1:0/1540093114 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9642+0+0 (secure 0 0 0) 0x7f1ed40135b0 con 0x5563c81641a0 2024-04-26T21:44:48.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.261+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1edc0a4f60 con 0x5563c81641a0 2024-04-26T21:44:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.263+0000 7f1ecf400640 1 -- 172.21.15.1:0/1540093114 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1ed40137f0 con 0x5563c81641a0 2024-04-26T21:44:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.263+0000 7f1ef3419740 10 client.26626 fetch_fsmap learned FSMap version 44 2024-04-26T21:44:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.263+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5563c8078cb0 con 0x5563c81641a0 2024-04-26T21:44:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.264+0000 7f1ecf400640 1 -- 172.21.15.1:0/1540093114 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1edc0a4f60 con 0x5563c81641a0 2024-04-26T21:44:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.265+0000 7f1ecf400640 1 -- 172.21.15.1:0/1540093114 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7f1ed401d030 con 0x5563c81641a0 2024-04-26T21:44:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.265+0000 7f1ef3419740 10 client.26626 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T21:44:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.265+0000 7f1ef3419740 10 client.26626 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:44:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.265+0000 7f1ef3419740 20 client.26626 populate_metadata read hostname 'smithi001' 2024-04-26T21:44:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.265+0000 7f1ef3419740 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5563c81da500 0x5563c81dc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:44:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.265+0000 7f1ef3419740 4 client.26626 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:44:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.265+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5563c80c5e00 con 0x5563c81da500 2024-04-26T21:44:48.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.266+0000 7f1eedc00640 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5563c81da500 0x5563c81dc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:44:48.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.266+0000 7f1eedc00640 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5563c81da500 0x5563c81dc960 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b08be2671d4ce90e server_cookie=756d5945e64c60ee in_seq=0 out_seq=0 2024-04-26T21:44:48.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:48.266+0000 7f1ecf400640 10 client.26626 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:48 smithi001 ceph-mon[28054]: pgmap v453: 129 pgs: 129 active+clean; 730 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-04-26T21:44:48.604 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:48 smithi038 ceph-mon[32724]: pgmap v453: 129 pgs: 129 active+clean; 730 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-04-26T21:44:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:48 smithi186 ceph-mon[32991]: pgmap v453: 129 pgs: 129 active+clean; 730 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-04-26T21:44:49.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 20020 27.15 MB/sec warmup 46 sec latency 501.598 ms 2024-04-26T21:44:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.201+0000 7f1ecf400640 1 -- 172.21.15.1:0/1540093114 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5563c80c5e00 con 0x5563c81da500 2024-04-26T21:44:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.201+0000 7f1ecf400640 10 client.26626 handle_command_reply: tid=0 2024-04-26T21:44:49.201 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.201+0000 7f1ef3419740 1 client.26626 shutdown 2024-04-26T21:44:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.202+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 msgr2=0x5563c815e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.202+0000 7f1ef3419740 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c815e9a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4016b60 tx=0x7f1ed4014780 comp rx=0 tx=0).stop 2024-04-26T21:44:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.202+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 shutdown_connections 2024-04-26T21:44:49.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.202+0000 7f1ef3419740 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5563c81da500 0x5563c81dc960 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.202+0000 7f1ef3419740 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5563c81641a0 0x5563c815e9a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.202+0000 7f1ef3419740 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5563c81634b0 0x5563c815e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.202+0000 7f1ef3419740 1 --2- 172.21.15.1:0/1540093114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5563c8158fc0 0x5563c815df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.202+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 >> 172.21.15.1:0/1540093114 conn(0x5563c81554d0 msgr2=0x5563c8153da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.202+0000 7f1ecf400640 0 client.26626 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:44:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.202+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 shutdown_connections 2024-04-26T21:44:49.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.203+0000 7f1ef3419740 1 -- 172.21.15.1:0/1540093114 wait complete. 2024-04-26T21:44:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.203+0000 7f1ef3419740 20 client.26626 trim_cache size 0 max 16384 2024-04-26T21:44:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:44:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:44:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:44:49.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.204+0000 7f1eef000640 1 -- 172.21.15.1:0/734795825 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1ed00a3da0 msgr2=0x7f1ed00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.204+0000 7f1eef000640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1ed00a3da0 0x7f1ed00a6260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8052070 tx=0x7f1ee8068bf0 comp rx=0 tx=0).stop 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.204+0000 7f1eef000640 1 -- 172.21.15.1:0/734795825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee0098160 msgr2=0x7f1ee00a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.204+0000 7f1eef000640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee0098160 0x7f1ee00a1f80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00b450 tx=0x7f1edc00b910 comp rx=0 tx=0).stop 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.204+0000 7f1eef000640 1 -- 172.21.15.1:0/734795825 shutdown_connections 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.204+0000 7f1eef000640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f1ed00a3da0 0x7f1ed00a6260 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.204+0000 7f1eef000640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ee00a2a00 0x7f1ee013eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.204+0000 7f1eef000640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1ee009d230 0x7f1ee00a24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.204+0000 7f1eef000640 1 --2- 172.21.15.1:0/734795825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ee0098160 0x7f1ee00a1f80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.204+0000 7f1eef000640 1 -- 172.21.15.1:0/734795825 >> 172.21.15.1:0/734795825 conn(0x7f1ee00939c0 msgr2=0x7f1ee0094d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:44:49.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.205+0000 7f1eef000640 1 -- 172.21.15.1:0/734795825 shutdown_connections 2024-04-26T21:44:49.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:44:49.205+0000 7f1eef000640 1 -- 172.21.15.1:0/734795825 wait complete. 2024-04-26T21:44:50.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 20435 26.89 MB/sec warmup 47 sec latency 103.490 ms 2024-04-26T21:44:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:50 smithi001 ceph-mon[28054]: pgmap v454: 129 pgs: 129 active+clean; 730 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-04-26T21:44:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:50 smithi038 ceph-mon[32724]: pgmap v454: 129 pgs: 129 active+clean; 730 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-04-26T21:44:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:50 smithi186 ceph-mon[32991]: pgmap v454: 129 pgs: 129 active+clean; 730 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-04-26T21:44:51.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 21327 27.42 MB/sec warmup 48 sec latency 59.652 ms 2024-04-26T21:44:52.121 INFO:tasks.workunit.client.0.smithi001.stdout: 10 22161 27.96 MB/sec warmup 49 sec latency 52.959 ms 2024-04-26T21:44:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:52 smithi001 ceph-mon[28054]: pgmap v455: 129 pgs: 129 active+clean; 746 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.0 MiB/s wr, 709 op/s 2024-04-26T21:44:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:52 smithi038 ceph-mon[32724]: pgmap v455: 129 pgs: 129 active+clean; 746 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.0 MiB/s wr, 709 op/s 2024-04-26T21:44:52.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:52 smithi186 ceph-mon[32991]: pgmap v455: 129 pgs: 129 active+clean; 746 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.0 MiB/s wr, 709 op/s 2024-04-26T21:44:53.122 INFO:tasks.workunit.client.0.smithi001.stdout: 10 22676 28.13 MB/sec warmup 50 sec latency 44.847 ms 2024-04-26T21:44:54.125 INFO:tasks.workunit.client.0.smithi001.stdout: 10 22709 27.58 MB/sec warmup 51 sec latency 657.285 ms 2024-04-26T21:44:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:54 smithi001 ceph-mon[28054]: pgmap v456: 129 pgs: 129 active+clean; 746 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 6.1 MiB/s wr, 472 op/s 2024-04-26T21:44:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:54 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:44:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:54 smithi038 ceph-mon[32724]: pgmap v456: 129 pgs: 129 active+clean; 746 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 6.1 MiB/s wr, 472 op/s 2024-04-26T21:44:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:54 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:44:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:54 smithi186 ceph-mon[32991]: pgmap v456: 129 pgs: 129 active+clean; 746 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 6.1 MiB/s wr, 472 op/s 2024-04-26T21:44:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:54 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:44:55.123 INFO:tasks.workunit.client.0.smithi001.stdout: 10 22785 27.13 MB/sec warmup 52 sec latency 828.322 ms 2024-04-26T21:44:56.123 INFO:tasks.workunit.client.0.smithi001.stdout: 10 22969 26.72 MB/sec warmup 53 sec latency 324.131 ms 2024-04-26T21:44:56.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:56 smithi001 ceph-mon[28054]: pgmap v457: 129 pgs: 129 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 726 op/s 2024-04-26T21:44:56.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:56 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:44:56.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:56 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:44:56.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:56 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:44:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:56 smithi038 ceph-mon[32724]: pgmap v457: 129 pgs: 129 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 726 op/s 2024-04-26T21:44:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:56 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:44:56.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:56 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:44:56.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:56 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:44:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:56 smithi186 ceph-mon[32991]: pgmap v457: 129 pgs: 129 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 726 op/s 2024-04-26T21:44:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:56 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:44:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:56 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:44:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:56 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:44:57.123 INFO:tasks.workunit.client.0.smithi001.stdout: 10 23258 26.46 MB/sec warmup 54 sec latency 99.374 ms 2024-04-26T21:44:58.123 INFO:tasks.workunit.client.0.smithi001.stdout: 10 23995 26.51 MB/sec warmup 55 sec latency 70.595 ms 2024-04-26T21:44:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:44:58 smithi001 ceph-mon[28054]: pgmap v458: 129 pgs: 129 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 552 op/s 2024-04-26T21:44:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:44:58 smithi038 ceph-mon[32724]: pgmap v458: 129 pgs: 129 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 552 op/s 2024-04-26T21:44:58.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:44:58 smithi186 ceph-mon[32991]: pgmap v458: 129 pgs: 129 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 552 op/s 2024-04-26T21:44:59.123 INFO:tasks.workunit.client.0.smithi001.stdout: 10 24815 26.89 MB/sec warmup 56 sec latency 68.962 ms 2024-04-26T21:45:00.123 INFO:tasks.workunit.client.0.smithi001.stdout: 10 25600 27.26 MB/sec warmup 57 sec latency 101.342 ms 2024-04-26T21:45:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:00 smithi001 ceph-mon[28054]: pgmap v459: 129 pgs: 129 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 552 op/s 2024-04-26T21:45:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:00 smithi038 ceph-mon[32724]: pgmap v459: 129 pgs: 129 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 552 op/s 2024-04-26T21:45:00.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:00 smithi186 ceph-mon[32991]: pgmap v459: 129 pgs: 129 active+clean; 836 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 552 op/s 2024-04-26T21:45:01.123 INFO:tasks.workunit.client.0.smithi001.stdout: 10 25876 27.17 MB/sec warmup 58 sec latency 522.577 ms 2024-04-26T21:45:02.123 INFO:tasks.workunit.client.0.smithi001.stdout: 10 25876 26.71 MB/sec warmup 59 sec latency 1522.677 ms 2024-04-26T21:45:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.423+0000 7f0cb4000640 1 -- 172.21.15.1:0/4252030481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cac113950 msgr2=0x7f0cac115d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:02.423 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.423+0000 7f0cb4000640 1 --2- 172.21.15.1:0/4252030481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cac113950 0x7f0cac115d40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009ca0 tx=0x7f0ca402a950 comp rx=0 tx=0).stop 2024-04-26T21:45:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.423+0000 7f0cb4000640 1 -- 172.21.15.1:0/4252030481 shutdown_connections 2024-04-26T21:45:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.423+0000 7f0cb4000640 1 --2- 172.21.15.1:0/4252030481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac0755f0 0x7f0cac113410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.423+0000 7f0cb4000640 1 --2- 172.21.15.1:0/4252030481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cac113950 0x7f0cac115d40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.423+0000 7f0cb4000640 1 --2- 172.21.15.1:0/4252030481 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cac074cd0 0x7f0cac0750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.423+0000 7f0cb4000640 1 -- 172.21.15.1:0/4252030481 >> 172.21.15.1:0/4252030481 conn(0x7f0cac0fe3f0 msgr2=0x7f0cac100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.425+0000 7f0cb4000640 1 -- 172.21.15.1:0/4252030481 shutdown_connections 2024-04-26T21:45:02.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.425+0000 7f0cb4000640 1 -- 172.21.15.1:0/4252030481 wait complete. 2024-04-26T21:45:02.426 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.426+0000 7f0cb4000640 1 Processor -- start 2024-04-26T21:45:02.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.427+0000 7f0cb4000640 1 -- start start 2024-04-26T21:45:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.427+0000 7f0cb4000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cac074cd0 0x7f0cac1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.427+0000 7f0cb4000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cac0755f0 0x7f0cac1a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.427+0000 7f0cb4000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac113950 0x7f0cac19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.427+0000 7f0cb4000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac119270 con 0x7f0cac074cd0 2024-04-26T21:45:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.427+0000 7f0cb4000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac1190f0 con 0x7f0cac0755f0 2024-04-26T21:45:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.427+0000 7f0cb4000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac1193f0 con 0x7f0cac113950 2024-04-26T21:45:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.427+0000 7f0cb2000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac113950 0x7f0cac19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb2000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac113950 0x7f0cac19f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:35884/0 (socket says 172.21.15.1:35884) 2024-04-26T21:45:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb2000640 1 -- 172.21.15.1:0/3246658655 learned_addr learned my addr 172.21.15.1:0/3246658655 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cac0755f0 0x7f0cac1a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb1600640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cac074cd0 0x7f0cac1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb0c00640 1 -- 172.21.15.1:0/3246658655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac113950 msgr2=0x7f0cac19f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:02.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac113950 0x7f0cac19f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb0c00640 1 -- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cac074cd0 msgr2=0x7f0cac1a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cac074cd0 0x7f0cac1a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb0c00640 1 -- 172.21.15.1:0/3246658655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac19fd20 con 0x7f0cac0755f0 2024-04-26T21:45:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb2000640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac113950 0x7f0cac19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:45:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.428+0000 7f0cb1600640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cac074cd0 0x7f0cac1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:45:02.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.429+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cac0755f0 0x7f0cac1a5960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0c9800b420 tx=0x7f0c9800b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.432+0000 7f0caa000640 1 -- 172.21.15.1:0/3246658655 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c98011830 con 0x7f0cac0755f0 2024-04-26T21:45:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.432+0000 7f0caa000640 1 -- 172.21.15.1:0/3246658655 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c98016070 con 0x7f0cac0755f0 2024-04-26T21:45:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.432+0000 7f0caa000640 1 -- 172.21.15.1:0/3246658655 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c98010dd0 con 0x7f0cac0755f0 2024-04-26T21:45:02.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.432+0000 7f0cb4000640 1 -- 172.21.15.1:0/3246658655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cac1a0010 con 0x7f0cac0755f0 2024-04-26T21:45:02.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.434+0000 7f0cb4000640 1 -- 172.21.15.1:0/3246658655 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cac106be0 con 0x7f0cac0755f0 2024-04-26T21:45:02.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.438+0000 7f0caa000640 1 -- 172.21.15.1:0/3246658655 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0c980119d0 con 0x7f0cac0755f0 2024-04-26T21:45:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.441+0000 7f0caa000640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0c940a3e60 0x7f0c940a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.441 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.441+0000 7f0caa000640 1 -- 172.21.15.1:0/3246658655 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 9642+0+0 (secure 0 0 0) 0x7f0c980cfb70 con 0x7f0cac0755f0 2024-04-26T21:45:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.442+0000 7f0cb6f7b740 1 Processor -- start 2024-04-26T21:45:02.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.442+0000 7f0cb6f7b740 1 -- start start 2024-04-26T21:45:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb1600640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0c940a3e60 0x7f0c940a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dcdd0 0x563e030dd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dd6f0 0x563e030e7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030ea3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x563e03042380 con 0x563e030dd6f0 2024-04-26T21:45:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563e02fdec30 con 0x563e030e7fb0 2024-04-26T21:45:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563e0303e340 con 0x563e030dcdd0 2024-04-26T21:45:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb0c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dd6f0 0x563e030e7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030ea3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb2000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030ea3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:39562/0 (socket says 172.21.15.1:39562) 2024-04-26T21:45:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.443+0000 7f0cb2000640 1 -- 172.21.15.1:0/3320317488 learned_addr learned my addr 172.21.15.1:0/3320317488 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.444+0000 7f0cb2000640 1 -- 172.21.15.1:0/3320317488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dcdd0 msgr2=0x563e030dd1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:45:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.444+0000 7f0cb2000640 1 --2- 172.21.15.1:0/3320317488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dcdd0 0x563e030dd1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.444+0000 7f0cb2000640 1 -- 172.21.15.1:0/3320317488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dd6f0 msgr2=0x563e030e7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:02.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.444+0000 7f0cb2000640 1 --2- 172.21.15.1:0/3320317488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dd6f0 0x563e030e7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.444+0000 7f0cb2000640 1 -- 172.21.15.1:0/3320317488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e03000940 con 0x563e030e7fb0 2024-04-26T21:45:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.444+0000 7f0cb2000640 1 --2- 172.21.15.1:0/3320317488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030ea3a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4003a20 tx=0x7f0ca4003a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af8074c8f6e75e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:02.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.445+0000 7f0c9e600640 1 -- 172.21.15.1:0/3320317488 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4048070 con 0x563e030e7fb0 2024-04-26T21:45:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.445+0000 7f0c9e600640 1 -- 172.21.15.1:0/3320317488 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca402fd20 con 0x563e030e7fb0 2024-04-26T21:45:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.445+0000 7f0c9e600640 1 -- 172.21.15.1:0/3320317488 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4034040 con 0x563e030e7fb0 2024-04-26T21:45:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.446+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/3320317488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 msgr2=0x563e030ea3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:02.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.446+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/3320317488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030ea3a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4003a20 tx=0x7f0ca4003a50 comp rx=0 tx=0).stop 2024-04-26T21:45:02.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.448+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/3320317488 shutdown_connections 2024-04-26T21:45:02.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.448+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/3320317488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030ea3a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.448+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/3320317488 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dd6f0 0x563e030e7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.448+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/3320317488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dcdd0 0x563e030dd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.448 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.448+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/3320317488 >> 172.21.15.1:0/3320317488 conn(0x563e030d92e0 msgr2=0x563e030d96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.449+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/3320317488 shutdown_connections 2024-04-26T21:45:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.449+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/3320317488 wait complete. 2024-04-26T21:45:02.449 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.449+0000 7f0cb1600640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0c940a3e60 0x7f0c940a6320 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0011500 tx=0x7f0ca0005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:02.450 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.450+0000 7f0cb6f7b740 1 -- start start 2024-04-26T21:45:02.451 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.451+0000 7f0cb6f7b740 1 Processor -- start 2024-04-26T21:45:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.452+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030e4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.452+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dd6f0 0x563e030e5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.452+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030e5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.452+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x563e030eab80 con 0x563e030dcdd0 2024-04-26T21:45:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.452+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563e02fc9c60 con 0x563e030e7fb0 2024-04-26T21:45:02.453 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.452+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563e030ead00 con 0x563e030dd6f0 2024-04-26T21:45:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.452+0000 7f0cb6f7b740 10 client.0 _ll_register_callbacks cb 0x563e02ff5e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:45:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.452+0000 7f0cb0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dd6f0 0x563e030e5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.453+0000 7f0cb0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dd6f0 0x563e030e5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:35900/0 (socket says 172.21.15.1:35900) 2024-04-26T21:45:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.453+0000 7f0cb0c00640 1 -- 172.21.15.1:0/1749708734 learned_addr learned my addr 172.21.15.1:0/1749708734 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.453+0000 7f0cb0c00640 1 -- 172.21.15.1:0/1749708734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 msgr2=0x563e030e5990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:45:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.454+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030e5990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.454+0000 7f0cb0c00640 1 -- 172.21.15.1:0/1749708734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 msgr2=0x563e030e4f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:45:02.455 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.455+0000 7f0cb1600640 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030e4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.456+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030e4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.456+0000 7f0cb0c00640 1 -- 172.21.15.1:0/1749708734 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e02fd8510 con 0x563e030dd6f0 2024-04-26T21:45:02.456 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.456+0000 7f0cb1600640 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030e4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:45:02.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.457+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dd6f0 0x563e030e5450 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0c980ccfd0 tx=0x7f0c9809bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be5de125915c5772 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:02.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.457+0000 7f0c92a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:45:02.457 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.457+0000 7f0c92a00640 1 -- 172.21.15.1:0/1749708734 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c980a4430 con 0x563e030dd6f0 2024-04-26T21:45:02.458 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.458+0000 7f0c92a00640 1 -- 172.21.15.1:0/1749708734 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9642+0+0 (secure 0 0 0) 0x7f0c9809ac10 con 0x563e030dd6f0 2024-04-26T21:45:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.458+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/1749708734 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ca402fd20 con 0x563e030dd6f0 2024-04-26T21:45:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.458+0000 7f0c92a00640 1 -- 172.21.15.1:0/1749708734 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f0c9809ae10 con 0x563e030dd6f0 2024-04-26T21:45:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.458+0000 7f0cb6f7b740 10 client.26632 fetch_fsmap learned FSMap version 44 2024-04-26T21:45:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.458+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/1749708734 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x563e02d54210 con 0x563e030dd6f0 2024-04-26T21:45:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.459+0000 7f0c92a00640 1 -- 172.21.15.1:0/1749708734 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca402fd20 con 0x563e030dd6f0 2024-04-26T21:45:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.459+0000 7f0c92a00640 1 -- 172.21.15.1:0/1749708734 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7f0c9809a680 con 0x563e030dd6f0 2024-04-26T21:45:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.459+0000 7f0cb6f7b740 10 client.26632 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T21:45:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.460+0000 7f0cb6f7b740 10 client.26632 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:45:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.460+0000 7f0cb6f7b740 20 client.26632 populate_metadata read hostname 'smithi001' 2024-04-26T21:45:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.460+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x563e03108c80 0x563e0310b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.460+0000 7f0cb6f7b740 4 client.26632 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:45:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.460+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/1749708734 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e0304ff90 con 0x563e03108c80 2024-04-26T21:45:02.461 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.461+0000 7f0cb1600640 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x563e03108c80 0x563e0310b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.464+0000 7f0cb1600640 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x563e03108c80 0x563e0310b0d0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c32e2e110cef5c9c server_cookie=1d3b6852598ebefc in_seq=0 out_seq=0 2024-04-26T21:45:02.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.465+0000 7f0c92a00640 10 client.26632 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:02.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.465+0000 7f0c92a00640 1 -- 172.21.15.1:0/1749708734 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e0304ff90 con 0x563e03108c80 2024-04-26T21:45:02.466 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.465+0000 7f0c92a00640 10 client.26632 handle_command_reply: tid=0 2024-04-26T21:45:02.466 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.465+0000 7f0cb6f7b740 1 client.26632 shutdown 2024-04-26T21:45:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.467+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/1749708734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dd6f0 msgr2=0x563e030e5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.467+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dd6f0 0x563e030e5450 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0c980ccfd0 tx=0x7f0c9809bec0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.468+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/1749708734 shutdown_connections 2024-04-26T21:45:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.468+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x563e03108c80 0x563e0310b0d0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.468+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030dd6f0 0x563e030e5450 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.468+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030e5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.468+0000 7f0c92a00640 0 client.26632 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.468+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/1749708734 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030e4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.468+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/1749708734 >> 172.21.15.1:0/1749708734 conn(0x563e030d92e0 msgr2=0x563e030df060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.468+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/1749708734 shutdown_connections 2024-04-26T21:45:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.469+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/1749708734 wait complete. 2024-04-26T21:45:02.469 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.469+0000 7f0cb6f7b740 20 client.26632 trim_cache size 0 max 16384 2024-04-26T21:45:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.502+0000 7f0cb6f7b740 1 Processor -- start 2024-04-26T21:45:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.503+0000 7f0cb6f7b740 1 -- start start 2024-04-26T21:45:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.503+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030d8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.503+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030e75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.503+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e031109e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.503+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac070780 con 0x563e030dcdd0 2024-04-26T21:45:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.503+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac06df90 con 0x563e030e7fb0 2024-04-26T21:45:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.503+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac10eea0 con 0x563e030df7e0 2024-04-26T21:45:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.503+0000 7f0cb2000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e031109e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.504+0000 7f0cb2000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e031109e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:35914/0 (socket says 172.21.15.1:35914) 2024-04-26T21:45:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.504+0000 7f0cb2000640 1 -- 172.21.15.1:0/2936584806 learned_addr learned my addr 172.21.15.1:0/2936584806 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.504+0000 7f0cb1600640 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030d8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.504+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030e75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.505+0000 7f0cb2000640 1 -- 172.21.15.1:0/2936584806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 msgr2=0x563e030e75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.505+0000 7f0cb2000640 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030e75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.505+0000 7f0cb2000640 1 -- 172.21.15.1:0/2936584806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 msgr2=0x563e030d8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.505+0000 7f0cb2000640 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030d8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.505+0000 7f0cb2000640 1 -- 172.21.15.1:0/2936584806 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e030ea710 con 0x563e030df7e0 2024-04-26T21:45:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.505+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030e75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:45:02.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.505+0000 7f0cb1600640 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030d8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:45:02.507 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.506+0000 7f0cb2000640 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e031109e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4004af0 tx=0x7f0ca402f7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f0d51af2aa887dd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.508+0000 7f0c9dc00640 1 -- 172.21.15.1:0/2936584806 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4051070 con 0x563e030df7e0 2024-04-26T21:45:02.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.508+0000 7f0c9dc00640 1 -- 172.21.15.1:0/2936584806 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca402fd20 con 0x563e030df7e0 2024-04-26T21:45:02.509 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.508+0000 7f0c9dc00640 1 -- 172.21.15.1:0/2936584806 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4048030 con 0x563e030df7e0 2024-04-26T21:45:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.509+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/2936584806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 msgr2=0x563e031109e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.509+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e031109e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4004af0 tx=0x7f0ca402f7b0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.510+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/2936584806 shutdown_connections 2024-04-26T21:45:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.510+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e031109e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.510+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030e75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.510+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/2936584806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030d8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.510+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/2936584806 >> 172.21.15.1:0/2936584806 conn(0x563e030d92e0 msgr2=0x563e030e6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.510+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/2936584806 shutdown_connections 2024-04-26T21:45:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.510+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/2936584806 wait complete. 2024-04-26T21:45:02.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.510+0000 7f0cb6f7b740 1 -- start start 2024-04-26T21:45:02.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.510+0000 7f0cb6f7b740 1 Processor -- start 2024-04-26T21:45:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.511+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030efec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.511+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e030f0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.511+0000 7f0cb6f7b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030ed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.511+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac06c470 con 0x563e030dcdd0 2024-04-26T21:45:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.511+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac06d8c0 con 0x563e030e7fb0 2024-04-26T21:45:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.511+0000 7f0cb6f7b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0cac06cb20 con 0x563e030df7e0 2024-04-26T21:45:02.512 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.512+0000 7f0cb6f7b740 10 client.0 _ll_register_callbacks cb 0x7f0c98097a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:45:02.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.512+0000 7f0cb0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e030f0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.512+0000 7f0cb0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e030f0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:35920/0 (socket says 172.21.15.1:35920) 2024-04-26T21:45:02.513 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.512+0000 7f0cb0c00640 1 -- 172.21.15.1:0/909255911 learned_addr learned my addr 172.21.15.1:0/909255911 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:02.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.513+0000 7f0cb2000640 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030ed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.513+0000 7f0cb0c00640 1 -- 172.21.15.1:0/909255911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 msgr2=0x563e030ed5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:02.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.513+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030ed5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.513+0000 7f0cb1600640 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030efec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.514 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.514+0000 7f0cb0c00640 1 -- 172.21.15.1:0/909255911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 msgr2=0x563e030efec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:02.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.514+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030efec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.514+0000 7f0cb0c00640 1 -- 172.21.15.1:0/909255911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e02e20620 con 0x563e030df7e0 2024-04-26T21:45:02.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.514+0000 7f0cb1600640 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:45:02.515 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.515+0000 7f0cb0c00640 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e030f0400 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0c98010560 tx=0x7f0c98098040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f20c2c2fb224dea5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.515+0000 7f0c92a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:45:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.515+0000 7f0c92a00640 1 -- 172.21.15.1:0/909255911 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9809b300 con 0x563e030df7e0 2024-04-26T21:45:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.515+0000 7f0c92a00640 1 -- 172.21.15.1:0/909255911 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 9642+0+0 (secure 0 0 0) 0x7f0c98097720 con 0x563e030df7e0 2024-04-26T21:45:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.515+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/909255911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ca402fd20 con 0x563e030df7e0 2024-04-26T21:45:02.516 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.516+0000 7f0c92a00640 1 -- 172.21.15.1:0/909255911 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f0c9809caf0 con 0x563e030df7e0 2024-04-26T21:45:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.516+0000 7f0cb6f7b740 10 client.26638 fetch_fsmap learned FSMap version 44 2024-04-26T21:45:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.516+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/909255911 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x563e02ffccb0 con 0x563e030df7e0 2024-04-26T21:45:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.517+0000 7f0c92a00640 1 -- 172.21.15.1:0/909255911 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ca402fd20 con 0x563e030df7e0 2024-04-26T21:45:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.517+0000 7f0c92a00640 1 -- 172.21.15.1:0/909255911 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7f0c980059c0 con 0x563e030df7e0 2024-04-26T21:45:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.517+0000 7f0cb6f7b740 10 client.26638 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T21:45:02.517 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.517+0000 7f0cb6f7b740 10 client.26638 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:45:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.517+0000 7f0cb6f7b740 20 client.26638 populate_metadata read hostname 'smithi001' 2024-04-26T21:45:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.517+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x563e03171180 0x563e031735a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.517+0000 7f0cb6f7b740 4 client.26638 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:45:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.517+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/909255911 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e03049e00 con 0x563e03171180 2024-04-26T21:45:02.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.518+0000 7f0cb1600640 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x563e03171180 0x563e031735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.519+0000 7f0cb1600640 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x563e03171180 0x563e031735a0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=299ec6a5c78cca54 server_cookie=badeb2fc79f9bf5c in_seq=0 out_seq=0 2024-04-26T21:45:02.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.519+0000 7f0c92a00640 10 client.26638 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:02 smithi001 ceph-mon[28054]: pgmap v460: 129 pgs: 129 active+clean; 769 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 15 MiB/s wr, 882 op/s 2024-04-26T21:45:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:02 smithi038 ceph-mon[32724]: pgmap v460: 129 pgs: 129 active+clean; 769 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 15 MiB/s wr, 882 op/s 2024-04-26T21:45:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:02 smithi186 ceph-mon[32991]: pgmap v460: 129 pgs: 129 active+clean; 769 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 15 MiB/s wr, 882 op/s 2024-04-26T21:45:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.865+0000 7f0c92a00640 1 -- 172.21.15.1:0/909255911 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563e03049e00 con 0x563e03171180 2024-04-26T21:45:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.865+0000 7f0c92a00640 10 client.26638 handle_command_reply: tid=0 2024-04-26T21:45:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.865+0000 7f0cb6f7b740 1 client.26638 shutdown 2024-04-26T21:45:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.865+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/909255911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 msgr2=0x563e030f0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.865+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e030f0400 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0c98010560 tx=0x7f0c98098040 comp rx=0 tx=0).stop 2024-04-26T21:45:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.866+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/909255911 shutdown_connections 2024-04-26T21:45:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.866+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x563e03171180 0x563e031735a0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.866+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e030df7e0 0x563e030f0400 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.866+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563e030e7fb0 0x563e030ed5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.866+0000 7f0cb6f7b740 1 --2- 172.21.15.1:0/909255911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563e030dcdd0 0x563e030efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.866+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/909255911 >> 172.21.15.1:0/909255911 conn(0x563e030d92e0 msgr2=0x563e030e65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.866+0000 7f0c92a00640 0 client.26638 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.866+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/909255911 shutdown_connections 2024-04-26T21:45:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.867+0000 7f0cb6f7b740 1 -- 172.21.15.1:0/909255911 wait complete. 2024-04-26T21:45:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.867+0000 7f0cb6f7b740 20 client.26638 trim_cache size 0 max 16384 2024-04-26T21:45:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:45:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:45:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:45:02.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:45:02.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.868+0000 7f0cb4000640 1 -- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0c940a3e60 msgr2=0x7f0c940a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.868+0000 7f0cb4000640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0c940a3e60 0x7f0c940a6320 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0011500 tx=0x7f0ca0005f90 comp rx=0 tx=0).stop 2024-04-26T21:45:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.868+0000 7f0cb4000640 1 -- 172.21.15.1:0/3246658655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cac0755f0 msgr2=0x7f0cac1a5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.868+0000 7f0cb4000640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cac0755f0 0x7f0cac1a5960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0c9800b420 tx=0x7f0c9800b8e0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.870+0000 7f0cb4000640 1 -- 172.21.15.1:0/3246658655 shutdown_connections 2024-04-26T21:45:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.870+0000 7f0cb4000640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0c940a3e60 0x7f0c940a6320 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.870+0000 7f0cb4000640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cac113950 0x7f0cac19f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.870+0000 7f0cb4000640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0cac0755f0 0x7f0cac1a5960 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.870+0000 7f0cb4000640 1 --2- 172.21.15.1:0/3246658655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cac074cd0 0x7f0cac1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.870+0000 7f0cb4000640 1 -- 172.21.15.1:0/3246658655 >> 172.21.15.1:0/3246658655 conn(0x7f0cac0fe3f0 msgr2=0x7f0cac0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.870+0000 7f0cb4000640 1 -- 172.21.15.1:0/3246658655 shutdown_connections 2024-04-26T21:45:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:02.870+0000 7f0cb4000640 1 -- 172.21.15.1:0/3246658655 wait complete. 2024-04-26T21:45:03.123 INFO:tasks.workunit.client.0.smithi001.stdout: 10 26109 26.38 MB/sec warmup 60 sec latency 1555.791 ms 2024-04-26T21:45:04.123 INFO:tasks.workunit.client.0.smithi001.stdout: 10 26290 26.04 MB/sec warmup 61 sec latency 298.627 ms 2024-04-26T21:45:04.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:04 smithi001 ceph-mon[28054]: pgmap v461: 129 pgs: 129 active+clean; 769 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 11 MiB/s wr, 583 op/s 2024-04-26T21:45:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:04 smithi038 ceph-mon[32724]: pgmap v461: 129 pgs: 129 active+clean; 769 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 11 MiB/s wr, 583 op/s 2024-04-26T21:45:04.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:04 smithi186 ceph-mon[32991]: pgmap v461: 129 pgs: 129 active+clean; 769 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 11 MiB/s wr, 583 op/s 2024-04-26T21:45:05.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 26421 25.74 MB/sec warmup 62 sec latency 297.643 ms 2024-04-26T21:45:06.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 26535 25.50 MB/sec warmup 63 sec latency 356.664 ms 2024-04-26T21:45:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:06 smithi001 ceph-mon[28054]: pgmap v462: 129 pgs: 129 active+clean; 772 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 895 op/s 2024-04-26T21:45:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:06 smithi038 ceph-mon[32724]: pgmap v462: 129 pgs: 129 active+clean; 772 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 895 op/s 2024-04-26T21:45:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:06 smithi186 ceph-mon[32991]: pgmap v462: 129 pgs: 129 active+clean; 772 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 895 op/s 2024-04-26T21:45:07.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 26869 25.40 MB/sec warmup 64 sec latency 259.492 ms 2024-04-26T21:45:08.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 27149 25.16 MB/sec warmup 65 sec latency 180.200 ms 2024-04-26T21:45:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:08 smithi001 ceph-mon[28054]: pgmap v463: 129 pgs: 129 active+clean; 772 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.9 MiB/s wr, 641 op/s 2024-04-26T21:45:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:08 smithi001 ceph-mon[28054]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T21:45:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:08 smithi038 ceph-mon[32724]: pgmap v463: 129 pgs: 129 active+clean; 772 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.9 MiB/s wr, 641 op/s 2024-04-26T21:45:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:08 smithi038 ceph-mon[32724]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T21:45:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:08 smithi186 ceph-mon[32991]: pgmap v463: 129 pgs: 129 active+clean; 772 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.9 MiB/s wr, 641 op/s 2024-04-26T21:45:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:08 smithi186 ceph-mon[32991]: osdmap e113: 12 total, 12 up, 12 in 2024-04-26T21:45:09.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 27698 25.05 MB/sec warmup 66 sec latency 105.566 ms 2024-04-26T21:45:10.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 28489 25.56 MB/sec warmup 67 sec latency 41.065 ms 2024-04-26T21:45:10.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:10 smithi001 ceph-mon[28054]: pgmap v465: 129 pgs: 129 active+clean; 772 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 4.7 MiB/s wr, 769 op/s 2024-04-26T21:45:10.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:10 smithi038 ceph-mon[32724]: pgmap v465: 129 pgs: 129 active+clean; 772 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 4.7 MiB/s wr, 769 op/s 2024-04-26T21:45:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:10 smithi186 ceph-mon[32991]: pgmap v465: 129 pgs: 129 active+clean; 772 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 4.7 MiB/s wr, 769 op/s 2024-04-26T21:45:11.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 29128 25.72 MB/sec warmup 68 sec latency 53.046 ms 2024-04-26T21:45:12.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 29657 25.79 MB/sec warmup 69 sec latency 80.644 ms 2024-04-26T21:45:12.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:12 smithi001 ceph-mon[28054]: pgmap v466: 129 pgs: 129 active+clean; 787 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.9 MiB/s wr, 628 op/s 2024-04-26T21:45:12.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:12 smithi038 ceph-mon[32724]: pgmap v466: 129 pgs: 129 active+clean; 787 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.9 MiB/s wr, 628 op/s 2024-04-26T21:45:12.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:12 smithi186 ceph-mon[32991]: pgmap v466: 129 pgs: 129 active+clean; 787 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.9 MiB/s wr, 628 op/s 2024-04-26T21:45:13.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 30151 25.70 MB/sec warmup 70 sec latency 66.953 ms 2024-04-26T21:45:14.125 INFO:tasks.workunit.client.0.smithi001.stdout: 10 30885 25.77 MB/sec warmup 71 sec latency 41.592 ms 2024-04-26T21:45:14.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:14 smithi001 ceph-mon[28054]: pgmap v467: 129 pgs: 129 active+clean; 787 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.9 MiB/s wr, 628 op/s 2024-04-26T21:45:14.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:14 smithi001 ceph-mon[28054]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T21:45:14.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:14 smithi038 ceph-mon[32724]: pgmap v467: 129 pgs: 129 active+clean; 787 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.9 MiB/s wr, 628 op/s 2024-04-26T21:45:14.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:14 smithi038 ceph-mon[32724]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T21:45:14.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:14 smithi186 ceph-mon[32991]: pgmap v467: 129 pgs: 129 active+clean; 787 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.9 MiB/s wr, 628 op/s 2024-04-26T21:45:14.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:14 smithi186 ceph-mon[32991]: osdmap e114: 12 total, 12 up, 12 in 2024-04-26T21:45:15.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 31265 25.69 MB/sec warmup 72 sec latency 328.822 ms 2024-04-26T21:45:15.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:15 smithi001 ceph-mon[28054]: pgmap v469: 129 pgs: 129 active+clean; 837 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.7 MiB/s wr, 613 op/s 2024-04-26T21:45:15.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:15 smithi038 ceph-mon[32724]: pgmap v469: 129 pgs: 129 active+clean; 837 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.7 MiB/s wr, 613 op/s 2024-04-26T21:45:15.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:15 smithi186 ceph-mon[32991]: pgmap v469: 129 pgs: 129 active+clean; 837 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.7 MiB/s wr, 613 op/s 2024-04-26T21:45:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.049+0000 7f0975600640 1 -- 172.21.15.1:0/1563436727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0970101280 msgr2=0x7f0970101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.049+0000 7f0975600640 1 --2- 172.21.15.1:0/1563436727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0970101280 0x7f0970101700 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f095c009930 tx=0x7f095c02a440 comp rx=0 tx=0).stop 2024-04-26T21:45:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.049+0000 7f0975600640 1 -- 172.21.15.1:0/1563436727 shutdown_connections 2024-04-26T21:45:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.049+0000 7f0975600640 1 --2- 172.21.15.1:0/1563436727 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0970101c40 0x7f097010f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.049+0000 7f0975600640 1 --2- 172.21.15.1:0/1563436727 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0970101280 0x7f0970101700 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.049+0000 7f0975600640 1 --2- 172.21.15.1:0/1563436727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0970107960 0x7f0970107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.049+0000 7f0975600640 1 -- 172.21.15.1:0/1563436727 >> 172.21.15.1:0/1563436727 conn(0x7f0970076be0 msgr2=0x7f0970076ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.050+0000 7f0975600640 1 -- 172.21.15.1:0/1563436727 shutdown_connections 2024-04-26T21:45:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.050+0000 7f0975600640 1 -- 172.21.15.1:0/1563436727 wait complete. 2024-04-26T21:45:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.051+0000 7f0975600640 1 Processor -- start 2024-04-26T21:45:16.051 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.051+0000 7f0975600640 1 -- start start 2024-04-26T21:45:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.052+0000 7f0975600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0970101280 0x7f0970199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.052+0000 7f0975600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0970101c40 0x7f0970199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.052+0000 7f0975600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0970107960 0x7f097019daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.052+0000 7f0975600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0970112d30 con 0x7f0970101280 2024-04-26T21:45:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.052+0000 7f0975600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0970112bb0 con 0x7f0970101c40 2024-04-26T21:45:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.052+0000 7f0975600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0970112eb0 con 0x7f0970107960 2024-04-26T21:45:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.052+0000 7f096ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0970101280 0x7f0970199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.052+0000 7f096ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0970101280 0x7f0970199230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45722/0 (socket says 172.21.15.1:45722) 2024-04-26T21:45:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.052+0000 7f096ea00640 1 -- 172.21.15.1:0/553854704 learned_addr learned my addr 172.21.15.1:0/553854704 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.053+0000 7f096ea00640 1 -- 172.21.15.1:0/553854704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0970107960 msgr2=0x7f097019daf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T21:45:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.053+0000 7f096ea00640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0970107960 0x7f097019daf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.053+0000 7f096ea00640 1 -- 172.21.15.1:0/553854704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0970101c40 msgr2=0x7f0970199770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T21:45:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.053+0000 7f096ea00640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0970101c40 0x7f0970199770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.053+0000 7f096ea00640 1 -- 172.21.15.1:0/553854704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097019e220 con 0x7f0970101280 2024-04-26T21:45:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.053+0000 7f096ea00640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0970101280 0x7f0970199230 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0960009de0 tx=0x7f096000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.053+0000 7f0967400640 1 -- 172.21.15.1:0/553854704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096001f070 con 0x7f0970101280 2024-04-26T21:45:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.053+0000 7f0967400640 1 -- 172.21.15.1:0/553854704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09600124e0 con 0x7f0970101280 2024-04-26T21:45:16.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.054+0000 7f0967400640 1 -- 172.21.15.1:0/553854704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096001a6b0 con 0x7f0970101280 2024-04-26T21:45:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.054+0000 7f0975600640 1 -- 172.21.15.1:0/553854704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097019e510 con 0x7f0970101280 2024-04-26T21:45:16.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.054+0000 7f0975600640 1 -- 172.21.15.1:0/553854704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0970102390 con 0x7f0970101280 2024-04-26T21:45:16.057 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.057+0000 7f0978435740 1 Processor -- start 2024-04-26T21:45:16.058 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f0978435740 1 -- start start 2024-04-26T21:45:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f0967400640 1 -- 172.21.15.1:0/553854704 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0960022070 con 0x7f0970101280 2024-04-26T21:45:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f0978435740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04cdd0 0x56484c04d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f0978435740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c057a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f0978435740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c057fb0 0x56484c05a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f0978435740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x56484bfb2380 con 0x56484c057fb0 2024-04-26T21:45:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f0978435740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56484bf4ec30 con 0x56484c04cdd0 2024-04-26T21:45:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f0978435740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56484bfae340 con 0x56484c04d6f0 2024-04-26T21:45:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f096ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c057a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f096ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c057a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51934/0 (socket says 172.21.15.1:51934) 2024-04-26T21:45:16.062 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.058+0000 7f096ea00640 1 -- 172.21.15.1:0/3532843644 learned_addr learned my addr 172.21.15.1:0/3532843644 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.059+0000 7f096e000640 1 --2- 172.21.15.1:0/3532843644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c057fb0 0x56484c05a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.059+0000 7f096e000640 1 -- 172.21.15.1:0/3532843644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 msgr2=0x56484c057a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.059+0000 7f096e000640 1 --2- 172.21.15.1:0/3532843644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c057a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.059+0000 7f096e000640 1 -- 172.21.15.1:0/3532843644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04cdd0 msgr2=0x56484c04d1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.059+0000 7f096e000640 1 --2- 172.21.15.1:0/3532843644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04cdd0 0x56484c04d1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.059+0000 7f096e000640 1 -- 172.21.15.1:0/3532843644 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56484bf70940 con 0x56484c057fb0 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.059+0000 7f096ea00640 1 --2- 172.21.15.1:0/3532843644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c057a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.060+0000 7f096e000640 1 --2- 172.21.15.1:0/3532843644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c057fb0 0x56484c05a3a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f095c02a960 tx=0x7f095c0095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37aa7b6b54706bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.060+0000 7f0967400640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09540a3d10 0x7f09540a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.060+0000 7f095b400640 1 -- 172.21.15.1:0/3532843644 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c04f070 con 0x56484c057fb0 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.060+0000 7f095b400640 1 -- 172.21.15.1:0/3532843644 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f095c004400 con 0x56484c057fb0 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.060+0000 7f095b400640 1 -- 172.21.15.1:0/3532843644 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c035040 con 0x56484c057fb0 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.060+0000 7f0967400640 1 -- 172.21.15.1:0/553854704 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9642+0+0 (secure 0 0 0) 0x7f09600d4550 con 0x7f0970101280 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.061+0000 7f096f400640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09540a3d10 0x7f09540a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.061+0000 7f0978435740 1 -- 172.21.15.1:0/3532843644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c057fb0 msgr2=0x56484c05a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:16.063 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.061+0000 7f0978435740 1 --2- 172.21.15.1:0/3532843644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c057fb0 0x56484c05a3a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f095c02a960 tx=0x7f095c0095d0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.062+0000 7f0978435740 1 -- 172.21.15.1:0/3532843644 shutdown_connections 2024-04-26T21:45:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.062+0000 7f0978435740 1 --2- 172.21.15.1:0/3532843644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c057a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.062+0000 7f0978435740 1 --2- 172.21.15.1:0/3532843644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c057fb0 0x56484c05a3a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.062+0000 7f0978435740 1 --2- 172.21.15.1:0/3532843644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04cdd0 0x56484c04d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.062+0000 7f0978435740 1 -- 172.21.15.1:0/3532843644 >> 172.21.15.1:0/3532843644 conn(0x56484c0492e0 msgr2=0x56484c0496c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.062+0000 7f0978435740 1 -- 172.21.15.1:0/3532843644 shutdown_connections 2024-04-26T21:45:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.062+0000 7f096f400640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09540a3d10 0x7f09540a61d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0968005ee0 tx=0x7f0968009440 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.062+0000 7f0978435740 1 -- 172.21.15.1:0/3532843644 wait complete. 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.063+0000 7f0978435740 1 -- start start 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.063+0000 7f0978435740 1 Processor -- start 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f0978435740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c054ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f0978435740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c055430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f0978435740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c057fb0 0x56484c055970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f0978435740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x56484c07ca40 con 0x56484c04cdd0 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f0978435740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56484bf39c60 con 0x56484c057fb0 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f0978435740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56484c07cbc0 con 0x56484c04d6f0 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f0978435740 10 client.0 _ll_register_callbacks cb 0x56484bf65e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c054ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096e000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c055430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c057fb0 0x56484c055970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c054ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45734/0 (socket says 172.21.15.1:45734) 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096e000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c055430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51940/0 (socket says 172.21.15.1:51940) 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096ea00640 1 -- 172.21.15.1:0/1765298996 learned_addr learned my addr 172.21.15.1:0/1765298996 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096e000640 1 -- 172.21.15.1:0/1765298996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c057fb0 msgr2=0x56484c055970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096e000640 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c057fb0 0x56484c055970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096e000640 1 -- 172.21.15.1:0/1765298996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 msgr2=0x56484c054ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096e000640 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c054ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.064+0000 7f096e000640 1 -- 172.21.15.1:0/1765298996 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56484bf48510 con 0x56484c04d6f0 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.065+0000 7f096e000640 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c055430 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f095c031440 tx=0x7f095c0048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f90617df99fe16ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.065+0000 7f0978435740 1 -- 172.21.15.1:0/1765298996 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f095c004400 con 0x56484c04d6f0 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.065+0000 7f094fe00640 10 client.26647 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.065+0000 7f094fe00640 1 -- 172.21.15.1:0/1765298996 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c03e020 con 0x56484c04d6f0 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.065+0000 7f094fe00640 1 -- 172.21.15.1:0/1765298996 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9642+0+0 (secure 0 0 0) 0x7f095c005460 con 0x56484c04d6f0 2024-04-26T21:45:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.065+0000 7f094fe00640 1 -- 172.21.15.1:0/1765298996 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f095c035070 con 0x56484c04d6f0 2024-04-26T21:45:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.065+0000 7f0978435740 10 client.26647 fetch_fsmap learned FSMap version 44 2024-04-26T21:45:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.065+0000 7f0978435740 1 -- 172.21.15.1:0/1765298996 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=44}) -- 0x56484bcc4210 con 0x56484c04d6f0 2024-04-26T21:45:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.066+0000 7f094fe00640 1 -- 172.21.15.1:0/1765298996 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7f095c04a040 con 0x56484c04d6f0 2024-04-26T21:45:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.066+0000 7f0978435740 10 client.26647 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T21:45:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.066+0000 7f0978435740 10 client.26647 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:45:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.066+0000 7f0978435740 20 client.26647 populate_metadata read hostname 'smithi001' 2024-04-26T21:45:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.066+0000 7f0978435740 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56484c089a40 0x56484c08be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.066+0000 7f0978435740 4 client.26647 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:45:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.066+0000 7f0978435740 1 -- 172.21.15.1:0/1765298996 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56484bfbff90 con 0x56484c089a40 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.066+0000 7f096ea00640 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56484c089a40 0x56484c08be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.067+0000 7f096ea00640 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56484c089a40 0x56484c08be30 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c17f47241034d02 server_cookie=b998be644f97421e in_seq=0 out_seq=0 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.067+0000 7f094fe00640 10 client.26647 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f094fe00640 1 -- 172.21.15.1:0/1765298996 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f096007ee50 con 0x56484c089a40 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f094fe00640 10 client.26647 handle_command_reply: tid=0 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f0978435740 1 client.26647 shutdown 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f0978435740 1 -- 172.21.15.1:0/1765298996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 msgr2=0x56484c055430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f0978435740 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c055430 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f095c031440 tx=0x7f095c0048f0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f0978435740 1 -- 172.21.15.1:0/1765298996 shutdown_connections 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f0978435740 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56484c089a40 0x56484c08be30 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f0978435740 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c055430 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f0978435740 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c057fb0 0x56484c055970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.071 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f0978435740 1 --2- 172.21.15.1:0/1765298996 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c054ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f0978435740 1 -- 172.21.15.1:0/1765298996 >> 172.21.15.1:0/1765298996 conn(0x56484c0492e0 msgr2=0x56484c04f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.068+0000 7f094fe00640 0 client.26647 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.069+0000 7f0978435740 1 -- 172.21.15.1:0/1765298996 shutdown_connections 2024-04-26T21:45:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.069+0000 7f0978435740 1 -- 172.21.15.1:0/1765298996 wait complete. 2024-04-26T21:45:16.072 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.069+0000 7f0978435740 20 client.26647 trim_cache size 0 max 16384 2024-04-26T21:45:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.094+0000 7f0978435740 1 Processor -- start 2024-04-26T21:45:16.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.095+0000 7f0978435740 1 -- start start 2024-04-26T21:45:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.095+0000 7f0978435740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04cdd0 0x56484c053ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.095+0000 7f0978435740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c08ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.095+0000 7f0978435740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c057fb0 0x56484c08f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f0978435740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f09701a7de0 con 0x56484c057fb0 2024-04-26T21:45:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f0978435740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09700743b0 con 0x56484c04cdd0 2024-04-26T21:45:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f0978435740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0970072220 con 0x56484c04d6f0 2024-04-26T21:45:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f096ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04cdd0 0x56484c053ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f096ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04cdd0 0x56484c053ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:58432/0 (socket says 172.21.15.1:58432) 2024-04-26T21:45:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f096e000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c08ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f096ea00640 1 -- 172.21.15.1:0/187036068 learned_addr learned my addr 172.21.15.1:0/187036068 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f096e000640 1 -- 172.21.15.1:0/187036068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04cdd0 msgr2=0x56484c053ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f096e000640 1 --2- 172.21.15.1:0/187036068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04cdd0 0x56484c053ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f096e000640 1 -- 172.21.15.1:0/187036068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c057fb0 msgr2=0x56484c08f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f096e000640 1 --2- 172.21.15.1:0/187036068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c057fb0 0x56484c08f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.096+0000 7f096e000640 1 -- 172.21.15.1:0/187036068 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56484c04d520 con 0x56484c04d6f0 2024-04-26T21:45:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.097+0000 7f096e000640 1 --2- 172.21.15.1:0/187036068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c08ee50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f095c02abf0 tx=0x7f095c037000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81c9729819e1389c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.098+0000 7f095aa00640 1 -- 172.21.15.1:0/187036068 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c057070 con 0x56484c04d6f0 2024-04-26T21:45:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.098+0000 7f095aa00640 1 -- 172.21.15.1:0/187036068 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f095c004400 con 0x56484c04d6f0 2024-04-26T21:45:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.098+0000 7f095aa00640 1 -- 172.21.15.1:0/187036068 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c03e040 con 0x56484c04d6f0 2024-04-26T21:45:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.099+0000 7f0978435740 1 -- 172.21.15.1:0/187036068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 msgr2=0x56484c08ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.099+0000 7f0978435740 1 --2- 172.21.15.1:0/187036068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c08ee50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f095c02abf0 tx=0x7f095c037000 comp rx=0 tx=0).stop 2024-04-26T21:45:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.099+0000 7f0978435740 1 -- 172.21.15.1:0/187036068 shutdown_connections 2024-04-26T21:45:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.099+0000 7f0978435740 1 --2- 172.21.15.1:0/187036068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c04d6f0 0x56484c08ee50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.099+0000 7f0978435740 1 --2- 172.21.15.1:0/187036068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c057fb0 0x56484c08f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.099+0000 7f0978435740 1 --2- 172.21.15.1:0/187036068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04cdd0 0x56484c053ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.099+0000 7f0978435740 1 -- 172.21.15.1:0/187036068 >> 172.21.15.1:0/187036068 conn(0x56484c0492e0 msgr2=0x56484c0537c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.099+0000 7f0978435740 1 -- 172.21.15.1:0/187036068 shutdown_connections 2024-04-26T21:45:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.099+0000 7f0978435740 1 -- 172.21.15.1:0/187036068 wait complete. 2024-04-26T21:45:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.100+0000 7f0978435740 1 -- start start 2024-04-26T21:45:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.100+0000 7f0978435740 1 Processor -- start 2024-04-26T21:45:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f0978435740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c08f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f0978435740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04d6f0 0x56484c08c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f0978435740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c057fb0 0x56484c08ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f0978435740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0970072870 con 0x56484c04cdd0 2024-04-26T21:45:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f0978435740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0970073ce0 con 0x56484c04d6f0 2024-04-26T21:45:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f0978435740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0970072f40 con 0x56484c057fb0 2024-04-26T21:45:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f0978435740 10 client.0 _ll_register_callbacks cb 0x7f09600d7ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:45:16.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f096ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c08f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f096ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c08f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45756/0 (socket says 172.21.15.1:45756) 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f096ea00640 1 -- 172.21.15.1:0/1289269635 learned_addr learned my addr 172.21.15.1:0/1289269635 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f096f400640 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c057fb0 0x56484c08ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.101+0000 7f096e000640 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04d6f0 0x56484c08c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.102+0000 7f096ea00640 1 -- 172.21.15.1:0/1289269635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c057fb0 msgr2=0x56484c08ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.102+0000 7f096ea00640 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c057fb0 0x56484c08ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.102+0000 7f096ea00640 1 -- 172.21.15.1:0/1289269635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04d6f0 msgr2=0x56484c08c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.102+0000 7f096ea00640 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04d6f0 0x56484c08c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.102+0000 7f096ea00640 1 -- 172.21.15.1:0/1289269635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56484bd90620 con 0x56484c04cdd0 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.102+0000 7f096ea00640 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c08f180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f096009ca00 tx=0x7f09600da060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ceedd32783e1d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.102+0000 7f094fe00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:45:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.103+0000 7f094fe00640 1 -- 172.21.15.1:0/1289269635 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096009f990 con 0x56484c04cdd0 2024-04-26T21:45:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.103+0000 7f094fe00640 1 -- 172.21.15.1:0/1289269635 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9642+0+0 (secure 0 0 0) 0x7f09600a40d0 con 0x56484c04cdd0 2024-04-26T21:45:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.103+0000 7f0978435740 1 -- 172.21.15.1:0/1289269635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f095c004400 con 0x56484c04cdd0 2024-04-26T21:45:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.104+0000 7f094fe00640 1 -- 172.21.15.1:0/1289269635 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f09600a0730 con 0x56484c04cdd0 2024-04-26T21:45:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.104+0000 7f0978435740 10 client.16464 fetch_fsmap learned FSMap version 44 2024-04-26T21:45:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.104+0000 7f0978435740 1 -- 172.21.15.1:0/1289269635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x56484bf6ccb0 con 0x56484c04cdd0 2024-04-26T21:45:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.105+0000 7f094fe00640 1 -- 172.21.15.1:0/1289269635 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f095c004400 con 0x56484c04cdd0 2024-04-26T21:45:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.105+0000 7f094fe00640 1 -- 172.21.15.1:0/1289269635 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7f09600d1a60 con 0x56484c04cdd0 2024-04-26T21:45:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.105+0000 7f0978435740 10 client.16464 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T21:45:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.105+0000 7f0978435740 10 client.16464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:45:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.105+0000 7f0978435740 20 client.16464 populate_metadata read hostname 'smithi001' 2024-04-26T21:45:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.106+0000 7f0978435740 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56484c0e6ef0 0x56484c0e9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.106+0000 7f0978435740 4 client.16464 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:45:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.106+0000 7f0978435740 1 -- 172.21.15.1:0/1289269635 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56484bfb9e00 con 0x56484c0e6ef0 2024-04-26T21:45:16.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.108+0000 7f096e000640 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56484c0e6ef0 0x56484c0e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.109+0000 7f096e000640 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56484c0e6ef0 0x56484c0e9310 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff09f8cdd3372d4a server_cookie=8b57fe5fce9cc094 in_seq=0 out_seq=0 2024-04-26T21:45:16.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.109+0000 7f094fe00640 10 client.16464 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:16.124 INFO:tasks.workunit.client.0.smithi001.stdout: 10 31863 25.89 MB/sec warmup 73 sec latency 80.183 ms 2024-04-26T21:45:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.601+0000 7f094fe00640 1 -- 172.21.15.1:0/1289269635 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56484bfb9e00 con 0x56484c0e6ef0 2024-04-26T21:45:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.601+0000 7f094fe00640 10 client.16464 handle_command_reply: tid=0 2024-04-26T21:45:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.601+0000 7f0978435740 1 client.16464 shutdown 2024-04-26T21:45:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.601+0000 7f0978435740 1 -- 172.21.15.1:0/1289269635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 msgr2=0x56484c08f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.601+0000 7f0978435740 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c08f180 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f096009ca00 tx=0x7f09600da060 comp rx=0 tx=0).stop 2024-04-26T21:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.602+0000 7f0978435740 1 -- 172.21.15.1:0/1289269635 shutdown_connections 2024-04-26T21:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.602+0000 7f0978435740 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56484c0e6ef0 0x56484c0e9310 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.602+0000 7f0978435740 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56484c057fb0 0x56484c08ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.602+0000 7f0978435740 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56484c04d6f0 0x56484c08c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.602+0000 7f0978435740 1 --2- 172.21.15.1:0/1289269635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56484c04cdd0 0x56484c08f180 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.602+0000 7f0978435740 1 -- 172.21.15.1:0/1289269635 >> 172.21.15.1:0/1289269635 conn(0x56484c0492e0 msgr2=0x56484c04e780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.602+0000 7f094fe00640 0 client.16464 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.602+0000 7f0978435740 1 -- 172.21.15.1:0/1289269635 shutdown_connections 2024-04-26T21:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.602+0000 7f0978435740 1 -- 172.21.15.1:0/1289269635 wait complete. 2024-04-26T21:45:16.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.602+0000 7f0978435740 20 client.16464 trim_cache size 0 max 16384 2024-04-26T21:45:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:45:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:45:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:45:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:45:16.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.603+0000 7f0975600640 1 -- 172.21.15.1:0/553854704 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09540a3d10 msgr2=0x7f09540a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.603+0000 7f0975600640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09540a3d10 0x7f09540a61d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0968005ee0 tx=0x7f0968009440 comp rx=0 tx=0).stop 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.603+0000 7f0975600640 1 -- 172.21.15.1:0/553854704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0970101280 msgr2=0x7f0970199230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.603+0000 7f0975600640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0970101280 0x7f0970199230 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0960009de0 tx=0x7f096000c6a0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.604+0000 7f0975600640 1 -- 172.21.15.1:0/553854704 shutdown_connections 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.604+0000 7f0975600640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09540a3d10 0x7f09540a61d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.604+0000 7f0975600640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0970107960 0x7f097019daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.604+0000 7f0975600640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0970101c40 0x7f0970199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.604+0000 7f0975600640 1 --2- 172.21.15.1:0/553854704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0970101280 0x7f0970199230 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.604+0000 7f0975600640 1 -- 172.21.15.1:0/553854704 >> 172.21.15.1:0/553854704 conn(0x7f0970076be0 msgr2=0x7f09700ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.604+0000 7f0975600640 1 -- 172.21.15.1:0/553854704 shutdown_connections 2024-04-26T21:45:16.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:16.604+0000 7f0975600640 1 -- 172.21.15.1:0/553854704 wait complete. 2024-04-26T21:45:17.125 INFO:tasks.workunit.client.0.smithi001.stdout: 10 32391 25.93 MB/sec warmup 74 sec latency 275.752 ms 2024-04-26T21:45:18.126 INFO:tasks.workunit.client.0.smithi001.stdout: 10 32881 26.01 MB/sec warmup 75 sec latency 180.822 ms 2024-04-26T21:45:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:18 smithi186 ceph-mon[32991]: pgmap v470: 129 pgs: 129 active+clean; 837 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 8.0 MiB/s wr, 503 op/s 2024-04-26T21:45:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:18 smithi001 ceph-mon[28054]: pgmap v470: 129 pgs: 129 active+clean; 837 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 8.0 MiB/s wr, 503 op/s 2024-04-26T21:45:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:18 smithi038 ceph-mon[32724]: pgmap v470: 129 pgs: 129 active+clean; 837 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 8.0 MiB/s wr, 503 op/s 2024-04-26T21:45:19.126 INFO:tasks.workunit.client.0.smithi001.stdout: 10 33218 25.93 MB/sec warmup 76 sec latency 182.998 ms 2024-04-26T21:45:20.127 INFO:tasks.workunit.client.0.smithi001.stdout: 10 33342 25.71 MB/sec warmup 77 sec latency 746.701 ms 2024-04-26T21:45:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:20 smithi186 ceph-mon[32991]: pgmap v471: 129 pgs: 129 active+clean; 837 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.8 MiB/s wr, 490 op/s 2024-04-26T21:45:20.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:20 smithi001 ceph-mon[28054]: pgmap v471: 129 pgs: 129 active+clean; 837 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.8 MiB/s wr, 490 op/s 2024-04-26T21:45:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:20 smithi038 ceph-mon[32724]: pgmap v471: 129 pgs: 129 active+clean; 837 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.8 MiB/s wr, 490 op/s 2024-04-26T21:45:21.127 INFO:tasks.workunit.client.0.smithi001.stdout: 10 33561 25.47 MB/sec warmup 78 sec latency 819.137 ms 2024-04-26T21:45:22.127 INFO:tasks.workunit.client.0.smithi001.stdout: 10 33975 25.38 MB/sec warmup 79 sec latency 115.305 ms 2024-04-26T21:45:22.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:22 smithi038 ceph-mon[32724]: pgmap v472: 129 pgs: 129 active+clean; 857 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 12 MiB/s wr, 672 op/s 2024-04-26T21:45:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:22 smithi186 ceph-mon[32991]: pgmap v472: 129 pgs: 129 active+clean; 857 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 12 MiB/s wr, 672 op/s 2024-04-26T21:45:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:22 smithi001 ceph-mon[28054]: pgmap v472: 129 pgs: 129 active+clean; 857 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 12 MiB/s wr, 672 op/s 2024-04-26T21:45:23.127 INFO:tasks.workunit.client.0.smithi001.stdout: 10 34582 25.45 MB/sec warmup 80 sec latency 75.238 ms 2024-04-26T21:45:24.127 INFO:tasks.workunit.client.0.smithi001.stdout: 10 35278 25.58 MB/sec warmup 81 sec latency 109.227 ms 2024-04-26T21:45:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:24 smithi186 ceph-mon[32991]: pgmap v473: 129 pgs: 129 active+clean; 857 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 12 MiB/s wr, 672 op/s 2024-04-26T21:45:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:24 smithi001 ceph-mon[28054]: pgmap v473: 129 pgs: 129 active+clean; 857 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 12 MiB/s wr, 672 op/s 2024-04-26T21:45:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:24 smithi038 ceph-mon[32724]: pgmap v473: 129 pgs: 129 active+clean; 857 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 12 MiB/s wr, 672 op/s 2024-04-26T21:45:25.127 INFO:tasks.workunit.client.0.smithi001.stdout: 10 36048 25.78 MB/sec warmup 82 sec latency 40.071 ms 2024-04-26T21:45:26.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 36691 26.00 MB/sec warmup 83 sec latency 49.977 ms 2024-04-26T21:45:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:26 smithi186 ceph-mon[32991]: pgmap v474: 129 pgs: 129 active+clean; 859 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 913 op/s 2024-04-26T21:45:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:26 smithi001 ceph-mon[28054]: pgmap v474: 129 pgs: 129 active+clean; 859 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 913 op/s 2024-04-26T21:45:26.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:26 smithi038 ceph-mon[32724]: pgmap v474: 129 pgs: 129 active+clean; 859 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 15 MiB/s wr, 913 op/s 2024-04-26T21:45:27.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 37329 26.07 MB/sec warmup 84 sec latency 37.540 ms 2024-04-26T21:45:28.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 37750 26.01 MB/sec warmup 85 sec latency 83.273 ms 2024-04-26T21:45:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:28 smithi186 ceph-mon[32991]: pgmap v475: 129 pgs: 129 active+clean; 859 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.6 MiB/s wr, 660 op/s 2024-04-26T21:45:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:28 smithi001 ceph-mon[28054]: pgmap v475: 129 pgs: 129 active+clean; 859 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.6 MiB/s wr, 660 op/s 2024-04-26T21:45:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:28 smithi038 ceph-mon[32724]: pgmap v475: 129 pgs: 129 active+clean; 859 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.6 MiB/s wr, 660 op/s 2024-04-26T21:45:29.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 38273 26.02 MB/sec warmup 86 sec latency 46.667 ms 2024-04-26T21:45:29.789 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.788+0000 7fd14f200640 1 -- 172.21.15.1:0/3190879042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd14810e390 msgr2=0x7fd14810e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.788+0000 7fd14f200640 1 --2- 172.21.15.1:0/3190879042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd14810e390 0x7fd14810e770 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd138009cd0 tx=0x7fd13802a8f0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.788+0000 7fd14f200640 1 -- 172.21.15.1:0/3190879042 shutdown_connections 2024-04-26T21:45:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.788+0000 7fd14f200640 1 --2- 172.21.15.1:0/3190879042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14810a2f0 0x7fd148116220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.788+0000 7fd14f200640 1 --2- 172.21.15.1:0/3190879042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd148109930 0x7fd148109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.788+0000 7fd14f200640 1 --2- 172.21.15.1:0/3190879042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd14810e390 0x7fd14810e770 secure :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd138009cd0 tx=0x7fd13802a8f0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.788+0000 7fd14f200640 1 -- 172.21.15.1:0/3190879042 >> 172.21.15.1:0/3190879042 conn(0x7fd14806da10 msgr2=0x7fd14806de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:29.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.788+0000 7fd14f200640 1 -- 172.21.15.1:0/3190879042 shutdown_connections 2024-04-26T21:45:29.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.789+0000 7fd14f200640 1 -- 172.21.15.1:0/3190879042 wait complete. 2024-04-26T21:45:29.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.790+0000 7fd14f200640 1 Processor -- start 2024-04-26T21:45:29.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.790+0000 7fd14f200640 1 -- start start 2024-04-26T21:45:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.791+0000 7fd14f200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd148109930 0x7fd1481a80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.791+0000 7fd14f200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14810a2f0 0x7fd1481a8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.791+0000 7fd14f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1481ac990 0x7fd1481ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.791+0000 7fd14f200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1481191a0 con 0x7fd148109930 2024-04-26T21:45:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.791+0000 7fd14f200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd148119020 con 0x7fd14810a2f0 2024-04-26T21:45:29.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.791+0000 7fd14f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd148119320 con 0x7fd1481ac990 2024-04-26T21:45:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.791+0000 7fd14d200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1481ac990 0x7fd1481ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.791+0000 7fd14d200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1481ac990 0x7fd1481ace40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50444/0 (socket says 172.21.15.1:50444) 2024-04-26T21:45:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.791+0000 7fd14d200640 1 -- 172.21.15.1:0/2896897461 learned_addr learned my addr 172.21.15.1:0/2896897461 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.792+0000 7fd147400640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14810a2f0 0x7fd1481a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.792+0000 7fd147e00640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd148109930 0x7fd1481a80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.792+0000 7fd14d200640 1 -- 172.21.15.1:0/2896897461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14810a2f0 msgr2=0x7fd1481a8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.792+0000 7fd14d200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14810a2f0 0x7fd1481a8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.792+0000 7fd14d200640 1 -- 172.21.15.1:0/2896897461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd148109930 msgr2=0x7fd1481a80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.792+0000 7fd14d200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd148109930 0x7fd1481a80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.792+0000 7fd14d200640 1 -- 172.21.15.1:0/2896897461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1481ad480 con 0x7fd1481ac990 2024-04-26T21:45:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.792+0000 7fd14d200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1481ac990 0x7fd1481ace40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd140009750 tx=0x7fd140009c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.793+0000 7fd144c00640 1 -- 172.21.15.1:0/2896897461 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140018030 con 0x7fd1481ac990 2024-04-26T21:45:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.793+0000 7fd144c00640 1 -- 172.21.15.1:0/2896897461 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd140009e60 con 0x7fd1481ac990 2024-04-26T21:45:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.793+0000 7fd144c00640 1 -- 172.21.15.1:0/2896897461 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1400047a0 con 0x7fd1481ac990 2024-04-26T21:45:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.793+0000 7fd14f200640 1 -- 172.21.15.1:0/2896897461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1481ad770 con 0x7fd1481ac990 2024-04-26T21:45:29.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.793+0000 7fd14f200640 1 -- 172.21.15.1:0/2896897461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1481b4f10 con 0x7fd1481ac990 2024-04-26T21:45:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.797+0000 7fd144c00640 1 -- 172.21.15.1:0/2896897461 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd14001b070 con 0x7fd1481ac990 2024-04-26T21:45:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.798+0000 7fd15201a740 1 Processor -- start 2024-04-26T21:45:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.799+0000 7fd15201a740 1 -- start start 2024-04-26T21:45:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.799+0000 7fd15201a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563621652dd0 0x5636216531b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.799+0000 7fd15201a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636216536f0 0x56362165da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.799+0000 7fd15201a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56362165dfb0 0x5636216603a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.799+0000 7fd15201a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5636215b8380 con 0x563621652dd0 2024-04-26T21:45:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.799+0000 7fd15201a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x563621554c30 con 0x56362165dfb0 2024-04-26T21:45:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.799+0000 7fd15201a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5636215b4340 con 0x5636216536f0 2024-04-26T21:45:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.800+0000 7fd147e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636216536f0 0x56362165da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.800+0000 7fd147e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636216536f0 0x56362165da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50450/0 (socket says 172.21.15.1:50450) 2024-04-26T21:45:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.800+0000 7fd147e00640 1 -- 172.21.15.1:0/2980898185 learned_addr learned my addr 172.21.15.1:0/2980898185 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:29.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.800+0000 7fd147400640 1 --2- 172.21.15.1:0/2980898185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563621652dd0 0x5636216531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.800+0000 7fd144c00640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd12c0a3d90 0x7fd12c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.800+0000 7fd147e00640 1 -- 172.21.15.1:0/2980898185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56362165dfb0 msgr2=0x5636216603a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:45:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.800+0000 7fd147e00640 1 --2- 172.21.15.1:0/2980898185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56362165dfb0 0x5636216603a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.800+0000 7fd147e00640 1 -- 172.21.15.1:0/2980898185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563621652dd0 msgr2=0x5636216531b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.800+0000 7fd147e00640 1 --2- 172.21.15.1:0/2980898185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563621652dd0 0x5636216531b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.800+0000 7fd147e00640 1 -- 172.21.15.1:0/2980898185 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563621576940 con 0x5636216536f0 2024-04-26T21:45:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.801+0000 7fd144c00640 1 -- 172.21.15.1:0/2896897461 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 9642+0+0 (secure 0 0 0) 0x7fd1400d0640 con 0x7fd1481ac990 2024-04-26T21:45:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.801+0000 7fd14d200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd12c0a3d90 0x7fd12c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.801+0000 7fd147e00640 1 --2- 172.21.15.1:0/2980898185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636216536f0 0x56362165da70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd138035000 tx=0x7fd138009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60193e44322a7679 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.802+0000 7fd13d600640 1 -- 172.21.15.1:0/2980898185 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1380305a0 con 0x5636216536f0 2024-04-26T21:45:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.802+0000 7fd13d600640 1 -- 172.21.15.1:0/2980898185 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd138036070 con 0x5636216536f0 2024-04-26T21:45:29.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.802+0000 7fd13d600640 1 -- 172.21.15.1:0/2980898185 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1380042a0 con 0x5636216536f0 2024-04-26T21:45:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.803+0000 7fd15201a740 1 -- 172.21.15.1:0/2980898185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636216536f0 msgr2=0x56362165da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.803+0000 7fd15201a740 1 --2- 172.21.15.1:0/2980898185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636216536f0 0x56362165da70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd138035000 tx=0x7fd138009a20 comp rx=0 tx=0).stop 2024-04-26T21:45:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.803+0000 7fd14d200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd12c0a3d90 0x7fd12c0a6250 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd140009720 tx=0x7fd1400046c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:29.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.803+0000 7fd15201a740 1 -- 172.21.15.1:0/2980898185 shutdown_connections 2024-04-26T21:45:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.803+0000 7fd15201a740 1 --2- 172.21.15.1:0/2980898185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636216536f0 0x56362165da70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.803+0000 7fd15201a740 1 --2- 172.21.15.1:0/2980898185 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56362165dfb0 0x5636216603a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.803+0000 7fd15201a740 1 --2- 172.21.15.1:0/2980898185 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563621652dd0 0x5636216531b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.803+0000 7fd15201a740 1 -- 172.21.15.1:0/2980898185 >> 172.21.15.1:0/2980898185 conn(0x56362164f2e0 msgr2=0x56362164f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.803+0000 7fd15201a740 1 -- 172.21.15.1:0/2980898185 shutdown_connections 2024-04-26T21:45:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.803+0000 7fd15201a740 1 -- 172.21.15.1:0/2980898185 wait complete. 2024-04-26T21:45:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.804+0000 7fd15201a740 1 -- start start 2024-04-26T21:45:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.804+0000 7fd15201a740 1 Processor -- start 2024-04-26T21:45:29.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd15201a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563621652dd0 0x56362165af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd15201a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd15201a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56362165dfb0 0x56362165b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd15201a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x563621660b80 con 0x56362165dfb0 2024-04-26T21:45:29.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd15201a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56362153fc60 con 0x5636216536f0 2024-04-26T21:45:29.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd15201a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563621660d00 con 0x563621652dd0 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd15201a740 10 client.0 _ll_register_callbacks cb 0x56362156be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd147e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd147e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50372/0 (socket says 172.21.15.1:50372) 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd147e00640 1 -- 172.21.15.1:0/4190731759 learned_addr learned my addr 172.21.15.1:0/4190731759 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.805+0000 7fd147400640 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563621652dd0 0x56362165af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.806+0000 7fd147e00640 1 -- 172.21.15.1:0/4190731759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563621652dd0 msgr2=0x56362165af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.806+0000 7fd147400640 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56362165dfb0 0x56362165b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.806+0000 7fd147e00640 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563621652dd0 0x56362165af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.806+0000 7fd147e00640 1 -- 172.21.15.1:0/4190731759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56362165dfb0 msgr2=0x56362165b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.806+0000 7fd147e00640 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56362165dfb0 0x56362165b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.806+0000 7fd147e00640 1 -- 172.21.15.1:0/4190731759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56362154e510 con 0x5636216536f0 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.806+0000 7fd147400640 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56362165dfb0 0x56362165b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.806+0000 7fd147e00640 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b460 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd138005e30 tx=0x7fd1380098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=510d70aa4e8b2a86 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:29.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.806+0000 7fd132000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:45:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.807+0000 7fd132000640 1 -- 172.21.15.1:0/4190731759 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1380305b0 con 0x5636216536f0 2024-04-26T21:45:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.807+0000 7fd132000640 1 -- 172.21.15.1:0/4190731759 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9642+0+0 (secure 0 0 0) 0x7fd138049040 con 0x5636216536f0 2024-04-26T21:45:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.807+0000 7fd15201a740 1 -- 172.21.15.1:0/4190731759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd138036070 con 0x5636216536f0 2024-04-26T21:45:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.807+0000 7fd132000640 1 -- 172.21.15.1:0/4190731759 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fd138058050 con 0x5636216536f0 2024-04-26T21:45:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.807+0000 7fd15201a740 10 client.26693 fetch_fsmap learned FSMap version 44 2024-04-26T21:45:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.807+0000 7fd15201a740 1 -- 172.21.15.1:0/4190731759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5636212ca210 con 0x5636216536f0 2024-04-26T21:45:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.807+0000 7fd132000640 1 -- 172.21.15.1:0/4190731759 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd138036070 con 0x5636216536f0 2024-04-26T21:45:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.807+0000 7fd132000640 1 -- 172.21.15.1:0/4190731759 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7fd1380044a0 con 0x5636216536f0 2024-04-26T21:45:29.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.807+0000 7fd15201a740 10 client.26693 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T21:45:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.807+0000 7fd15201a740 10 client.26693 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:45:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.808+0000 7fd15201a740 20 client.26693 populate_metadata read hostname 'smithi001' 2024-04-26T21:45:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.808+0000 7fd15201a740 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56362166db00 0x56362166fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.808+0000 7fd15201a740 4 client.26693 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:45:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.808+0000 7fd15201a740 1 -- 172.21.15.1:0/4190731759 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5636215c5f90 con 0x56362166db00 2024-04-26T21:45:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.809+0000 7fd147400640 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56362166db00 0x56362166fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.810+0000 7fd147400640 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56362166db00 0x56362166fef0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77ee0c7bfb34d31b server_cookie=1e9018ad12484ba8 in_seq=0 out_seq=0 2024-04-26T21:45:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.810+0000 7fd132000640 10 client.26693 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.811+0000 7fd132000640 1 -- 172.21.15.1:0/4190731759 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5636215c5f90 con 0x56362166db00 2024-04-26T21:45:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.811+0000 7fd132000640 10 client.26693 handle_command_reply: tid=0 2024-04-26T21:45:29.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.812+0000 7fd15201a740 1 client.26693 shutdown 2024-04-26T21:45:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 1 -- 172.21.15.1:0/4190731759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 msgr2=0x56362165b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b460 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd138005e30 tx=0x7fd1380098f0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 1 -- 172.21.15.1:0/4190731759 shutdown_connections 2024-04-26T21:45:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56362166db00 0x56362166fef0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56362165dfb0 0x56362165b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b460 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 1 --2- 172.21.15.1:0/4190731759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563621652dd0 0x56362165af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 1 -- 172.21.15.1:0/4190731759 >> 172.21.15.1:0/4190731759 conn(0x56362164f2e0 msgr2=0x563621655010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd132000640 0 client.26693 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 1 -- 172.21.15.1:0/4190731759 shutdown_connections 2024-04-26T21:45:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 1 -- 172.21.15.1:0/4190731759 wait complete. 2024-04-26T21:45:29.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.813+0000 7fd15201a740 20 client.26693 trim_cache size 0 max 16384 2024-04-26T21:45:29.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.838+0000 7fd15201a740 1 Processor -- start 2024-04-26T21:45:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.838+0000 7fd15201a740 1 -- start start 2024-04-26T21:45:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.838+0000 7fd15201a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563621652dd0 0x563621669a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.839+0000 7fd15201a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.839+0000 7fd15201a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56362165dfb0 0x563621675490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.839+0000 7fd15201a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1481c1ce0 con 0x563621652dd0 2024-04-26T21:45:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.839+0000 7fd15201a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1481bf4f0 con 0x5636216536f0 2024-04-26T21:45:29.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.839+0000 7fd15201a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1481bd360 con 0x56362165dfb0 2024-04-26T21:45:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.839+0000 7fd147e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.839+0000 7fd147e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50380/0 (socket says 172.21.15.1:50380) 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.839+0000 7fd147e00640 1 -- 172.21.15.1:0/1986185017 learned_addr learned my addr 172.21.15.1:0/1986185017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.839+0000 7fd147400640 1 --2- 172.21.15.1:0/1986185017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56362165dfb0 0x563621675490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.840+0000 7fd147e00640 1 -- 172.21.15.1:0/1986185017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56362165dfb0 msgr2=0x563621675490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.840+0000 7fd147e00640 1 --2- 172.21.15.1:0/1986185017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56362165dfb0 0x563621675490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.840+0000 7fd147e00640 1 -- 172.21.15.1:0/1986185017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563621652dd0 msgr2=0x563621669a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.840+0000 7fd147e00640 1 --2- 172.21.15.1:0/1986185017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563621652dd0 0x563621669a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.840+0000 7fd147e00640 1 -- 172.21.15.1:0/1986185017 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563621653520 con 0x5636216536f0 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.840+0000 7fd147400640 1 --2- 172.21.15.1:0/1986185017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56362165dfb0 0x563621675490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.840+0000 7fd147e00640 1 --2- 172.21.15.1:0/1986185017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b7c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd13802a930 tx=0x7fd138009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=583d3391d315ff50 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.840+0000 7fd13cc00640 1 -- 172.21.15.1:0/1986185017 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd138005c60 con 0x5636216536f0 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.840+0000 7fd13cc00640 1 -- 172.21.15.1:0/1986185017 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd138036040 con 0x5636216536f0 2024-04-26T21:45:29.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.840+0000 7fd13cc00640 1 -- 172.21.15.1:0/1986185017 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd138031580 con 0x5636216536f0 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.841+0000 7fd15201a740 1 -- 172.21.15.1:0/1986185017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 msgr2=0x56362165b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.841+0000 7fd15201a740 1 --2- 172.21.15.1:0/1986185017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b7c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd13802a930 tx=0x7fd138009a20 comp rx=0 tx=0).stop 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.841+0000 7fd15201a740 1 -- 172.21.15.1:0/1986185017 shutdown_connections 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.841+0000 7fd15201a740 1 --2- 172.21.15.1:0/1986185017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56362165dfb0 0x563621675490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.841+0000 7fd15201a740 1 --2- 172.21.15.1:0/1986185017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5636216536f0 0x56362165b7c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.841+0000 7fd15201a740 1 --2- 172.21.15.1:0/1986185017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x563621652dd0 0x563621669a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.841+0000 7fd15201a740 1 -- 172.21.15.1:0/1986185017 >> 172.21.15.1:0/1986185017 conn(0x56362164f2e0 msgr2=0x563621656330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.841+0000 7fd15201a740 1 -- 172.21.15.1:0/1986185017 shutdown_connections 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.842+0000 7fd15201a740 1 -- 172.21.15.1:0/1986185017 wait complete. 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.842+0000 7fd15201a740 1 -- start start 2024-04-26T21:45:29.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.842+0000 7fd15201a740 1 Processor -- start 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd15201a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563621652dd0 0x5636216663e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd15201a740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5636216536f0 0x563621661750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd15201a740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56362165dfb0 0x563621661d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd15201a740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1481bd9b0 con 0x5636216536f0 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd15201a740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1481bee20 con 0x56362165dfb0 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd15201a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1481be080 con 0x563621652dd0 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd15201a740 10 client.0 _ll_register_callbacks cb 0x7fd140005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd147e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5636216536f0 0x563621661750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd147e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5636216536f0 0x563621661750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41710/0 (socket says 172.21.15.1:41710) 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd147e00640 1 -- 172.21.15.1:0/950082386 learned_addr learned my addr 172.21.15.1:0/950082386 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:29.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.843+0000 7fd147400640 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563621652dd0 0x5636216663e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.844+0000 7fd147e00640 1 -- 172.21.15.1:0/950082386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563621652dd0 msgr2=0x5636216663e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.844+0000 7fd147e00640 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563621652dd0 0x5636216663e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.844+0000 7fd147e00640 1 -- 172.21.15.1:0/950082386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56362165dfb0 msgr2=0x563621661d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.844+0000 7fd147e00640 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56362165dfb0 0x563621661d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.844+0000 7fd147e00640 1 -- 172.21.15.1:0/950082386 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563621396620 con 0x5636216536f0 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.844+0000 7fd147e00640 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5636216536f0 0x563621661750 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd138009960 tx=0x7fd138030ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6867092ae7addf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.844+0000 7fd132000640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.844+0000 7fd132000640 1 -- 172.21.15.1:0/950082386 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd138030ad0 con 0x5636216536f0 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.844+0000 7fd132000640 1 -- 172.21.15.1:0/950082386 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(114..114 src has 1..114) ==== 9642+0+0 (secure 0 0 0) 0x7fd138031490 con 0x5636216536f0 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.844+0000 7fd15201a740 1 -- 172.21.15.1:0/950082386 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd138036040 con 0x5636216536f0 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd132000640 1 -- 172.21.15.1:0/950082386 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fd138030c70 con 0x5636216536f0 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd15201a740 10 client.16473 fetch_fsmap learned FSMap version 44 2024-04-26T21:45:29.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd15201a740 1 -- 172.21.15.1:0/950082386 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x563621572cb0 con 0x5636216536f0 2024-04-26T21:45:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd132000640 1 -- 172.21.15.1:0/950082386 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd138036040 con 0x5636216536f0 2024-04-26T21:45:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd132000640 1 -- 172.21.15.1:0/950082386 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 44) ==== 8370+0+0 (secure 0 0 0) 0x7fd138064030 con 0x5636216536f0 2024-04-26T21:45:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd15201a740 10 client.16473 fetch_fsmap finished waiting for FSMap version 44 2024-04-26T21:45:29.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd15201a740 10 client.16473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:45:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd15201a740 20 client.16473 populate_metadata read hostname 'smithi001' 2024-04-26T21:45:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd15201a740 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5636216d40a0 0x5636216d64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd15201a740 4 client.16473 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:45:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.845+0000 7fd15201a740 1 -- 172.21.15.1:0/950082386 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5636215bfe00 con 0x5636216d40a0 2024-04-26T21:45:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.846+0000 7fd147400640 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5636216d40a0 0x5636216d64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.846+0000 7fd147400640 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5636216d40a0 0x5636216d64c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fec517107fcafa6 server_cookie=f433c14212de6fb4 in_seq=0 out_seq=0 2024-04-26T21:45:29.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:29.846+0000 7fd132000640 10 client.16473 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:30.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 38821 25.91 MB/sec warmup 87 sec latency 180.068 ms 2024-04-26T21:45:30.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.247+0000 7fd132000640 1 -- 172.21.15.1:0/950082386 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5636215bfe00 con 0x5636216d40a0 2024-04-26T21:45:30.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.247+0000 7fd132000640 10 client.16473 handle_command_reply: tid=0 2024-04-26T21:45:30.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.248+0000 7fd15201a740 1 client.16473 shutdown 2024-04-26T21:45:30.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.248+0000 7fd15201a740 1 -- 172.21.15.1:0/950082386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5636216536f0 msgr2=0x563621661750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:30.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.248+0000 7fd15201a740 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5636216536f0 0x563621661750 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd138009960 tx=0x7fd138030ff0 comp rx=0 tx=0).stop 2024-04-26T21:45:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.248+0000 7fd15201a740 1 -- 172.21.15.1:0/950082386 shutdown_connections 2024-04-26T21:45:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.248+0000 7fd15201a740 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5636216d40a0 0x5636216d64c0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.248+0000 7fd15201a740 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56362165dfb0 0x563621661d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:30.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.248+0000 7fd15201a740 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5636216536f0 0x563621661750 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.248+0000 7fd15201a740 1 --2- 172.21.15.1:0/950082386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563621652dd0 0x5636216663e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.248+0000 7fd15201a740 1 -- 172.21.15.1:0/950082386 >> 172.21.15.1:0/950082386 conn(0x56362164f2e0 msgr2=0x563621654780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.248+0000 7fd132000640 0 client.16473 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.249+0000 7fd15201a740 1 -- 172.21.15.1:0/950082386 shutdown_connections 2024-04-26T21:45:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.249+0000 7fd15201a740 1 -- 172.21.15.1:0/950082386 wait complete. 2024-04-26T21:45:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.249+0000 7fd15201a740 20 client.16473 trim_cache size 0 max 16384 2024-04-26T21:45:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:45:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:45:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:45:30.251 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:45:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.250+0000 7fd14f200640 1 -- 172.21.15.1:0/2896897461 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd12c0a3d90 msgr2=0x7fd12c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.250+0000 7fd14f200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd12c0a3d90 0x7fd12c0a6250 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd140009720 tx=0x7fd1400046c0 comp rx=0 tx=0).stop 2024-04-26T21:45:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.250+0000 7fd14f200640 1 -- 172.21.15.1:0/2896897461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1481ac990 msgr2=0x7fd1481ace40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.250+0000 7fd14f200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1481ac990 0x7fd1481ace40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd140009750 tx=0x7fd140009c10 comp rx=0 tx=0).stop 2024-04-26T21:45:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.250+0000 7fd14f200640 1 -- 172.21.15.1:0/2896897461 shutdown_connections 2024-04-26T21:45:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.250+0000 7fd14f200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd12c0a3d90 0x7fd12c0a6250 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.250+0000 7fd14f200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1481ac990 0x7fd1481ace40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:30.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.250+0000 7fd14f200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd14810a2f0 0x7fd1481a8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.250+0000 7fd14f200640 1 --2- 172.21.15.1:0/2896897461 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd148109930 0x7fd1481a80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.250+0000 7fd14f200640 1 -- 172.21.15.1:0/2896897461 >> 172.21.15.1:0/2896897461 conn(0x7fd14806da10 msgr2=0x7fd1480723b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.251+0000 7fd14f200640 1 -- 172.21.15.1:0/2896897461 shutdown_connections 2024-04-26T21:45:30.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:30.251+0000 7fd14f200640 1 -- 172.21.15.1:0/2896897461 wait complete. 2024-04-26T21:45:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:30 smithi186 ceph-mon[32991]: pgmap v476: 129 pgs: 129 active+clean; 859 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.6 MiB/s wr, 660 op/s 2024-04-26T21:45:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:30 smithi001 ceph-mon[28054]: pgmap v476: 129 pgs: 129 active+clean; 859 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.6 MiB/s wr, 660 op/s 2024-04-26T21:45:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:30 smithi038 ceph-mon[32724]: pgmap v476: 129 pgs: 129 active+clean; 859 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.6 MiB/s wr, 660 op/s 2024-04-26T21:45:30.954 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:45:30 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: ignoring --setuser ceph since I am not root 2024-04-26T21:45:30.954 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:45:30 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: ignoring --setgroup ceph since I am not root 2024-04-26T21:45:30.954 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:45:30 smithi186 ceph-mds[63876]: -- 172.21.15.186:0/4080889173 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b836a7a40 con 0x559b8367a800 2024-04-26T21:45:30.955 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:45:30 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: starting mds.i at 2024-04-26T21:45:31.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 39537 26.09 MB/sec warmup 88 sec latency 255.536 ms 2024-04-26T21:45:31.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:31 smithi186 ceph-mon[32991]: daemon mds.i restarted 2024-04-26T21:45:31.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:31 smithi001 ceph-mon[28054]: daemon mds.i restarted 2024-04-26T21:45:31.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:31 smithi038 ceph-mon[32724]: daemon mds.i restarted 2024-04-26T21:45:32.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 40248 26.30 MB/sec warmup 89 sec latency 66.069 ms 2024-04-26T21:45:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:32 smithi186 ceph-mon[32991]: pgmap v477: 129 pgs: 129 active+clean; 883 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 856 op/s 2024-04-26T21:45:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:32 smithi186 ceph-mon[32991]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T21:45:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:32 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.186:6838/2448285392,v1:172.21.15.186:6839/2448285392] up:boot 2024-04-26T21:45:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:32 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T21:45:32.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:32 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:45:32.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:32 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:45:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:32 smithi001 ceph-mon[28054]: pgmap v477: 129 pgs: 129 active+clean; 883 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 856 op/s 2024-04-26T21:45:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:32 smithi001 ceph-mon[28054]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T21:45:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:32 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.186:6838/2448285392,v1:172.21.15.186:6839/2448285392] up:boot 2024-04-26T21:45:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:32 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T21:45:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:32 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:45:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:32 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:45:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:32 smithi038 ceph-mon[32724]: pgmap v477: 129 pgs: 129 active+clean; 883 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 856 op/s 2024-04-26T21:45:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:32 smithi038 ceph-mon[32724]: osdmap e115: 12 total, 12 up, 12 in 2024-04-26T21:45:32.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:32 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.186:6838/2448285392,v1:172.21.15.186:6839/2448285392] up:boot 2024-04-26T21:45:32.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:32 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T21:45:32.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:32 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:45:32.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:32 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:45:33.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 40817 26.40 MB/sec warmup 90 sec latency 76.674 ms 2024-04-26T21:45:34.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 41282 26.39 MB/sec warmup 91 sec latency 155.476 ms 2024-04-26T21:45:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:34 smithi186 ceph-mon[32991]: pgmap v479: 129 pgs: 129 active+clean; 883 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.3 MiB/s wr, 591 op/s 2024-04-26T21:45:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:34 smithi001 ceph-mon[28054]: pgmap v479: 129 pgs: 129 active+clean; 883 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.3 MiB/s wr, 591 op/s 2024-04-26T21:45:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:34 smithi038 ceph-mon[32724]: pgmap v479: 129 pgs: 129 active+clean; 883 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.3 MiB/s wr, 591 op/s 2024-04-26T21:45:35.129 INFO:tasks.workunit.client.0.smithi001.stdout: 10 41722 26.36 MB/sec warmup 92 sec latency 104.426 ms 2024-04-26T21:45:36.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 42351 26.40 MB/sec warmup 93 sec latency 42.209 ms 2024-04-26T21:45:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:36 smithi186 ceph-mon[32991]: pgmap v480: 129 pgs: 129 active+clean; 889 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 528 op/s 2024-04-26T21:45:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:36 smithi001 ceph-mon[28054]: pgmap v480: 129 pgs: 129 active+clean; 889 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 528 op/s 2024-04-26T21:45:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:36 smithi038 ceph-mon[32724]: pgmap v480: 129 pgs: 129 active+clean; 889 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 528 op/s 2024-04-26T21:45:37.128 INFO:tasks.workunit.client.0.smithi001.stdout: 10 43049 26.50 MB/sec warmup 94 sec latency 48.361 ms 2024-04-26T21:45:38.129 INFO:tasks.workunit.client.0.smithi001.stdout: 10 43722 26.62 MB/sec warmup 95 sec latency 92.535 ms 2024-04-26T21:45:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:38 smithi186 ceph-mon[32991]: pgmap v481: 129 pgs: 129 active+clean; 893 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 544 op/s 2024-04-26T21:45:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:38 smithi001 ceph-mon[28054]: pgmap v481: 129 pgs: 129 active+clean; 893 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 544 op/s 2024-04-26T21:45:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:38 smithi038 ceph-mon[32724]: pgmap v481: 129 pgs: 129 active+clean; 893 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 544 op/s 2024-04-26T21:45:39.129 INFO:tasks.workunit.client.0.smithi001.stdout: 10 44260 26.68 MB/sec warmup 96 sec latency 174.471 ms 2024-04-26T21:45:40.129 INFO:tasks.workunit.client.0.smithi001.stdout: 10 44893 26.74 MB/sec warmup 97 sec latency 89.106 ms 2024-04-26T21:45:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:40 smithi186 ceph-mon[32991]: pgmap v482: 129 pgs: 129 active+clean; 893 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 544 op/s 2024-04-26T21:45:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:40 smithi001 ceph-mon[28054]: pgmap v482: 129 pgs: 129 active+clean; 893 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 544 op/s 2024-04-26T21:45:40.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:40 smithi038 ceph-mon[32724]: pgmap v482: 129 pgs: 129 active+clean; 893 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.2 MiB/s wr, 544 op/s 2024-04-26T21:45:41.129 INFO:tasks.workunit.client.0.smithi001.stdout: 10 45042 26.56 MB/sec warmup 98 sec latency 525.323 ms 2024-04-26T21:45:42.129 INFO:tasks.workunit.client.0.smithi001.stdout: 10 45350 26.40 MB/sec warmup 99 sec latency 96.094 ms 2024-04-26T21:45:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:42 smithi186 ceph-mon[32991]: pgmap v483: 129 pgs: 129 active+clean; 916 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.0 MiB/s wr, 508 op/s 2024-04-26T21:45:42.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:42 smithi001 ceph-mon[28054]: pgmap v483: 129 pgs: 129 active+clean; 916 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.0 MiB/s wr, 508 op/s 2024-04-26T21:45:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:42 smithi038 ceph-mon[32724]: pgmap v483: 129 pgs: 129 active+clean; 916 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.0 MiB/s wr, 508 op/s 2024-04-26T21:45:43.129 INFO:tasks.workunit.client.0.smithi001.stdout: 10 45928 26.48 MB/sec warmup 100 sec latency 76.331 ms 2024-04-26T21:45:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.466+0000 7fede0600640 1 -- 172.21.15.1:0/1248606458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedd810cf90 msgr2=0x7fedd810f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.466+0000 7fede0600640 1 --2- 172.21.15.1:0/1248606458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedd810cf90 0x7fedd810f3c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009be0 tx=0x7fedd002a770 comp rx=0 tx=0).stop 2024-04-26T21:45:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.466+0000 7fede0600640 1 -- 172.21.15.1:0/1248606458 shutdown_connections 2024-04-26T21:45:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.466+0000 7fede0600640 1 --2- 172.21.15.1:0/1248606458 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedd810cf90 0x7fedd810f3c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.466+0000 7fede0600640 1 --2- 172.21.15.1:0/1248606458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd81044e0 0x7fedd810c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.467+0000 7fede0600640 1 --2- 172.21.15.1:0/1248606458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd8103bc0 0x7fedd8103fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.467+0000 7fede0600640 1 -- 172.21.15.1:0/1248606458 >> 172.21.15.1:0/1248606458 conn(0x7fedd8078950 msgr2=0x7fedd8078d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.467+0000 7fede0600640 1 -- 172.21.15.1:0/1248606458 shutdown_connections 2024-04-26T21:45:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.467+0000 7fede0600640 1 -- 172.21.15.1:0/1248606458 wait complete. 2024-04-26T21:45:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.470+0000 7fede0600640 1 Processor -- start 2024-04-26T21:45:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.470+0000 7fede0600640 1 -- start start 2024-04-26T21:45:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.471+0000 7fede0600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedd8103bc0 0x7fedd8108d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.471+0000 7fede0600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd81044e0 0x7fedd8109250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.471+0000 7fedddc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedd8103bc0 0x7fedd8108d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.471+0000 7fedddc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedd8103bc0 0x7fedd8108d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34984/0 (socket says 172.21.15.1:34984) 2024-04-26T21:45:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.471+0000 7fedddc00640 1 -- 172.21.15.1:0/1032249394 learned_addr learned my addr 172.21.15.1:0/1032249394 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.471+0000 7feddd200640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd81044e0 0x7fedd8109250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.472+0000 7fede0600640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd810cf90 0x7fedd8109790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.473+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8102520 con 0x7fedd810cf90 2024-04-26T21:45:43.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.473+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fedd81023a0 con 0x7fedd8103bc0 2024-04-26T21:45:43.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.473+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fedd81026a0 con 0x7fedd81044e0 2024-04-26T21:45:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.474+0000 7feddd200640 1 -- 172.21.15.1:0/1032249394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedd8103bc0 msgr2=0x7fedd8108d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.474+0000 7feddd200640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedd8103bc0 0x7fedd8108d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.474+0000 7feddd200640 1 -- 172.21.15.1:0/1032249394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd810cf90 msgr2=0x7fedd8109790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.474+0000 7feddd200640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd810cf90 0x7fedd8109790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.474+0000 7feddd200640 1 -- 172.21.15.1:0/1032249394 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd81ab9e0 con 0x7fedd81044e0 2024-04-26T21:45:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.475+0000 7feddd200640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd81044e0 0x7fedd8109250 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fedc400b420 tx=0x7fedc400b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.475+0000 7fedd6000640 1 -- 172.21.15.1:0/1032249394 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4011830 con 0x7fedd81044e0 2024-04-26T21:45:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.476+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd81abcd0 con 0x7fedd81044e0 2024-04-26T21:45:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.476+0000 7fedd6000640 1 -- 172.21.15.1:0/1032249394 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedc4016070 con 0x7fedd81044e0 2024-04-26T21:45:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.476+0000 7fedd6000640 1 -- 172.21.15.1:0/1032249394 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4010dd0 con 0x7fedd81044e0 2024-04-26T21:45:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.476+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd81ac210 con 0x7fedd81044e0 2024-04-26T21:45:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.486+0000 7fede3428740 1 Processor -- start 2024-04-26T21:45:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.487+0000 7fede3428740 1 -- start start 2024-04-26T21:45:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.488+0000 7fede3428740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 0x5565a20a81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.489+0000 7fedde600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 0x5565a20a81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.489+0000 7fedde600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 0x5565a20a81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:40286/0 (socket says 172.21.15.1:40286) 2024-04-26T21:45:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.489+0000 7fede3428740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a86f0 0x5565a20b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.489+0000 7fede3428740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 0x5565a20b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.489+0000 7fede3428740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5565a200d380 con 0x5565a20a86f0 2024-04-26T21:45:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.489+0000 7fede3428740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5565a1fa9c30 con 0x5565a20b2fb0 2024-04-26T21:45:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.489+0000 7fede3428740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5565a2009340 con 0x5565a20a7dd0 2024-04-26T21:45:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.489+0000 7fedde600640 1 -- 172.21.15.1:0/1742409724 learned_addr learned my addr 172.21.15.1:0/1742409724 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.490+0000 7fedde600640 1 -- 172.21.15.1:0/1742409724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 msgr2=0x5565a20b53a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:45:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.490+0000 7fedde600640 1 --2- 172.21.15.1:0/1742409724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 0x5565a20b53a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.490+0000 7fedde600640 1 -- 172.21.15.1:0/1742409724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a86f0 msgr2=0x5565a20b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.490+0000 7fedde600640 1 --2- 172.21.15.1:0/1742409724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a86f0 0x5565a20b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.490+0000 7fedde600640 1 -- 172.21.15.1:0/1742409724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565a1fcb940 con 0x5565a20a7dd0 2024-04-26T21:45:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.491+0000 7fedd6000640 1 -- 172.21.15.1:0/1032249394 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fedc40119d0 con 0x7fedd81044e0 2024-04-26T21:45:43.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.493+0000 7fedde600640 1 --2- 172.21.15.1:0/1742409724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 0x5565a20a81b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fedd0034040 tx=0x7fedd0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7ed957e5e300dfc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.493+0000 7fedcaa00640 1 -- 172.21.15.1:0/1742409724 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0040020 con 0x5565a20a7dd0 2024-04-26T21:45:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.494+0000 7fedd6000640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fedc00a3e10 0x7fedc00a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.494+0000 7fedcaa00640 1 -- 172.21.15.1:0/1742409724 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd0003e20 con 0x5565a20a7dd0 2024-04-26T21:45:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.494+0000 7fedd6000640 1 -- 172.21.15.1:0/1032249394 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7fedc40d00b0 con 0x7fedd81044e0 2024-04-26T21:45:43.494 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.494+0000 7fedcaa00640 1 -- 172.21.15.1:0/1742409724 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0036040 con 0x5565a20a7dd0 2024-04-26T21:45:43.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.494+0000 7fedddc00640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fedc00a3e10 0x7fedc00a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.495+0000 7fede3428740 1 -- 172.21.15.1:0/1742409724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 msgr2=0x5565a20a81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.495+0000 7fede3428740 1 --2- 172.21.15.1:0/1742409724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 0x5565a20a81b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fedd0034040 tx=0x7fedd0009310 comp rx=0 tx=0).stop 2024-04-26T21:45:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.495+0000 7fede3428740 1 -- 172.21.15.1:0/1742409724 shutdown_connections 2024-04-26T21:45:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.495+0000 7fede3428740 1 --2- 172.21.15.1:0/1742409724 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 0x5565a20b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.495+0000 7fede3428740 1 --2- 172.21.15.1:0/1742409724 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a86f0 0x5565a20b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.495+0000 7fede3428740 1 --2- 172.21.15.1:0/1742409724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 0x5565a20a81b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.495+0000 7fede3428740 1 -- 172.21.15.1:0/1742409724 >> 172.21.15.1:0/1742409724 conn(0x5565a20a42e0 msgr2=0x5565a20a46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.495+0000 7fedddc00640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fedc00a3e10 0x7fedc00a62d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00b770 tx=0x7fedcc00b700 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.496+0000 7fede3428740 1 -- 172.21.15.1:0/1742409724 shutdown_connections 2024-04-26T21:45:43.496 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.496+0000 7fede3428740 1 -- 172.21.15.1:0/1742409724 wait complete. 2024-04-26T21:45:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.496+0000 7fede3428740 1 -- start start 2024-04-26T21:45:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.497+0000 7fede3428740 1 Processor -- start 2024-04-26T21:45:43.497 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.497+0000 7fede3428740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a7dd0 0x5565a20affe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.497+0000 7fede3428740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a86f0 0x5565a20b0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.498 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.497+0000 7fede3428740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20b2fb0 0x5565a20b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.497+0000 7fede3428740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5565a20c6b90 con 0x5565a20b2fb0 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.497+0000 7fede3428740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5565a1f94c60 con 0x5565a20a7dd0 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.497+0000 7fede3428740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5565a20c6d10 con 0x5565a20a86f0 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.497+0000 7fede3428740 10 client.0 _ll_register_callbacks cb 0x5565a1fc0e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.498+0000 7fedddc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a7dd0 0x5565a20affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.498+0000 7fedddc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a7dd0 0x5565a20affe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:34988/0 (socket says 172.21.15.1:34988) 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.498+0000 7fedddc00640 1 -- 172.21.15.1:0/1212682056 learned_addr learned my addr 172.21.15.1:0/1212682056 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.498+0000 7feddd200640 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a86f0 0x5565a20b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.498+0000 7feddd200640 1 -- 172.21.15.1:0/1212682056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a7dd0 msgr2=0x5565a20affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.498+0000 7feddd200640 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a7dd0 0x5565a20affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.498+0000 7feddd200640 1 -- 172.21.15.1:0/1212682056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20b2fb0 msgr2=0x5565a20b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.498+0000 7feddd200640 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20b2fb0 0x5565a20b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.498+0000 7feddd200640 1 -- 172.21.15.1:0/1212682056 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565a1fa3510 con 0x5565a20a86f0 2024-04-26T21:45:43.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.498+0000 7feddd200640 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a86f0 0x5565a20b0520 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fedc40cd4a0 tx=0x7fedc409b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ad1df5125f2a5d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:43.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.499+0000 7fedbf400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.499+0000 7fedbf400640 1 -- 172.21.15.1:0/1212682056 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc40a00a0 con 0x5565a20a86f0 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.499+0000 7fedbf400640 1 -- 172.21.15.1:0/1212682056 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7fedc4011cc0 con 0x5565a20a86f0 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.499+0000 7fede3428740 1 -- 172.21.15.1:0/1212682056 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedd0003e20 con 0x5565a20a86f0 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.500+0000 7fedbf400640 1 -- 172.21.15.1:0/1212682056 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fedc40a4960 con 0x5565a20a86f0 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.500+0000 7fede3428740 10 client.26689 fetch_fsmap learned FSMap version 46 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.500+0000 7fede3428740 1 -- 172.21.15.1:0/1212682056 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5565a1d1f210 con 0x5565a20a86f0 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.500+0000 7fedbf400640 1 -- 172.21.15.1:0/1212682056 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedd0003e20 con 0x5565a20a86f0 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.500+0000 7fedbf400640 1 -- 172.21.15.1:0/1212682056 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 46) ==== 8370+0+0 (secure 0 0 0) 0x7fedc40cdae0 con 0x5565a20a86f0 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.501+0000 7fede3428740 10 client.26689 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.501+0000 7fede3428740 10 client.26689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:45:43.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.501+0000 7fede3428740 20 client.26689 populate_metadata read hostname 'smithi001' 2024-04-26T21:45:43.502 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.501+0000 7fede3428740 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5565a20d3d50 0x5565a20d6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.502 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.501+0000 7fede3428740 4 client.26689 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:45:43.502 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.501+0000 7fede3428740 1 -- 172.21.15.1:0/1212682056 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565a201af90 con 0x5565a20d3d50 2024-04-26T21:45:43.502 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.501+0000 7fedddc00640 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5565a20d3d50 0x5565a20d6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.502+0000 7fedddc00640 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5565a20d3d50 0x5565a20d6140 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4db63a70b695cc54 server_cookie=385c9c52aec4661b in_seq=0 out_seq=0 2024-04-26T21:45:43.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.502+0000 7fedbf400640 10 client.26689 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.508+0000 7fedbf400640 1 -- 172.21.15.1:0/1212682056 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5565a201af90 con 0x5565a20d3d50 2024-04-26T21:45:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.508+0000 7fedbf400640 10 client.26689 handle_command_reply: tid=0 2024-04-26T21:45:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.508+0000 7fede3428740 1 client.26689 shutdown 2024-04-26T21:45:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.508+0000 7fede3428740 1 -- 172.21.15.1:0/1212682056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a86f0 msgr2=0x5565a20b0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.508+0000 7fede3428740 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a86f0 0x5565a20b0520 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fedc40cd4a0 tx=0x7fedc409b940 comp rx=0 tx=0).stop 2024-04-26T21:45:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.509+0000 7fede3428740 1 -- 172.21.15.1:0/1212682056 shutdown_connections 2024-04-26T21:45:43.509 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.509+0000 7fede3428740 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5565a20d3d50 0x5565a20d6140 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.509+0000 7fede3428740 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a86f0 0x5565a20b0520 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.509+0000 7fede3428740 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20b2fb0 0x5565a20b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.509+0000 7fede3428740 1 --2- 172.21.15.1:0/1212682056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a7dd0 0x5565a20affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.509+0000 7fedbf400640 0 client.26689 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.509+0000 7fede3428740 1 -- 172.21.15.1:0/1212682056 >> 172.21.15.1:0/1212682056 conn(0x5565a20a42e0 msgr2=0x5565a20aa060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.509+0000 7fede3428740 1 -- 172.21.15.1:0/1212682056 shutdown_connections 2024-04-26T21:45:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.509+0000 7fede3428740 1 -- 172.21.15.1:0/1212682056 wait complete. 2024-04-26T21:45:43.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.509+0000 7fede3428740 20 client.26689 trim_cache size 0 max 16384 2024-04-26T21:45:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.539+0000 7fede3428740 1 Processor -- start 2024-04-26T21:45:43.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.539+0000 7fede3428740 1 -- start start 2024-04-26T21:45:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.540+0000 7fede3428740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a7dd0 0x5565a20cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.540+0000 7fede3428740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a86f0 0x5565a20b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.540+0000 7fede3428740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20b2fb0 0x5565a20dba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.540+0000 7fede3428740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fedd81b04d0 con 0x5565a20a7dd0 2024-04-26T21:45:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.540+0000 7fede3428740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fedd81adce0 con 0x5565a20a86f0 2024-04-26T21:45:43.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.540+0000 7fede3428740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8074190 con 0x5565a20b2fb0 2024-04-26T21:45:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.540+0000 7fedde600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20b2fb0 0x5565a20dba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.540+0000 7fedde600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20b2fb0 0x5565a20dba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:40312/0 (socket says 172.21.15.1:40312) 2024-04-26T21:45:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.540+0000 7fedde600640 1 -- 172.21.15.1:0/2432679538 learned_addr learned my addr 172.21.15.1:0/2432679538 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.540+0000 7feddd200640 1 --2- 172.21.15.1:0/2432679538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a86f0 0x5565a20b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.541+0000 7fedddc00640 1 --2- 172.21.15.1:0/2432679538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a7dd0 0x5565a20cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.541+0000 7feddd200640 1 -- 172.21.15.1:0/2432679538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20b2fb0 msgr2=0x5565a20dba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.541+0000 7feddd200640 1 --2- 172.21.15.1:0/2432679538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20b2fb0 0x5565a20dba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.541+0000 7feddd200640 1 -- 172.21.15.1:0/2432679538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a7dd0 msgr2=0x5565a20cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.541+0000 7feddd200640 1 --2- 172.21.15.1:0/2432679538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a7dd0 0x5565a20cfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.541+0000 7feddd200640 1 -- 172.21.15.1:0/2432679538 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565a1deb620 con 0x5565a20a86f0 2024-04-26T21:45:43.542 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.541+0000 7fedddc00640 1 --2- 172.21.15.1:0/2432679538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a7dd0 0x5565a20cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:45:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.541+0000 7feddd200640 1 --2- 172.21.15.1:0/2432679538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a86f0 0x5565a20b0880 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fedc400b3f0 tx=0x7fedc409b830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f7ef7ae73733c15 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.542+0000 7fedca000640 1 -- 172.21.15.1:0/2432679538 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc40bc0e0 con 0x5565a20a86f0 2024-04-26T21:45:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.542+0000 7fedca000640 1 -- 172.21.15.1:0/2432679538 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedc40a00a0 con 0x5565a20a86f0 2024-04-26T21:45:43.543 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.542+0000 7fedca000640 1 -- 172.21.15.1:0/2432679538 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc4098360 con 0x5565a20a86f0 2024-04-26T21:45:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.543+0000 7fede3428740 1 -- 172.21.15.1:0/2432679538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a86f0 msgr2=0x5565a20b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.543+0000 7fede3428740 1 --2- 172.21.15.1:0/2432679538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a86f0 0x5565a20b0880 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fedc400b3f0 tx=0x7fedc409b830 comp rx=0 tx=0).stop 2024-04-26T21:45:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.543+0000 7fede3428740 1 -- 172.21.15.1:0/2432679538 shutdown_connections 2024-04-26T21:45:43.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.544+0000 7fede3428740 1 --2- 172.21.15.1:0/2432679538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20b2fb0 0x5565a20dba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.544+0000 7fede3428740 1 --2- 172.21.15.1:0/2432679538 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20a86f0 0x5565a20b0880 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.544+0000 7fede3428740 1 --2- 172.21.15.1:0/2432679538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a7dd0 0x5565a20cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.544+0000 7fede3428740 1 -- 172.21.15.1:0/2432679538 >> 172.21.15.1:0/2432679538 conn(0x5565a20a42e0 msgr2=0x5565a20a3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.544+0000 7fede3428740 1 -- 172.21.15.1:0/2432679538 shutdown_connections 2024-04-26T21:45:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.544+0000 7fede3428740 1 -- 172.21.15.1:0/2432679538 wait complete. 2024-04-26T21:45:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.545+0000 7fede3428740 1 -- start start 2024-04-26T21:45:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.545+0000 7fede3428740 1 Processor -- start 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.546+0000 7fede3428740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 0x5565a20ad3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.546+0000 7fede3428740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a86f0 0x5565a20ad910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.546+0000 7fede3428740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 0x5565a2139270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.546+0000 7fede3428740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8074800 con 0x5565a20a86f0 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.546+0000 7fede3428740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fedd81ad610 con 0x5565a20b2fb0 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.546+0000 7fede3428740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fedd81ac870 con 0x5565a20a7dd0 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.546+0000 7fede3428740 10 client.0 _ll_register_callbacks cb 0x7fedc409bbb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.546+0000 7fedde600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 0x5565a2139270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.546+0000 7fedde600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 0x5565a2139270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:35012/0 (socket says 172.21.15.1:35012) 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.546+0000 7fedde600640 1 -- 172.21.15.1:0/3905236222 learned_addr learned my addr 172.21.15.1:0/3905236222 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.547+0000 7fedddc00640 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 0x5565a20ad3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.547+0000 7fedde600640 1 -- 172.21.15.1:0/3905236222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 msgr2=0x5565a20ad3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.547+0000 7fedde600640 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 0x5565a20ad3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.547+0000 7fedde600640 1 -- 172.21.15.1:0/3905236222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a86f0 msgr2=0x5565a20ad910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.547+0000 7feddd200640 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a86f0 0x5565a20ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.547+0000 7fedde600640 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a86f0 0x5565a20ad910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.548+0000 7fedde600640 1 -- 172.21.15.1:0/3905236222 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565a1fc7cb0 con 0x5565a20b2fb0 2024-04-26T21:45:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.548+0000 7feddd200640 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a86f0 0x5565a20ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:45:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.550+0000 7fedde600640 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 0x5565a2139270 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009560 tx=0x7fedd0003af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ec4cc3059609b34 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.550+0000 7fedbf400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:45:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.551+0000 7fedbf400640 1 -- 172.21.15.1:0/3905236222 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0052060 con 0x5565a20b2fb0 2024-04-26T21:45:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.552+0000 7fede3428740 1 -- 172.21.15.1:0/3905236222 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedc40a00a0 con 0x5565a20b2fb0 2024-04-26T21:45:43.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.552+0000 7fedbf400640 1 -- 172.21.15.1:0/3905236222 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7fedd0030b00 con 0x5565a20b2fb0 2024-04-26T21:45:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.553+0000 7fedbf400640 1 -- 172.21.15.1:0/3905236222 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fedd0040040 con 0x5565a20b2fb0 2024-04-26T21:45:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.553+0000 7fede3428740 10 client.26717 fetch_fsmap learned FSMap version 46 2024-04-26T21:45:43.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.553+0000 7fede3428740 1 -- 172.21.15.1:0/3905236222 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5565a214b820 con 0x5565a20b2fb0 2024-04-26T21:45:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.554+0000 7fedbf400640 1 -- 172.21.15.1:0/3905236222 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fedc40a00a0 con 0x5565a20b2fb0 2024-04-26T21:45:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.554+0000 7fedbf400640 1 -- 172.21.15.1:0/3905236222 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 46) ==== 8370+0+0 (secure 0 0 0) 0x7fedd003c470 con 0x5565a20b2fb0 2024-04-26T21:45:43.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.554+0000 7fede3428740 10 client.26717 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T21:45:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.554+0000 7fede3428740 10 client.26717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:45:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.554+0000 7fede3428740 20 client.26717 populate_metadata read hostname 'smithi001' 2024-04-26T21:45:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.554+0000 7fede3428740 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5565a214bb80 0x5565a214dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.554+0000 7fede3428740 4 client.26717 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:45:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.554+0000 7fede3428740 1 -- 172.21.15.1:0/3905236222 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5565a2014e00 con 0x5565a214bb80 2024-04-26T21:45:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.555+0000 7fedddc00640 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5565a214bb80 0x5565a214dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:43.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.555+0000 7fedbf400640 10 client.26717 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:43.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.557+0000 7fedddc00640 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5565a214bb80 0x5565a214dfa0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec365cd7993ce139 server_cookie=2a2b48f5d6d96604 in_seq=0 out_seq=0 2024-04-26T21:45:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.962+0000 7fedbf400640 1 -- 172.21.15.1:0/3905236222 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5565a2014e00 con 0x5565a214bb80 2024-04-26T21:45:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.962+0000 7fedbf400640 10 client.26717 handle_command_reply: tid=0 2024-04-26T21:45:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.962+0000 7fede3428740 1 client.26717 shutdown 2024-04-26T21:45:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.963+0000 7fede3428740 1 -- 172.21.15.1:0/3905236222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 msgr2=0x5565a2139270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.963+0000 7fede3428740 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 0x5565a2139270 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009560 tx=0x7fedd0003af0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.963+0000 7fede3428740 1 -- 172.21.15.1:0/3905236222 shutdown_connections 2024-04-26T21:45:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.963+0000 7fede3428740 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5565a214bb80 0x5565a214dfa0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.963+0000 7fede3428740 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5565a20b2fb0 0x5565a2139270 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.963+0000 7fede3428740 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5565a20a86f0 0x5565a20ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.963+0000 7fede3428740 1 --2- 172.21.15.1:0/3905236222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565a20a7dd0 0x5565a20ad3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.963+0000 7fede3428740 1 -- 172.21.15.1:0/3905236222 >> 172.21.15.1:0/3905236222 conn(0x5565a20a42e0 msgr2=0x5565a20a9780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.963+0000 7fedbf400640 0 client.26717 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.964+0000 7fede3428740 1 -- 172.21.15.1:0/3905236222 shutdown_connections 2024-04-26T21:45:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.964+0000 7fede3428740 1 -- 172.21.15.1:0/3905236222 wait complete. 2024-04-26T21:45:43.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.964+0000 7fede3428740 20 client.26717 trim_cache size 0 max 16384 2024-04-26T21:45:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:45:43.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:45:43.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:45:43.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:45:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.969+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fedc00a3e10 msgr2=0x7fedc00a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.969+0000 7fede0600640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fedc00a3e10 0x7fedc00a62d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00b770 tx=0x7fedcc00b700 comp rx=0 tx=0).stop 2024-04-26T21:45:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.969+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd81044e0 msgr2=0x7fedd8109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.969+0000 7fede0600640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd81044e0 0x7fedd8109250 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fedc400b420 tx=0x7fedc400b8e0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.970+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 shutdown_connections 2024-04-26T21:45:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.970+0000 7fede0600640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fedc00a3e10 0x7fedc00a62d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.970+0000 7fede0600640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd81044e0 0x7fedd8109250 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.970+0000 7fede0600640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fedd810cf90 0x7fedd8109790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.970+0000 7fede0600640 1 --2- 172.21.15.1:0/1032249394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedd8103bc0 0x7fedd8108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.970+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 >> 172.21.15.1:0/1032249394 conn(0x7fedd8078950 msgr2=0x7fedd810ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.970+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 shutdown_connections 2024-04-26T21:45:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:43.970+0000 7fede0600640 1 -- 172.21.15.1:0/1032249394 wait complete. 2024-04-26T21:45:44.129 INFO:tasks.workunit.client.0.smithi001.stdout: 10 46331 26.41 MB/sec warmup 101 sec latency 277.980 ms 2024-04-26T21:45:44.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:44 smithi186 ceph-mon[32991]: pgmap v484: 129 pgs: 129 active+clean; 916 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.0 MiB/s wr, 426 op/s 2024-04-26T21:45:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:44 smithi001 ceph-mon[28054]: pgmap v484: 129 pgs: 129 active+clean; 916 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.0 MiB/s wr, 426 op/s 2024-04-26T21:45:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:44 smithi038 ceph-mon[32724]: pgmap v484: 129 pgs: 129 active+clean; 916 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.0 MiB/s wr, 426 op/s 2024-04-26T21:45:45.130 INFO:tasks.workunit.client.0.smithi001.stdout: 10 46876 26.46 MB/sec warmup 102 sec latency 430.005 ms 2024-04-26T21:45:46.130 INFO:tasks.workunit.client.0.smithi001.stdout: 10 47392 26.47 MB/sec warmup 103 sec latency 248.661 ms 2024-04-26T21:45:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:46 smithi001 ceph-mon[28054]: pgmap v485: 129 pgs: 129 active+clean; 915 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.9 MiB/s wr, 704 op/s 2024-04-26T21:45:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:46 smithi038 ceph-mon[32724]: pgmap v485: 129 pgs: 129 active+clean; 915 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.9 MiB/s wr, 704 op/s 2024-04-26T21:45:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:46 smithi186 ceph-mon[32991]: pgmap v485: 129 pgs: 129 active+clean; 915 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.9 MiB/s wr, 704 op/s 2024-04-26T21:45:47.130 INFO:tasks.workunit.client.0.smithi001.stdout: 10 47494 26.32 MB/sec warmup 104 sec latency 605.232 ms 2024-04-26T21:45:48.130 INFO:tasks.workunit.client.0.smithi001.stdout: 10 47701 26.14 MB/sec warmup 105 sec latency 290.936 ms 2024-04-26T21:45:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:48 smithi001 ceph-mon[28054]: pgmap v486: 129 pgs: 129 active+clean; 920 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.1 MiB/s wr, 574 op/s 2024-04-26T21:45:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:48 smithi038 ceph-mon[32724]: pgmap v486: 129 pgs: 129 active+clean; 920 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.1 MiB/s wr, 574 op/s 2024-04-26T21:45:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:48 smithi186 ceph-mon[32991]: pgmap v486: 129 pgs: 129 active+clean; 920 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.1 MiB/s wr, 574 op/s 2024-04-26T21:45:49.130 INFO:tasks.workunit.client.0.smithi001.stdout: 10 48008 26.11 MB/sec warmup 106 sec latency 178.577 ms 2024-04-26T21:45:50.130 INFO:tasks.workunit.client.0.smithi001.stdout: 10 48525 26.09 MB/sec warmup 107 sec latency 84.043 ms 2024-04-26T21:45:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:50 smithi001 ceph-mon[28054]: pgmap v487: 129 pgs: 129 active+clean; 920 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.6 MiB/s wr, 537 op/s 2024-04-26T21:45:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:50 smithi038 ceph-mon[32724]: pgmap v487: 129 pgs: 129 active+clean; 920 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.6 MiB/s wr, 537 op/s 2024-04-26T21:45:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:50 smithi186 ceph-mon[32991]: pgmap v487: 129 pgs: 129 active+clean; 920 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.6 MiB/s wr, 537 op/s 2024-04-26T21:45:51.130 INFO:tasks.workunit.client.0.smithi001.stdout: 10 49116 26.05 MB/sec warmup 108 sec latency 78.474 ms 2024-04-26T21:45:52.130 INFO:tasks.workunit.client.0.smithi001.stdout: 10 49763 26.19 MB/sec warmup 109 sec latency 81.103 ms 2024-04-26T21:45:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:52 smithi001 ceph-mon[28054]: pgmap v488: 129 pgs: 129 active+clean; 926 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 16 MiB/s wr, 907 op/s 2024-04-26T21:45:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:52 smithi038 ceph-mon[32724]: pgmap v488: 129 pgs: 129 active+clean; 926 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 16 MiB/s wr, 907 op/s 2024-04-26T21:45:52.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:52 smithi186 ceph-mon[32991]: pgmap v488: 129 pgs: 129 active+clean; 926 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 16 MiB/s wr, 907 op/s 2024-04-26T21:45:53.130 INFO:tasks.workunit.client.0.smithi001.stdout: 10 50516 26.31 MB/sec warmup 110 sec latency 54.364 ms 2024-04-26T21:45:54.130 INFO:tasks.workunit.client.0.smithi001.stdout: 10 51021 26.31 MB/sec warmup 111 sec latency 131.888 ms 2024-04-26T21:45:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:54 smithi001 ceph-mon[28054]: pgmap v489: 129 pgs: 129 active+clean; 926 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 14 MiB/s wr, 741 op/s 2024-04-26T21:45:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:54 smithi038 ceph-mon[32724]: pgmap v489: 129 pgs: 129 active+clean; 926 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 14 MiB/s wr, 741 op/s 2024-04-26T21:45:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:54 smithi186 ceph-mon[32991]: pgmap v489: 129 pgs: 129 active+clean; 926 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 14 MiB/s wr, 741 op/s 2024-04-26T21:45:55.137 INFO:tasks.workunit.client.0.smithi001.stdout: 10 51536 26.39 MB/sec warmup 112 sec latency 213.570 ms 2024-04-26T21:45:56.136 INFO:tasks.workunit.client.0.smithi001.stdout: 10 52078 26.39 MB/sec warmup 113 sec latency 51.327 ms 2024-04-26T21:45:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:56 smithi001 ceph-mon[28054]: pgmap v490: 129 pgs: 129 active+clean; 941 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 15 MiB/s wr, 855 op/s 2024-04-26T21:45:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:56 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:45:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:56 smithi038 ceph-mon[32724]: pgmap v490: 129 pgs: 129 active+clean; 941 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 15 MiB/s wr, 855 op/s 2024-04-26T21:45:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:56 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:45:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:56 smithi186 ceph-mon[32991]: pgmap v490: 129 pgs: 129 active+clean; 941 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 15 MiB/s wr, 855 op/s 2024-04-26T21:45:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:56 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:45:57.138 INFO:tasks.workunit.client.0.smithi001.stdout: 10 52647 26.40 MB/sec warmup 114 sec latency 54.027 ms 2024-04-26T21:45:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.169+0000 7fd612800640 1 -- 172.21.15.1:0/3325874254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd60c102020 msgr2=0x7fd60c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.169+0000 7fd612800640 1 --2- 172.21.15.1:0/3325874254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd60c102020 0x7fd60c1064e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd600009900 tx=0x7fd60002a470 comp rx=0 tx=0).stop 2024-04-26T21:45:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.169+0000 7fd612800640 1 -- 172.21.15.1:0/3325874254 shutdown_connections 2024-04-26T21:45:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.169+0000 7fd612800640 1 --2- 172.21.15.1:0/3325874254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd60c106b00 0x7fd60c106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.169+0000 7fd612800640 1 --2- 172.21.15.1:0/3325874254 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd60c102020 0x7fd60c1064e0 secure :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd600009900 tx=0x7fd60002a470 comp rx=0 tx=0).stop 2024-04-26T21:45:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.169+0000 7fd612800640 1 --2- 172.21.15.1:0/3325874254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd60c101700 0x7fd60c101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.169+0000 7fd612800640 1 -- 172.21.15.1:0/3325874254 >> 172.21.15.1:0/3325874254 conn(0x7fd60c0fd110 msgr2=0x7fd60c0ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.170+0000 7fd612800640 1 -- 172.21.15.1:0/3325874254 shutdown_connections 2024-04-26T21:45:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.170+0000 7fd612800640 1 -- 172.21.15.1:0/3325874254 wait complete. 2024-04-26T21:45:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.171+0000 7fd612800640 1 Processor -- start 2024-04-26T21:45:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.171+0000 7fd612800640 1 -- start start 2024-04-26T21:45:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd612800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd60c101700 0x7fd60c19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd612800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd60c106b00 0x7fd60c19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd612800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd60c1a3fd0 0x7fd60c1a4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd612800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c119000 con 0x7fd60c1a3fd0 2024-04-26T21:45:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd612800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c118e80 con 0x7fd60c101700 2024-04-26T21:45:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd612800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c119180 con 0x7fd60c106b00 2024-04-26T21:45:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd611400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd60c101700 0x7fd60c19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd610a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd60c106b00 0x7fd60c19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd611400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd60c101700 0x7fd60c19f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60516/0 (socket says 172.21.15.1:60516) 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd610a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd60c106b00 0x7fd60c19fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:59054/0 (socket says 172.21.15.1:59054) 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.172+0000 7fd611400640 1 -- 172.21.15.1:0/2162558578 learned_addr learned my addr 172.21.15.1:0/2162558578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.173+0000 7fd610a00640 1 -- 172.21.15.1:0/2162558578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd60c101700 msgr2=0x7fd60c19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.173+0000 7fd610a00640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd60c101700 0x7fd60c19f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.173+0000 7fd610a00640 1 -- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd60c1a3fd0 msgr2=0x7fd60c1a4480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.173+0000 7fd611e00640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd60c1a3fd0 0x7fd60c1a4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.173+0000 7fd610a00640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd60c1a3fd0 0x7fd60c1a4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.173+0000 7fd610a00640 1 -- 172.21.15.1:0/2162558578 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60c1a4ac0 con 0x7fd60c106b00 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.173+0000 7fd611e00640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd60c1a3fd0 0x7fd60c1a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:45:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.174+0000 7fd610a00640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd60c106b00 0x7fd60c19fc50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd600008000 tx=0x7fd600003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.174+0000 7fd60a000640 1 -- 172.21.15.1:0/2162558578 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60003f020 con 0x7fd60c106b00 2024-04-26T21:45:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.174+0000 7fd60a000640 1 -- 172.21.15.1:0/2162558578 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd600031b50 con 0x7fd60c106b00 2024-04-26T21:45:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.174+0000 7fd60a000640 1 -- 172.21.15.1:0/2162558578 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd600035070 con 0x7fd60c106b00 2024-04-26T21:45:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.174+0000 7fd612800640 1 -- 172.21.15.1:0/2162558578 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60c1a4d50 con 0x7fd60c106b00 2024-04-26T21:45:57.176 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.174+0000 7fd612800640 1 -- 172.21.15.1:0/2162558578 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60c107c90 con 0x7fd60c106b00 2024-04-26T21:45:57.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.178+0000 7fd616c6f740 1 Processor -- start 2024-04-26T21:45:57.178 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.178+0000 7fd616c6f740 1 -- start start 2024-04-26T21:45:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.179+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e803c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.179+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 0x5588e6e8ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.179+0000 7fd611e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e803c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.179+0000 7fd611e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e803c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60532/0 (socket says 172.21.15.1:60532) 2024-04-26T21:45:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.179+0000 7fd611400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 0x5588e6e8ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.179+0000 7fd611400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 0x5588e6e8ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41034/0 (socket says 172.21.15.1:41034) 2024-04-26T21:45:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.180+0000 7fd60a000640 1 -- 172.21.15.1:0/2162558578 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd600042020 con 0x7fd60c106b00 2024-04-26T21:45:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.181+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6e8d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.181+0000 7fd616c6f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5588e6de5380 con 0x5588e6e80900 2024-04-26T21:45:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.181+0000 7fd616c6f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5588e6d81c30 con 0x5588e6e7ffe0 2024-04-26T21:45:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.181+0000 7fd616c6f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5588e6de1340 con 0x5588e6e8b1c0 2024-04-26T21:45:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.181+0000 7fd611e00640 1 -- 172.21.15.1:0/3928792575 learned_addr learned my addr 172.21.15.1:0/3928792575 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.182+0000 7fd611e00640 1 -- 172.21.15.1:0/3928792575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 msgr2=0x5588e6e8d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.182+0000 7fd610a00640 1 --2- 172.21.15.1:0/3928792575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6e8d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.182+0000 7fd611e00640 1 --2- 172.21.15.1:0/3928792575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6e8d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.182+0000 7fd611e00640 1 -- 172.21.15.1:0/3928792575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 msgr2=0x5588e6e8ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.182+0000 7fd611e00640 1 --2- 172.21.15.1:0/3928792575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 0x5588e6e8ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.182+0000 7fd611e00640 1 -- 172.21.15.1:0/3928792575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588e6da3940 con 0x5588e6e7ffe0 2024-04-26T21:45:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.182+0000 7fd60a000640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd5f40a3de0 0x7fd5f40a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.182+0000 7fd611400640 1 --2- 172.21.15.1:0/3928792575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 0x5588e6e8ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-26T21:45:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.183+0000 7fd60a000640 1 -- 172.21.15.1:0/2162558578 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7fd6000f10a0 con 0x7fd60c106b00 2024-04-26T21:45:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.183+0000 7fd611400640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd5f40a3de0 0x7fd5f40a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.184+0000 7fd610a00640 1 --2- 172.21.15.1:0/3928792575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6e8d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:45:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.184+0000 7fd611e00640 1 --2- 172.21.15.1:0/3928792575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e803c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd604008de0 tx=0x7fd60400d710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a781cbe7d702425 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.185+0000 7fd5fe600640 1 -- 172.21.15.1:0/3928792575 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60400f830 con 0x5588e6e7ffe0 2024-04-26T21:45:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.185+0000 7fd611400640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd5f40a3de0 0x7fd5f40a62a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5588e6e81580 tx=0x7fd5f8005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.185+0000 7fd5fe600640 1 -- 172.21.15.1:0/3928792575 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd60400de20 con 0x5588e6e7ffe0 2024-04-26T21:45:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.185+0000 7fd5fe600640 1 -- 172.21.15.1:0/3928792575 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60400e3b0 con 0x5588e6e7ffe0 2024-04-26T21:45:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.188+0000 7fd616c6f740 1 -- 172.21.15.1:0/3928792575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 msgr2=0x5588e6e803c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.188+0000 7fd616c6f740 1 --2- 172.21.15.1:0/3928792575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e803c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd604008de0 tx=0x7fd60400d710 comp rx=0 tx=0).stop 2024-04-26T21:45:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.188+0000 7fd616c6f740 1 -- 172.21.15.1:0/3928792575 shutdown_connections 2024-04-26T21:45:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.188+0000 7fd616c6f740 1 --2- 172.21.15.1:0/3928792575 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6e8d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.188+0000 7fd616c6f740 1 --2- 172.21.15.1:0/3928792575 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 0x5588e6e8ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.188+0000 7fd616c6f740 1 --2- 172.21.15.1:0/3928792575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e803c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.188+0000 7fd616c6f740 1 -- 172.21.15.1:0/3928792575 >> 172.21.15.1:0/3928792575 conn(0x5588e6e7c4f0 msgr2=0x5588e6e7c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.188+0000 7fd616c6f740 1 -- 172.21.15.1:0/3928792575 shutdown_connections 2024-04-26T21:45:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.188+0000 7fd616c6f740 1 -- 172.21.15.1:0/3928792575 wait complete. 2024-04-26T21:45:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.189+0000 7fd616c6f740 1 -- start start 2024-04-26T21:45:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.189+0000 7fd616c6f740 1 Processor -- start 2024-04-26T21:45:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e881a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 0x5588e6e886e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6e88c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd616c6f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5588e6e8ddf0 con 0x5588e6e80900 2024-04-26T21:45:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd616c6f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5588e6d6cc60 con 0x5588e6e7ffe0 2024-04-26T21:45:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd616c6f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5588e6e8df70 con 0x5588e6e8b1c0 2024-04-26T21:45:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd616c6f740 10 client.0 _ll_register_callbacks cb 0x5588e6d98e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:45:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd611400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e881a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd611400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e881a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60546/0 (socket says 172.21.15.1:60546) 2024-04-26T21:45:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd611400640 1 -- 172.21.15.1:0/755032118 learned_addr learned my addr 172.21.15.1:0/755032118 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.190+0000 7fd611e00640 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6e88c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.191+0000 7fd611400640 1 -- 172.21.15.1:0/755032118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 msgr2=0x5588e6e88c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.191+0000 7fd611400640 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6e88c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.191+0000 7fd611400640 1 -- 172.21.15.1:0/755032118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 msgr2=0x5588e6e886e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.191+0000 7fd611400640 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 0x5588e6e886e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.191+0000 7fd611400640 1 -- 172.21.15.1:0/755032118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588e6d7b510 con 0x5588e6e7ffe0 2024-04-26T21:45:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.192+0000 7fd611400640 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e881a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009720 tx=0x7fd5f80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20e4892ef44317fa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.192+0000 7fd5f2a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:45:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.192+0000 7fd5f2a00640 1 -- 172.21.15.1:0/755032118 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f800f890 con 0x5588e6e7ffe0 2024-04-26T21:45:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.192+0000 7fd616c6f740 1 -- 172.21.15.1:0/755032118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd60400de20 con 0x5588e6e7ffe0 2024-04-26T21:45:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.192+0000 7fd5f2a00640 1 -- 172.21.15.1:0/755032118 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7fd5f800fa30 con 0x5588e6e7ffe0 2024-04-26T21:45:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd5f2a00640 1 -- 172.21.15.1:0/755032118 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fd5f800fc30 con 0x5588e6e7ffe0 2024-04-26T21:45:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd616c6f740 10 client.26726 fetch_fsmap learned FSMap version 46 2024-04-26T21:45:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd616c6f740 1 -- 172.21.15.1:0/755032118 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5588e6af7210 con 0x5588e6e7ffe0 2024-04-26T21:45:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd5f2a00640 1 -- 172.21.15.1:0/755032118 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd60400de20 con 0x5588e6e7ffe0 2024-04-26T21:45:57.194 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd5f2a00640 1 -- 172.21.15.1:0/755032118 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 46) ==== 8370+0+0 (secure 0 0 0) 0x7fd5f8027430 con 0x5588e6e7ffe0 2024-04-26T21:45:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd616c6f740 10 client.26726 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T21:45:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd616c6f740 10 client.26726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:45:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd616c6f740 20 client.26726 populate_metadata read hostname 'smithi001' 2024-04-26T21:45:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd616c6f740 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5588e6eac3e0 0x5588e6eae880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd616c6f740 4 client.26726 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:45:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.193+0000 7fd616c6f740 1 -- 172.21.15.1:0/755032118 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588e6df2f90 con 0x5588e6eac3e0 2024-04-26T21:45:57.195 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.194+0000 7fd610a00640 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5588e6eac3e0 0x5588e6eae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.194+0000 7fd610a00640 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5588e6eac3e0 0x5588e6eae880 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9549cccfeb2608cd server_cookie=cbc8f0793b26cf7f in_seq=0 out_seq=0 2024-04-26T21:45:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.194+0000 7fd5f2a00640 10 client.26726 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.195+0000 7fd5f2a00640 1 -- 172.21.15.1:0/755032118 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5588e6df2f90 con 0x5588e6eac3e0 2024-04-26T21:45:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.195+0000 7fd5f2a00640 10 client.26726 handle_command_reply: tid=0 2024-04-26T21:45:57.196 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.195+0000 7fd616c6f740 1 client.26726 shutdown 2024-04-26T21:45:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.196+0000 7fd616c6f740 1 -- 172.21.15.1:0/755032118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 msgr2=0x5588e6e881a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.196+0000 7fd616c6f740 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e881a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009720 tx=0x7fd5f80046c0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.196+0000 7fd616c6f740 1 -- 172.21.15.1:0/755032118 shutdown_connections 2024-04-26T21:45:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.196+0000 7fd616c6f740 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5588e6eac3e0 0x5588e6eae880 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.196+0000 7fd616c6f740 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6e88c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.196+0000 7fd616c6f740 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e80900 0x5588e6e886e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.196+0000 7fd616c6f740 1 --2- 172.21.15.1:0/755032118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e7ffe0 0x5588e6e881a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.197 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.196+0000 7fd616c6f740 1 -- 172.21.15.1:0/755032118 >> 172.21.15.1:0/755032118 conn(0x5588e6e7c4f0 msgr2=0x5588e6e82270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.196+0000 7fd5f2a00640 0 client.26726 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.197+0000 7fd616c6f740 1 -- 172.21.15.1:0/755032118 shutdown_connections 2024-04-26T21:45:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.197+0000 7fd616c6f740 1 -- 172.21.15.1:0/755032118 wait complete. 2024-04-26T21:45:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.197+0000 7fd616c6f740 20 client.26726 trim_cache size 0 max 16384 2024-04-26T21:45:57.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.227+0000 7fd616c6f740 1 Processor -- start 2024-04-26T21:45:57.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.227+0000 7fd616c6f740 1 -- start start 2024-04-26T21:45:57.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.227+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e96cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.227+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e80900 0x5588e6e829f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.227+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6eb4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.227+0000 7fd616c6f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c10d2e0 con 0x5588e6e7ffe0 2024-04-26T21:45:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.227+0000 7fd616c6f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c10ab30 con 0x5588e6e80900 2024-04-26T21:45:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.227+0000 7fd616c6f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c1a0a60 con 0x5588e6e8b1c0 2024-04-26T21:45:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.227+0000 7fd610a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e80900 0x5588e6e829f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.228+0000 7fd610a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e80900 0x5588e6e829f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60548/0 (socket says 172.21.15.1:60548) 2024-04-26T21:45:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.228+0000 7fd610a00640 1 -- 172.21.15.1:0/4270358098 learned_addr learned my addr 172.21.15.1:0/4270358098 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.228+0000 7fd611e00640 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6eb4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.229 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.228+0000 7fd611400640 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e96cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.229+0000 7fd610a00640 1 -- 172.21.15.1:0/4270358098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 msgr2=0x5588e6eb4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.229+0000 7fd610a00640 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6eb4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.229+0000 7fd610a00640 1 -- 172.21.15.1:0/4270358098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 msgr2=0x5588e6e96cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.229+0000 7fd610a00640 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e96cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.229+0000 7fd610a00640 1 -- 172.21.15.1:0/4270358098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588e6bc3620 con 0x5588e6e80900 2024-04-26T21:45:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.229+0000 7fd611e00640 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6eb4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:45:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.229+0000 7fd611400640 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e96cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:45:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.229+0000 7fd610a00640 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e80900 0x5588e6e829f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd6000b9030 tx=0x7fd6000f34f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b943aa9d2e1a69 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.230+0000 7fd5fdc00640 1 -- 172.21.15.1:0/4270358098 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6000bd2d0 con 0x5588e6e80900 2024-04-26T21:45:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.230+0000 7fd5fdc00640 1 -- 172.21.15.1:0/4270358098 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6000bcbd0 con 0x5588e6e80900 2024-04-26T21:45:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.230+0000 7fd5fdc00640 1 -- 172.21.15.1:0/4270358098 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6000dd0b0 con 0x5588e6e80900 2024-04-26T21:45:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.231+0000 7fd616c6f740 1 -- 172.21.15.1:0/4270358098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e80900 msgr2=0x5588e6e829f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.231+0000 7fd616c6f740 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e80900 0x5588e6e829f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd6000b9030 tx=0x7fd6000f34f0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.231+0000 7fd616c6f740 1 -- 172.21.15.1:0/4270358098 shutdown_connections 2024-04-26T21:45:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.231+0000 7fd616c6f740 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e8b1c0 0x5588e6eb4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.231+0000 7fd616c6f740 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e80900 0x5588e6e829f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.231+0000 7fd616c6f740 1 --2- 172.21.15.1:0/4270358098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e96cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.231+0000 7fd616c6f740 1 -- 172.21.15.1:0/4270358098 >> 172.21.15.1:0/4270358098 conn(0x5588e6e7c4f0 msgr2=0x5588e6e88160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.231+0000 7fd616c6f740 1 -- 172.21.15.1:0/4270358098 shutdown_connections 2024-04-26T21:45:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.231+0000 7fd616c6f740 1 -- 172.21.15.1:0/4270358098 wait complete. 2024-04-26T21:45:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.232+0000 7fd616c6f740 1 -- start start 2024-04-26T21:45:57.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.232+0000 7fd616c6f740 1 Processor -- start 2024-04-26T21:45:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.232+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e93550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.232+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e80900 0x5588e6e8e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.233+0000 7fd616c6f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e8b1c0 0x5588e6e8ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.233+0000 7fd616c6f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c074980 con 0x5588e6e7ffe0 2024-04-26T21:45:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.233+0000 7fd616c6f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c10a460 con 0x5588e6e8b1c0 2024-04-26T21:45:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.233+0000 7fd616c6f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c1096c0 con 0x5588e6e80900 2024-04-26T21:45:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.233+0000 7fd616c6f740 10 client.0 _ll_register_callbacks cb 0x7fd5f8009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:45:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.233+0000 7fd611400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e93550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.233+0000 7fd611400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e93550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41052/0 (socket says 172.21.15.1:41052) 2024-04-26T21:45:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.233+0000 7fd611400640 1 -- 172.21.15.1:0/2659061557 learned_addr learned my addr 172.21.15.1:0/2659061557 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:45:57.234 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.234+0000 7fd610a00640 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e80900 0x5588e6e8e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.234+0000 7fd611e00640 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e8b1c0 0x5588e6e8ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.234+0000 7fd611400640 1 -- 172.21.15.1:0/2659061557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e80900 msgr2=0x5588e6e8e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.234+0000 7fd611400640 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e80900 0x5588e6e8e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.234+0000 7fd611400640 1 -- 172.21.15.1:0/2659061557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e8b1c0 msgr2=0x5588e6e8ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:45:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.234+0000 7fd611400640 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e8b1c0 0x5588e6e8ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.234+0000 7fd611400640 1 -- 172.21.15.1:0/2659061557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588e6d9fcb0 con 0x5588e6e7ffe0 2024-04-26T21:45:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.235+0000 7fd611400640 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e93550 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009bd0 tx=0x7fd5f800fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fae610ba411f617 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:45:57.235 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.235+0000 7fd5f2a00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:45:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.235+0000 7fd5f2a00640 1 -- 172.21.15.1:0/2659061557 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8027e60 con 0x5588e6e7ffe0 2024-04-26T21:45:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.235+0000 7fd5f2a00640 1 -- 172.21.15.1:0/2659061557 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(115..115 src has 1..115) ==== 9554+0+0 (secure 0 0 0) 0x7fd5f8027a90 con 0x5588e6e7ffe0 2024-04-26T21:45:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.236+0000 7fd616c6f740 1 -- 172.21.15.1:0/2659061557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6000bcbd0 con 0x5588e6e7ffe0 2024-04-26T21:45:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.236+0000 7fd5f2a00640 1 -- 172.21.15.1:0/2659061557 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fd5f800fc30 con 0x5588e6e7ffe0 2024-04-26T21:45:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.236+0000 7fd616c6f740 10 client.16482 fetch_fsmap learned FSMap version 46 2024-04-26T21:45:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.236+0000 7fd616c6f740 1 -- 172.21.15.1:0/2659061557 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5588e6f12880 con 0x5588e6e7ffe0 2024-04-26T21:45:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.237+0000 7fd5f2a00640 1 -- 172.21.15.1:0/2659061557 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6000bcbd0 con 0x5588e6e7ffe0 2024-04-26T21:45:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.237+0000 7fd5f2a00640 1 -- 172.21.15.1:0/2659061557 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 46) ==== 8370+0+0 (secure 0 0 0) 0x7fd5f8004850 con 0x5588e6e7ffe0 2024-04-26T21:45:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.238+0000 7fd616c6f740 10 client.16482 fetch_fsmap finished waiting for FSMap version 46 2024-04-26T21:45:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.238+0000 7fd616c6f740 10 client.16482 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:45:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.238+0000 7fd616c6f740 20 client.16482 populate_metadata read hostname 'smithi001' 2024-04-26T21:45:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.238+0000 7fd616c6f740 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5588e6f13000 0x5588e6f15420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:45:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.238+0000 7fd616c6f740 4 client.16482 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:45:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.238+0000 7fd616c6f740 1 -- 172.21.15.1:0/2659061557 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5588e6dece00 con 0x5588e6f13000 2024-04-26T21:45:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.239+0000 7fd610a00640 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5588e6f13000 0x5588e6f15420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:45:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.240+0000 7fd610a00640 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5588e6f13000 0x5588e6f15420 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be1726adcb7f0879 server_cookie=82f5a3a08dbebfc5 in_seq=0 out_seq=0 2024-04-26T21:45:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.240+0000 7fd5f2a00640 10 client.16482 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:57.689 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.688+0000 7fd5f2a00640 1 -- 172.21.15.1:0/2659061557 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5588e6dece00 con 0x5588e6f13000 2024-04-26T21:45:57.689 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.688+0000 7fd5f2a00640 10 client.16482 handle_command_reply: tid=0 2024-04-26T21:45:57.689 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd616c6f740 1 client.16482 shutdown 2024-04-26T21:45:57.689 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd616c6f740 1 -- 172.21.15.1:0/2659061557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 msgr2=0x5588e6e93550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:57.689 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd616c6f740 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e93550 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009bd0 tx=0x7fd5f800fa30 comp rx=0 tx=0).stop 2024-04-26T21:45:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd616c6f740 1 -- 172.21.15.1:0/2659061557 shutdown_connections 2024-04-26T21:45:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd616c6f740 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5588e6f13000 0x5588e6f15420 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd616c6f740 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5588e6e80900 0x5588e6e8e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd616c6f740 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5588e6e8b1c0 0x5588e6e8ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd5f2a00640 0 client.16482 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:45:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd616c6f740 1 --2- 172.21.15.1:0/2659061557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5588e6e7ffe0 0x5588e6e93550 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd616c6f740 1 -- 172.21.15.1:0/2659061557 >> 172.21.15.1:0/2659061557 conn(0x5588e6e7c4f0 msgr2=0x5588e6e88980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.689+0000 7fd616c6f740 1 -- 172.21.15.1:0/2659061557 shutdown_connections 2024-04-26T21:45:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.690+0000 7fd616c6f740 1 -- 172.21.15.1:0/2659061557 wait complete. 2024-04-26T21:45:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.690+0000 7fd616c6f740 20 client.16482 trim_cache size 0 max 16384 2024-04-26T21:45:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:45:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:45:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:45:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:45:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 -- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd5f40a3de0 msgr2=0x7fd5f40a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd5f40a3de0 0x7fd5f40a62a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5588e6e81580 tx=0x7fd5f8005f90 comp rx=0 tx=0).stop 2024-04-26T21:45:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 -- 172.21.15.1:0/2162558578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd60c106b00 msgr2=0x7fd60c19fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:45:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd60c106b00 0x7fd60c19fc50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd600008000 tx=0x7fd600003b90 comp rx=0 tx=0).stop 2024-04-26T21:45:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 -- 172.21.15.1:0/2162558578 shutdown_connections 2024-04-26T21:45:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd5f40a3de0 0x7fd5f40a62a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd60c106b00 0x7fd60c19fc50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd60c1a3fd0 0x7fd60c1a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 --2- 172.21.15.1:0/2162558578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd60c101700 0x7fd60c19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:45:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 -- 172.21.15.1:0/2162558578 >> 172.21.15.1:0/2162558578 conn(0x7fd60c0fd110 msgr2=0x7fd60c0fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:45:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.691+0000 7fd612800640 1 -- 172.21.15.1:0/2162558578 shutdown_connections 2024-04-26T21:45:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:45:57.692+0000 7fd612800640 1 -- 172.21.15.1:0/2162558578 wait complete. 2024-04-26T21:45:58.138 INFO:tasks.workunit.client.0.smithi001.stdout: 10 53048 26.34 MB/sec warmup 115 sec latency 239.090 ms 2024-04-26T21:45:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:58 smithi001 ceph-mon[28054]: pgmap v491: 129 pgs: 129 active+clean; 947 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 12 MiB/s wr, 624 op/s 2024-04-26T21:45:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:58 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:45:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:58 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:45:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:45:58 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:45:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:58 smithi038 ceph-mon[32724]: pgmap v491: 129 pgs: 129 active+clean; 947 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 12 MiB/s wr, 624 op/s 2024-04-26T21:45:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:58 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:45:58.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:58 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:45:58.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:45:58 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:45:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:58 smithi186 ceph-mon[32991]: pgmap v491: 129 pgs: 129 active+clean; 947 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 12 MiB/s wr, 624 op/s 2024-04-26T21:45:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:58 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:45:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:58 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:45:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:45:58 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:45:59.138 INFO:tasks.workunit.client.0.smithi001.stdout: 10 53555 26.40 MB/sec warmup 116 sec latency 126.707 ms 2024-04-26T21:46:00.139 INFO:tasks.workunit.client.0.smithi001.stdout: 10 54189 26.44 MB/sec warmup 117 sec latency 57.276 ms 2024-04-26T21:46:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:00 smithi001 ceph-mon[28054]: pgmap v492: 129 pgs: 129 active+clean; 947 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 533 op/s 2024-04-26T21:46:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:00 smithi038 ceph-mon[32724]: pgmap v492: 129 pgs: 129 active+clean; 947 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 533 op/s 2024-04-26T21:46:00.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:00 smithi186 ceph-mon[32991]: pgmap v492: 129 pgs: 129 active+clean; 947 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 533 op/s 2024-04-26T21:46:01.138 INFO:tasks.workunit.client.0.smithi001.stdout: 10 54598 26.33 MB/sec warmup 118 sec latency 245.830 ms 2024-04-26T21:46:02.138 INFO:tasks.workunit.client.0.smithi001.stdout: 10 54807 26.24 MB/sec warmup 119 sec latency 446.240 ms 2024-04-26T21:46:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:02 smithi001 ceph-mon[28054]: pgmap v493: 129 pgs: 129 active+clean; 958 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 799 op/s 2024-04-26T21:46:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:02 smithi038 ceph-mon[32724]: pgmap v493: 129 pgs: 129 active+clean; 958 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 799 op/s 2024-04-26T21:46:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:02 smithi186 ceph-mon[32991]: pgmap v493: 129 pgs: 129 active+clean; 958 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 799 op/s 2024-04-26T21:46:03.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:03 smithi001 ceph-mon[28054]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T21:46:03.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:03 smithi038 ceph-mon[32724]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T21:46:03.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:03 smithi186 ceph-mon[32991]: osdmap e116: 12 total, 12 up, 12 in 2024-04-26T21:46:04.138 INFO:tasks.workunit.client.0.smithi001.stdout: 10 55179 11.70 MB/sec execute 1 sec latency 305.000 ms 2024-04-26T21:46:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:04 smithi001 ceph-mon[28054]: pgmap v495: 129 pgs: 129 active+clean; 958 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.7 MiB/s wr, 515 op/s 2024-04-26T21:46:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:04 smithi038 ceph-mon[32724]: pgmap v495: 129 pgs: 129 active+clean; 958 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.7 MiB/s wr, 515 op/s 2024-04-26T21:46:04.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:04 smithi186 ceph-mon[32991]: pgmap v495: 129 pgs: 129 active+clean; 958 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 4.7 MiB/s wr, 515 op/s 2024-04-26T21:46:05.139 INFO:tasks.workunit.client.0.smithi001.stdout: 10 55570 15.61 MB/sec execute 2 sec latency 297.493 ms 2024-04-26T21:46:06.139 INFO:tasks.workunit.client.0.smithi001.stdout: 10 56079 18.33 MB/sec execute 3 sec latency 63.530 ms 2024-04-26T21:46:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:06 smithi001 ceph-mon[28054]: pgmap v496: 129 pgs: 6 active+clean+snaptrim, 116 active+clean, 7 active+clean+snaptrim_wait; 965 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.6 MiB/s wr, 591 op/s 2024-04-26T21:46:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:06 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:06 smithi038 ceph-mon[32724]: pgmap v496: 129 pgs: 6 active+clean+snaptrim, 116 active+clean, 7 active+clean+snaptrim_wait; 965 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.6 MiB/s wr, 591 op/s 2024-04-26T21:46:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:06 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:06 smithi186 ceph-mon[32991]: pgmap v496: 129 pgs: 6 active+clean+snaptrim, 116 active+clean, 7 active+clean+snaptrim_wait; 965 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.6 MiB/s wr, 591 op/s 2024-04-26T21:46:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:06 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:07.140 INFO:tasks.workunit.client.0.smithi001.stdout: 10 56522 21.89 MB/sec execute 4 sec latency 273.346 ms 2024-04-26T21:46:08.140 INFO:tasks.workunit.client.0.smithi001.stdout: 10 56590 18.15 MB/sec execute 5 sec latency 804.726 ms 2024-04-26T21:46:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:08 smithi001 ceph-mon[28054]: pgmap v497: 129 pgs: 7 active+clean+snaptrim, 115 active+clean, 7 active+clean+snaptrim_wait; 968 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.0 MiB/s wr, 607 op/s 2024-04-26T21:46:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:08 smithi038 ceph-mon[32724]: pgmap v497: 129 pgs: 7 active+clean+snaptrim, 115 active+clean, 7 active+clean+snaptrim_wait; 968 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.0 MiB/s wr, 607 op/s 2024-04-26T21:46:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:08 smithi186 ceph-mon[32991]: pgmap v497: 129 pgs: 7 active+clean+snaptrim, 115 active+clean, 7 active+clean+snaptrim_wait; 968 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.0 MiB/s wr, 607 op/s 2024-04-26T21:46:09.140 INFO:tasks.workunit.client.0.smithi001.stdout: 10 56876 18.53 MB/sec execute 6 sec latency 80.487 ms 2024-04-26T21:46:10.140 INFO:tasks.workunit.client.0.smithi001.stdout: 10 57131 17.47 MB/sec execute 7 sec latency 177.288 ms 2024-04-26T21:46:10.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:10 smithi001 ceph-mon[28054]: pgmap v498: 129 pgs: 7 active+clean+snaptrim, 115 active+clean, 7 active+clean+snaptrim_wait; 968 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.0 MiB/s wr, 607 op/s 2024-04-26T21:46:10.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:10 smithi038 ceph-mon[32724]: pgmap v498: 129 pgs: 7 active+clean+snaptrim, 115 active+clean, 7 active+clean+snaptrim_wait; 968 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.0 MiB/s wr, 607 op/s 2024-04-26T21:46:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:10 smithi186 ceph-mon[32991]: pgmap v498: 129 pgs: 7 active+clean+snaptrim, 115 active+clean, 7 active+clean+snaptrim_wait; 968 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 7.0 MiB/s wr, 607 op/s 2024-04-26T21:46:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.864+0000 7f507ea00640 1 -- 172.21.15.1:0/3033963826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5078074ce0 msgr2=0x7f50780750c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.864+0000 7f507ea00640 1 --2- 172.21.15.1:0/3033963826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5078074ce0 0x7f50780750c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5068009900 tx=0x7f506802a480 comp rx=0 tx=0).stop 2024-04-26T21:46:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.864+0000 7f507ea00640 1 -- 172.21.15.1:0/3033963826 shutdown_connections 2024-04-26T21:46:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.864+0000 7f507ea00640 1 --2- 172.21.15.1:0/3033963826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f507810f1e0 0x7f50781115d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.864+0000 7f507ea00640 1 --2- 172.21.15.1:0/3033963826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5078075600 0x7f507810eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.864+0000 7f507ea00640 1 --2- 172.21.15.1:0/3033963826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5078074ce0 0x7f50780750c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.864+0000 7f507ea00640 1 -- 172.21.15.1:0/3033963826 >> 172.21.15.1:0/3033963826 conn(0x7f50780fe3f0 msgr2=0x7f5078100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.865+0000 7f507ea00640 1 -- 172.21.15.1:0/3033963826 shutdown_connections 2024-04-26T21:46:10.866 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.865+0000 7f507ea00640 1 -- 172.21.15.1:0/3033963826 wait complete. 2024-04-26T21:46:10.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.867+0000 7f507ea00640 1 Processor -- start 2024-04-26T21:46:10.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f507ea00640 1 -- start start 2024-04-26T21:46:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f507ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5078074ce0 0x7f507810d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.869 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f507ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5078075600 0x7f507810d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f507ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f507810f1e0 0x7f50781ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f507ea00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f50781035d0 con 0x7f5078074ce0 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f507ea00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5078103450 con 0x7f5078075600 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f507ea00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5078103750 con 0x7f507810f1e0 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f5077400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5078075600 0x7f507810d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f5077400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5078075600 0x7f507810d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54686/0 (socket says 172.21.15.1:54686) 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f5077400640 1 -- 172.21.15.1:0/4163382164 learned_addr learned my addr 172.21.15.1:0/4163382164 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.868+0000 7f5077e00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5078074ce0 0x7f507810d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.869+0000 7f507ca00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f507810f1e0 0x7f50781ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.869+0000 7f5077400640 1 -- 172.21.15.1:0/4163382164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f507810f1e0 msgr2=0x7f50781ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.869+0000 7f5077400640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f507810f1e0 0x7f50781ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.869+0000 7f5077400640 1 -- 172.21.15.1:0/4163382164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5078074ce0 msgr2=0x7f507810d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.869+0000 7f5077400640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5078074ce0 0x7f507810d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.869+0000 7f5077400640 1 -- 172.21.15.1:0/4163382164 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50781abb30 con 0x7f5078075600 2024-04-26T21:46:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.869+0000 7f507ca00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f507810f1e0 0x7f50781ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:46:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.869+0000 7f5077400640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5078075600 0x7f507810d700 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f506400d950 tx=0x7f506400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.870+0000 7f5074c00640 1 -- 172.21.15.1:0/4163382164 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5064017070 con 0x7f5078075600 2024-04-26T21:46:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.871+0000 7f507ea00640 1 -- 172.21.15.1:0/4163382164 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50781abd00 con 0x7f5078075600 2024-04-26T21:46:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.871+0000 7f5074c00640 1 -- 172.21.15.1:0/4163382164 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5064009c30 con 0x7f5078075600 2024-04-26T21:46:10.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.871+0000 7f507ea00640 1 -- 172.21.15.1:0/4163382164 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50781ac1e0 con 0x7f5078075600 2024-04-26T21:46:10.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.871+0000 7f5074c00640 1 -- 172.21.15.1:0/4163382164 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506400f620 con 0x7f5078075600 2024-04-26T21:46:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.876+0000 7f5074c00640 1 -- 172.21.15.1:0/4163382164 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f506400f7c0 con 0x7f5078075600 2024-04-26T21:46:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.877+0000 7f50817af740 1 Processor -- start 2024-04-26T21:46:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.877+0000 7f50817af740 1 -- start start 2024-04-26T21:46:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.877+0000 7f50817af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.877+0000 7f50817af740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.877+0000 7f50817af740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6bc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.877+0000 7f50817af740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb614380 con 0x55e9cb6af6f0 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.877+0000 7f50817af740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb5b0c30 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.877+0000 7f50817af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb610340 con 0x55e9cb6aedd0 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f5077e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f507ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f5077400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6bc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f5077e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54552/0 (socket says 172.21.15.1:54552) 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f5077e00640 1 -- 172.21.15.1:0/544024806 learned_addr learned my addr 172.21.15.1:0/544024806 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f507ca00640 1 --2- 172.21.15.1:0/544024806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6af1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43722/0 (socket says 172.21.15.1:43722) 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f5077400640 1 -- 172.21.15.1:0/544024806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 msgr2=0x55e9cb6af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f5077400640 1 --2- 172.21.15.1:0/544024806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f5077400640 1 -- 172.21.15.1:0/544024806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 msgr2=0x55e9cb6b9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f5077400640 1 --2- 172.21.15.1:0/544024806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f5077400640 1 -- 172.21.15.1:0/544024806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9cb5d2940 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.878+0000 7f507ca00640 1 --2- 172.21.15.1:0/544024806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:46:10.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.879+0000 7f5077400640 1 --2- 172.21.15.1:0/544024806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6bc3a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f50640cd980 tx=0x7f50640d8670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8440c3095eb21b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:10.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.879+0000 7f5074c00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f505c0a3d90 0x7f505c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.879+0000 7f506d200640 1 -- 172.21.15.1:0/544024806 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50640df070 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.879+0000 7f506d200640 1 -- 172.21.15.1:0/544024806 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50640a0580 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.879+0000 7f506d200640 1 -- 172.21.15.1:0/544024806 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506409d730 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.879+0000 7f5074c00640 1 -- 172.21.15.1:0/4163382164 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 9730+0+0 (secure 0 0 0) 0x7f50640d0750 con 0x7f5078075600 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.879+0000 7f507ca00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f505c0a3d90 0x7f505c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.880+0000 7f50817af740 1 -- 172.21.15.1:0/544024806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 msgr2=0x55e9cb6bc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.880+0000 7f50817af740 1 --2- 172.21.15.1:0/544024806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6bc3a0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f50640cd980 tx=0x7f50640d8670 comp rx=0 tx=0).stop 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.880+0000 7f50817af740 1 -- 172.21.15.1:0/544024806 shutdown_connections 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.880+0000 7f50817af740 1 --2- 172.21.15.1:0/544024806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6bc3a0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.880+0000 7f50817af740 1 --2- 172.21.15.1:0/544024806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.880+0000 7f50817af740 1 --2- 172.21.15.1:0/544024806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.880+0000 7f50817af740 1 -- 172.21.15.1:0/544024806 >> 172.21.15.1:0/544024806 conn(0x55e9cb6ab2e0 msgr2=0x55e9cb6ab6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:10.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.880+0000 7f50817af740 1 -- 172.21.15.1:0/544024806 shutdown_connections 2024-04-26T21:46:10.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.881+0000 7f507ca00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f505c0a3d90 0x7f505c0a6250 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5070005fd0 tx=0x7f5070009600 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:10.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.881+0000 7f50817af740 1 -- 172.21.15.1:0/544024806 wait complete. 2024-04-26T21:46:10.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.881+0000 7f50817af740 1 -- start start 2024-04-26T21:46:10.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.881+0000 7f50817af740 1 Processor -- start 2024-04-26T21:46:10.885 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.882+0000 7f50817af740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6b6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.885 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.882+0000 7f50817af740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.885 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.882+0000 7f50817af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.882+0000 7f50817af740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb6bcbe0 con 0x55e9cb6aedd0 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.882+0000 7f50817af740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb59bc60 con 0x55e9cb6af6f0 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.882+0000 7f50817af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e9cb6bcd60 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.882+0000 7f50817af740 10 client.0 _ll_register_callbacks cb 0x55e9cb5c7e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.882+0000 7f507ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.882+0000 7f507ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43734/0 (socket says 172.21.15.1:43734) 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.882+0000 7f507ca00640 1 -- 172.21.15.1:0/518071706 learned_addr learned my addr 172.21.15.1:0/518071706 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.883+0000 7f5077e00640 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6b6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.883+0000 7f507ca00640 1 -- 172.21.15.1:0/518071706 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6af6f0 msgr2=0x55e9cb6b73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.883+0000 7f507ca00640 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.883+0000 7f507ca00640 1 -- 172.21.15.1:0/518071706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6aedd0 msgr2=0x55e9cb6b6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.883+0000 7f507ca00640 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6b6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.883+0000 7f507ca00640 1 -- 172.21.15.1:0/518071706 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9cb5aa510 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.883+0000 7f5077e00640 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6b6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.883+0000 7f507ca00640 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b7910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5070010770 tx=0x7f5070004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5130cbd8706c0ff6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.883+0000 7f5061600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.884+0000 7f5061600640 1 -- 172.21.15.1:0/518071706 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070028670 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.884+0000 7f5061600640 1 -- 172.21.15.1:0/518071706 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9730+0+0 (secure 0 0 0) 0x7f507001f420 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.884+0000 7f50817af740 1 -- 172.21.15.1:0/518071706 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50640a0580 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.884+0000 7f5061600640 1 -- 172.21.15.1:0/518071706 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5070028810 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.884+0000 7f50817af740 10 client.26722 fetch_fsmap learned FSMap version 47 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.884+0000 7f50817af740 1 -- 172.21.15.1:0/518071706 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e9cb326210 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.884+0000 7f5061600640 1 -- 172.21.15.1:0/518071706 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50640a0580 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.885+0000 7f5061600640 1 -- 172.21.15.1:0/518071706 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 47) ==== 8366+0+0 (secure 0 0 0) 0x7f507000f040 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.885+0000 7f50817af740 10 client.26722 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.885+0000 7f50817af740 10 client.26722 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:46:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.885+0000 7f50817af740 20 client.26722 populate_metadata read hostname 'smithi001' 2024-04-26T21:46:10.888 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.885+0000 7f50817af740 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e9cb6c9df0 0x55e9cb6cc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.888 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.885+0000 7f50817af740 4 client.26722 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:46:10.888 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.885+0000 7f50817af740 1 -- 172.21.15.1:0/518071706 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9cb621f90 con 0x55e9cb6c9df0 2024-04-26T21:46:10.888 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.885+0000 7f5077e00640 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e9cb6c9df0 0x55e9cb6cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.886+0000 7f5077e00640 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e9cb6c9df0 0x55e9cb6cc280 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=157c8dc1f28b97c5 server_cookie=295a0cc68891f298 in_seq=0 out_seq=0 2024-04-26T21:46:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.886+0000 7f5061600640 10 client.26722 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f5061600640 1 -- 172.21.15.1:0/518071706 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e9cb621f90 con 0x55e9cb6c9df0 2024-04-26T21:46:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f5061600640 10 client.26722 handle_command_reply: tid=0 2024-04-26T21:46:10.889 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f50817af740 1 client.26722 shutdown 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f50817af740 1 -- 172.21.15.1:0/518071706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 msgr2=0x55e9cb6b7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f50817af740 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b7910 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5070010770 tx=0x7f5070004a40 comp rx=0 tx=0).stop 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f50817af740 1 -- 172.21.15.1:0/518071706 shutdown_connections 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f50817af740 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e9cb6c9df0 0x55e9cb6cc280 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f50817af740 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b7910 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f50817af740 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f50817af740 1 --2- 172.21.15.1:0/518071706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6b6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.887+0000 7f50817af740 1 -- 172.21.15.1:0/518071706 >> 172.21.15.1:0/518071706 conn(0x55e9cb6ab2e0 msgr2=0x55e9cb6b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.888+0000 7f5061600640 0 client.26722 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.888+0000 7f50817af740 1 -- 172.21.15.1:0/518071706 shutdown_connections 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.888+0000 7f50817af740 1 -- 172.21.15.1:0/518071706 wait complete. 2024-04-26T21:46:10.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.888+0000 7f50817af740 20 client.26722 trim_cache size 0 max 16384 2024-04-26T21:46:10.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.914+0000 7f50817af740 1 Processor -- start 2024-04-26T21:46:10.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.914+0000 7f50817af740 1 -- start start 2024-04-26T21:46:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.915+0000 7f50817af740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.915+0000 7f50817af740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.915+0000 7f50817af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.915+0000 7f50817af740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5058012ca0 con 0x55e9cb6af6f0 2024-04-26T21:46:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.915+0000 7f50817af740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f50580104e0 con 0x55e9cb6aedd0 2024-04-26T21:46:10.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.915+0000 7f50817af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f505800e350 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.915+0000 7f507ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.915+0000 7f507ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43736/0 (socket says 172.21.15.1:43736) 2024-04-26T21:46:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.915+0000 7f507ca00640 1 -- 172.21.15.1:0/1715041852 learned_addr learned my addr 172.21.15.1:0/1715041852 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:10.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.915+0000 7f5077e00640 1 --2- 172.21.15.1:0/1715041852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6c5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.916+0000 7f507ca00640 1 -- 172.21.15.1:0/1715041852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6aedd0 msgr2=0x55e9cb6c5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.916+0000 7f507ca00640 1 --2- 172.21.15.1:0/1715041852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6c5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.916+0000 7f507ca00640 1 -- 172.21.15.1:0/1715041852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 msgr2=0x55e9cb6b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.916+0000 7f507ca00640 1 --2- 172.21.15.1:0/1715041852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.916+0000 7f507ca00640 1 -- 172.21.15.1:0/1715041852 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9cb3f2620 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.916+0000 7f507ca00640 1 --2- 172.21.15.1:0/1715041852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b5ff0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f507001fc30 tx=0x7f5070005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a9d5957f635a174 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.917+0000 7f5063e00640 1 -- 172.21.15.1:0/1715041852 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070028670 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.917+0000 7f5063e00640 1 -- 172.21.15.1:0/1715041852 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5070023070 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.917+0000 7f5063e00640 1 -- 172.21.15.1:0/1715041852 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070032420 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.918+0000 7f50817af740 1 -- 172.21.15.1:0/1715041852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 msgr2=0x55e9cb6b5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.918+0000 7f50817af740 1 --2- 172.21.15.1:0/1715041852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b5ff0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f507001fc30 tx=0x7f5070005d30 comp rx=0 tx=0).stop 2024-04-26T21:46:10.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.918+0000 7f50817af740 1 -- 172.21.15.1:0/1715041852 shutdown_connections 2024-04-26T21:46:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.918+0000 7f50817af740 1 --2- 172.21.15.1:0/1715041852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6b5ff0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.918+0000 7f50817af740 1 --2- 172.21.15.1:0/1715041852 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.918+0000 7f50817af740 1 --2- 172.21.15.1:0/1715041852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.918+0000 7f50817af740 1 -- 172.21.15.1:0/1715041852 >> 172.21.15.1:0/1715041852 conn(0x55e9cb6ab2e0 msgr2=0x55e9cb6a9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.919+0000 7f50817af740 1 -- 172.21.15.1:0/1715041852 shutdown_connections 2024-04-26T21:46:10.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.919+0000 7f50817af740 1 -- 172.21.15.1:0/1715041852 wait complete. 2024-04-26T21:46:10.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.919+0000 7f50817af740 1 -- start start 2024-04-26T21:46:10.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.919+0000 7f50817af740 1 Processor -- start 2024-04-26T21:46:10.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.920+0000 7f50817af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6bcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.920+0000 7f50817af740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6bd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.920+0000 7f50817af740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6bda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.920+0000 7f50817af740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f505800e9a0 con 0x55e9cb6af6f0 2024-04-26T21:46:10.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.920+0000 7f50817af740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f505800fe10 con 0x55e9cb6b9fb0 2024-04-26T21:46:10.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.920+0000 7f50817af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f505800f070 con 0x55e9cb6aedd0 2024-04-26T21:46:10.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.920+0000 7f50817af740 10 client.0 _ll_register_callbacks cb 0x7f5070009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.920+0000 7f5077e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6bcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5077e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6bcfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43748/0 (socket says 172.21.15.1:43748) 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5077e00640 1 -- 172.21.15.1:0/4062202672 learned_addr learned my addr 172.21.15.1:0/4062202672 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f507ca00640 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6bda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5077400640 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6bd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5077e00640 1 -- 172.21.15.1:0/4062202672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 msgr2=0x55e9cb6bda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5077e00640 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6bda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5077e00640 1 -- 172.21.15.1:0/4062202672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 msgr2=0x55e9cb6bd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5077e00640 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6bd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5077e00640 1 -- 172.21.15.1:0/4062202672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9cb5cecb0 con 0x55e9cb6aedd0 2024-04-26T21:46:10.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5077400640 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6bd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:46:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5077e00640 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6bcfd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f506802a9a0 tx=0x7f5068009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3749c6853bea6f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.921+0000 7f5061600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:46:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.922+0000 7f5061600640 1 -- 172.21.15.1:0/4062202672 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068035040 con 0x55e9cb6aedd0 2024-04-26T21:46:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.922+0000 7f5061600640 1 -- 172.21.15.1:0/4062202672 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(116..116 src has 1..116) ==== 9730+0+0 (secure 0 0 0) 0x7f50680052c0 con 0x55e9cb6aedd0 2024-04-26T21:46:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.922+0000 7f50817af740 1 -- 172.21.15.1:0/4062202672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5070023070 con 0x55e9cb6aedd0 2024-04-26T21:46:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.922+0000 7f5061600640 1 -- 172.21.15.1:0/4062202672 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5068054050 con 0x55e9cb6aedd0 2024-04-26T21:46:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.923+0000 7f50817af740 10 client.26728 fetch_fsmap learned FSMap version 47 2024-04-26T21:46:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.923+0000 7f50817af740 1 -- 172.21.15.1:0/4062202672 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e9cb72f860 con 0x55e9cb6aedd0 2024-04-26T21:46:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.923+0000 7f5061600640 1 -- 172.21.15.1:0/4062202672 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5070023070 con 0x55e9cb6aedd0 2024-04-26T21:46:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.924+0000 7f5061600640 1 -- 172.21.15.1:0/4062202672 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 47) ==== 8366+0+0 (secure 0 0 0) 0x7f5068004770 con 0x55e9cb6aedd0 2024-04-26T21:46:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.924+0000 7f50817af740 10 client.26728 fetch_fsmap finished waiting for FSMap version 47 2024-04-26T21:46:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.924+0000 7f50817af740 10 client.26728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:46:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.924+0000 7f50817af740 20 client.26728 populate_metadata read hostname 'smithi001' 2024-04-26T21:46:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.924+0000 7f50817af740 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e9cb72ff30 0x55e9cb732350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.924+0000 7f50817af740 4 client.26728 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:46:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.924+0000 7f50817af740 1 -- 172.21.15.1:0/4062202672 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e9cb61be00 con 0x55e9cb72ff30 2024-04-26T21:46:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.924+0000 7f5077400640 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e9cb72ff30 0x55e9cb732350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.925+0000 7f5077400640 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e9cb72ff30 0x55e9cb732350 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=217ff2dda8443230 server_cookie=1fc63f45a1af7e7b in_seq=0 out_seq=0 2024-04-26T21:46:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:10.925+0000 7f5061600640 10 client.26728 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:11.141 INFO:tasks.workunit.client.0.smithi001.stdout: 10 57564 16.81 MB/sec execute 8 sec latency 120.479 ms 2024-04-26T21:46:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.220+0000 7f5061600640 1 -- 172.21.15.1:0/4062202672 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e9cb61be00 con 0x55e9cb72ff30 2024-04-26T21:46:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.220+0000 7f5061600640 10 client.26728 handle_command_reply: tid=0 2024-04-26T21:46:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.220+0000 7f50817af740 1 client.26728 shutdown 2024-04-26T21:46:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.220+0000 7f50817af740 1 -- 172.21.15.1:0/4062202672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 msgr2=0x55e9cb6bcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:11.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.220+0000 7f50817af740 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6bcfd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f506802a9a0 tx=0x7f5068009520 comp rx=0 tx=0).stop 2024-04-26T21:46:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.221+0000 7f50817af740 1 -- 172.21.15.1:0/4062202672 shutdown_connections 2024-04-26T21:46:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.221+0000 7f50817af740 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55e9cb72ff30 0x55e9cb732350 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.221+0000 7f50817af740 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9cb6b9fb0 0x55e9cb6bda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.221+0000 7f50817af740 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55e9cb6af6f0 0x55e9cb6bd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.221+0000 7f50817af740 1 --2- 172.21.15.1:0/4062202672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cb6aedd0 0x55e9cb6bcfd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:11.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.221+0000 7f50817af740 1 -- 172.21.15.1:0/4062202672 >> 172.21.15.1:0/4062202672 conn(0x55e9cb6ab2e0 msgr2=0x55e9cb6a9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.221+0000 7f5061600640 0 client.26728 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.221+0000 7f50817af740 1 -- 172.21.15.1:0/4062202672 shutdown_connections 2024-04-26T21:46:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.221+0000 7f50817af740 1 -- 172.21.15.1:0/4062202672 wait complete. 2024-04-26T21:46:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.221+0000 7f50817af740 20 client.26728 trim_cache size 0 max 16384 2024-04-26T21:46:11.222 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:46:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:46:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:46:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:46:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.223+0000 7f507ea00640 1 -- 172.21.15.1:0/4163382164 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f505c0a3d90 msgr2=0x7f505c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.223+0000 7f507ea00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f505c0a3d90 0x7f505c0a6250 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5070005fd0 tx=0x7f5070009600 comp rx=0 tx=0).stop 2024-04-26T21:46:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.223+0000 7f507ea00640 1 -- 172.21.15.1:0/4163382164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5078075600 msgr2=0x7f507810d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.223+0000 7f507ea00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5078075600 0x7f507810d700 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f506400d950 tx=0x7f506400de10 comp rx=0 tx=0).stop 2024-04-26T21:46:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.223+0000 7f507ea00640 1 -- 172.21.15.1:0/4163382164 shutdown_connections 2024-04-26T21:46:11.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.223+0000 7f507ea00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f505c0a3d90 0x7f505c0a6250 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.223+0000 7f507ea00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f507810f1e0 0x7f50781ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.223+0000 7f507ea00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5078075600 0x7f507810d700 secure :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f506400d950 tx=0x7f506400de10 comp rx=0 tx=0).stop 2024-04-26T21:46:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.223+0000 7f507ea00640 1 --2- 172.21.15.1:0/4163382164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5078074ce0 0x7f507810d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.223+0000 7f507ea00640 1 -- 172.21.15.1:0/4163382164 >> 172.21.15.1:0/4163382164 conn(0x7f50780fe3f0 msgr2=0x7f50780ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.224+0000 7f507ea00640 1 -- 172.21.15.1:0/4163382164 shutdown_connections 2024-04-26T21:46:11.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:11.224+0000 7f507ea00640 1 -- 172.21.15.1:0/4163382164 wait complete. 2024-04-26T21:46:11.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:11 smithi186 ceph-mon[32991]: pgmap v499: 129 pgs: 129 active+clean; 991 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 13 MiB/s wr, 631 op/s 2024-04-26T21:46:11.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:11 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:11.704 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:46:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: ignoring --setuser ceph since I am not root 2024-04-26T21:46:11.705 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:46:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: ignoring --setgroup ceph since I am not root 2024-04-26T21:46:11.705 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:46:11 smithi186 ceph-mds[63876]: -- 172.21.15.186:0/452559595 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f7d3eada40 con 0x55f7d3e80800 2024-04-26T21:46:11.705 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:46:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: starting mds.i at 2024-04-26T21:46:11.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:11 smithi001 ceph-mon[28054]: pgmap v499: 129 pgs: 129 active+clean; 991 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 13 MiB/s wr, 631 op/s 2024-04-26T21:46:11.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:11 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:11.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:11 smithi038 ceph-mon[32724]: pgmap v499: 129 pgs: 129 active+clean; 991 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 13 MiB/s wr, 631 op/s 2024-04-26T21:46:11.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:11 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:12.141 INFO:tasks.workunit.client.0.smithi001.stdout: 10 58095 18.37 MB/sec execute 9 sec latency 190.199 ms 2024-04-26T21:46:12.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:12 smithi186 ceph-mon[32991]: daemon mds.i restarted 2024-04-26T21:46:12.719 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:12 smithi001 ceph-mon[28054]: daemon mds.i restarted 2024-04-26T21:46:12.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:12 smithi038 ceph-mon[32724]: daemon mds.i restarted 2024-04-26T21:46:13.141 INFO:tasks.workunit.client.0.smithi001.stdout: 10 58863 20.59 MB/sec execute 10 sec latency 75.418 ms 2024-04-26T21:46:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:13 smithi186 ceph-mon[32991]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T21:46:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:13 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.186:6838/1748915282,v1:172.21.15.186:6839/1748915282] up:boot 2024-04-26T21:46:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:13 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T21:46:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:13 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:46:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:13 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:13.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:13 smithi186 ceph-mon[32991]: pgmap v501: 129 pgs: 129 active+clean; 991 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 13 MiB/s wr, 631 op/s 2024-04-26T21:46:13.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:13 smithi001 ceph-mon[28054]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T21:46:13.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:13 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.186:6838/1748915282,v1:172.21.15.186:6839/1748915282] up:boot 2024-04-26T21:46:13.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:13 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T21:46:13.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:13 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:46:13.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:13 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:13.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:13 smithi001 ceph-mon[28054]: pgmap v501: 129 pgs: 129 active+clean; 991 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 13 MiB/s wr, 631 op/s 2024-04-26T21:46:13.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:13 smithi038 ceph-mon[32724]: osdmap e117: 12 total, 12 up, 12 in 2024-04-26T21:46:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:13 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.186:6838/1748915282,v1:172.21.15.186:6839/1748915282] up:boot 2024-04-26T21:46:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:13 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T21:46:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:13 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:46:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:13 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:13 smithi038 ceph-mon[32724]: pgmap v501: 129 pgs: 129 active+clean; 991 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 13 MiB/s wr, 631 op/s 2024-04-26T21:46:14.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 59575 22.58 MB/sec execute 11 sec latency 63.813 ms 2024-04-26T21:46:15.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 60134 23.92 MB/sec execute 12 sec latency 72.445 ms 2024-04-26T21:46:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:15 smithi186 ceph-mon[32991]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T21:46:15.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:15 smithi001 ceph-mon[28054]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T21:46:15.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:15 smithi038 ceph-mon[32724]: osdmap e118: 12 total, 12 up, 12 in 2024-04-26T21:46:16.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 60530 23.43 MB/sec execute 13 sec latency 54.230 ms 2024-04-26T21:46:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:16 smithi186 ceph-mon[32991]: pgmap v503: 129 pgs: 129 active+clean; 1002 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 749 op/s 2024-04-26T21:46:16.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:16 smithi001 ceph-mon[28054]: pgmap v503: 129 pgs: 129 active+clean; 1002 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 749 op/s 2024-04-26T21:46:16.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:16 smithi038 ceph-mon[32724]: pgmap v503: 129 pgs: 129 active+clean; 1002 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 749 op/s 2024-04-26T21:46:17.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 60970 22.95 MB/sec execute 14 sec latency 78.534 ms 2024-04-26T21:46:18.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 61702 23.87 MB/sec execute 15 sec latency 51.690 ms 2024-04-26T21:46:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:18 smithi186 ceph-mon[32991]: pgmap v504: 129 pgs: 129 active+clean; 1004 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 795 op/s 2024-04-26T21:46:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:18 smithi001 ceph-mon[28054]: pgmap v504: 129 pgs: 129 active+clean; 1004 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 795 op/s 2024-04-26T21:46:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:18 smithi038 ceph-mon[32724]: pgmap v504: 129 pgs: 129 active+clean; 1004 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 795 op/s 2024-04-26T21:46:19.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 62195 23.97 MB/sec execute 16 sec latency 271.986 ms 2024-04-26T21:46:20.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 62907 25.11 MB/sec execute 17 sec latency 378.895 ms 2024-04-26T21:46:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:20 smithi186 ceph-mon[32991]: pgmap v505: 129 pgs: 129 active+clean; 1004 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.1 MiB/s wr, 367 op/s 2024-04-26T21:46:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:20 smithi001 ceph-mon[28054]: pgmap v505: 129 pgs: 129 active+clean; 1004 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.1 MiB/s wr, 367 op/s 2024-04-26T21:46:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:20 smithi038 ceph-mon[32724]: pgmap v505: 129 pgs: 129 active+clean; 1004 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.1 MiB/s wr, 367 op/s 2024-04-26T21:46:21.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 63508 26.01 MB/sec execute 18 sec latency 56.784 ms 2024-04-26T21:46:22.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 63889 25.64 MB/sec execute 19 sec latency 83.985 ms 2024-04-26T21:46:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:22 smithi186 ceph-mon[32991]: pgmap v506: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.4 MiB/s wr, 565 op/s 2024-04-26T21:46:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:22 smithi001 ceph-mon[28054]: pgmap v506: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.4 MiB/s wr, 565 op/s 2024-04-26T21:46:22.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:22 smithi038 ceph-mon[32724]: pgmap v506: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 7.4 MiB/s wr, 565 op/s 2024-04-26T21:46:23.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 64240 24.86 MB/sec execute 20 sec latency 193.130 ms 2024-04-26T21:46:24.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 64616 24.57 MB/sec execute 21 sec latency 176.617 ms 2024-04-26T21:46:24.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.410+0000 7f09e1200640 1 -- 172.21.15.1:0/4016553682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc075620 msgr2=0x7f09dc10ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:24.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.410+0000 7f09e1200640 1 --2- 172.21.15.1:0/4016553682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc075620 0x7f09dc10ec80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009900 tx=0x7f09c802a4a0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.412+0000 7f09e1200640 1 -- 172.21.15.1:0/4016553682 shutdown_connections 2024-04-26T21:46:24.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.412+0000 7f09e1200640 1 --2- 172.21.15.1:0/4016553682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc075620 0x7f09dc10ec80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.412+0000 7f09e1200640 1 --2- 172.21.15.1:0/4016553682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09dc10f1c0 0x7f09dc1115d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.412+0000 7f09e1200640 1 --2- 172.21.15.1:0/4016553682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09dc074d00 0x7f09dc0750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.412+0000 7f09e1200640 1 -- 172.21.15.1:0/4016553682 >> 172.21.15.1:0/4016553682 conn(0x7f09dc0fe3f0 msgr2=0x7f09dc100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:24.413 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.413+0000 7f09e1200640 1 -- 172.21.15.1:0/4016553682 shutdown_connections 2024-04-26T21:46:24.413 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.413+0000 7f09e1200640 1 -- 172.21.15.1:0/4016553682 wait complete. 2024-04-26T21:46:24.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.414+0000 7f09e1200640 1 Processor -- start 2024-04-26T21:46:24.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.414+0000 7f09e1200640 1 -- start start 2024-04-26T21:46:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09e1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09dc074d00 0x7f09dc19b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09e1200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09dc075620 0x7f09dc19b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09e1200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc10f1c0 0x7f09dc19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09e1200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc114e20 con 0x7f09dc075620 2024-04-26T21:46:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09e1200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc114ca0 con 0x7f09dc074d00 2024-04-26T21:46:24.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09e1200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc114fa0 con 0x7f09dc10f1c0 2024-04-26T21:46:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09da800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc10f1c0 0x7f09dc19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09d9e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09dc074d00 0x7f09dc19b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09da800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc10f1c0 0x7f09dc19fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33074/0 (socket says 172.21.15.1:33074) 2024-04-26T21:46:24.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09da800640 1 -- 172.21.15.1:0/1577200184 learned_addr learned my addr 172.21.15.1:0/1577200184 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.415+0000 7f09d9400640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09dc075620 0x7f09dc19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09d9e00640 1 -- 172.21.15.1:0/1577200184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc10f1c0 msgr2=0x7f09dc19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09d9e00640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc10f1c0 0x7f09dc19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09d9e00640 1 -- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09dc075620 msgr2=0x7f09dc19b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09d9e00640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09dc075620 0x7f09dc19b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09d9e00640 1 -- 172.21.15.1:0/1577200184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc1a03b0 con 0x7f09dc074d00 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09d9400640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09dc075620 0x7f09dc19b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09d9e00640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09dc074d00 0x7f09dc19b3c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00d6d0 tx=0x7f09cc00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09d2a00640 1 -- 172.21.15.1:0/1577200184 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc017070 con 0x7f09dc074d00 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09d2a00640 1 -- 172.21.15.1:0/1577200184 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09cc009820 con 0x7f09dc074d00 2024-04-26T21:46:24.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09d2a00640 1 -- 172.21.15.1:0/1577200184 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc004c40 con 0x7f09dc074d00 2024-04-26T21:46:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.416+0000 7f09e1200640 1 -- 172.21.15.1:0/1577200184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09dc1a06a0 con 0x7f09dc074d00 2024-04-26T21:46:24.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.417+0000 7f09e1200640 1 -- 172.21.15.1:0/1577200184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09dc1a7f30 con 0x7f09dc074d00 2024-04-26T21:46:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.421+0000 7f09d2a00640 1 -- 172.21.15.1:0/1577200184 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f09cc005220 con 0x7f09dc074d00 2024-04-26T21:46:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.421+0000 7f09e3f89740 1 Processor -- start 2024-04-26T21:46:24.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.422+0000 7f09e3f89740 1 -- start start 2024-04-26T21:46:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.422+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x55866801a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.422+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801af00 0x558668025280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.422+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668027bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.422+0000 7f09e3f89740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558667f77380 con 0x55866801af00 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.422+0000 7f09e3f89740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558667f13c30 con 0x55866801a5e0 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.422+0000 7f09e3f89740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558667f73340 con 0x5586680257c0 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.423+0000 7f09da800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x55866801a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.423+0000 7f09da800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x55866801a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:56718/0 (socket says 172.21.15.1:56718) 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.423+0000 7f09da800640 1 -- 172.21.15.1:0/623612511 learned_addr learned my addr 172.21.15.1:0/623612511 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.423+0000 7f09d9400640 1 --2- 172.21.15.1:0/623612511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668027bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.423+0000 7f09d9e00640 1 --2- 172.21.15.1:0/623612511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801af00 0x558668025280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.423+0000 7f09da800640 1 -- 172.21.15.1:0/623612511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 msgr2=0x558668027bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.423+0000 7f09da800640 1 --2- 172.21.15.1:0/623612511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668027bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.423+0000 7f09da800640 1 -- 172.21.15.1:0/623612511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801af00 msgr2=0x558668025280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:24.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.424+0000 7f09da800640 1 --2- 172.21.15.1:0/623612511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801af00 0x558668025280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.424+0000 7f09da800640 1 -- 172.21.15.1:0/623612511 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558667f35940 con 0x55866801a5e0 2024-04-26T21:46:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.424+0000 7f09d9e00640 1 --2- 172.21.15.1:0/623612511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801af00 0x558668025280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:46:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.424+0000 7f09d2a00640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09c00a3de0 0x7f09c00a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.424+0000 7f09d2a00640 1 -- 172.21.15.1:0/1577200184 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7f09cc0d01e0 con 0x7f09dc074d00 2024-04-26T21:46:24.426 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.424+0000 7f09da800640 1 --2- 172.21.15.1:0/623612511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x55866801a9c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f09d4007ee0 tx=0x7f09d40207a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e54a971f730ae7e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.425+0000 7f09c7400640 1 -- 172.21.15.1:0/623612511 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4003b70 con 0x55866801a5e0 2024-04-26T21:46:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.425+0000 7f09c7400640 1 -- 172.21.15.1:0/623612511 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09d4003d10 con 0x55866801a5e0 2024-04-26T21:46:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.425+0000 7f09c7400640 1 -- 172.21.15.1:0/623612511 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d40044c0 con 0x55866801a5e0 2024-04-26T21:46:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.425+0000 7f09d9e00640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09c00a3de0 0x7f09c00a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.427+0000 7f09d9e00640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09c00a3de0 0x7f09c00a62a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55866801bb80 tx=0x7f09cc0bc040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.427+0000 7f09e3f89740 1 -- 172.21.15.1:0/623612511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 msgr2=0x55866801a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:24.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.427+0000 7f09e3f89740 1 --2- 172.21.15.1:0/623612511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x55866801a9c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f09d4007ee0 tx=0x7f09d40207a0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.428+0000 7f09e3f89740 1 -- 172.21.15.1:0/623612511 shutdown_connections 2024-04-26T21:46:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.428+0000 7f09e3f89740 1 --2- 172.21.15.1:0/623612511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668027bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.428+0000 7f09e3f89740 1 --2- 172.21.15.1:0/623612511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801af00 0x558668025280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.428+0000 7f09e3f89740 1 --2- 172.21.15.1:0/623612511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x55866801a9c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.428+0000 7f09e3f89740 1 -- 172.21.15.1:0/623612511 >> 172.21.15.1:0/623612511 conn(0x558668016af0 msgr2=0x558668016ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.429+0000 7f09e3f89740 1 -- 172.21.15.1:0/623612511 shutdown_connections 2024-04-26T21:46:24.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.429+0000 7f09e3f89740 1 -- 172.21.15.1:0/623612511 wait complete. 2024-04-26T21:46:24.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.430+0000 7f09e3f89740 1 -- start start 2024-04-26T21:46:24.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.431+0000 7f09e3f89740 1 Processor -- start 2024-04-26T21:46:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.431+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801a5e0 0x5586680138b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.431+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x558668013df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.431+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668024270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.431+0000 7f09e3f89740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55866800ded0 con 0x55866801a5e0 2024-04-26T21:46:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.431+0000 7f09e3f89740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558667efec60 con 0x55866801af00 2024-04-26T21:46:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.431+0000 7f09e3f89740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55866800e050 con 0x5586680257c0 2024-04-26T21:46:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.431+0000 7f09e3f89740 10 client.0 _ll_register_callbacks cb 0x558667f2ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:46:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.432+0000 7f09d9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x558668013df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.433+0000 7f09d9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x558668013df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:56734/0 (socket says 172.21.15.1:56734) 2024-04-26T21:46:24.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.433+0000 7f09d9400640 1 -- 172.21.15.1:0/1643634101 learned_addr learned my addr 172.21.15.1:0/1643634101 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.433+0000 7f09d9400640 1 -- 172.21.15.1:0/1643634101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 msgr2=0x558668024270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:46:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.433+0000 7f09d9400640 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668024270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.433+0000 7f09d9400640 1 -- 172.21.15.1:0/1643634101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801a5e0 msgr2=0x5586680138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.433+0000 7f09d9400640 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801a5e0 0x5586680138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.433+0000 7f09d9400640 1 -- 172.21.15.1:0/1643634101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558667f0d510 con 0x55866801af00 2024-04-26T21:46:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.434+0000 7f09d9400640 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x558668013df0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009230 tx=0x7f09c80044f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a334997317bd6c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.434+0000 7f09bbe00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:46:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.435+0000 7f09bbe00640 1 -- 172.21.15.1:0/1643634101 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8003ac0 con 0x55866801af00 2024-04-26T21:46:24.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.436+0000 7f09bbe00640 1 -- 172.21.15.1:0/1643634101 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7f09c8003c60 con 0x55866801af00 2024-04-26T21:46:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.437+0000 7f09e3f89740 1 -- 172.21.15.1:0/1643634101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09d4003d10 con 0x55866801af00 2024-04-26T21:46:24.437 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.437+0000 7f09bbe00640 1 -- 172.21.15.1:0/1643634101 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f09c8003e60 con 0x55866801af00 2024-04-26T21:46:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.437+0000 7f09e3f89740 10 client.26762 fetch_fsmap learned FSMap version 50 2024-04-26T21:46:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.437+0000 7f09e3f89740 1 -- 172.21.15.1:0/1643634101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x558667c89210 con 0x55866801af00 2024-04-26T21:46:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.438+0000 7f09bbe00640 1 -- 172.21.15.1:0/1643634101 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09d4003d10 con 0x55866801af00 2024-04-26T21:46:24.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.438+0000 7f09bbe00640 1 -- 172.21.15.1:0/1643634101 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7f09c8044070 con 0x55866801af00 2024-04-26T21:46:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.439+0000 7f09e3f89740 10 client.26762 fetch_fsmap finished waiting for FSMap version 50 2024-04-26T21:46:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.439+0000 7f09e3f89740 10 client.26762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:46:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.439+0000 7f09e3f89740 20 client.26762 populate_metadata read hostname 'smithi001' 2024-04-26T21:46:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.439+0000 7f09e3f89740 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55866802ccb0 0x55866802f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.439+0000 7f09e3f89740 4 client.26762 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:46:24.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.439+0000 7f09e3f89740 1 -- 172.21.15.1:0/1643634101 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558667f84f90 con 0x55866802ccb0 2024-04-26T21:46:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.439+0000 7f09da800640 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55866802ccb0 0x55866802f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.440+0000 7f09da800640 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55866802ccb0 0x55866802f0e0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d72cea702eff39a server_cookie=a4f6c888f7f6c901 in_seq=0 out_seq=0 2024-04-26T21:46:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.440+0000 7f09bbe00640 10 client.26762 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.444+0000 7f09bbe00640 1 -- 172.21.15.1:0/1643634101 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558667f84f90 con 0x55866802ccb0 2024-04-26T21:46:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.444+0000 7f09bbe00640 10 client.26762 handle_command_reply: tid=0 2024-04-26T21:46:24.444 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.444+0000 7f09e3f89740 1 client.26762 shutdown 2024-04-26T21:46:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.444+0000 7f09e3f89740 1 -- 172.21.15.1:0/1643634101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 msgr2=0x558668013df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:24.445 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.444+0000 7f09e3f89740 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x558668013df0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009230 tx=0x7f09c80044f0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.445+0000 7f09e3f89740 1 -- 172.21.15.1:0/1643634101 shutdown_connections 2024-04-26T21:46:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.445+0000 7f09e3f89740 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55866802ccb0 0x55866802f0e0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.445+0000 7f09e3f89740 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668024270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.445+0000 7f09e3f89740 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x558668013df0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.445+0000 7f09e3f89740 1 --2- 172.21.15.1:0/1643634101 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801a5e0 0x5586680138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.445+0000 7f09e3f89740 1 -- 172.21.15.1:0/1643634101 >> 172.21.15.1:0/1643634101 conn(0x558668016af0 msgr2=0x55866800d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.445+0000 7f09bbe00640 0 client.26762 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.446+0000 7f09e3f89740 1 -- 172.21.15.1:0/1643634101 shutdown_connections 2024-04-26T21:46:24.446 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.446+0000 7f09e3f89740 1 -- 172.21.15.1:0/1643634101 wait complete. 2024-04-26T21:46:24.447 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.447+0000 7f09e3f89740 20 client.26762 trim_cache size 0 max 16384 2024-04-26T21:46:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:24 smithi186 ceph-mon[32991]: pgmap v507: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 491 op/s 2024-04-26T21:46:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.473+0000 7f09e3f89740 1 Processor -- start 2024-04-26T21:46:24.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.473+0000 7f09e3f89740 1 -- start start 2024-04-26T21:46:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.474+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801a5e0 0x558668015fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.474+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x5586680349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.474+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668034f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.474+0000 7f09e3f89740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc1b07a0 con 0x55866801a5e0 2024-04-26T21:46:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.474+0000 7f09e3f89740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc1ae010 con 0x55866801af00 2024-04-26T21:46:24.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.474+0000 7f09e3f89740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc1abea0 con 0x5586680257c0 2024-04-26T21:46:24.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.474+0000 7f09d9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x5586680349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.474+0000 7f09da800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801a5e0 0x558668015fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.474+0000 7f09d9400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x5586680349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:56744/0 (socket says 172.21.15.1:56744) 2024-04-26T21:46:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.474+0000 7f09d9400640 1 -- 172.21.15.1:0/1457759528 learned_addr learned my addr 172.21.15.1:0/1457759528 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.475+0000 7f09da800640 1 --2- 172.21.15.1:0/1457759528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668034f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.475+0000 7f09d9400640 1 -- 172.21.15.1:0/1457759528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 msgr2=0x558668034f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.475+0000 7f09d9400640 1 --2- 172.21.15.1:0/1457759528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668034f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.475+0000 7f09d9400640 1 -- 172.21.15.1:0/1457759528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801a5e0 msgr2=0x558668015fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.475+0000 7f09d9400640 1 --2- 172.21.15.1:0/1457759528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801a5e0 0x558668015fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.475+0000 7f09d9400640 1 -- 172.21.15.1:0/1457759528 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558667d55620 con 0x55866801af00 2024-04-26T21:46:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.475+0000 7f09d9400640 1 --2- 172.21.15.1:0/1457759528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x5586680349f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f09c8004b50 tx=0x7f09c8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9aa4104ca7f2615 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.476+0000 7f09c6a00640 1 -- 172.21.15.1:0/1457759528 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c8057020 con 0x55866801af00 2024-04-26T21:46:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.476+0000 7f09c6a00640 1 -- 172.21.15.1:0/1457759528 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09c8052450 con 0x55866801af00 2024-04-26T21:46:24.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.476+0000 7f09c6a00640 1 -- 172.21.15.1:0/1457759528 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c80466a0 con 0x55866801af00 2024-04-26T21:46:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.477+0000 7f09e3f89740 1 -- 172.21.15.1:0/1457759528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 msgr2=0x5586680349f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.477+0000 7f09e3f89740 1 --2- 172.21.15.1:0/1457759528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x5586680349f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f09c8004b50 tx=0x7f09c8009260 comp rx=0 tx=0).stop 2024-04-26T21:46:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.477+0000 7f09e3f89740 1 -- 172.21.15.1:0/1457759528 shutdown_connections 2024-04-26T21:46:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.477+0000 7f09e3f89740 1 --2- 172.21.15.1:0/1457759528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668034f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.477+0000 7f09e3f89740 1 --2- 172.21.15.1:0/1457759528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801af00 0x5586680349f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.477+0000 7f09e3f89740 1 --2- 172.21.15.1:0/1457759528 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801a5e0 0x558668015fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.477+0000 7f09e3f89740 1 -- 172.21.15.1:0/1457759528 >> 172.21.15.1:0/1457759528 conn(0x558668016af0 msgr2=0x558668013870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.477+0000 7f09e3f89740 1 -- 172.21.15.1:0/1457759528 shutdown_connections 2024-04-26T21:46:24.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.477+0000 7f09e3f89740 1 -- 172.21.15.1:0/1457759528 wait complete. 2024-04-26T21:46:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.477+0000 7f09e3f89740 1 -- start start 2024-04-26T21:46:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.478+0000 7f09e3f89740 1 Processor -- start 2024-04-26T21:46:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.478+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x558668020170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.479+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801af00 0x5586680206b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.479+0000 7f09e3f89740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668092270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.479+0000 7f09e3f89740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc1ac4f0 con 0x55866801af00 2024-04-26T21:46:24.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.479+0000 7f09e3f89740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc1ad940 con 0x55866801a5e0 2024-04-26T21:46:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.479+0000 7f09e3f89740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09dc1acba0 con 0x5586680257c0 2024-04-26T21:46:24.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.479+0000 7f09e3f89740 10 client.0 _ll_register_callbacks cb 0x7f09c8009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:46:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.479+0000 7f09da800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x558668020170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.479+0000 7f09da800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x558668020170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:56746/0 (socket says 172.21.15.1:56746) 2024-04-26T21:46:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.479+0000 7f09da800640 1 -- 172.21.15.1:0/2751498522 learned_addr learned my addr 172.21.15.1:0/2751498522 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.479+0000 7f09da800640 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668092270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.480+0000 7f09da800640 1 -- 172.21.15.1:0/2751498522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 msgr2=0x558668092270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.480+0000 7f09da800640 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668092270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.480+0000 7f09da800640 1 -- 172.21.15.1:0/2751498522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801af00 msgr2=0x5586680206b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.480+0000 7f09da800640 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801af00 0x5586680206b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.480+0000 7f09da800640 1 -- 172.21.15.1:0/2751498522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558667f31cb0 con 0x55866801a5e0 2024-04-26T21:46:24.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.480+0000 7f09da800640 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x558668020170 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f09d4005b40 tx=0x7f09d4005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a52052bf45b4ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.480+0000 7f09bbe00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:46:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.480+0000 7f09bbe00640 1 -- 172.21.15.1:0/2751498522 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d402e040 con 0x55866801a5e0 2024-04-26T21:46:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.480+0000 7f09bbe00640 1 -- 172.21.15.1:0/2751498522 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7f09d4003cc0 con 0x55866801a5e0 2024-04-26T21:46:24.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.481+0000 7f09e3f89740 1 -- 172.21.15.1:0/2751498522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09c8052450 con 0x55866801a5e0 2024-04-26T21:46:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.482+0000 7f09bbe00640 1 -- 172.21.15.1:0/2751498522 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f09d40390e0 con 0x55866801a5e0 2024-04-26T21:46:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.483+0000 7f09e3f89740 10 client.26768 fetch_fsmap learned FSMap version 50 2024-04-26T21:46:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.483+0000 7f09e3f89740 1 -- 172.21.15.1:0/2751498522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5586680938b0 con 0x55866801a5e0 2024-04-26T21:46:24.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.483+0000 7f09bbe00640 1 -- 172.21.15.1:0/2751498522 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09c8052450 con 0x55866801a5e0 2024-04-26T21:46:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.484+0000 7f09bbe00640 1 -- 172.21.15.1:0/2751498522 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7f09d4043030 con 0x55866801a5e0 2024-04-26T21:46:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.485+0000 7f09e3f89740 10 client.26768 fetch_fsmap finished waiting for FSMap version 50 2024-04-26T21:46:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.485+0000 7f09e3f89740 10 client.26768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:46:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.485+0000 7f09e3f89740 20 client.26768 populate_metadata read hostname 'smithi001' 2024-04-26T21:46:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.485+0000 7f09e3f89740 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558668093d50 0x558668096180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.485+0000 7f09e3f89740 4 client.26768 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:46:24.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.485+0000 7f09e3f89740 1 -- 172.21.15.1:0/2751498522 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558667f7ee00 con 0x558668093d50 2024-04-26T21:46:24.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.485+0000 7f09d9400640 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558668093d50 0x558668096180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:24.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.486+0000 7f09d9400640 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558668093d50 0x558668096180 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aebc4983dc0e8c95 server_cookie=1d5f8984e0fd2f3a in_seq=0 out_seq=0 2024-04-26T21:46:24.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.486+0000 7f09bbe00640 10 client.26768 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:24 smithi001 ceph-mon[28054]: pgmap v507: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 491 op/s 2024-04-26T21:46:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:24 smithi038 ceph-mon[32724]: pgmap v507: 129 pgs: 129 active+clean; 1.0 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 491 op/s 2024-04-26T21:46:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09bbe00640 1 -- 172.21.15.1:0/2751498522 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558667f7ee00 con 0x558668093d50 2024-04-26T21:46:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09bbe00640 10 client.26768 handle_command_reply: tid=0 2024-04-26T21:46:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09e3f89740 1 client.26768 shutdown 2024-04-26T21:46:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09e3f89740 1 -- 172.21.15.1:0/2751498522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 msgr2=0x558668020170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09e3f89740 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x558668020170 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f09d4005b40 tx=0x7f09d4005f80 comp rx=0 tx=0).stop 2024-04-26T21:46:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09e3f89740 1 -- 172.21.15.1:0/2751498522 shutdown_connections 2024-04-26T21:46:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09e3f89740 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558668093d50 0x558668096180 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09e3f89740 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586680257c0 0x558668092270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09e3f89740 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55866801af00 0x5586680206b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09bbe00640 0 client.26768 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09e3f89740 1 --2- 172.21.15.1:0/2751498522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55866801a5e0 0x558668020170 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.913+0000 7f09e3f89740 1 -- 172.21.15.1:0/2751498522 >> 172.21.15.1:0/2751498522 conn(0x558668016af0 msgr2=0x55866800cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.914+0000 7f09e3f89740 1 -- 172.21.15.1:0/2751498522 shutdown_connections 2024-04-26T21:46:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.914+0000 7f09e3f89740 1 -- 172.21.15.1:0/2751498522 wait complete. 2024-04-26T21:46:24.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.917+0000 7f09e3f89740 20 client.26768 trim_cache size 0 max 16384 2024-04-26T21:46:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:46:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:46:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:46:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:46:24.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.918+0000 7f09e1200640 1 -- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09c00a3de0 msgr2=0x7f09c00a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.918+0000 7f09e1200640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09c00a3de0 0x7f09c00a62a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x55866801bb80 tx=0x7f09cc0bc040 comp rx=0 tx=0).stop 2024-04-26T21:46:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.918+0000 7f09e1200640 1 -- 172.21.15.1:0/1577200184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09dc074d00 msgr2=0x7f09dc19b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:24.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.918+0000 7f09e1200640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09dc074d00 0x7f09dc19b3c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00d6d0 tx=0x7f09cc00db90 comp rx=0 tx=0).stop 2024-04-26T21:46:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.919+0000 7f09e1200640 1 -- 172.21.15.1:0/1577200184 shutdown_connections 2024-04-26T21:46:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.919+0000 7f09e1200640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f09c00a3de0 0x7f09c00a62a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.919+0000 7f09e1200640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09dc10f1c0 0x7f09dc19fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.919+0000 7f09e1200640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09dc075620 0x7f09dc19b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.919+0000 7f09e1200640 1 --2- 172.21.15.1:0/1577200184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f09dc074d00 0x7f09dc19b3c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.919+0000 7f09e1200640 1 -- 172.21.15.1:0/1577200184 >> 172.21.15.1:0/1577200184 conn(0x7f09dc0fe3f0 msgr2=0x7f09dc0ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.920+0000 7f09e1200640 1 -- 172.21.15.1:0/1577200184 shutdown_connections 2024-04-26T21:46:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:24.920+0000 7f09e1200640 1 -- 172.21.15.1:0/1577200184 wait complete. 2024-04-26T21:46:25.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 65156 24.54 MB/sec execute 22 sec latency 272.024 ms 2024-04-26T21:46:26.142 INFO:tasks.workunit.client.0.smithi001.stdout: 10 65997 25.80 MB/sec execute 23 sec latency 33.710 ms 2024-04-26T21:46:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:26 smithi186 ceph-mon[32991]: pgmap v508: 129 pgs: 129 active+clean; 1014 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 9.5 MiB/s wr, 702 op/s 2024-04-26T21:46:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:26 smithi001 ceph-mon[28054]: pgmap v508: 129 pgs: 129 active+clean; 1014 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 9.5 MiB/s wr, 702 op/s 2024-04-26T21:46:26.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:26 smithi038 ceph-mon[32724]: pgmap v508: 129 pgs: 129 active+clean; 1014 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 9.5 MiB/s wr, 702 op/s 2024-04-26T21:46:27.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 66700 26.61 MB/sec execute 24 sec latency 50.234 ms 2024-04-26T21:46:28.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 67027 26.26 MB/sec execute 25 sec latency 277.096 ms 2024-04-26T21:46:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:28 smithi186 ceph-mon[32991]: pgmap v509: 129 pgs: 129 active+clean; 1006 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 7.1 MiB/s wr, 490 op/s 2024-04-26T21:46:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:28 smithi001 ceph-mon[28054]: pgmap v509: 129 pgs: 129 active+clean; 1006 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 7.1 MiB/s wr, 490 op/s 2024-04-26T21:46:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:28 smithi038 ceph-mon[32724]: pgmap v509: 129 pgs: 129 active+clean; 1006 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 7.1 MiB/s wr, 490 op/s 2024-04-26T21:46:29.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 67114 25.45 MB/sec execute 26 sec latency 659.842 ms 2024-04-26T21:46:30.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 67303 24.83 MB/sec execute 27 sec latency 688.140 ms 2024-04-26T21:46:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:30 smithi186 ceph-mon[32991]: pgmap v510: 129 pgs: 129 active+clean; 1006 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 6.9 MiB/s wr, 460 op/s 2024-04-26T21:46:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:30 smithi001 ceph-mon[28054]: pgmap v510: 129 pgs: 129 active+clean; 1006 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 6.9 MiB/s wr, 460 op/s 2024-04-26T21:46:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:30 smithi038 ceph-mon[32724]: pgmap v510: 129 pgs: 129 active+clean; 1006 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 6.9 MiB/s wr, 460 op/s 2024-04-26T21:46:31.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 67610 24.40 MB/sec execute 28 sec latency 79.927 ms 2024-04-26T21:46:32.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 68215 24.35 MB/sec execute 29 sec latency 84.999 ms 2024-04-26T21:46:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:32 smithi186 ceph-mon[32991]: pgmap v511: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 748 op/s 2024-04-26T21:46:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:32 smithi001 ceph-mon[28054]: pgmap v511: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 748 op/s 2024-04-26T21:46:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:32 smithi038 ceph-mon[32724]: pgmap v511: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 748 op/s 2024-04-26T21:46:33.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 69036 25.05 MB/sec execute 30 sec latency 59.512 ms 2024-04-26T21:46:34.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 69746 25.63 MB/sec execute 31 sec latency 50.835 ms 2024-04-26T21:46:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:34 smithi186 ceph-mon[32991]: pgmap v512: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 11 MiB/s wr, 583 op/s 2024-04-26T21:46:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:34 smithi001 ceph-mon[28054]: pgmap v512: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 11 MiB/s wr, 583 op/s 2024-04-26T21:46:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:34 smithi038 ceph-mon[32724]: pgmap v512: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 11 MiB/s wr, 583 op/s 2024-04-26T21:46:35.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 70431 26.35 MB/sec execute 32 sec latency 44.075 ms 2024-04-26T21:46:36.145 INFO:tasks.workunit.client.0.smithi001.stdout: 10 70845 26.21 MB/sec execute 33 sec latency 60.754 ms 2024-04-26T21:46:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:36 smithi186 ceph-mon[32991]: pgmap v513: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 784 op/s 2024-04-26T21:46:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:36 smithi001 ceph-mon[28054]: pgmap v513: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 784 op/s 2024-04-26T21:46:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:36 smithi038 ceph-mon[32724]: pgmap v513: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 784 op/s 2024-04-26T21:46:37.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 71280 25.87 MB/sec execute 34 sec latency 67.130 ms 2024-04-26T21:46:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.107+0000 7fc44ba00640 1 -- 172.21.15.1:0/1319714641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc444105510 msgr2=0x7fc4441119d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.107+0000 7fc44ba00640 1 --2- 172.21.15.1:0/1319714641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc444105510 0x7fc4441119d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009cd0 tx=0x7fc43c02a8f0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.107+0000 7fc44ba00640 1 -- 172.21.15.1:0/1319714641 shutdown_connections 2024-04-26T21:46:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.107+0000 7fc44ba00640 1 --2- 172.21.15.1:0/1319714641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc444104b50 0x7fc444104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.107+0000 7fc44ba00640 1 --2- 172.21.15.1:0/1319714641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc444105510 0x7fc4441119d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.107+0000 7fc44ba00640 1 --2- 172.21.15.1:0/1319714641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc44410ac10 0x7fc44410aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.107+0000 7fc44ba00640 1 -- 172.21.15.1:0/1319714641 >> 172.21.15.1:0/1319714641 conn(0x7fc444100600 msgr2=0x7fc444102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.107+0000 7fc44ba00640 1 -- 172.21.15.1:0/1319714641 shutdown_connections 2024-04-26T21:46:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.107+0000 7fc44ba00640 1 -- 172.21.15.1:0/1319714641 wait complete. 2024-04-26T21:46:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.108+0000 7fc44ba00640 1 Processor -- start 2024-04-26T21:46:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.108+0000 7fc44ba00640 1 -- start start 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc44ba00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc444104b50 0x7fc44419b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc44ba00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc444105510 0x7fc44419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc44ba00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc44410ac10 0x7fc44419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc44ba00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc444114e50 con 0x7fc44410ac10 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc44ba00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc444114cd0 con 0x7fc444104b50 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc44ba00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc444114fd0 con 0x7fc444105510 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc449000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc444104b50 0x7fc44419b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc443e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc444105510 0x7fc44419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc449a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc44410ac10 0x7fc44419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc449000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc444104b50 0x7fc44419b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51862/0 (socket says 172.21.15.1:51862) 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc449000640 1 -- 172.21.15.1:0/3289920443 learned_addr learned my addr 172.21.15.1:0/3289920443 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc449000640 1 -- 172.21.15.1:0/3289920443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc444105510 msgr2=0x7fc44419b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc449000640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc444105510 0x7fc44419b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.109+0000 7fc449000640 1 -- 172.21.15.1:0/3289920443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc44410ac10 msgr2=0x7fc44419fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.110+0000 7fc449000640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc44410ac10 0x7fc44419fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.110+0000 7fc449000640 1 -- 172.21.15.1:0/3289920443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4441a0260 con 0x7fc444104b50 2024-04-26T21:46:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.110+0000 7fc449a00640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc44410ac10 0x7fc44419fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:46:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.110+0000 7fc449000640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc444104b50 0x7fc44419b270 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc43800ea60 tx=0x7fc43800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.110+0000 7fc441600640 1 -- 172.21.15.1:0/3289920443 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43800cde0 con 0x7fc444104b50 2024-04-26T21:46:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.110+0000 7fc441600640 1 -- 172.21.15.1:0/3289920443 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc4380035f0 con 0x7fc444104b50 2024-04-26T21:46:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.110+0000 7fc44ba00640 1 -- 172.21.15.1:0/3289920443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4441a0550 con 0x7fc444104b50 2024-04-26T21:46:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.110+0000 7fc441600640 1 -- 172.21.15.1:0/3289920443 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438010690 con 0x7fc444104b50 2024-04-26T21:46:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.110+0000 7fc44ba00640 1 -- 172.21.15.1:0/3289920443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4441a7de0 con 0x7fc444104b50 2024-04-26T21:46:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.114+0000 7fc44e80b740 1 Processor -- start 2024-04-26T21:46:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.114+0000 7fc44e80b740 1 -- start start 2024-04-26T21:46:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.114+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 0x562d320da0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320ed280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320ed7c0 0x562d320efbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc44e80b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562d3203f380 con 0x562d320da630 2024-04-26T21:46:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc44e80b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562d31fdbc30 con 0x562d320ed7c0 2024-04-26T21:46:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc44e80b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562d3203b340 con 0x562d320d9d10 2024-04-26T21:46:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc443e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320ed7c0 0x562d320efbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc443e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320ed7c0 0x562d320efbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51872/0 (socket says 172.21.15.1:51872) 2024-04-26T21:46:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc443e00640 1 -- 172.21.15.1:0/986125116 learned_addr learned my addr 172.21.15.1:0/986125116 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc449a00640 1 --2- 172.21.15.1:0/986125116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 0x562d320da0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc443e00640 1 -- 172.21.15.1:0/986125116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 msgr2=0x562d320da0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc443e00640 1 --2- 172.21.15.1:0/986125116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 0x562d320da0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc443e00640 1 -- 172.21.15.1:0/986125116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 msgr2=0x562d320ed280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:46:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc443e00640 1 --2- 172.21.15.1:0/986125116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320ed280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.115+0000 7fc443e00640 1 -- 172.21.15.1:0/986125116 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d31ffd940 con 0x562d320ed7c0 2024-04-26T21:46:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.116+0000 7fc449a00640 1 --2- 172.21.15.1:0/986125116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 0x562d320da0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:46:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.116+0000 7fc441600640 1 -- 172.21.15.1:0/3289920443 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fc4380037a0 con 0x7fc444104b50 2024-04-26T21:46:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.117+0000 7fc443e00640 1 --2- 172.21.15.1:0/986125116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320ed7c0 0x562d320efbb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc430009ca0 tx=0x7fc430011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae5d986ccc10968e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.118+0000 7fc441600640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc42c0a3d90 0x7fc42c0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.118+0000 7fc436000640 1 -- 172.21.15.1:0/986125116 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43001e070 con 0x562d320ed7c0 2024-04-26T21:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.118+0000 7fc436000640 1 -- 172.21.15.1:0/986125116 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc430019ae0 con 0x562d320ed7c0 2024-04-26T21:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.118+0000 7fc436000640 1 -- 172.21.15.1:0/986125116 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43000e9b0 con 0x562d320ed7c0 2024-04-26T21:46:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.119+0000 7fc441600640 1 -- 172.21.15.1:0/3289920443 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7fc4380d0f60 con 0x7fc444104b50 2024-04-26T21:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.120+0000 7fc44e80b740 1 -- 172.21.15.1:0/986125116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320ed7c0 msgr2=0x562d320efbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.120+0000 7fc44e80b740 1 --2- 172.21.15.1:0/986125116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320ed7c0 0x562d320efbb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc430009ca0 tx=0x7fc430011810 comp rx=0 tx=0).stop 2024-04-26T21:46:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.120+0000 7fc449a00640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc42c0a3d90 0x7fc42c0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.121+0000 7fc449a00640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc42c0a3d90 0x7fc42c0a6250 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc43c004af0 tx=0x7fc43c005d80 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.121+0000 7fc44e80b740 1 -- 172.21.15.1:0/986125116 shutdown_connections 2024-04-26T21:46:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.121+0000 7fc44e80b740 1 --2- 172.21.15.1:0/986125116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320ed7c0 0x562d320efbb0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.121+0000 7fc44e80b740 1 --2- 172.21.15.1:0/986125116 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320ed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.121+0000 7fc44e80b740 1 --2- 172.21.15.1:0/986125116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 0x562d320da0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.121+0000 7fc44e80b740 1 -- 172.21.15.1:0/986125116 >> 172.21.15.1:0/986125116 conn(0x562d320d62e0 msgr2=0x562d320d66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.122+0000 7fc44e80b740 1 -- 172.21.15.1:0/986125116 shutdown_connections 2024-04-26T21:46:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.122+0000 7fc44e80b740 1 -- 172.21.15.1:0/986125116 wait complete. 2024-04-26T21:46:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.122+0000 7fc44e80b740 1 -- start start 2024-04-26T21:46:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.122+0000 7fc44e80b740 1 Processor -- start 2024-04-26T21:46:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.123+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.123+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.123+0000 7fc449000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.123+0000 7fc449000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320ea7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51876/0 (socket says 172.21.15.1:51876) 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.123+0000 7fc449000640 1 -- 172.21.15.1:0/3577095058 learned_addr learned my addr 172.21.15.1:0/3577095058 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.124+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320ed7c0 0x562d320eb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.124+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562d320f0390 con 0x562d320da630 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.124+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562d31fc6c60 con 0x562d320d9d10 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.124+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562d320f0510 con 0x562d320ed7c0 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.124+0000 7fc44e80b740 10 client.0 _ll_register_callbacks cb 0x562d31ff2e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.124+0000 7fc449a00640 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320ed7c0 0x562d320eb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.124+0000 7fc443e00640 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.124+0000 7fc449000640 1 -- 172.21.15.1:0/3577095058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320ed7c0 msgr2=0x562d320eb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.125+0000 7fc449000640 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320ed7c0 0x562d320eb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.125+0000 7fc449000640 1 -- 172.21.15.1:0/3577095058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 msgr2=0x562d320eacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.125+0000 7fc449000640 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320eacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.125+0000 7fc449000640 1 -- 172.21.15.1:0/3577095058 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d31fd5510 con 0x562d320d9d10 2024-04-26T21:46:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.125+0000 7fc449000640 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320ea7b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc438098ea0 tx=0x7fc43809c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b52ac8cf433ce80 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.125+0000 7fc42aa00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:46:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.125+0000 7fc42aa00640 1 -- 172.21.15.1:0/3577095058 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4380a0df0 con 0x562d320d9d10 2024-04-26T21:46:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.126+0000 7fc42aa00640 1 -- 172.21.15.1:0/3577095058 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7fc4380990b0 con 0x562d320d9d10 2024-04-26T21:46:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.126+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc430019ae0 con 0x562d320d9d10 2024-04-26T21:46:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.126+0000 7fc42aa00640 1 -- 172.21.15.1:0/3577095058 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fc4380a56b0 con 0x562d320d9d10 2024-04-26T21:46:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.127+0000 7fc44e80b740 10 client.26780 fetch_fsmap learned FSMap version 50 2024-04-26T21:46:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.127+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x562d31d51210 con 0x562d320d9d10 2024-04-26T21:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.129+0000 7fc42aa00640 1 -- 172.21.15.1:0/3577095058 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc430019ae0 con 0x562d320d9d10 2024-04-26T21:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.129+0000 7fc42aa00640 1 -- 172.21.15.1:0/3577095058 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7fc4380870e0 con 0x562d320d9d10 2024-04-26T21:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.129+0000 7fc44e80b740 10 client.26780 fetch_fsmap finished waiting for FSMap version 50 2024-04-26T21:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.129+0000 7fc44e80b740 10 client.26780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:46:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.129+0000 7fc44e80b740 20 client.26780 populate_metadata read hostname 'smithi001' 2024-04-26T21:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.129+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562d320fd530 0x562d320ff920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.129+0000 7fc44e80b740 4 client.26780 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.129+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d3204cf90 con 0x562d320fd530 2024-04-26T21:46:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.129+0000 7fc443e00640 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562d320fd530 0x562d320ff920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.130+0000 7fc443e00640 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562d320fd530 0x562d320ff920 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc498b5daf968390 server_cookie=8b019189fbf00a05 in_seq=0 out_seq=0 2024-04-26T21:46:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.130+0000 7fc42aa00640 10 client.26780 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.134+0000 7fc42aa00640 1 -- 172.21.15.1:0/3577095058 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d3204cf90 con 0x562d320fd530 2024-04-26T21:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.134+0000 7fc42aa00640 10 client.26780 handle_command_reply: tid=0 2024-04-26T21:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.134+0000 7fc44e80b740 1 client.26780 shutdown 2024-04-26T21:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.134+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 msgr2=0x562d320ea7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.134+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320ea7b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc438098ea0 tx=0x7fc43809c3a0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.134+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 shutdown_connections 2024-04-26T21:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.134+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562d320fd530 0x562d320ff920 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.135+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320ed7c0 0x562d320eb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.135+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320eacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.135+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3577095058 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320ea7b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.135+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 >> 172.21.15.1:0/3577095058 conn(0x562d320d62e0 msgr2=0x562d320dc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.135+0000 7fc42aa00640 0 client.26780 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.135+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 shutdown_connections 2024-04-26T21:46:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.135+0000 7fc44e80b740 1 -- 172.21.15.1:0/3577095058 wait complete. 2024-04-26T21:46:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.135+0000 7fc44e80b740 20 client.26780 trim_cache size 0 max 16384 2024-04-26T21:46:38.143 INFO:tasks.workunit.client.0.smithi001.stdout: 10 71887 25.85 MB/sec execute 35 sec latency 56.958 ms 2024-04-26T21:46:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.156+0000 7fc44e80b740 1 Processor -- start 2024-04-26T21:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.156+0000 7fc44e80b740 1 -- start start 2024-04-26T21:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.157+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 0x562d320f9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.157+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320da630 0x562d320ec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.157+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320ed7c0 0x562d32105190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.157+0000 7fc44e80b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441b0650 con 0x562d320ed7c0 2024-04-26T21:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.157+0000 7fc44e80b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441adec0 con 0x562d320da630 2024-04-26T21:46:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.157+0000 7fc44e80b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441abd50 con 0x562d320d9d10 2024-04-26T21:46:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.157+0000 7fc449a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320ed7c0 0x562d32105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.157+0000 7fc449a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320ed7c0 0x562d32105190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35208/0 (socket says 172.21.15.1:35208) 2024-04-26T21:46:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.157+0000 7fc449a00640 1 -- 172.21.15.1:0/3720871728 learned_addr learned my addr 172.21.15.1:0/3720871728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.157+0000 7fc449000640 1 --2- 172.21.15.1:0/3720871728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 0x562d320f9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.158+0000 7fc449a00640 1 -- 172.21.15.1:0/3720871728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 msgr2=0x562d320f9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.158+0000 7fc449a00640 1 --2- 172.21.15.1:0/3720871728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 0x562d320f9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.158+0000 7fc449a00640 1 -- 172.21.15.1:0/3720871728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320da630 msgr2=0x562d320ec5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.158+0000 7fc449a00640 1 --2- 172.21.15.1:0/3720871728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320da630 0x562d320ec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.158+0000 7fc449a00640 1 -- 172.21.15.1:0/3720871728 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d320d1940 con 0x562d320ed7c0 2024-04-26T21:46:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.159+0000 7fc449a00640 1 --2- 172.21.15.1:0/3720871728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320ed7c0 0x562d32105190 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009cd0 tx=0x7fc43c041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4e60c82a31b3de server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.159+0000 7fc435600640 1 -- 172.21.15.1:0/3720871728 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c003a30 con 0x562d320ed7c0 2024-04-26T21:46:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.159+0000 7fc435600640 1 -- 172.21.15.1:0/3720871728 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc43c036050 con 0x562d320ed7c0 2024-04-26T21:46:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.160+0000 7fc435600640 1 -- 172.21.15.1:0/3720871728 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c049420 con 0x562d320ed7c0 2024-04-26T21:46:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.160+0000 7fc44e80b740 1 -- 172.21.15.1:0/3720871728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320ed7c0 msgr2=0x562d32105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.160+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3720871728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320ed7c0 0x562d32105190 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009cd0 tx=0x7fc43c041000 comp rx=0 tx=0).stop 2024-04-26T21:46:38.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.161+0000 7fc44e80b740 1 -- 172.21.15.1:0/3720871728 shutdown_connections 2024-04-26T21:46:38.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.161+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3720871728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320ed7c0 0x562d32105190 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.161+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3720871728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320da630 0x562d320ec5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.161+0000 7fc44e80b740 1 --2- 172.21.15.1:0/3720871728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320d9d10 0x562d320f9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.161+0000 7fc44e80b740 1 -- 172.21.15.1:0/3720871728 >> 172.21.15.1:0/3720871728 conn(0x562d320d62e0 msgr2=0x562d320ebed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:38.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.161+0000 7fc44e80b740 1 -- 172.21.15.1:0/3720871728 shutdown_connections 2024-04-26T21:46:38.163 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.162+0000 7fc44e80b740 1 -- 172.21.15.1:0/3720871728 wait complete. 2024-04-26T21:46:38.164 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.163+0000 7fc44e80b740 1 -- start start 2024-04-26T21:46:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.164+0000 7fc44e80b740 1 Processor -- start 2024-04-26T21:46:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.164+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320f47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.164+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.164+0000 7fc44e80b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320ed7c0 0x562d320f5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.165+0000 7fc44e80b740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441ac3a0 con 0x562d320da630 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.165+0000 7fc44e80b740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441ad7f0 con 0x562d320d9d10 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.165+0000 7fc44e80b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441aca50 con 0x562d320ed7c0 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.165+0000 7fc44e80b740 10 client.0 _ll_register_callbacks cb 0x7fc438003d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.165+0000 7fc449a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320ed7c0 0x562d320f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.165+0000 7fc449000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320f47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.165+0000 7fc449000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320f47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:51884/0 (socket says 172.21.15.1:51884) 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.165+0000 7fc449000640 1 -- 172.21.15.1:0/1929293915 learned_addr learned my addr 172.21.15.1:0/1929293915 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.165+0000 7fc443e00640 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.166+0000 7fc449000640 1 -- 172.21.15.1:0/1929293915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320ed7c0 msgr2=0x562d320f5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.166+0000 7fc449000640 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320ed7c0 0x562d320f5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.166+0000 7fc449000640 1 -- 172.21.15.1:0/1929293915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 msgr2=0x562d320f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.166+0000 7fc449000640 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.166+0000 7fc449000640 1 -- 172.21.15.1:0/1929293915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d31e1d620 con 0x562d320d9d10 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.166+0000 7fc449000640 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320f47a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc438087450 tx=0x7fc43809c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c94dda75874ea571 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.166+0000 7fc42aa00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.166+0000 7fc42aa00640 1 -- 172.21.15.1:0/1929293915 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4380a0df0 con 0x562d320d9d10 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.166+0000 7fc42aa00640 1 -- 172.21.15.1:0/1929293915 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7fc4380ce470 con 0x562d320d9d10 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.166+0000 7fc44e80b740 1 -- 172.21.15.1:0/1929293915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc43c036050 con 0x562d320d9d10 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc42aa00640 1 -- 172.21.15.1:0/1929293915 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7fc4380ce670 con 0x562d320d9d10 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc44e80b740 10 client.26783 fetch_fsmap learned FSMap version 50 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc44e80b740 1 -- 172.21.15.1:0/1929293915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x562d31ff9cb0 con 0x562d320d9d10 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc42aa00640 1 -- 172.21.15.1:0/1929293915 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc438080020 con 0x562d320d9d10 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc42aa00640 1 -- 172.21.15.1:0/1929293915 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 50) ==== 8362+0+0 (secure 0 0 0) 0x7fc4380a1f80 con 0x562d320d9d10 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc44e80b740 10 client.26783 fetch_fsmap finished waiting for FSMap version 50 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc44e80b740 10 client.26783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc44e80b740 20 client.26783 populate_metadata read hostname 'smithi001' 2024-04-26T21:46:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc44e80b740 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562d320dcf90 0x562d320df3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc44e80b740 4 client.26783 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:46:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.167+0000 7fc44e80b740 1 -- 172.21.15.1:0/1929293915 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562d32046e00 con 0x562d320dcf90 2024-04-26T21:46:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.168+0000 7fc443e00640 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562d320dcf90 0x562d320df3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.168+0000 7fc443e00640 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562d320dcf90 0x562d320df3b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7130cd0db10aeac0 server_cookie=674e980d14884cf8 in_seq=0 out_seq=0 2024-04-26T21:46:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.168+0000 7fc42aa00640 10 client.26783 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:38 smithi186 ceph-mon[32991]: pgmap v514: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 580 op/s 2024-04-26T21:46:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:38 smithi001 ceph-mon[28054]: pgmap v514: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 580 op/s 2024-04-26T21:46:38.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:38 smithi038 ceph-mon[32724]: pgmap v514: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 10 MiB/s wr, 580 op/s 2024-04-26T21:46:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.650+0000 7fc42aa00640 1 -- 172.21.15.1:0/1929293915 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562d32046e00 con 0x562d320dcf90 2024-04-26T21:46:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.650+0000 7fc42aa00640 10 client.26783 handle_command_reply: tid=0 2024-04-26T21:46:38.651 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.650+0000 7fc44e80b740 1 client.26783 shutdown 2024-04-26T21:46:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.651+0000 7fc44e80b740 1 -- 172.21.15.1:0/1929293915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 msgr2=0x562d320f47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.651+0000 7fc44e80b740 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320f47a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc438087450 tx=0x7fc43809c4d0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.653+0000 7fc44e80b740 1 -- 172.21.15.1:0/1929293915 shutdown_connections 2024-04-26T21:46:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.653+0000 7fc44e80b740 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562d320dcf90 0x562d320df3b0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.653+0000 7fc44e80b740 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562d320ed7c0 0x562d320f5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.653+0000 7fc44e80b740 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562d320da630 0x562d320f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.653+0000 7fc44e80b740 1 --2- 172.21.15.1:0/1929293915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562d320d9d10 0x562d320f47a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.654 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.653+0000 7fc44e80b740 1 -- 172.21.15.1:0/1929293915 >> 172.21.15.1:0/1929293915 conn(0x562d320d62e0 msgr2=0x562d320dbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:38.655 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.654+0000 7fc42aa00640 0 client.26783 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.654+0000 7fc44e80b740 1 -- 172.21.15.1:0/1929293915 shutdown_connections 2024-04-26T21:46:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.654+0000 7fc44e80b740 1 -- 172.21.15.1:0/1929293915 wait complete. 2024-04-26T21:46:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.654+0000 7fc44e80b740 20 client.26783 trim_cache size 0 max 16384 2024-04-26T21:46:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:46:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:46:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:46:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:46:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 -- 172.21.15.1:0/3289920443 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc42c0a3d90 msgr2=0x7fc42c0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc42c0a3d90 0x7fc42c0a6250 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc43c004af0 tx=0x7fc43c005d80 comp rx=0 tx=0).stop 2024-04-26T21:46:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 -- 172.21.15.1:0/3289920443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc444104b50 msgr2=0x7fc44419b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc444104b50 0x7fc44419b270 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc43800ea60 tx=0x7fc43800ef20 comp rx=0 tx=0).stop 2024-04-26T21:46:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 -- 172.21.15.1:0/3289920443 shutdown_connections 2024-04-26T21:46:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc42c0a3d90 0x7fc42c0a6250 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc444105510 0x7fc44419b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc44410ac10 0x7fc44419fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 --2- 172.21.15.1:0/3289920443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc444104b50 0x7fc44419b270 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 -- 172.21.15.1:0/3289920443 >> 172.21.15.1:0/3289920443 conn(0x7fc444100600 msgr2=0x7fc444101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 -- 172.21.15.1:0/3289920443 shutdown_connections 2024-04-26T21:46:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:38.656+0000 7fc44ba00640 1 -- 172.21.15.1:0/3289920443 wait complete. 2024-04-26T21:46:39.144 INFO:tasks.workunit.client.0.smithi001.stdout: 10 72187 25.69 MB/sec execute 36 sec latency 327.813 ms 2024-04-26T21:46:40.144 INFO:tasks.workunit.client.0.smithi001.stdout: 10 72820 25.92 MB/sec execute 37 sec latency 372.062 ms 2024-04-26T21:46:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:40 smithi186 ceph-mon[32991]: pgmap v515: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.2 MiB/s wr, 514 op/s 2024-04-26T21:46:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:40 smithi001 ceph-mon[28054]: pgmap v515: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.2 MiB/s wr, 514 op/s 2024-04-26T21:46:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:40 smithi038 ceph-mon[32724]: pgmap v515: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.2 MiB/s wr, 514 op/s 2024-04-26T21:46:41.144 INFO:tasks.workunit.client.0.smithi001.stdout: 10 73536 26.29 MB/sec execute 38 sec latency 66.825 ms 2024-04-26T21:46:42.144 INFO:tasks.workunit.client.0.smithi001.stdout: 10 74095 26.54 MB/sec execute 39 sec latency 53.966 ms 2024-04-26T21:46:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:42 smithi186 ceph-mon[32991]: pgmap v516: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 781 op/s 2024-04-26T21:46:42.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:42 smithi001 ceph-mon[28054]: pgmap v516: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 781 op/s 2024-04-26T21:46:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:42 smithi038 ceph-mon[32724]: pgmap v516: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 781 op/s 2024-04-26T21:46:43.144 INFO:tasks.workunit.client.0.smithi001.stdout: 10 74669 26.63 MB/sec execute 40 sec latency 73.301 ms 2024-04-26T21:46:43.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:43 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:43.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:43 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:43.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:43 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:44.145 INFO:tasks.workunit.client.0.smithi001.stdout: 10 75075 26.49 MB/sec execute 41 sec latency 274.644 ms 2024-04-26T21:46:44.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:44 smithi001 ceph-mon[28054]: pgmap v517: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.4 MiB/s wr, 493 op/s 2024-04-26T21:46:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:44 smithi038 ceph-mon[32724]: pgmap v517: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.4 MiB/s wr, 493 op/s 2024-04-26T21:46:44.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:44 smithi186 ceph-mon[32991]: pgmap v517: 129 pgs: 129 active+clean; 1.0 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.4 MiB/s wr, 493 op/s 2024-04-26T21:46:45.145 INFO:tasks.workunit.client.0.smithi001.stdout: 10 75316 26.10 MB/sec execute 42 sec latency 665.747 ms 2024-04-26T21:46:46.145 INFO:tasks.workunit.client.0.smithi001.stdout: 10 75754 26.05 MB/sec execute 43 sec latency 69.960 ms 2024-04-26T21:46:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:46 smithi001 ceph-mon[28054]: pgmap v518: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.0 MiB/s wr, 660 op/s 2024-04-26T21:46:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:46 smithi038 ceph-mon[32724]: pgmap v518: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.0 MiB/s wr, 660 op/s 2024-04-26T21:46:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:46 smithi186 ceph-mon[32991]: pgmap v518: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.0 MiB/s wr, 660 op/s 2024-04-26T21:46:47.145 INFO:tasks.workunit.client.0.smithi001.stdout: 10 76473 26.22 MB/sec execute 44 sec latency 84.938 ms 2024-04-26T21:46:48.145 INFO:tasks.workunit.client.0.smithi001.stdout: 10 77067 26.45 MB/sec execute 45 sec latency 51.054 ms 2024-04-26T21:46:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:48 smithi001 ceph-mon[28054]: pgmap v519: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 7.9 MiB/s wr, 500 op/s 2024-04-26T21:46:48.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:48 smithi038 ceph-mon[32724]: pgmap v519: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 7.9 MiB/s wr, 500 op/s 2024-04-26T21:46:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:48 smithi186 ceph-mon[32991]: pgmap v519: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 7.9 MiB/s wr, 500 op/s 2024-04-26T21:46:49.147 INFO:tasks.workunit.client.0.smithi001.stdout: 10 77719 26.72 MB/sec execute 46 sec latency 72.479 ms 2024-04-26T21:46:50.147 INFO:tasks.workunit.client.0.smithi001.stdout: 10 77889 26.38 MB/sec execute 47 sec latency 519.042 ms 2024-04-26T21:46:50.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:50 smithi001 ceph-mon[28054]: pgmap v520: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.7 MiB/s wr, 475 op/s 2024-04-26T21:46:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:50 smithi038 ceph-mon[32724]: pgmap v520: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.7 MiB/s wr, 475 op/s 2024-04-26T21:46:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:50 smithi186 ceph-mon[32991]: pgmap v520: 129 pgs: 129 active+clean; 1.1 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.7 MiB/s wr, 475 op/s 2024-04-26T21:46:51.148 INFO:tasks.workunit.client.0.smithi001.stdout: 10 78111 26.02 MB/sec execute 48 sec latency 363.101 ms 2024-04-26T21:46:51.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:51 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:51.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:51 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:51.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:51 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:46:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.883+0000 7f5294a00640 1 -- 172.21.15.1:0/297604318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5290100dc0 msgr2=0x7f52901011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.883+0000 7f5294a00640 1 --2- 172.21.15.1:0/297604318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5290100dc0 0x7f52901011a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5280009900 tx=0x7f528002a490 comp rx=0 tx=0).stop 2024-04-26T21:46:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.884+0000 7f5294a00640 1 -- 172.21.15.1:0/297604318 shutdown_connections 2024-04-26T21:46:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.884+0000 7f5294a00640 1 --2- 172.21.15.1:0/297604318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f529010f300 0x7f52901116f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.885 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.884+0000 7f5294a00640 1 --2- 172.21.15.1:0/297604318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f52901016e0 0x7f529010edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.884+0000 7f5294a00640 1 --2- 172.21.15.1:0/297604318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5290100dc0 0x7f52901011a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.884+0000 7f5294a00640 1 -- 172.21.15.1:0/297604318 >> 172.21.15.1:0/297604318 conn(0x7f5290076740 msgr2=0x7f5290076b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:51.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.885+0000 7f5294a00640 1 -- 172.21.15.1:0/297604318 shutdown_connections 2024-04-26T21:46:51.889 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.888+0000 7f5294a00640 1 -- 172.21.15.1:0/297604318 wait complete. 2024-04-26T21:46:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.889+0000 7f5294a00640 1 Processor -- start 2024-04-26T21:46:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.889+0000 7f5294a00640 1 -- start start 2024-04-26T21:46:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.891+0000 7f5294a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290100dc0 0x7f52900740d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.891+0000 7f5294a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52901016e0 0x7f5290074610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.891+0000 7f5294a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529010f300 0x7f529006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.891+0000 7f5294a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5290114de0 con 0x7f5290100dc0 2024-04-26T21:46:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.891+0000 7f5294a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5290114c60 con 0x7f52901016e0 2024-04-26T21:46:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.891+0000 7f5294a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5290114f60 con 0x7f529010f300 2024-04-26T21:46:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.891+0000 7f528d400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52901016e0 0x7f5290074610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.891+0000 7f528d400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52901016e0 0x7f5290074610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42668/0 (socket says 172.21.15.1:42668) 2024-04-26T21:46:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.891+0000 7f528de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290100dc0 0x7f52900740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.891+0000 7f528d400640 1 -- 172.21.15.1:0/2170136629 learned_addr learned my addr 172.21.15.1:0/2170136629 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.892+0000 7f528e800640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529010f300 0x7f529006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.892+0000 7f528d400640 1 -- 172.21.15.1:0/2170136629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529010f300 msgr2=0x7f529006d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.892+0000 7f528d400640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529010f300 0x7f529006d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.892+0000 7f528d400640 1 -- 172.21.15.1:0/2170136629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290100dc0 msgr2=0x7f52900740d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:51.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.892+0000 7f528d400640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290100dc0 0x7f52900740d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.892+0000 7f528d400640 1 -- 172.21.15.1:0/2170136629 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f529006d8d0 con 0x7f52901016e0 2024-04-26T21:46:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.892+0000 7f528de00640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290100dc0 0x7f52900740d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:46:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.893+0000 7f528d400640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52901016e0 0x7f5290074610 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f527c01d980 tx=0x7f527c01de40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.894+0000 7f5286a00640 1 -- 172.21.15.1:0/2170136629 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c025070 con 0x7f52901016e0 2024-04-26T21:46:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.894+0000 7f5286a00640 1 -- 172.21.15.1:0/2170136629 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527c004440 con 0x7f52901016e0 2024-04-26T21:46:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.894+0000 7f5286a00640 1 -- 172.21.15.1:0/2170136629 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c020460 con 0x7f52901016e0 2024-04-26T21:46:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.894+0000 7f5294a00640 1 -- 172.21.15.1:0/2170136629 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f529006db60 con 0x7f52901016e0 2024-04-26T21:46:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.894+0000 7f5294a00640 1 -- 172.21.15.1:0/2170136629 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5290071bf0 con 0x7f52901016e0 2024-04-26T21:46:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.902+0000 7f52977cf740 1 Processor -- start 2024-04-26T21:46:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.904+0000 7f52977cf740 1 -- start start 2024-04-26T21:46:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.905+0000 7f5286a00640 1 -- 172.21.15.1:0/2170136629 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f527c028070 con 0x7f52901016e0 2024-04-26T21:46:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.906+0000 7f52977cf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x5643583541b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.906+0000 7f52977cf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435835ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.906+0000 7f52977cf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56435835efb0 0x5643583613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.906+0000 7f52977cf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5643582b9380 con 0x56435835efb0 2024-04-26T21:46:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.906+0000 7f52977cf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564358255c30 con 0x564358353dd0 2024-04-26T21:46:51.907 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.906+0000 7f52977cf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5643582b5340 con 0x5643583546f0 2024-04-26T21:46:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.906+0000 7f528de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435835ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.906+0000 7f528de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435835ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:36674/0 (socket says 172.21.15.1:36674) 2024-04-26T21:46:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.906+0000 7f528de00640 1 -- 172.21.15.1:0/3101933442 learned_addr learned my addr 172.21.15.1:0/3101933442 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.906+0000 7f528e800640 1 --2- 172.21.15.1:0/3101933442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x5643583541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.907+0000 7f528de00640 1 -- 172.21.15.1:0/3101933442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 msgr2=0x5643583541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.907+0000 7f528de00640 1 --2- 172.21.15.1:0/3101933442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x5643583541b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.907+0000 7f528de00640 1 -- 172.21.15.1:0/3101933442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56435835efb0 msgr2=0x5643583613a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:46:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.907+0000 7f528de00640 1 --2- 172.21.15.1:0/3101933442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56435835efb0 0x5643583613a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.907+0000 7f528de00640 1 -- 172.21.15.1:0/3101933442 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564358277940 con 0x5643583546f0 2024-04-26T21:46:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.908+0000 7f528de00640 1 --2- 172.21.15.1:0/3101933442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435835ea70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5280009230 tx=0x7f5280004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2b6bc5e6b3a3772 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.909+0000 7f5286a00640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f52740a3ee0 0x7f52740a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.909+0000 7f527b000640 1 -- 172.21.15.1:0/3101933442 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280035040 con 0x5643583546f0 2024-04-26T21:46:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.910+0000 7f527b000640 1 -- 172.21.15.1:0/3101933442 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5280005500 con 0x5643583546f0 2024-04-26T21:46:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.910+0000 7f527b000640 1 -- 172.21.15.1:0/3101933442 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280003be0 con 0x5643583546f0 2024-04-26T21:46:51.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.911+0000 7f52977cf740 1 -- 172.21.15.1:0/3101933442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 msgr2=0x56435835ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:51.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.911+0000 7f52977cf740 1 --2- 172.21.15.1:0/3101933442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435835ea70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5280009230 tx=0x7f5280004610 comp rx=0 tx=0).stop 2024-04-26T21:46:51.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.911+0000 7f5286a00640 1 -- 172.21.15.1:0/2170136629 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7f527c0e1790 con 0x7f52901016e0 2024-04-26T21:46:51.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.911+0000 7f528e800640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f52740a3ee0 0x7f52740a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.914+0000 7f52977cf740 1 -- 172.21.15.1:0/3101933442 shutdown_connections 2024-04-26T21:46:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.914+0000 7f52977cf740 1 --2- 172.21.15.1:0/3101933442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435835ea70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.914+0000 7f52977cf740 1 --2- 172.21.15.1:0/3101933442 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56435835efb0 0x5643583613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.914+0000 7f52977cf740 1 --2- 172.21.15.1:0/3101933442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x5643583541b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.914+0000 7f52977cf740 1 -- 172.21.15.1:0/3101933442 >> 172.21.15.1:0/3101933442 conn(0x5643583502e0 msgr2=0x5643583506c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.915+0000 7f528e800640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f52740a3ee0 0x7f52740a63a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f52880092f0 tx=0x7f5288009280 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.915+0000 7f52977cf740 1 -- 172.21.15.1:0/3101933442 shutdown_connections 2024-04-26T21:46:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.915+0000 7f52977cf740 1 -- 172.21.15.1:0/3101933442 wait complete. 2024-04-26T21:46:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.916+0000 7f52977cf740 1 -- start start 2024-04-26T21:46:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.916+0000 7f52977cf740 1 Processor -- start 2024-04-26T21:46:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.921+0000 7f52977cf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x56435835bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.921+0000 7f52977cf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5643583546f0 0x56435835c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.921+0000 7f52977cf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x56435835ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.921+0000 7f528de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x56435835bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.921+0000 7f528de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x56435835bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:42684/0 (socket says 172.21.15.1:42684) 2024-04-26T21:46:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.922+0000 7f52977cf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564358361b80 con 0x5643583546f0 2024-04-26T21:46:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.922+0000 7f52977cf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564358240c60 con 0x564358353dd0 2024-04-26T21:46:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.922+0000 7f52977cf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564358361d00 con 0x56435835efb0 2024-04-26T21:46:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.922+0000 7f52977cf740 10 client.0 _ll_register_callbacks cb 0x56435826ce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:46:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.922+0000 7f528de00640 1 -- 172.21.15.1:0/3346423503 learned_addr learned my addr 172.21.15.1:0/3346423503 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.922+0000 7f528e800640 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x56435835ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.923+0000 7f528e800640 1 -- 172.21.15.1:0/3346423503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 msgr2=0x56435835bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.923+0000 7f528e800640 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x56435835bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.923+0000 7f528e800640 1 -- 172.21.15.1:0/3346423503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5643583546f0 msgr2=0x56435835c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.923+0000 7f528e800640 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5643583546f0 0x56435835c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.923+0000 7f528e800640 1 -- 172.21.15.1:0/3346423503 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56435824f510 con 0x56435835efb0 2024-04-26T21:46:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.923+0000 7f528e800640 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x56435835ca60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f52880127d0 tx=0x7f5288004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74f2630856bfba29 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.924+0000 7f526f400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:46:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.924+0000 7f526f400640 1 -- 172.21.15.1:0/3346423503 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528800ead0 con 0x56435835efb0 2024-04-26T21:46:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.924+0000 7f526f400640 1 -- 172.21.15.1:0/3346423503 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7f528800ec70 con 0x56435835efb0 2024-04-26T21:46:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.924+0000 7f52977cf740 1 -- 172.21.15.1:0/3346423503 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5280005500 con 0x56435835efb0 2024-04-26T21:46:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.924+0000 7f526f400640 1 -- 172.21.15.1:0/3346423503 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f5288030050 con 0x56435835efb0 2024-04-26T21:46:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.924+0000 7f52977cf740 10 client.26779 fetch_fsmap learned FSMap version 52 2024-04-26T21:46:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.924+0000 7f52977cf740 1 -- 172.21.15.1:0/3346423503 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x564357fcb210 con 0x56435835efb0 2024-04-26T21:46:51.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.925+0000 7f526f400640 1 -- 172.21.15.1:0/3346423503 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5280005500 con 0x56435835efb0 2024-04-26T21:46:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.926+0000 7f526f400640 1 -- 172.21.15.1:0/3346423503 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 52) ==== 8354+0+0 (secure 0 0 0) 0x7f528800f040 con 0x56435835efb0 2024-04-26T21:46:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.926+0000 7f52977cf740 10 client.26779 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T21:46:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.926+0000 7f52977cf740 10 client.26779 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:46:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.926+0000 7f52977cf740 20 client.26779 populate_metadata read hostname 'smithi001' 2024-04-26T21:46:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.926+0000 7f52977cf740 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56435836eda0 0x5643583711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.926+0000 7f52977cf740 4 client.26779 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:46:51.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.926+0000 7f52977cf740 1 -- 172.21.15.1:0/3346423503 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643582c6f90 con 0x56435836eda0 2024-04-26T21:46:51.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.926+0000 7f528de00640 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56435836eda0 0x5643583711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.927+0000 7f528de00640 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56435836eda0 0x5643583711b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75a82aa810720d7e server_cookie=f0ecff066b05d4a5 in_seq=0 out_seq=0 2024-04-26T21:46:51.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.927+0000 7f526f400640 10 client.26779 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.932+0000 7f526f400640 1 -- 172.21.15.1:0/3346423503 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5643582c6f90 con 0x56435836eda0 2024-04-26T21:46:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.932+0000 7f526f400640 10 client.26779 handle_command_reply: tid=0 2024-04-26T21:46:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.932+0000 7f52977cf740 1 client.26779 shutdown 2024-04-26T21:46:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.934+0000 7f52977cf740 1 -- 172.21.15.1:0/3346423503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 msgr2=0x56435835ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.934+0000 7f52977cf740 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x56435835ca60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f52880127d0 tx=0x7f5288004a40 comp rx=0 tx=0).stop 2024-04-26T21:46:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.935+0000 7f52977cf740 1 -- 172.21.15.1:0/3346423503 shutdown_connections 2024-04-26T21:46:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.935+0000 7f52977cf740 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56435836eda0 0x5643583711b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.935+0000 7f52977cf740 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x56435835ca60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.935+0000 7f526f400640 0 client.26779 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.935+0000 7f52977cf740 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5643583546f0 0x56435835c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.935+0000 7f52977cf740 1 --2- 172.21.15.1:0/3346423503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x56435835bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.935+0000 7f52977cf740 1 -- 172.21.15.1:0/3346423503 >> 172.21.15.1:0/3346423503 conn(0x5643583502e0 msgr2=0x564358356010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.936+0000 7f52977cf740 1 -- 172.21.15.1:0/3346423503 shutdown_connections 2024-04-26T21:46:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.936+0000 7f52977cf740 1 -- 172.21.15.1:0/3346423503 wait complete. 2024-04-26T21:46:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.936+0000 7f52977cf740 20 client.26779 trim_cache size 0 max 16384 2024-04-26T21:46:51.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.968+0000 7f52977cf740 1 Processor -- start 2024-04-26T21:46:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.968+0000 7f52977cf740 1 -- start start 2024-04-26T21:46:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.969+0000 7f52977cf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x56435836aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.969+0000 7f52977cf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435834fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.969+0000 7f52977cf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56435835efb0 0x564358355e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.969+0000 7f52977cf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f52901b8d50 con 0x56435835efb0 2024-04-26T21:46:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.969+0000 7f52977cf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52901b6620 con 0x564358353dd0 2024-04-26T21:46:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.969+0000 7f52977cf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52901b2480 con 0x5643583546f0 2024-04-26T21:46:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.969+0000 7f528d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435834fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.970+0000 7f528d400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435834fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:36694/0 (socket says 172.21.15.1:36694) 2024-04-26T21:46:51.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.970+0000 7f528d400640 1 -- 172.21.15.1:0/205431697 learned_addr learned my addr 172.21.15.1:0/205431697 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.970+0000 7f528e800640 1 --2- 172.21.15.1:0/205431697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56435835efb0 0x564358355e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.970+0000 7f528de00640 1 --2- 172.21.15.1:0/205431697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x56435836aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.970+0000 7f528d400640 1 -- 172.21.15.1:0/205431697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 msgr2=0x56435836aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.970+0000 7f528d400640 1 --2- 172.21.15.1:0/205431697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x56435836aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.970+0000 7f528d400640 1 -- 172.21.15.1:0/205431697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56435835efb0 msgr2=0x564358355e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.970+0000 7f528d400640 1 --2- 172.21.15.1:0/205431697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56435835efb0 0x564358355e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.970+0000 7f528d400640 1 -- 172.21.15.1:0/205431697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564358097620 con 0x5643583546f0 2024-04-26T21:46:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.970+0000 7f528d400640 1 --2- 172.21.15.1:0/205431697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435834fbc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f527c004630 tx=0x7f527c0aceb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c832bec9d7d529 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.972+0000 7f527a600640 1 -- 172.21.15.1:0/205431697 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c0c6070 con 0x5643583546f0 2024-04-26T21:46:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.972+0000 7f527a600640 1 -- 172.21.15.1:0/205431697 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527c0b1620 con 0x5643583546f0 2024-04-26T21:46:51.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.972+0000 7f527a600640 1 -- 172.21.15.1:0/205431697 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f527c0e0040 con 0x5643583546f0 2024-04-26T21:46:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.973+0000 7f52977cf740 1 -- 172.21.15.1:0/205431697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 msgr2=0x56435834fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.973+0000 7f52977cf740 1 --2- 172.21.15.1:0/205431697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435834fbc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f527c004630 tx=0x7f527c0aceb0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.973+0000 7f52977cf740 1 -- 172.21.15.1:0/205431697 shutdown_connections 2024-04-26T21:46:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.973+0000 7f52977cf740 1 --2- 172.21.15.1:0/205431697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643583546f0 0x56435834fbc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.973+0000 7f52977cf740 1 --2- 172.21.15.1:0/205431697 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56435835efb0 0x564358355e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.973+0000 7f52977cf740 1 --2- 172.21.15.1:0/205431697 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564358353dd0 0x56435836aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.973+0000 7f52977cf740 1 -- 172.21.15.1:0/205431697 >> 172.21.15.1:0/205431697 conn(0x5643583502e0 msgr2=0x564358355780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:51.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.973+0000 7f52977cf740 1 -- 172.21.15.1:0/205431697 shutdown_connections 2024-04-26T21:46:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.973+0000 7f52977cf740 1 -- 172.21.15.1:0/205431697 wait complete. 2024-04-26T21:46:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.973+0000 7f52977cf740 1 -- start start 2024-04-26T21:46:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.974+0000 7f52977cf740 1 Processor -- start 2024-04-26T21:46:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.975+0000 7f52977cf740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564358353dd0 0x564358362ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.975+0000 7f52977cf740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5643583546f0 0x564358363530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.975+0000 7f52977cf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x5643583d4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.975+0000 7f52977cf740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f52901b2ad0 con 0x564358353dd0 2024-04-26T21:46:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.975+0000 7f52977cf740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f52901b5f50 con 0x5643583546f0 2024-04-26T21:46:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.975+0000 7f52977cf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52901b31a0 con 0x56435835efb0 2024-04-26T21:46:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.975+0000 7f52977cf740 10 client.0 _ll_register_callbacks cb 0x7f5288012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:46:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.975+0000 7f528de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564358353dd0 0x564358362ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.975+0000 7f528e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x5643583d4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.975+0000 7f528e800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x5643583d4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:36708/0 (socket says 172.21.15.1:36708) 2024-04-26T21:46:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.976+0000 7f528e800640 1 -- 172.21.15.1:0/1766801018 learned_addr learned my addr 172.21.15.1:0/1766801018 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:46:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.976+0000 7f528e800640 1 -- 172.21.15.1:0/1766801018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5643583546f0 msgr2=0x564358363530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:46:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.976+0000 7f528e800640 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5643583546f0 0x564358363530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.976+0000 7f528e800640 1 -- 172.21.15.1:0/1766801018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564358353dd0 msgr2=0x564358362ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:46:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.976+0000 7f528e800640 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564358353dd0 0x564358362ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.976+0000 7f528e800640 1 -- 172.21.15.1:0/1766801018 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564358273cb0 con 0x56435835efb0 2024-04-26T21:46:51.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.976+0000 7f528e800640 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x5643583d4330 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5288005be0 tx=0x7f52880279a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25920781960e19c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:46:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.976+0000 7f526f400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:46:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.977+0000 7f526f400640 1 -- 172.21.15.1:0/1766801018 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528802d040 con 0x56435835efb0 2024-04-26T21:46:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.977+0000 7f526f400640 1 -- 172.21.15.1:0/1766801018 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7f528800e720 con 0x56435835efb0 2024-04-26T21:46:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.977+0000 7f52977cf740 1 -- 172.21.15.1:0/1766801018 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f527c0b1620 con 0x56435835efb0 2024-04-26T21:46:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.978+0000 7f526f400640 1 -- 172.21.15.1:0/1766801018 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f528800e920 con 0x56435835efb0 2024-04-26T21:46:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.978+0000 7f52977cf740 10 client.26785 fetch_fsmap learned FSMap version 52 2024-04-26T21:46:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.978+0000 7f52977cf740 1 -- 172.21.15.1:0/1766801018 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x5643583d5910 con 0x56435835efb0 2024-04-26T21:46:51.980 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.979+0000 7f526f400640 1 -- 172.21.15.1:0/1766801018 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f527c0b1620 con 0x56435835efb0 2024-04-26T21:46:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.979+0000 7f526f400640 1 -- 172.21.15.1:0/1766801018 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 52) ==== 8354+0+0 (secure 0 0 0) 0x7f528803d990 con 0x56435835efb0 2024-04-26T21:46:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.979+0000 7f52977cf740 10 client.26785 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T21:46:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.979+0000 7f52977cf740 10 client.26785 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:46:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.979+0000 7f52977cf740 20 client.26785 populate_metadata read hostname 'smithi001' 2024-04-26T21:46:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.979+0000 7f52977cf740 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56435836eda0 0x5643583d8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:46:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.979+0000 7f52977cf740 4 client.26785 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:46:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.979+0000 7f52977cf740 1 -- 172.21.15.1:0/1766801018 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5643582c0e00 con 0x56435836eda0 2024-04-26T21:46:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.980+0000 7f528de00640 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56435836eda0 0x5643583d8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:46:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.982+0000 7f528de00640 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56435836eda0 0x5643583d8070 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7be585a643f9982 server_cookie=7bdfb89e4a2640a0 in_seq=0 out_seq=0 2024-04-26T21:46:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:51.982+0000 7f526f400640 10 client.26785 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:52.149 INFO:tasks.workunit.client.0.smithi001.stdout: 10 78379 25.80 MB/sec execute 49 sec latency 152.100 ms 2024-04-26T21:46:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.293+0000 7f526f400640 1 -- 172.21.15.1:0/1766801018 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5643582c0e00 con 0x56435836eda0 2024-04-26T21:46:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.293+0000 7f526f400640 10 client.26785 handle_command_reply: tid=0 2024-04-26T21:46:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.293+0000 7f52977cf740 1 client.26785 shutdown 2024-04-26T21:46:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.294+0000 7f52977cf740 1 -- 172.21.15.1:0/1766801018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 msgr2=0x5643583d4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.294+0000 7f52977cf740 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x5643583d4330 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5288005be0 tx=0x7f52880279a0 comp rx=0 tx=0).stop 2024-04-26T21:46:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.294+0000 7f52977cf740 1 -- 172.21.15.1:0/1766801018 shutdown_connections 2024-04-26T21:46:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.294+0000 7f52977cf740 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56435836eda0 0x5643583d8070 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.294+0000 7f52977cf740 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56435835efb0 0x5643583d4330 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.294+0000 7f52977cf740 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5643583546f0 0x564358363530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.294+0000 7f52977cf740 1 --2- 172.21.15.1:0/1766801018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564358353dd0 0x564358362ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.294+0000 7f52977cf740 1 -- 172.21.15.1:0/1766801018 >> 172.21.15.1:0/1766801018 conn(0x5643583502e0 msgr2=0x564358356010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.295+0000 7f526f400640 0 client.26785 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:46:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.295+0000 7f52977cf740 1 -- 172.21.15.1:0/1766801018 shutdown_connections 2024-04-26T21:46:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.296+0000 7f52977cf740 1 -- 172.21.15.1:0/1766801018 wait complete. 2024-04-26T21:46:52.297 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.296+0000 7f52977cf740 20 client.26785 trim_cache size 0 max 16384 2024-04-26T21:46:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:46:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:46:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:46:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:46:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.297+0000 7f5294a00640 1 -- 172.21.15.1:0/2170136629 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f52740a3ee0 msgr2=0x7f52740a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:52.298 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.297+0000 7f5294a00640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f52740a3ee0 0x7f52740a63a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f52880092f0 tx=0x7f5288009280 comp rx=0 tx=0).stop 2024-04-26T21:46:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.297+0000 7f5294a00640 1 -- 172.21.15.1:0/2170136629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52901016e0 msgr2=0x7f5290074610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:46:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.297+0000 7f5294a00640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52901016e0 0x7f5290074610 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f527c01d980 tx=0x7f527c01de40 comp rx=0 tx=0).stop 2024-04-26T21:46:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.298+0000 7f5294a00640 1 -- 172.21.15.1:0/2170136629 shutdown_connections 2024-04-26T21:46:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.298+0000 7f5294a00640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f52740a3ee0 0x7f52740a63a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.298+0000 7f5294a00640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529010f300 0x7f529006d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.298+0000 7f5294a00640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52901016e0 0x7f5290074610 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.298+0000 7f5294a00640 1 --2- 172.21.15.1:0/2170136629 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5290100dc0 0x7f52900740d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:46:52.299 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.298+0000 7f5294a00640 1 -- 172.21.15.1:0/2170136629 >> 172.21.15.1:0/2170136629 conn(0x7f5290076740 msgr2=0x7f529010f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:46:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.301+0000 7f5294a00640 1 -- 172.21.15.1:0/2170136629 shutdown_connections 2024-04-26T21:46:52.302 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:46:52.301+0000 7f5294a00640 1 -- 172.21.15.1:0/2170136629 wait complete. 2024-04-26T21:46:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:52 smithi001 ceph-mon[28054]: pgmap v521: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 737 op/s 2024-04-26T21:46:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:52 smithi038 ceph-mon[32724]: pgmap v521: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 737 op/s 2024-04-26T21:46:52.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:52 smithi186 ceph-mon[32991]: pgmap v521: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 737 op/s 2024-04-26T21:46:53.148 INFO:tasks.workunit.client.0.smithi001.stdout: 10 78819 25.58 MB/sec execute 50 sec latency 194.752 ms 2024-04-26T21:46:54.149 INFO:tasks.workunit.client.0.smithi001.stdout: 10 79304 25.72 MB/sec execute 51 sec latency 229.402 ms 2024-04-26T21:46:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:54 smithi001 ceph-mon[28054]: pgmap v522: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 11 MiB/s wr, 470 op/s 2024-04-26T21:46:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:54 smithi038 ceph-mon[32724]: pgmap v522: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 11 MiB/s wr, 470 op/s 2024-04-26T21:46:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:54 smithi186 ceph-mon[32991]: pgmap v522: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 11 MiB/s wr, 470 op/s 2024-04-26T21:46:55.148 INFO:tasks.workunit.client.0.smithi001.stdout: 10 79996 25.82 MB/sec execute 52 sec latency 232.408 ms 2024-04-26T21:46:56.149 INFO:tasks.workunit.client.0.smithi001.stdout: 10 80675 26.13 MB/sec execute 53 sec latency 46.919 ms 2024-04-26T21:46:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:56 smithi001 ceph-mon[28054]: pgmap v523: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 14 MiB/s wr, 694 op/s 2024-04-26T21:46:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:56 smithi038 ceph-mon[32724]: pgmap v523: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 14 MiB/s wr, 694 op/s 2024-04-26T21:46:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:56 smithi186 ceph-mon[32991]: pgmap v523: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 14 MiB/s wr, 694 op/s 2024-04-26T21:46:57.149 INFO:tasks.workunit.client.0.smithi001.stdout: 10 81272 26.25 MB/sec execute 54 sec latency 51.363 ms 2024-04-26T21:46:58.149 INFO:tasks.workunit.client.0.smithi001.stdout: 10 81766 26.33 MB/sec execute 55 sec latency 67.346 ms 2024-04-26T21:46:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:58 smithi001 ceph-mon[28054]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.6 MiB/s wr, 598 op/s 2024-04-26T21:46:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:46:58 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:46:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:58 smithi038 ceph-mon[32724]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.6 MiB/s wr, 598 op/s 2024-04-26T21:46:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:46:58 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:46:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:58 smithi186 ceph-mon[32991]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.6 MiB/s wr, 598 op/s 2024-04-26T21:46:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:46:58 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:46:59.149 INFO:tasks.workunit.client.0.smithi001.stdout: 10 82275 26.27 MB/sec execute 56 sec latency 138.690 ms 2024-04-26T21:47:00.149 INFO:tasks.workunit.client.0.smithi001.stdout: 10 82558 26.06 MB/sec execute 57 sec latency 394.076 ms 2024-04-26T21:47:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:00 smithi001 ceph-mon[28054]: pgmap v525: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 8.6 MiB/s wr, 557 op/s 2024-04-26T21:47:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:00 smithi038 ceph-mon[32724]: pgmap v525: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 8.6 MiB/s wr, 557 op/s 2024-04-26T21:47:00.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:00 smithi186 ceph-mon[32991]: pgmap v525: 129 pgs: 129 active+clean; 1.1 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 8.6 MiB/s wr, 557 op/s 2024-04-26T21:47:01.149 INFO:tasks.workunit.client.0.smithi001.stdout: 10 82851 25.89 MB/sec execute 58 sec latency 559.494 ms 2024-04-26T21:47:02.149 INFO:tasks.workunit.client.0.smithi001.stdout: 10 82851 25.45 MB/sec execute 59 sec latency 1559.628 ms 2024-04-26T21:47:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:02 smithi001 ceph-mon[28054]: pgmap v526: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 711 op/s 2024-04-26T21:47:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:02 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:47:02.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:02 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:47:02.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:02 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:47:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:02 smithi038 ceph-mon[32724]: pgmap v526: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 711 op/s 2024-04-26T21:47:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:02 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:47:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:02 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:47:02.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:02 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:47:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:02 smithi186 ceph-mon[32991]: pgmap v526: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 11 MiB/s wr, 711 op/s 2024-04-26T21:47:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:02 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:47:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:02 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:47:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:02 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:47:03.149 INFO:tasks.workunit.client.0.smithi001.stdout: 10 82851 25.02 MB/sec execute 60 sec latency 2559.801 ms 2024-04-26T21:47:04.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 82851 24.61 MB/sec execute 61 sec latency 3559.959 ms 2024-04-26T21:47:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:04 smithi001 ceph-mon[28054]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.6 MiB/s wr, 449 op/s 2024-04-26T21:47:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:04 smithi038 ceph-mon[32724]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.6 MiB/s wr, 449 op/s 2024-04-26T21:47:04.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:04 smithi186 ceph-mon[32991]: pgmap v527: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.6 MiB/s wr, 449 op/s 2024-04-26T21:47:05.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 83183 24.46 MB/sec execute 62 sec latency 3950.312 ms 2024-04-26T21:47:05.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.528+0000 7f15ea600640 1 -- 172.21.15.1:0/1612043825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e41137d0 msgr2=0x7f15e4115bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:05.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.528+0000 7f15ea600640 1 --2- 172.21.15.1:0/1612043825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e41137d0 0x7f15e4115bc0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009cd0 tx=0x7f15dc02a8e0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.528+0000 7f15ea600640 1 -- 172.21.15.1:0/1612043825 shutdown_connections 2024-04-26T21:47:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.528+0000 7f15ea600640 1 --2- 172.21.15.1:0/1612043825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e41137d0 0x7f15e4115bc0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.528+0000 7f15ea600640 1 --2- 172.21.15.1:0/1612043825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15e4077780 0x7f15e4113290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.528+0000 7f15ea600640 1 --2- 172.21.15.1:0/1612043825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e4076e60 0x7f15e4077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.528+0000 7f15ea600640 1 -- 172.21.15.1:0/1612043825 >> 172.21.15.1:0/1612043825 conn(0x7f15e4100620 msgr2=0x7f15e4102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:05.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.529+0000 7f15ea600640 1 -- 172.21.15.1:0/1612043825 shutdown_connections 2024-04-26T21:47:05.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.529+0000 7f15ea600640 1 -- 172.21.15.1:0/1612043825 wait complete. 2024-04-26T21:47:05.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.530+0000 7f15ea600640 1 Processor -- start 2024-04-26T21:47:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.531+0000 7f15ea600640 1 -- start start 2024-04-26T21:47:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.531+0000 7f15ea600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e4076e60 0x7f15e419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.531+0000 7f15ea600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15e4077780 0x7f15e419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.531+0000 7f15ea600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e41137d0 0x7f15e41a4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.531+0000 7f15ea600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41191a0 con 0x7f15e41137d0 2024-04-26T21:47:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.531+0000 7f15ea600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15e4119020 con 0x7f15e4077780 2024-04-26T21:47:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.531+0000 7f15ea600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15e4119320 con 0x7f15e4076e60 2024-04-26T21:47:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.532+0000 7f15e3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e4076e60 0x7f15e419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.532+0000 7f15e3e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e41137d0 0x7f15e41a4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.532+0000 7f15e3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e4076e60 0x7f15e419f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54484/0 (socket says 172.21.15.1:54484) 2024-04-26T21:47:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.532+0000 7f15e3400640 1 -- 172.21.15.1:0/3809324712 learned_addr learned my addr 172.21.15.1:0/3809324712 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.533+0000 7f15e3e00640 1 -- 172.21.15.1:0/3809324712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e4076e60 msgr2=0x7f15e419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.533+0000 7f15e3e00640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e4076e60 0x7f15e419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.533+0000 7f15e3e00640 1 -- 172.21.15.1:0/3809324712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15e4077780 msgr2=0x7f15e419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.532+0000 7f15e2a00640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15e4077780 0x7f15e419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.534+0000 7f15e3e00640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15e4077780 0x7f15e419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.534+0000 7f15e3e00640 1 -- 172.21.15.1:0/3809324712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e41a4710 con 0x7f15e41137d0 2024-04-26T21:47:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.534+0000 7f15e2a00640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15e4077780 0x7f15e419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:47:05.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.535+0000 7f15e3e00640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e41137d0 0x7f15e41a4010 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f15dc02ae00 tx=0x7f15dc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.537+0000 7f15dbe00640 1 -- 172.21.15.1:0/3809324712 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc004240 con 0x7f15e41137d0 2024-04-26T21:47:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.537+0000 7f15dbe00640 1 -- 172.21.15.1:0/3809324712 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15dc0305a0 con 0x7f15e41137d0 2024-04-26T21:47:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.537+0000 7f15dbe00640 1 -- 172.21.15.1:0/3809324712 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc0466c0 con 0x7f15e41137d0 2024-04-26T21:47:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.537+0000 7f15ea600640 1 -- 172.21.15.1:0/3809324712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e41a49a0 con 0x7f15e41137d0 2024-04-26T21:47:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.538+0000 7f15ea600640 1 -- 172.21.15.1:0/3809324712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e406c350 con 0x7f15e41137d0 2024-04-26T21:47:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.541+0000 7f15dbe00640 1 -- 172.21.15.1:0/3809324712 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f15dc030a30 con 0x7f15e41137d0 2024-04-26T21:47:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.542+0000 7f15ed449740 1 Processor -- start 2024-04-26T21:47:05.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.544+0000 7f15ed449740 1 -- start start 2024-04-26T21:47:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.544+0000 7f15ed449740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 0x55a6c77901b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.544+0000 7f15ed449740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c77906f0 0x55a6c779aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.544+0000 7f15ed449740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c779d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.545+0000 7f15ed449740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a6c76f5380 con 0x55a6c778fdd0 2024-04-26T21:47:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.545+0000 7f15ed449740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a6c7691c30 con 0x55a6c779afb0 2024-04-26T21:47:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.545+0000 7f15ed449740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a6c76f1340 con 0x55a6c77906f0 2024-04-26T21:47:05.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.545+0000 7f15dbe00640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f15c80a3e60 0x7f15c80a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.545+0000 7f15e3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c77906f0 0x55a6c779aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.545+0000 7f15e3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c77906f0 0x55a6c779aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54492/0 (socket says 172.21.15.1:54492) 2024-04-26T21:47:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.545+0000 7f15e3400640 1 -- 172.21.15.1:0/3542391621 learned_addr learned my addr 172.21.15.1:0/3542391621 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.545+0000 7f15e2a00640 1 --2- 172.21.15.1:0/3542391621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c779d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.546+0000 7f15dbe00640 1 -- 172.21.15.1:0/3809324712 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7f15dc02e030 con 0x7f15e41137d0 2024-04-26T21:47:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.546+0000 7f15e2a00640 1 -- 172.21.15.1:0/3542391621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c77906f0 msgr2=0x55a6c779aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.546+0000 7f15e2a00640 1 --2- 172.21.15.1:0/3542391621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c77906f0 0x55a6c779aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.546+0000 7f15e2a00640 1 -- 172.21.15.1:0/3542391621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 msgr2=0x55a6c77901b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.546+0000 7f15e2a00640 1 --2- 172.21.15.1:0/3542391621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 0x55a6c77901b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.546+0000 7f15e2a00640 1 -- 172.21.15.1:0/3542391621 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6c76b3940 con 0x55a6c779afb0 2024-04-26T21:47:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.546+0000 7f15e2a00640 1 --2- 172.21.15.1:0/3542391621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c779d3a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f15d0005fd0 tx=0x7f15d0010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbaf45c5b161ef11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.546+0000 7f15d8c00640 1 -- 172.21.15.1:0/3542391621 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0004020 con 0x55a6c779afb0 2024-04-26T21:47:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.547+0000 7f15d8c00640 1 -- 172.21.15.1:0/3542391621 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15d00041c0 con 0x55a6c779afb0 2024-04-26T21:47:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.547+0000 7f15d8c00640 1 -- 172.21.15.1:0/3542391621 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d00126b0 con 0x55a6c779afb0 2024-04-26T21:47:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.547+0000 7f15e3e00640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f15c80a3e60 0x7f15c80a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.548+0000 7f15ed449740 1 -- 172.21.15.1:0/3542391621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 msgr2=0x55a6c779d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:05.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.548+0000 7f15ed449740 1 --2- 172.21.15.1:0/3542391621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c779d3a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f15d0005fd0 tx=0x7f15d0010740 comp rx=0 tx=0).stop 2024-04-26T21:47:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.548+0000 7f15ed449740 1 -- 172.21.15.1:0/3542391621 shutdown_connections 2024-04-26T21:47:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.548+0000 7f15ed449740 1 --2- 172.21.15.1:0/3542391621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c77906f0 0x55a6c779aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.548+0000 7f15ed449740 1 --2- 172.21.15.1:0/3542391621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c779d3a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.548+0000 7f15ed449740 1 --2- 172.21.15.1:0/3542391621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 0x55a6c77901b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.548+0000 7f15ed449740 1 -- 172.21.15.1:0/3542391621 >> 172.21.15.1:0/3542391621 conn(0x55a6c778c2e0 msgr2=0x55a6c778c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.548+0000 7f15e3e00640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f15c80a3e60 0x7f15c80a6320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f15dc0bdc50 tx=0x7f15dc0de040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.548+0000 7f15ed449740 1 -- 172.21.15.1:0/3542391621 shutdown_connections 2024-04-26T21:47:05.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.548+0000 7f15ed449740 1 -- 172.21.15.1:0/3542391621 wait complete. 2024-04-26T21:47:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.549+0000 7f15ed449740 1 -- start start 2024-04-26T21:47:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.549+0000 7f15ed449740 1 Processor -- start 2024-04-26T21:47:05.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.550+0000 7f15ed449740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c7797f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.550+0000 7f15ed449740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 0x55a6c7798450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.550+0000 7f15ed449740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c7798990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.550+0000 7f15ed449740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a6c779dc30 con 0x55a6c77906f0 2024-04-26T21:47:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.550+0000 7f15ed449740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a6c767cc60 con 0x55a6c779afb0 2024-04-26T21:47:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.550+0000 7f15ed449740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a6c779ddb0 con 0x55a6c778fdd0 2024-04-26T21:47:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.550+0000 7f15ed449740 10 client.0 _ll_register_callbacks cb 0x55a6c76a8e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:47:05.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.550+0000 7f15e3400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 0x55a6c7798450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.551+0000 7f15e2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c7797f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.551+0000 7f15e2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c7797f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54504/0 (socket says 172.21.15.1:54504) 2024-04-26T21:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.551+0000 7f15e2a00640 1 -- 172.21.15.1:0/2371401229 learned_addr learned my addr 172.21.15.1:0/2371401229 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.551+0000 7f15e2a00640 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c7798990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.551+0000 7f15e3400640 1 -- 172.21.15.1:0/2371401229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 msgr2=0x55a6c7797f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.551+0000 7f15e3400640 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c7797f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.551+0000 7f15e3400640 1 -- 172.21.15.1:0/2371401229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 msgr2=0x55a6c7798990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:05.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.551+0000 7f15e3400640 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c7798990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.551+0000 7f15e3400640 1 -- 172.21.15.1:0/2371401229 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6c768b510 con 0x55a6c77906f0 2024-04-26T21:47:05.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.552+0000 7f15e2a00640 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c7798990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:47:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.553+0000 7f15e3400640 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 0x55a6c7798450 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f15d4012700 tx=0x7f15d4012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ab334eaac15963 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.553+0000 7f15cd600640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:47:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.554+0000 7f15cd600640 1 -- 172.21.15.1:0/2371401229 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d4004230 con 0x55a6c77906f0 2024-04-26T21:47:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.554+0000 7f15cd600640 1 -- 172.21.15.1:0/2371401229 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7f15d40043d0 con 0x55a6c77906f0 2024-04-26T21:47:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.554+0000 7f15ed449740 1 -- 172.21.15.1:0/2371401229 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15d00041c0 con 0x55a6c77906f0 2024-04-26T21:47:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.555+0000 7f15cd600640 1 -- 172.21.15.1:0/2371401229 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f15d4024050 con 0x55a6c77906f0 2024-04-26T21:47:05.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.555+0000 7f15ed449740 10 client.16533 fetch_fsmap learned FSMap version 52 2024-04-26T21:47:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.555+0000 7f15ed449740 1 -- 172.21.15.1:0/2371401229 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x55a6c7407210 con 0x55a6c77906f0 2024-04-26T21:47:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.556+0000 7f15cd600640 1 -- 172.21.15.1:0/2371401229 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15d00041c0 con 0x55a6c77906f0 2024-04-26T21:47:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.557+0000 7f15cd600640 1 -- 172.21.15.1:0/2371401229 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 52) ==== 8354+0+0 (secure 0 0 0) 0x7f15d4019020 con 0x55a6c77906f0 2024-04-26T21:47:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.557+0000 7f15ed449740 10 client.16533 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T21:47:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.557+0000 7f15ed449740 10 client.16533 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:47:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.557+0000 7f15ed449740 20 client.16533 populate_metadata read hostname 'smithi001' 2024-04-26T21:47:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.557+0000 7f15ed449740 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a6c77aac00 0x55a6c77acff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.557+0000 7f15ed449740 4 client.16533 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:47:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.557+0000 7f15ed449740 1 -- 172.21.15.1:0/2371401229 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6c7702f90 con 0x55a6c77aac00 2024-04-26T21:47:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.557+0000 7f15e2a00640 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a6c77aac00 0x55a6c77acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.558+0000 7f15cd600640 10 client.16533 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.559+0000 7f15e2a00640 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a6c77aac00 0x55a6c77acff0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69397f8ed6e67ede server_cookie=96038127281f81e in_seq=0 out_seq=0 2024-04-26T21:47:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.563+0000 7f15cd600640 1 -- 172.21.15.1:0/2371401229 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a6c7702f90 con 0x55a6c77aac00 2024-04-26T21:47:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.563+0000 7f15cd600640 10 client.16533 handle_command_reply: tid=0 2024-04-26T21:47:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15ed449740 1 client.16533 shutdown 2024-04-26T21:47:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15ed449740 1 -- 172.21.15.1:0/2371401229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 msgr2=0x55a6c7798450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15ed449740 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 0x55a6c7798450 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f15d4012700 tx=0x7f15d4012bc0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15ed449740 1 -- 172.21.15.1:0/2371401229 shutdown_connections 2024-04-26T21:47:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15ed449740 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a6c77aac00 0x55a6c77acff0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15ed449740 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c7798990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15ed449740 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 0x55a6c7798450 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15ed449740 1 --2- 172.21.15.1:0/2371401229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c7797f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15ed449740 1 -- 172.21.15.1:0/2371401229 >> 172.21.15.1:0/2371401229 conn(0x55a6c778c2e0 msgr2=0x55a6c7792060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15cd600640 0 client.16533 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.566+0000 7f15ed449740 1 -- 172.21.15.1:0/2371401229 shutdown_connections 2024-04-26T21:47:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.567+0000 7f15ed449740 1 -- 172.21.15.1:0/2371401229 wait complete. 2024-04-26T21:47:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.567+0000 7f15ed449740 20 client.16533 trim_cache size 0 max 16384 2024-04-26T21:47:05.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.593+0000 7f15ed449740 1 Processor -- start 2024-04-26T21:47:05.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.593+0000 7f15ed449740 1 -- start start 2024-04-26T21:47:05.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15ed449740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c77a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15ed449740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 0x55a6c7792790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15ed449740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c7799d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15ed449740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41c15e0 con 0x55a6c77906f0 2024-04-26T21:47:05.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15ed449740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41beeb0 con 0x55a6c779afb0 2024-04-26T21:47:05.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15ed449740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15e40745b0 con 0x55a6c778fdd0 2024-04-26T21:47:05.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15e2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c77a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15e2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c77a6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54506/0 (socket says 172.21.15.1:54506) 2024-04-26T21:47:05.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15e2a00640 1 -- 172.21.15.1:0/2391799132 learned_addr learned my addr 172.21.15.1:0/2391799132 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:05.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15e3400640 1 --2- 172.21.15.1:0/2391799132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 0x55a6c7792790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15e2a00640 1 --2- 172.21.15.1:0/2391799132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c7799d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15e2a00640 1 -- 172.21.15.1:0/2391799132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 msgr2=0x55a6c7799d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.594+0000 7f15e2a00640 1 --2- 172.21.15.1:0/2391799132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c7799d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.595+0000 7f15e2a00640 1 -- 172.21.15.1:0/2391799132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 msgr2=0x55a6c7792790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.595+0000 7f15e2a00640 1 --2- 172.21.15.1:0/2391799132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 0x55a6c7792790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.595+0000 7f15e2a00640 1 -- 172.21.15.1:0/2391799132 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6c77ad360 con 0x55a6c778fdd0 2024-04-26T21:47:05.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.597+0000 7f15e2a00640 1 --2- 172.21.15.1:0/2391799132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c77a6af0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f15d0005fd0 tx=0x7f15d0003e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c958da9eab9a22c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.597+0000 7f15cfe00640 1 -- 172.21.15.1:0/2391799132 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0024040 con 0x55a6c778fdd0 2024-04-26T21:47:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.597+0000 7f15cfe00640 1 -- 172.21.15.1:0/2391799132 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15d00035f0 con 0x55a6c778fdd0 2024-04-26T21:47:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.597+0000 7f15cfe00640 1 -- 172.21.15.1:0/2391799132 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0005280 con 0x55a6c778fdd0 2024-04-26T21:47:05.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.598+0000 7f15ed449740 1 -- 172.21.15.1:0/2391799132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 msgr2=0x55a6c77a6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.598+0000 7f15ed449740 1 --2- 172.21.15.1:0/2391799132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c77a6af0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f15d0005fd0 tx=0x7f15d0003e40 comp rx=0 tx=0).stop 2024-04-26T21:47:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.598+0000 7f15ed449740 1 -- 172.21.15.1:0/2391799132 shutdown_connections 2024-04-26T21:47:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.598+0000 7f15ed449740 1 --2- 172.21.15.1:0/2391799132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c7799d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.598+0000 7f15ed449740 1 --2- 172.21.15.1:0/2391799132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c77906f0 0x55a6c7792790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.598+0000 7f15ed449740 1 --2- 172.21.15.1:0/2391799132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c778fdd0 0x55a6c77a6af0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.598+0000 7f15ed449740 1 -- 172.21.15.1:0/2391799132 >> 172.21.15.1:0/2391799132 conn(0x55a6c778c2e0 msgr2=0x55a6c7799630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.599+0000 7f15ed449740 1 -- 172.21.15.1:0/2391799132 shutdown_connections 2024-04-26T21:47:05.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.599+0000 7f15ed449740 1 -- 172.21.15.1:0/2391799132 wait complete. 2024-04-26T21:47:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.599+0000 7f15ed449740 1 -- start start 2024-04-26T21:47:05.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.600+0000 7f15ed449740 1 Processor -- start 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.600+0000 7f15ed449740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 0x55a6c779eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15ed449740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c77906f0 0x55a6c779f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15ed449740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c779f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15ed449740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41bd370 con 0x55a6c778fdd0 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15ed449740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41be7e0 con 0x55a6c779afb0 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15ed449740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41bda40 con 0x55a6c77906f0 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15ed449740 10 client.0 _ll_register_callbacks cb 0x7f15d4005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15e2a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 0x55a6c779eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15e2a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 0x55a6c779eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43848/0 (socket says 172.21.15.1:43848) 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15e2a00640 1 -- 172.21.15.1:0/3208007375 learned_addr learned my addr 172.21.15.1:0/3208007375 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:05.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15e2a00640 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c779f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15e2a00640 1 -- 172.21.15.1:0/3208007375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c77906f0 msgr2=0x55a6c779f0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:47:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15e2a00640 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c77906f0 0x55a6c779f0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15e2a00640 1 -- 172.21.15.1:0/3208007375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 msgr2=0x55a6c779f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15e2a00640 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c779f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.601+0000 7f15e2a00640 1 -- 172.21.15.1:0/3208007375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6c74d3620 con 0x55a6c778fdd0 2024-04-26T21:47:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.602+0000 7f15e2a00640 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 0x55a6c779eb80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f15d0005940 tx=0x7f15d0005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb4c496d6ddef06 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.602+0000 7f15cd600640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:47:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.602+0000 7f15cd600640 1 -- 172.21.15.1:0/3208007375 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d0003830 con 0x55a6c778fdd0 2024-04-26T21:47:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.602+0000 7f15cd600640 1 -- 172.21.15.1:0/3208007375 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(118..118 src has 1..118) ==== 9642+0+0 (secure 0 0 0) 0x7f15d0033030 con 0x55a6c778fdd0 2024-04-26T21:47:05.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.603+0000 7f15ed449740 1 -- 172.21.15.1:0/3208007375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15d00035f0 con 0x55a6c778fdd0 2024-04-26T21:47:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.603+0000 7f15cd600640 1 -- 172.21.15.1:0/3208007375 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f15d00039d0 con 0x55a6c778fdd0 2024-04-26T21:47:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.603+0000 7f15ed449740 10 client.16539 fetch_fsmap learned FSMap version 52 2024-04-26T21:47:05.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.603+0000 7f15ed449740 1 -- 172.21.15.1:0/3208007375 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) -- 0x55a6c76afcb0 con 0x55a6c778fdd0 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.604+0000 7f15cd600640 1 -- 172.21.15.1:0/3208007375 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15d00035f0 con 0x55a6c778fdd0 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.604+0000 7f15cd600640 1 -- 172.21.15.1:0/3208007375 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 52) ==== 8354+0+0 (secure 0 0 0) 0x7f15d000f040 con 0x55a6c778fdd0 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.604+0000 7f15ed449740 10 client.16539 fetch_fsmap finished waiting for FSMap version 52 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.604+0000 7f15ed449740 10 client.16539 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.604+0000 7f15ed449740 20 client.16539 populate_metadata read hostname 'smithi001' 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.604+0000 7f15ed449740 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a6c7811650 0x55a6c7813a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.604+0000 7f15ed449740 4 client.16539 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.604+0000 7f15ed449740 1 -- 172.21.15.1:0/3208007375 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a6c76fce00 con 0x55a6c7811650 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.605+0000 7f15e3400640 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a6c7811650 0x55a6c7813a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.605+0000 7f15e3400640 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a6c7811650 0x55a6c7813a40 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=643182634c0a08a2 server_cookie=d4e6ea5bb4b03d78 in_seq=0 out_seq=0 2024-04-26T21:47:05.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:05.605+0000 7f15cd600640 10 client.16539 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:06.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.042+0000 7f15cd600640 1 -- 172.21.15.1:0/3208007375 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a6c76fce00 con 0x55a6c7811650 2024-04-26T21:47:06.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.042+0000 7f15cd600640 10 client.16539 handle_command_reply: tid=0 2024-04-26T21:47:06.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.042+0000 7f15ed449740 1 client.16539 shutdown 2024-04-26T21:47:06.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.042+0000 7f15ed449740 1 -- 172.21.15.1:0/3208007375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 msgr2=0x55a6c779eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:06.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.042+0000 7f15ed449740 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 0x55a6c779eb80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f15d0005940 tx=0x7f15d0005d20 comp rx=0 tx=0).stop 2024-04-26T21:47:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.044+0000 7f15ed449740 1 -- 172.21.15.1:0/3208007375 shutdown_connections 2024-04-26T21:47:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.044+0000 7f15ed449740 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a6c7811650 0x55a6c7813a40 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.044+0000 7f15ed449740 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a6c77906f0 0x55a6c779f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.044+0000 7f15ed449740 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a6c779afb0 0x55a6c779f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.044+0000 7f15ed449740 1 --2- 172.21.15.1:0/3208007375 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a6c778fdd0 0x55a6c779eb80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.044+0000 7f15ed449740 1 -- 172.21.15.1:0/3208007375 >> 172.21.15.1:0/3208007375 conn(0x55a6c778c2e0 msgr2=0x55a6c7798740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.044+0000 7f15cd600640 0 client.16539 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.044+0000 7f15ed449740 1 -- 172.21.15.1:0/3208007375 shutdown_connections 2024-04-26T21:47:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.044+0000 7f15ed449740 1 -- 172.21.15.1:0/3208007375 wait complete. 2024-04-26T21:47:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.044+0000 7f15ed449740 20 client.16539 trim_cache size 0 max 16384 2024-04-26T21:47:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:47:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:47:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:47:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:47:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.045+0000 7f15ea600640 1 -- 172.21.15.1:0/3809324712 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f15c80a3e60 msgr2=0x7f15c80a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.045+0000 7f15ea600640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f15c80a3e60 0x7f15c80a6320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f15dc0bdc50 tx=0x7f15dc0de040 comp rx=0 tx=0).stop 2024-04-26T21:47:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.045+0000 7f15ea600640 1 -- 172.21.15.1:0/3809324712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e41137d0 msgr2=0x7f15e41a4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.045+0000 7f15ea600640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e41137d0 0x7f15e41a4010 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f15dc02ae00 tx=0x7f15dc003ba0 comp rx=0 tx=0).stop 2024-04-26T21:47:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.046+0000 7f15ea600640 1 -- 172.21.15.1:0/3809324712 shutdown_connections 2024-04-26T21:47:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.046+0000 7f15ea600640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f15c80a3e60 0x7f15c80a6320 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.046+0000 7f15ea600640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15e41137d0 0x7f15e41a4010 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.046+0000 7f15ea600640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15e4077780 0x7f15e419fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.046+0000 7f15ea600640 1 --2- 172.21.15.1:0/3809324712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e4076e60 0x7f15e419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.046+0000 7f15ea600640 1 -- 172.21.15.1:0/3809324712 >> 172.21.15.1:0/3809324712 conn(0x7f15e4100620 msgr2=0x7f15e4114fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.046+0000 7f15ea600640 1 -- 172.21.15.1:0/3809324712 shutdown_connections 2024-04-26T21:47:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:06.046+0000 7f15ea600640 1 -- 172.21.15.1:0/3809324712 wait complete. 2024-04-26T21:47:06.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 83495 24.14 MB/sec execute 63 sec latency 565.235 ms 2024-04-26T21:47:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:06 smithi001 ceph-mon[28054]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 5.8 MiB/s wr, 473 op/s 2024-04-26T21:47:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:06 smithi038 ceph-mon[32724]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 5.8 MiB/s wr, 473 op/s 2024-04-26T21:47:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:06 smithi186 ceph-mon[32991]: pgmap v528: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 5.8 MiB/s wr, 473 op/s 2024-04-26T21:47:07.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 83542 23.82 MB/sec execute 64 sec latency 554.099 ms 2024-04-26T21:47:08.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 83549 23.48 MB/sec execute 65 sec latency 832.450 ms 2024-04-26T21:47:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:08 smithi001 ceph-mon[28054]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.4 MiB/s wr, 286 op/s 2024-04-26T21:47:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:08 smithi001 ceph-mon[28054]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T21:47:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:08 smithi038 ceph-mon[32724]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.4 MiB/s wr, 286 op/s 2024-04-26T21:47:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:08 smithi038 ceph-mon[32724]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T21:47:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:08 smithi186 ceph-mon[32991]: pgmap v529: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.4 MiB/s wr, 286 op/s 2024-04-26T21:47:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:08 smithi186 ceph-mon[32991]: osdmap e119: 12 total, 12 up, 12 in 2024-04-26T21:47:09.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 83552 23.14 MB/sec execute 66 sec latency 1181.407 ms 2024-04-26T21:47:10.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 83611 22.87 MB/sec execute 67 sec latency 1115.401 ms 2024-04-26T21:47:10.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:10 smithi001 ceph-mon[28054]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.3 MiB/s wr, 257 op/s 2024-04-26T21:47:10.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:10 smithi038 ceph-mon[32724]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.3 MiB/s wr, 257 op/s 2024-04-26T21:47:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:10 smithi186 ceph-mon[32991]: pgmap v531: 129 pgs: 129 active+clean; 1.1 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.3 MiB/s wr, 257 op/s 2024-04-26T21:47:11.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 83796 22.70 MB/sec execute 68 sec latency 529.367 ms 2024-04-26T21:47:12.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 84196 22.80 MB/sec execute 69 sec latency 395.291 ms 2024-04-26T21:47:12.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:12 smithi038 ceph-mon[32724]: pgmap v532: 129 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 115 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 421 op/s 2024-04-26T21:47:12.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:12 smithi186 ceph-mon[32991]: pgmap v532: 129 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 115 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 421 op/s 2024-04-26T21:47:12.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:12 smithi001 ceph-mon[28054]: pgmap v532: 129 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 115 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 421 op/s 2024-04-26T21:47:13.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 84716 22.99 MB/sec execute 70 sec latency 51.922 ms 2024-04-26T21:47:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:13 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:47:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:13 smithi186 ceph-mon[32991]: pgmap v533: 129 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 115 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 421 op/s 2024-04-26T21:47:13.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:13 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:47:13.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:13 smithi001 ceph-mon[28054]: pgmap v533: 129 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 115 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 421 op/s 2024-04-26T21:47:13.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:13 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:47:13.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:13 smithi038 ceph-mon[32724]: pgmap v533: 129 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 115 active+clean; 1.1 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 421 op/s 2024-04-26T21:47:14.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 85283 23.01 MB/sec execute 71 sec latency 54.018 ms 2024-04-26T21:47:15.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 85782 23.01 MB/sec execute 72 sec latency 69.335 ms 2024-04-26T21:47:16.150 INFO:tasks.workunit.client.0.smithi001.stdout: 10 86033 22.95 MB/sec execute 73 sec latency 404.831 ms 2024-04-26T21:47:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:16 smithi186 ceph-mon[32991]: pgmap v534: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.2 MiB/s wr, 520 op/s 2024-04-26T21:47:16.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:16 smithi001 ceph-mon[28054]: pgmap v534: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.2 MiB/s wr, 520 op/s 2024-04-26T21:47:16.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:16 smithi038 ceph-mon[32724]: pgmap v534: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.2 MiB/s wr, 520 op/s 2024-04-26T21:47:17.151 INFO:tasks.workunit.client.0.smithi001.stdout: 10 86368 22.91 MB/sec execute 74 sec latency 352.997 ms 2024-04-26T21:47:18.151 INFO:tasks.workunit.client.0.smithi001.stdout: 10 86763 22.86 MB/sec execute 75 sec latency 149.612 ms 2024-04-26T21:47:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:18 smithi186 ceph-mon[32991]: pgmap v535: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 541 op/s 2024-04-26T21:47:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:18 smithi001 ceph-mon[28054]: pgmap v535: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 541 op/s 2024-04-26T21:47:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:18 smithi038 ceph-mon[32724]: pgmap v535: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.8 MiB/s wr, 541 op/s 2024-04-26T21:47:19.151 INFO:tasks.workunit.client.0.smithi001.stdout: 10 87389 23.00 MB/sec execute 76 sec latency 56.933 ms 2024-04-26T21:47:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.227+0000 7fc18cc00640 1 -- 172.21.15.1:0/485105141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc188115a80 msgr2=0x7fc188117e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.227+0000 7fc18cc00640 1 --2- 172.21.15.1:0/485105141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc188115a80 0x7fc188117e70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc180009ca0 tx=0x7fc18002a950 comp rx=0 tx=0).stop 2024-04-26T21:47:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.228+0000 7fc18cc00640 1 -- 172.21.15.1:0/485105141 shutdown_connections 2024-04-26T21:47:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.228+0000 7fc18cc00640 1 --2- 172.21.15.1:0/485105141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc18810a460 0x7fc188115540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.228+0000 7fc18cc00640 1 --2- 172.21.15.1:0/485105141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc188115a80 0x7fc188117e70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.228+0000 7fc18cc00640 1 --2- 172.21.15.1:0/485105141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc188109b40 0x7fc188109f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.228+0000 7fc18cc00640 1 -- 172.21.15.1:0/485105141 >> 172.21.15.1:0/485105141 conn(0x7fc18806da10 msgr2=0x7fc18806de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.228+0000 7fc18cc00640 1 -- 172.21.15.1:0/485105141 shutdown_connections 2024-04-26T21:47:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.228+0000 7fc18cc00640 1 -- 172.21.15.1:0/485105141 wait complete. 2024-04-26T21:47:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.230+0000 7fc18cc00640 1 Processor -- start 2024-04-26T21:47:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.230+0000 7fc18cc00640 1 -- start start 2024-04-26T21:47:19.230 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.230+0000 7fc18cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc188109b40 0x7fc1881a9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.230+0000 7fc18cc00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc18810a460 0x7fc1881a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc18cc00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc188115a80 0x7fc1881a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc18cc00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc18811d650 con 0x7fc188115a80 2024-04-26T21:47:19.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc18cc00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc18811d4d0 con 0x7fc18810a460 2024-04-26T21:47:19.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc18cc00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc18811d7d0 con 0x7fc188109b40 2024-04-26T21:47:19.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc185400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc18810a460 0x7fc1881a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc185400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc18810a460 0x7fc1881a9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:33198/0 (socket says 172.21.15.1:33198) 2024-04-26T21:47:19.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc185e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc188109b40 0x7fc1881a9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc185400640 1 -- 172.21.15.1:0/1830861628 learned_addr learned my addr 172.21.15.1:0/1830861628 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:19.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc185400640 1 -- 172.21.15.1:0/1830861628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc188109b40 msgr2=0x7fc1881a9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:19.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc185400640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc188109b40 0x7fc1881a9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc185400640 1 -- 172.21.15.1:0/1830861628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc188115a80 msgr2=0x7fc1881a3930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:47:19.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc185400640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc188115a80 0x7fc1881a3930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.231+0000 7fc185400640 1 -- 172.21.15.1:0/1830861628 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1881a40b0 con 0x7fc18810a460 2024-04-26T21:47:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.232+0000 7fc185400640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc18810a460 0x7fc1881a9cc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc17400e9f0 tx=0x7fc17400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.232+0000 7fc17ea00640 1 -- 172.21.15.1:0/1830861628 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17400ccc0 con 0x7fc18810a460 2024-04-26T21:47:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.232+0000 7fc18cc00640 1 -- 172.21.15.1:0/1830861628 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1881a43a0 con 0x7fc18810a460 2024-04-26T21:47:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.232+0000 7fc17ea00640 1 -- 172.21.15.1:0/1830861628 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc17400ce60 con 0x7fc18810a460 2024-04-26T21:47:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.232+0000 7fc17ea00640 1 -- 172.21.15.1:0/1830861628 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc174010430 con 0x7fc18810a460 2024-04-26T21:47:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.232+0000 7fc18cc00640 1 -- 172.21.15.1:0/1830861628 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1881b0640 con 0x7fc18810a460 2024-04-26T21:47:19.235 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.235+0000 7fc18f9aa740 1 Processor -- start 2024-04-26T21:47:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.236+0000 7fc18f9aa740 1 -- start start 2024-04-26T21:47:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.236+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 0x55feb0f6a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.236+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f6a6f0 0x55feb0f74a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.236+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f74fb0 0x55feb0f773a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.236 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.236+0000 7fc18f9aa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55feb0ecf380 con 0x55feb0f69dd0 2024-04-26T21:47:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.236+0000 7fc18f9aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55feb0e6bc30 con 0x55feb0f74fb0 2024-04-26T21:47:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.236+0000 7fc18f9aa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55feb0ecb340 con 0x55feb0f6a6f0 2024-04-26T21:47:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.237+0000 7fc186800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 0x55feb0f6a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.237+0000 7fc185e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f6a6f0 0x55feb0f74a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.237+0000 7fc186800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 0x55feb0f6a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51740/0 (socket says 172.21.15.1:51740) 2024-04-26T21:47:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.237+0000 7fc186800640 1 -- 172.21.15.1:0/2416581935 learned_addr learned my addr 172.21.15.1:0/2416581935 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.237+0000 7fc185e00640 1 -- 172.21.15.1:0/2416581935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f74fb0 msgr2=0x55feb0f773a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:47:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.237+0000 7fc185e00640 1 --2- 172.21.15.1:0/2416581935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f74fb0 0x55feb0f773a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.237+0000 7fc185e00640 1 -- 172.21.15.1:0/2416581935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 msgr2=0x55feb0f6a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.237+0000 7fc185e00640 1 --2- 172.21.15.1:0/2416581935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 0x55feb0f6a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.237+0000 7fc185e00640 1 -- 172.21.15.1:0/2416581935 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55feb0e8d940 con 0x55feb0f6a6f0 2024-04-26T21:47:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.237+0000 7fc185e00640 1 --2- 172.21.15.1:0/2416581935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f6a6f0 0x55feb0f74a70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc17800aba0 tx=0x7fc17800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=376585a6f6bdd365 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:19.238 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.238+0000 7fc173000640 1 -- 172.21.15.1:0/2416581935 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178005220 con 0x55feb0f6a6f0 2024-04-26T21:47:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.238+0000 7fc173000640 1 -- 172.21.15.1:0/2416581935 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc178004c50 con 0x55feb0f6a6f0 2024-04-26T21:47:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.238+0000 7fc173000640 1 -- 172.21.15.1:0/2416581935 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178018800 con 0x55feb0f6a6f0 2024-04-26T21:47:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.239+0000 7fc17ea00640 1 -- 172.21.15.1:0/1830861628 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fc1740105d0 con 0x7fc18810a460 2024-04-26T21:47:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.239+0000 7fc18f9aa740 1 -- 172.21.15.1:0/2416581935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f6a6f0 msgr2=0x55feb0f74a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.239+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/2416581935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f6a6f0 0x55feb0f74a70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc17800aba0 tx=0x7fc17800f790 comp rx=0 tx=0).stop 2024-04-26T21:47:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.239+0000 7fc18f9aa740 1 -- 172.21.15.1:0/2416581935 shutdown_connections 2024-04-26T21:47:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.239+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/2416581935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f6a6f0 0x55feb0f74a70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.239+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/2416581935 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f74fb0 0x55feb0f773a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.239+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/2416581935 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 0x55feb0f6a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.239+0000 7fc18f9aa740 1 -- 172.21.15.1:0/2416581935 >> 172.21.15.1:0/2416581935 conn(0x55feb0f662e0 msgr2=0x55feb0f666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:19.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.242+0000 7fc18f9aa740 1 -- 172.21.15.1:0/2416581935 shutdown_connections 2024-04-26T21:47:19.242 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.242+0000 7fc18f9aa740 1 -- 172.21.15.1:0/2416581935 wait complete. 2024-04-26T21:47:19.243 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.242+0000 7fc17ea00640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc16c0a3d70 0x7fc16c0a6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.243 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.243+0000 7fc17ea00640 1 -- 172.21.15.1:0/1830861628 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7fc1740d1060 con 0x7fc18810a460 2024-04-26T21:47:19.243 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.243+0000 7fc18f9aa740 1 -- start start 2024-04-26T21:47:19.244 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.243+0000 7fc18f9aa740 1 Processor -- start 2024-04-26T21:47:19.244 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.244+0000 7fc185e00640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc16c0a3d70 0x7fc16c0a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.244 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.244+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 0x55feb0f71ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.244 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.244+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f6a6f0 0x55feb0f72410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.245 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.245+0000 7fc185400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f6a6f0 0x55feb0f72410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.245 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.245+0000 7fc185400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f6a6f0 0x55feb0f72410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:33210/0 (socket says 172.21.15.1:33210) 2024-04-26T21:47:19.245 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.245+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f72950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.245+0000 7fc185400640 1 -- 172.21.15.1:0/681041352 learned_addr learned my addr 172.21.15.1:0/681041352 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.245+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55feb0f77be0 con 0x55feb0f69dd0 2024-04-26T21:47:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.245+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55feb0e56c60 con 0x55feb0f6a6f0 2024-04-26T21:47:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.245+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55feb0f77d60 con 0x55feb0f74fb0 2024-04-26T21:47:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.245+0000 7fc185e00640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc16c0a3d70 0x7fc16c0a6230 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc1780056f0 tx=0x7fc178012040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.245+0000 7fc185e00640 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 0x55feb0f71ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.246+0000 7fc18f9aa740 10 client.0 _ll_register_callbacks cb 0x55feb0e82e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:47:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.246+0000 7fc186800640 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f72950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.246+0000 7fc185e00640 1 -- 172.21.15.1:0/681041352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 msgr2=0x55feb0f72950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.246+0000 7fc185e00640 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f72950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.246+0000 7fc185e00640 1 -- 172.21.15.1:0/681041352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f6a6f0 msgr2=0x55feb0f72410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.246+0000 7fc185e00640 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f6a6f0 0x55feb0f72410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.246+0000 7fc185e00640 1 -- 172.21.15.1:0/681041352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55feb0e65510 con 0x55feb0f69dd0 2024-04-26T21:47:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.246+0000 7fc185400640 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f6a6f0 0x55feb0f72410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:47:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.246+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc178004c50 con 0x55feb0f69dd0 2024-04-26T21:47:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.247+0000 7fc185e00640 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 0x55feb0f71ed0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc178005790 tx=0x7fc17800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932a7ee14939fe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.247+0000 7fc167400640 10 client.16545 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:47:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.248+0000 7fc167400640 1 -- 172.21.15.1:0/681041352 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178018810 con 0x55feb0f69dd0 2024-04-26T21:47:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.248+0000 7fc167400640 1 -- 172.21.15.1:0/681041352 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7fc1780189b0 con 0x55feb0f69dd0 2024-04-26T21:47:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.248+0000 7fc167400640 1 -- 172.21.15.1:0/681041352 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc178018bf0 con 0x55feb0f69dd0 2024-04-26T21:47:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.248+0000 7fc18f9aa740 10 client.16545 fetch_fsmap learned FSMap version 53 2024-04-26T21:47:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.248+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({fsmap=53}) -- 0x55feb0be1210 con 0x55feb0f69dd0 2024-04-26T21:47:19.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.249+0000 7fc167400640 1 -- 172.21.15.1:0/681041352 <== mon.0 v2:172.21.15.1:3300/0 4 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7fc178039450 con 0x55feb0f69dd0 2024-04-26T21:47:19.249 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.249+0000 7fc18f9aa740 10 client.16545 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T21:47:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.249+0000 7fc18f9aa740 10 client.16545 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:47:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.249+0000 7fc18f9aa740 20 client.16545 populate_metadata read hostname 'smithi001' 2024-04-26T21:47:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.249+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55feb0f8d2c0 0x55feb0f8f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.249+0000 7fc18f9aa740 4 client.16545 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:47:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.249+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55feb0edcf90 con 0x55feb0f8d2c0 2024-04-26T21:47:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.250+0000 7fc185400640 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55feb0f8d2c0 0x55feb0f8f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.252+0000 7fc185400640 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55feb0f8d2c0 0x55feb0f8f6b0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4385b9cdf2bd6b68 server_cookie=696b89a200856335 in_seq=0 out_seq=0 2024-04-26T21:47:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.252+0000 7fc167400640 10 client.16545 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.253+0000 7fc167400640 1 -- 172.21.15.1:0/681041352 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55feb0edcf90 con 0x55feb0f8d2c0 2024-04-26T21:47:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.253+0000 7fc167400640 10 client.16545 handle_command_reply: tid=0 2024-04-26T21:47:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.253+0000 7fc18f9aa740 1 client.16545 shutdown 2024-04-26T21:47:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.254+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 msgr2=0x55feb0f71ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.254+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 0x55feb0f71ed0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc178005790 tx=0x7fc17800f790 comp rx=0 tx=0).stop 2024-04-26T21:47:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.254+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 shutdown_connections 2024-04-26T21:47:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.254+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55feb0f8d2c0 0x55feb0f8f6b0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.254+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f72950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.254+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f6a6f0 0x55feb0f72410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.254+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/681041352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f69dd0 0x55feb0f71ed0 secure :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc178005790 tx=0x7fc17800f790 comp rx=0 tx=0).stop 2024-04-26T21:47:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.254+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 >> 172.21.15.1:0/681041352 conn(0x55feb0f662e0 msgr2=0x55feb0f6c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.254+0000 7fc167400640 0 client.16545 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.255+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 shutdown_connections 2024-04-26T21:47:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.255+0000 7fc18f9aa740 1 -- 172.21.15.1:0/681041352 wait complete. 2024-04-26T21:47:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.255+0000 7fc18f9aa740 20 client.16545 trim_cache size 0 max 16384 2024-04-26T21:47:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.279+0000 7fc18f9aa740 1 Processor -- start 2024-04-26T21:47:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.279+0000 7fc18f9aa740 1 -- start start 2024-04-26T21:47:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.279+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f735f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.279+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f73b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.279+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 0x55feb0f74070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.279+0000 7fc18f9aa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881b8de0 con 0x55feb0f6a6f0 2024-04-26T21:47:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.279+0000 7fc18f9aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881b66b0 con 0x55feb0f89360 2024-04-26T21:47:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.279+0000 7fc18f9aa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881b4520 con 0x55feb0f74fb0 2024-04-26T21:47:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc185e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc185400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f73b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc185e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51758/0 (socket says 172.21.15.1:51758) 2024-04-26T21:47:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc185e00640 1 -- 172.21.15.1:0/2879381152 learned_addr learned my addr 172.21.15.1:0/2879381152 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc186800640 1 --2- 172.21.15.1:0/2879381152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 0x55feb0f74070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc185e00640 1 -- 172.21.15.1:0/2879381152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 msgr2=0x55feb0f73b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc185e00640 1 --2- 172.21.15.1:0/2879381152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f73b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc185e00640 1 -- 172.21.15.1:0/2879381152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 msgr2=0x55feb0f74070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc185e00640 1 --2- 172.21.15.1:0/2879381152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 0x55feb0f74070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc185e00640 1 -- 172.21.15.1:0/2879381152 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55feb0cad620 con 0x55feb0f6a6f0 2024-04-26T21:47:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.280+0000 7fc186800640 1 --2- 172.21.15.1:0/2879381152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 0x55feb0f74070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:47:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.281+0000 7fc185e00640 1 --2- 172.21.15.1:0/2879381152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f735f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc17800a5c0 tx=0x7fc17802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2ede240d3b0c1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.281+0000 7fc172600640 1 -- 172.21.15.1:0/2879381152 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1780049d0 con 0x55feb0f6a6f0 2024-04-26T21:47:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.281+0000 7fc172600640 1 -- 172.21.15.1:0/2879381152 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc178048070 con 0x55feb0f6a6f0 2024-04-26T21:47:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.281+0000 7fc172600640 1 -- 172.21.15.1:0/2879381152 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178018810 con 0x55feb0f6a6f0 2024-04-26T21:47:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.282+0000 7fc18f9aa740 1 -- 172.21.15.1:0/2879381152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 msgr2=0x55feb0f735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.282+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/2879381152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f735f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc17800a5c0 tx=0x7fc17802e000 comp rx=0 tx=0).stop 2024-04-26T21:47:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.283+0000 7fc18f9aa740 1 -- 172.21.15.1:0/2879381152 shutdown_connections 2024-04-26T21:47:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.283+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/2879381152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f73b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.283+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/2879381152 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 0x55feb0f74070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.283+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/2879381152 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f735f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.283+0000 7fc18f9aa740 1 -- 172.21.15.1:0/2879381152 >> 172.21.15.1:0/2879381152 conn(0x55feb0f662e0 msgr2=0x55feb0f6d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.283+0000 7fc18f9aa740 1 -- 172.21.15.1:0/2879381152 shutdown_connections 2024-04-26T21:47:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.283+0000 7fc18f9aa740 1 -- 172.21.15.1:0/2879381152 wait complete. 2024-04-26T21:47:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.283+0000 7fc18f9aa740 1 -- start start 2024-04-26T21:47:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.283+0000 7fc18f9aa740 1 Processor -- start 2024-04-26T21:47:19.284 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.284+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f85870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f85db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc18f9aa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 0x55feb0f6e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.285 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc18f9aa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881b4b70 con 0x55feb0f6a6f0 2024-04-26T21:47:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc185400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f85db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc185e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f85870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc18f9aa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881b5fe0 con 0x55feb0f89360 2024-04-26T21:47:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc18f9aa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc1881b5240 con 0x55feb0f74fb0 2024-04-26T21:47:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc18f9aa740 10 client.0 _ll_register_callbacks cb 0x7fc174005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:47:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc185e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f85870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51762/0 (socket says 172.21.15.1:51762) 2024-04-26T21:47:19.286 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc185e00640 1 -- 172.21.15.1:0/1744723048 learned_addr learned my addr 172.21.15.1:0/1744723048 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:19.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.285+0000 7fc186800640 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 0x55feb0f6e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.286+0000 7fc185e00640 1 -- 172.21.15.1:0/1744723048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 msgr2=0x55feb0f85db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:19.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.286+0000 7fc185e00640 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f85db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.286+0000 7fc185e00640 1 -- 172.21.15.1:0/1744723048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 msgr2=0x55feb0f6e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:19.287 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.286+0000 7fc185e00640 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 0x55feb0f6e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.286+0000 7fc185e00640 1 -- 172.21.15.1:0/1744723048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55feb0e89cb0 con 0x55feb0f6a6f0 2024-04-26T21:47:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.286+0000 7fc185400640 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f85db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:47:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.286+0000 7fc186800640 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 0x55feb0f6e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:47:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.286+0000 7fc185e00640 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f85870 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc17800fcd0 tx=0x7fc178005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867ab42ba9acac8f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.286+0000 7fc167400640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:47:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.287+0000 7fc167400640 1 -- 172.21.15.1:0/1744723048 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178053070 con 0x55feb0f6a6f0 2024-04-26T21:47:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.287+0000 7fc167400640 1 -- 172.21.15.1:0/1744723048 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7fc178043480 con 0x55feb0f6a6f0 2024-04-26T21:47:19.288 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.287+0000 7fc18f9aa740 1 -- 172.21.15.1:0/1744723048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc178048070 con 0x55feb0f6a6f0 2024-04-26T21:47:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.288+0000 7fc167400640 1 -- 172.21.15.1:0/1744723048 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fc17800a400 con 0x55feb0f6a6f0 2024-04-26T21:47:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.288+0000 7fc18f9aa740 10 client.16551 fetch_fsmap learned FSMap version 53 2024-04-26T21:47:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.288+0000 7fc18f9aa740 1 -- 172.21.15.1:0/1744723048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55feb0ff2c70 con 0x55feb0f6a6f0 2024-04-26T21:47:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.288+0000 7fc167400640 1 -- 172.21.15.1:0/1744723048 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc178048070 con 0x55feb0f6a6f0 2024-04-26T21:47:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.288+0000 7fc167400640 1 -- 172.21.15.1:0/1744723048 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7fc178056030 con 0x55feb0f6a6f0 2024-04-26T21:47:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.289+0000 7fc18f9aa740 10 client.16551 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T21:47:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.289+0000 7fc18f9aa740 10 client.16551 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:47:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.289+0000 7fc18f9aa740 20 client.16551 populate_metadata read hostname 'smithi001' 2024-04-26T21:47:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.289+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55feb0ff32d0 0x55feb0ff56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.289+0000 7fc18f9aa740 4 client.16551 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:47:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.289+0000 7fc18f9aa740 1 -- 172.21.15.1:0/1744723048 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55feb0ed6e00 con 0x55feb0ff32d0 2024-04-26T21:47:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.292+0000 7fc185400640 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55feb0ff32d0 0x55feb0ff56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.292+0000 7fc167400640 10 client.16551 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.292+0000 7fc185400640 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55feb0ff32d0 0x55feb0ff56c0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de7cfbe3c509f09d server_cookie=e19fc635fac3ca in_seq=0 out_seq=0 2024-04-26T21:47:19.724 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.724+0000 7fc167400640 1 -- 172.21.15.1:0/1744723048 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55feb0ed6e00 con 0x55feb0ff32d0 2024-04-26T21:47:19.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.724+0000 7fc167400640 10 client.16551 handle_command_reply: tid=0 2024-04-26T21:47:19.725 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.725+0000 7fc18f9aa740 1 client.16551 shutdown 2024-04-26T21:47:19.726 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.726+0000 7fc18f9aa740 1 -- 172.21.15.1:0/1744723048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 msgr2=0x55feb0f85870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:19.726 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.726+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f85870 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc17800fcd0 tx=0x7fc178005c00 comp rx=0 tx=0).stop 2024-04-26T21:47:19.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.727+0000 7fc18f9aa740 1 -- 172.21.15.1:0/1744723048 shutdown_connections 2024-04-26T21:47:19.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.727+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55feb0ff32d0 0x55feb0ff56c0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.727+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55feb0f74fb0 0x55feb0f85db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.727+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55feb0f89360 0x55feb0f6e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.727+0000 7fc18f9aa740 1 --2- 172.21.15.1:0/1744723048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55feb0f6a6f0 0x55feb0f85870 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.727+0000 7fc18f9aa740 1 -- 172.21.15.1:0/1744723048 >> 172.21.15.1:0/1744723048 conn(0x55feb0f662e0 msgr2=0x55feb0f73580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:19.727 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.727+0000 7fc167400640 0 client.16551 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:19.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.727+0000 7fc18f9aa740 1 -- 172.21.15.1:0/1744723048 shutdown_connections 2024-04-26T21:47:19.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.728+0000 7fc18f9aa740 1 -- 172.21.15.1:0/1744723048 wait complete. 2024-04-26T21:47:19.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.728+0000 7fc18f9aa740 20 client.16551 trim_cache size 0 max 16384 2024-04-26T21:47:19.728 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:47:19.729 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:47:19.729 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:47:19.729 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:47:19.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.730+0000 7fc18cc00640 1 -- 172.21.15.1:0/1830861628 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc16c0a3d70 msgr2=0x7fc16c0a6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:19.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.730+0000 7fc18cc00640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc16c0a3d70 0x7fc16c0a6230 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc1780056f0 tx=0x7fc178012040 comp rx=0 tx=0).stop 2024-04-26T21:47:19.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.730+0000 7fc18cc00640 1 -- 172.21.15.1:0/1830861628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc18810a460 msgr2=0x7fc1881a9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:19.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.730+0000 7fc18cc00640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc18810a460 0x7fc1881a9cc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc17400e9f0 tx=0x7fc17400edf0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.731+0000 7fc18cc00640 1 -- 172.21.15.1:0/1830861628 shutdown_connections 2024-04-26T21:47:19.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.731+0000 7fc18cc00640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc16c0a3d70 0x7fc16c0a6230 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.731+0000 7fc18cc00640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc188115a80 0x7fc1881a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.731+0000 7fc18cc00640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc18810a460 0x7fc1881a9cc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.731+0000 7fc18cc00640 1 --2- 172.21.15.1:0/1830861628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc188109b40 0x7fc1881a9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:19.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.731+0000 7fc18cc00640 1 -- 172.21.15.1:0/1830861628 >> 172.21.15.1:0/1830861628 conn(0x7fc18806da10 msgr2=0x7fc1880715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:19.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.731+0000 7fc18cc00640 1 -- 172.21.15.1:0/1830861628 shutdown_connections 2024-04-26T21:47:19.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:19.732+0000 7fc18cc00640 1 -- 172.21.15.1:0/1830861628 wait complete. 2024-04-26T21:47:20.151 INFO:tasks.workunit.client.0.smithi001.stdout: 10 87786 22.88 MB/sec execute 77 sec latency 252.856 ms 2024-04-26T21:47:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:20 smithi186 ceph-mon[32991]: pgmap v536: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.2 MiB/s wr, 459 op/s 2024-04-26T21:47:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:20 smithi186 ceph-mon[32991]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T21:47:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:20 smithi001 ceph-mon[28054]: pgmap v536: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.2 MiB/s wr, 459 op/s 2024-04-26T21:47:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:20 smithi001 ceph-mon[28054]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T21:47:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:20 smithi038 ceph-mon[32724]: pgmap v536: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.2 MiB/s wr, 459 op/s 2024-04-26T21:47:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:20 smithi038 ceph-mon[32724]: osdmap e120: 12 total, 12 up, 12 in 2024-04-26T21:47:21.154 INFO:tasks.workunit.client.0.smithi001.stdout: 10 88346 23.10 MB/sec execute 78 sec latency 54.064 ms 2024-04-26T21:47:22.151 INFO:tasks.workunit.client.0.smithi001.stdout: 10 88433 22.87 MB/sec execute 79 sec latency 309.697 ms 2024-04-26T21:47:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:22 smithi186 ceph-mon[32991]: pgmap v538: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.2 MiB/s wr, 596 op/s 2024-04-26T21:47:22.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:22 smithi001 ceph-mon[28054]: pgmap v538: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.2 MiB/s wr, 596 op/s 2024-04-26T21:47:22.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:22 smithi038 ceph-mon[32724]: pgmap v538: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.2 MiB/s wr, 596 op/s 2024-04-26T21:47:23.151 INFO:tasks.workunit.client.0.smithi001.stdout: 10 88672 22.70 MB/sec execute 80 sec latency 476.122 ms 2024-04-26T21:47:24.151 INFO:tasks.workunit.client.0.smithi001.stdout: 10 89024 22.66 MB/sec execute 81 sec latency 130.989 ms 2024-04-26T21:47:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:24 smithi186 ceph-mon[32991]: pgmap v539: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.2 MiB/s wr, 596 op/s 2024-04-26T21:47:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:24 smithi001 ceph-mon[28054]: pgmap v539: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.2 MiB/s wr, 596 op/s 2024-04-26T21:47:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:24 smithi038 ceph-mon[32724]: pgmap v539: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.2 MiB/s wr, 596 op/s 2024-04-26T21:47:25.151 INFO:tasks.workunit.client.0.smithi001.stdout: 10 89495 22.69 MB/sec execute 82 sec latency 59.978 ms 2024-04-26T21:47:26.151 INFO:tasks.workunit.client.0.smithi001.stdout: 10 90105 22.90 MB/sec execute 83 sec latency 57.600 ms 2024-04-26T21:47:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:26 smithi186 ceph-mon[32991]: pgmap v540: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 13 MiB/s wr, 764 op/s 2024-04-26T21:47:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:26 smithi001 ceph-mon[28054]: pgmap v540: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 13 MiB/s wr, 764 op/s 2024-04-26T21:47:26.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:26 smithi038 ceph-mon[32724]: pgmap v540: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 13 MiB/s wr, 764 op/s 2024-04-26T21:47:27.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 90606 22.85 MB/sec execute 84 sec latency 81.554 ms 2024-04-26T21:47:28.151 INFO:tasks.workunit.client.0.smithi001.stdout: 10 91140 22.87 MB/sec execute 85 sec latency 121.599 ms 2024-04-26T21:47:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:28 smithi186 ceph-mon[32991]: pgmap v541: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 14 MiB/s wr, 776 op/s 2024-04-26T21:47:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:28 smithi001 ceph-mon[28054]: pgmap v541: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 14 MiB/s wr, 776 op/s 2024-04-26T21:47:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:28 smithi038 ceph-mon[32724]: pgmap v541: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 14 MiB/s wr, 776 op/s 2024-04-26T21:47:29.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 91703 23.06 MB/sec execute 86 sec latency 118.200 ms 2024-04-26T21:47:30.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 92332 23.07 MB/sec execute 87 sec latency 79.570 ms 2024-04-26T21:47:30.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:30 smithi186 ceph-mon[32991]: pgmap v542: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 14 MiB/s wr, 776 op/s 2024-04-26T21:47:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:30 smithi001 ceph-mon[28054]: pgmap v542: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 14 MiB/s wr, 776 op/s 2024-04-26T21:47:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:30 smithi038 ceph-mon[32724]: pgmap v542: 129 pgs: 129 active+clean; 1.1 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 14 MiB/s wr, 776 op/s 2024-04-26T21:47:31.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 92980 23.23 MB/sec execute 88 sec latency 42.385 ms 2024-04-26T21:47:32.153 INFO:tasks.workunit.client.0.smithi001.stdout: 10 93526 23.41 MB/sec execute 89 sec latency 68.318 ms 2024-04-26T21:47:32.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:32 smithi186 ceph-mon[32991]: pgmap v543: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 15 MiB/s wr, 822 op/s 2024-04-26T21:47:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:32 smithi001 ceph-mon[28054]: pgmap v543: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 15 MiB/s wr, 822 op/s 2024-04-26T21:47:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:32 smithi038 ceph-mon[32724]: pgmap v543: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 15 MiB/s wr, 822 op/s 2024-04-26T21:47:32.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.918+0000 7f72ec400640 1 -- 172.21.15.1:0/916142531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72e4103300 msgr2=0x7f72e410f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:32.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.918+0000 7f72ec400640 1 --2- 172.21.15.1:0/916142531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72e4103300 0x7f72e410f810 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009ca0 tx=0x7f72dc02a8b0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.919+0000 7f72ec400640 1 -- 172.21.15.1:0/916142531 shutdown_connections 2024-04-26T21:47:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.919+0000 7f72ec400640 1 --2- 172.21.15.1:0/916142531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e4102940 0x7f72e4102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.919+0000 7f72ec400640 1 --2- 172.21.15.1:0/916142531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72e4103300 0x7f72e410f810 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.919+0000 7f72ec400640 1 --2- 172.21.15.1:0/916142531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e4108950 0x7f72e4108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.919+0000 7f72ec400640 1 -- 172.21.15.1:0/916142531 >> 172.21.15.1:0/916142531 conn(0x7f72e40fe3f0 msgr2=0x7f72e4100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.920+0000 7f72ec400640 1 -- 172.21.15.1:0/916142531 shutdown_connections 2024-04-26T21:47:32.920 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.920+0000 7f72ec400640 1 -- 172.21.15.1:0/916142531 wait complete. 2024-04-26T21:47:32.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.921+0000 7f72ec400640 1 Processor -- start 2024-04-26T21:47:32.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.921+0000 7f72ec400640 1 -- start start 2024-04-26T21:47:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.921+0000 7f72ec400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72e4102940 0x7f72e4109870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.921+0000 7f72ec400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e4103300 0x7f72e4109db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.921+0000 7f72ec400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e4108950 0x7f72e410a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.921+0000 7f72ec400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f72e4112c60 con 0x7f72e4108950 2024-04-26T21:47:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.921+0000 7f72ec400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f72e4112ae0 con 0x7f72e4102940 2024-04-26T21:47:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.921+0000 7f72ec400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72e4112de0 con 0x7f72e4103300 2024-04-26T21:47:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72e9a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72e4102940 0x7f72e4109870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72e9000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e4103300 0x7f72e4109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72e9a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72e4102940 0x7f72e4109870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:44330/0 (socket says 172.21.15.1:44330) 2024-04-26T21:47:32.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72e9000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e4103300 0x7f72e4109db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52216/0 (socket says 172.21.15.1:52216) 2024-04-26T21:47:32.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72e9a00640 1 -- 172.21.15.1:0/3690151909 learned_addr learned my addr 172.21.15.1:0/3690151909 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72e9000640 1 -- 172.21.15.1:0/3690151909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72e4102940 msgr2=0x7f72e4109870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72e9000640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72e4102940 0x7f72e4109870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72e9000640 1 -- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e4108950 msgr2=0x7f72e410a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72ea400640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e4108950 0x7f72e410a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72e9000640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e4108950 0x7f72e410a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72e9000640 1 -- 172.21.15.1:0/3690151909 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e41ab8e0 con 0x7f72e4103300 2024-04-26T21:47:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.922+0000 7f72ea400640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e4108950 0x7f72e410a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:47:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.923+0000 7f72e9000640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e4103300 0x7f72e4109db0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f72d000e9f0 tx=0x7f72d000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.923+0000 7f72e2000640 1 -- 172.21.15.1:0/3690151909 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d000ccc0 con 0x7f72e4103300 2024-04-26T21:47:32.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.923+0000 7f72ec400640 1 -- 172.21.15.1:0/3690151909 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e41abbd0 con 0x7f72e4103300 2024-04-26T21:47:32.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.923+0000 7f72e2000640 1 -- 172.21.15.1:0/3690151909 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72d00108e0 con 0x7f72e4103300 2024-04-26T21:47:32.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.923+0000 7f72e2000640 1 -- 172.21.15.1:0/3690151909 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0018880 con 0x7f72e4103300 2024-04-26T21:47:32.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.923+0000 7f72ec400640 1 -- 172.21.15.1:0/3690151909 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e41ac170 con 0x7f72e4103300 2024-04-26T21:47:32.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.927+0000 7f72ef174740 1 Processor -- start 2024-04-26T21:47:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.928+0000 7f72ef174740 1 -- start start 2024-04-26T21:47:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.928+0000 7f72e2000640 1 -- 172.21.15.1:0/3690151909 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f72d0010430 con 0x7f72e4103300 2024-04-26T21:47:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.928+0000 7f72ef174740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 0x55f94b3b81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.928+0000 7f72ef174740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b86f0 0x55f94b3c2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.928+0000 7f72ef174740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3c2fb0 0x55f94b3c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.928+0000 7f72ef174740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55f94b31d380 con 0x55f94b3b86f0 2024-04-26T21:47:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.928+0000 7f72ef174740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f94b2b9c30 con 0x55f94b3c2fb0 2024-04-26T21:47:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.928+0000 7f72ef174740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f94b319340 con 0x55f94b3b7dd0 2024-04-26T21:47:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.929+0000 7f72e9000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3c2fb0 0x55f94b3c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.929+0000 7f72ea400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 0x55f94b3b81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.929+0000 7f72ea400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 0x55f94b3b81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52228/0 (socket says 172.21.15.1:52228) 2024-04-26T21:47:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.929+0000 7f72ea400640 1 -- 172.21.15.1:0/2552191151 learned_addr learned my addr 172.21.15.1:0/2552191151 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:32.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.929+0000 7f72e9a00640 1 --2- 172.21.15.1:0/2552191151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b86f0 0x55f94b3c2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.930+0000 7f72e9000640 1 -- 172.21.15.1:0/2552191151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 msgr2=0x55f94b3b81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:32.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.930+0000 7f72e9000640 1 --2- 172.21.15.1:0/2552191151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 0x55f94b3b81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.930+0000 7f72e9000640 1 -- 172.21.15.1:0/2552191151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b86f0 msgr2=0x55f94b3c2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:32.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.930+0000 7f72e9000640 1 --2- 172.21.15.1:0/2552191151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b86f0 0x55f94b3c2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.930+0000 7f72e9000640 1 -- 172.21.15.1:0/2552191151 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f94b2db940 con 0x55f94b3c2fb0 2024-04-26T21:47:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.930+0000 7f72e9000640 1 --2- 172.21.15.1:0/2552191151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3c2fb0 0x55f94b3c53a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f72d00ce2b0 tx=0x7f72d00d96f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c8cc1c547da7fb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.930+0000 7f72e2000640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f72cc0a3d90 0x7f72cc0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.930+0000 7f72e2000640 1 -- 172.21.15.1:0/3690151909 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7f72d00d0fc0 con 0x7f72e4103300 2024-04-26T21:47:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.931+0000 7f72ea400640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f72cc0a3d90 0x7f72cc0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.931+0000 7f72d6a00640 1 -- 172.21.15.1:0/2552191151 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0026030 con 0x55f94b3c2fb0 2024-04-26T21:47:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.931+0000 7f72d6a00640 1 -- 172.21.15.1:0/2552191151 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72d0010720 con 0x55f94b3c2fb0 2024-04-26T21:47:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.931+0000 7f72d6a00640 1 -- 172.21.15.1:0/2552191151 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d009c640 con 0x55f94b3c2fb0 2024-04-26T21:47:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.932+0000 7f72ef174740 1 -- 172.21.15.1:0/2552191151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3c2fb0 msgr2=0x55f94b3c53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.932+0000 7f72ef174740 1 --2- 172.21.15.1:0/2552191151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3c2fb0 0x55f94b3c53a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f72d00ce2b0 tx=0x7f72d00d96f0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.933+0000 7f72ea400640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f72cc0a3d90 0x7f72cc0a6250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f72dc02add0 tx=0x7f72dc009360 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:32.933 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.933+0000 7f72ef174740 1 -- 172.21.15.1:0/2552191151 shutdown_connections 2024-04-26T21:47:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.933+0000 7f72ef174740 1 --2- 172.21.15.1:0/2552191151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3c2fb0 0x55f94b3c53a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.933+0000 7f72ef174740 1 --2- 172.21.15.1:0/2552191151 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b86f0 0x55f94b3c2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.933+0000 7f72ef174740 1 --2- 172.21.15.1:0/2552191151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 0x55f94b3b81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.933+0000 7f72ef174740 1 -- 172.21.15.1:0/2552191151 >> 172.21.15.1:0/2552191151 conn(0x55f94b3b42e0 msgr2=0x55f94b3b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.934+0000 7f72ef174740 1 -- 172.21.15.1:0/2552191151 shutdown_connections 2024-04-26T21:47:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.934+0000 7f72ef174740 1 -- 172.21.15.1:0/2552191151 wait complete. 2024-04-26T21:47:32.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.934+0000 7f72ef174740 1 -- start start 2024-04-26T21:47:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.934+0000 7f72ef174740 1 Processor -- start 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.935+0000 7f72ef174740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b7dd0 0x55f94b3bff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.935+0000 7f72ef174740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b86f0 0x55f94b3c0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.935+0000 7f72ef174740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3c2fb0 0x55f94b3c0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.935+0000 7f72ef174740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55f94b3c5be0 con 0x55f94b3b7dd0 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.935+0000 7f72ef174740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55f94b2a4c60 con 0x55f94b3c2fb0 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.935+0000 7f72ef174740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f94b3c5d60 con 0x55f94b3b86f0 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.935+0000 7f72ef174740 10 client.0 _ll_register_callbacks cb 0x55f94b2d0e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.935+0000 7f72e9000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b86f0 0x55f94b3c0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.935+0000 7f72e9000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b86f0 0x55f94b3c0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52230/0 (socket says 172.21.15.1:52230) 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.935+0000 7f72e9000640 1 -- 172.21.15.1:0/1380851651 learned_addr learned my addr 172.21.15.1:0/1380851651 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.936+0000 7f72e9a00640 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b7dd0 0x55f94b3bff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.936+0000 7f72e9000640 1 -- 172.21.15.1:0/1380851651 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3c2fb0 msgr2=0x55f94b3c0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.936+0000 7f72e9000640 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3c2fb0 0x55f94b3c0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.936+0000 7f72e9000640 1 -- 172.21.15.1:0/1380851651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b7dd0 msgr2=0x55f94b3bff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.936+0000 7f72e9000640 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b7dd0 0x55f94b3bff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.936+0000 7f72e9000640 1 -- 172.21.15.1:0/1380851651 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f94b2b3510 con 0x55f94b3b86f0 2024-04-26T21:47:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.936+0000 7f72e9000640 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b86f0 0x55f94b3c0440 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f72d00ce8b0 tx=0x7f72d00d4720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3f9bb93112fdb6a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.936+0000 7f72cb400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:47:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.936+0000 7f72cb400640 1 -- 172.21.15.1:0/1380851651 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d009c460 con 0x55f94b3b86f0 2024-04-26T21:47:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.936+0000 7f72cb400640 1 -- 172.21.15.1:0/1380851651 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7f72d0010a90 con 0x55f94b3b86f0 2024-04-26T21:47:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.937+0000 7f72ef174740 1 -- 172.21.15.1:0/1380851651 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f94b32af90 con 0x55f94b3b86f0 2024-04-26T21:47:32.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.937+0000 7f72cb400640 1 -- 172.21.15.1:0/1380851651 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f72d009c640 con 0x55f94b3b86f0 2024-04-26T21:47:32.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.937+0000 7f72ef174740 10 client.26827 fetch_fsmap learned FSMap version 53 2024-04-26T21:47:32.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.937+0000 7f72ef174740 1 -- 172.21.15.1:0/1380851651 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55f94b02f210 con 0x55f94b3b86f0 2024-04-26T21:47:32.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.937+0000 7f72cb400640 1 -- 172.21.15.1:0/1380851651 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f94b32af90 con 0x55f94b3b86f0 2024-04-26T21:47:32.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.938+0000 7f72cb400640 1 -- 172.21.15.1:0/1380851651 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f72d009cd60 con 0x55f94b3b86f0 2024-04-26T21:47:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.938+0000 7f72ef174740 10 client.26827 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T21:47:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.938+0000 7f72ef174740 10 client.26827 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:47:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.938+0000 7f72ef174740 20 client.26827 populate_metadata read hostname 'smithi001' 2024-04-26T21:47:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.938+0000 7f72ef174740 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f94b3e3ba0 0x55f94b3e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.938+0000 7f72ef174740 4 client.26827 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:47:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.938+0000 7f72ef174740 1 -- 172.21.15.1:0/1380851651 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f94b324e00 con 0x55f94b3e3ba0 2024-04-26T21:47:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.938+0000 7f72e9a00640 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f94b3e3ba0 0x55f94b3e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.939+0000 7f72e9a00640 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f94b3e3ba0 0x55f94b3e5f90 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18d940865ad1d585 server_cookie=fe29c8d2d9dbdcd6 in_seq=0 out_seq=0 2024-04-26T21:47:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.939+0000 7f72cb400640 10 client.26827 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.942+0000 7f72cb400640 1 -- 172.21.15.1:0/1380851651 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f94b324e00 con 0x55f94b3e3ba0 2024-04-26T21:47:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.942+0000 7f72cb400640 10 client.26827 handle_command_reply: tid=0 2024-04-26T21:47:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.942+0000 7f72ef174740 1 client.26827 shutdown 2024-04-26T21:47:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.942+0000 7f72ef174740 1 -- 172.21.15.1:0/1380851651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b86f0 msgr2=0x55f94b3c0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.942+0000 7f72ef174740 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b86f0 0x55f94b3c0440 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f72d00ce8b0 tx=0x7f72d00d4720 comp rx=0 tx=0).stop 2024-04-26T21:47:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.943+0000 7f72ef174740 1 -- 172.21.15.1:0/1380851651 shutdown_connections 2024-04-26T21:47:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.943+0000 7f72ef174740 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f94b3e3ba0 0x55f94b3e5f90 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.943+0000 7f72ef174740 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b86f0 0x55f94b3c0440 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.943+0000 7f72ef174740 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3c2fb0 0x55f94b3c0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.943+0000 7f72ef174740 1 --2- 172.21.15.1:0/1380851651 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b7dd0 0x55f94b3bff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.943+0000 7f72ef174740 1 -- 172.21.15.1:0/1380851651 >> 172.21.15.1:0/1380851651 conn(0x55f94b3b42e0 msgr2=0x55f94b3ba060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.943+0000 7f72cb400640 0 client.26827 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.943+0000 7f72ef174740 1 -- 172.21.15.1:0/1380851651 shutdown_connections 2024-04-26T21:47:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.943+0000 7f72ef174740 1 -- 172.21.15.1:0/1380851651 wait complete. 2024-04-26T21:47:32.944 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.943+0000 7f72ef174740 20 client.26827 trim_cache size 0 max 16384 2024-04-26T21:47:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.969+0000 7f72ef174740 1 Processor -- start 2024-04-26T21:47:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.969+0000 7f72ef174740 1 -- start start 2024-04-26T21:47:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.969+0000 7f72ef174740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b7dd0 0x55f94b3c2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.969+0000 7f72ef174740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b86f0 0x55f94b3ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.969+0000 7f72ef174740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3c2fb0 0x55f94b3ba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.969+0000 7f72ef174740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41b0660 con 0x55f94b3b86f0 2024-04-26T21:47:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.969+0000 7f72ef174740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41adf30 con 0x55f94b3b7dd0 2024-04-26T21:47:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.969+0000 7f72ef174740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72e410ada0 con 0x55f94b3c2fb0 2024-04-26T21:47:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72ea400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3c2fb0 0x55f94b3ba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72ea400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3c2fb0 0x55f94b3ba790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:52242/0 (socket says 172.21.15.1:52242) 2024-04-26T21:47:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72ea400640 1 -- 172.21.15.1:0/2938104069 learned_addr learned my addr 172.21.15.1:0/2938104069 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72e9000640 1 --2- 172.21.15.1:0/2938104069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b86f0 0x55f94b3ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72e9a00640 1 --2- 172.21.15.1:0/2938104069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b7dd0 0x55f94b3c2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72ea400640 1 -- 172.21.15.1:0/2938104069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b7dd0 msgr2=0x55f94b3c2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72ea400640 1 --2- 172.21.15.1:0/2938104069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b7dd0 0x55f94b3c2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72ea400640 1 -- 172.21.15.1:0/2938104069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b86f0 msgr2=0x55f94b3ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72ea400640 1 --2- 172.21.15.1:0/2938104069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b86f0 0x55f94b3ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72ea400640 1 -- 172.21.15.1:0/2938104069 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f94b3e6300 con 0x55f94b3c2fb0 2024-04-26T21:47:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.970+0000 7f72e9a00640 1 --2- 172.21.15.1:0/2938104069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b7dd0 0x55f94b3c2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:47:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.971+0000 7f72ea400640 1 --2- 172.21.15.1:0/2938104069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3c2fb0 0x55f94b3ba790 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f72dc005e30 tx=0x7f72dc0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38f3c7c135b08027 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.971+0000 7f72d6000640 1 -- 172.21.15.1:0/2938104069 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc003970 con 0x55f94b3c2fb0 2024-04-26T21:47:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.971+0000 7f72d6000640 1 -- 172.21.15.1:0/2938104069 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72dc036050 con 0x55f94b3c2fb0 2024-04-26T21:47:32.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.971+0000 7f72d6000640 1 -- 172.21.15.1:0/2938104069 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72dc0526e0 con 0x55f94b3c2fb0 2024-04-26T21:47:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.972+0000 7f72ef174740 1 -- 172.21.15.1:0/2938104069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3c2fb0 msgr2=0x55f94b3ba790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.972+0000 7f72ef174740 1 --2- 172.21.15.1:0/2938104069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3c2fb0 0x55f94b3ba790 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f72dc005e30 tx=0x7f72dc0095c0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.972+0000 7f72ef174740 1 -- 172.21.15.1:0/2938104069 shutdown_connections 2024-04-26T21:47:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.972+0000 7f72ef174740 1 --2- 172.21.15.1:0/2938104069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3c2fb0 0x55f94b3ba790 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.972+0000 7f72ef174740 1 --2- 172.21.15.1:0/2938104069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3b86f0 0x55f94b3ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.972+0000 7f72ef174740 1 --2- 172.21.15.1:0/2938104069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b7dd0 0x55f94b3c2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.972+0000 7f72ef174740 1 -- 172.21.15.1:0/2938104069 >> 172.21.15.1:0/2938104069 conn(0x55f94b3b42e0 msgr2=0x55f94b3bb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.972+0000 7f72ef174740 1 -- 172.21.15.1:0/2938104069 shutdown_connections 2024-04-26T21:47:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.972+0000 7f72ef174740 1 -- 172.21.15.1:0/2938104069 wait complete. 2024-04-26T21:47:32.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.973+0000 7f72ef174740 1 -- start start 2024-04-26T21:47:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.973+0000 7f72ef174740 1 Processor -- start 2024-04-26T21:47:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.973+0000 7f72ef174740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 0x55f94b3c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.973+0000 7f72ef174740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b86f0 0x55f94b3c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.973+0000 7f72ef174740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3c2fb0 0x55f94b448100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.973+0000 7f72ef174740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41ac410 con 0x55f94b3c2fb0 2024-04-26T21:47:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72ef174740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41ad860 con 0x55f94b3b86f0 2024-04-26T21:47:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72ef174740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72e41acac0 con 0x55f94b3b7dd0 2024-04-26T21:47:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72ef174740 10 client.0 _ll_register_callbacks cb 0x7f72d8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:47:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72ea400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3c2fb0 0x55f94b448100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72e9a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 0x55f94b3c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72e9000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b86f0 0x55f94b3c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72ea400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3c2fb0 0x55f94b448100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33952/0 (socket says 172.21.15.1:33952) 2024-04-26T21:47:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72ea400640 1 -- 172.21.15.1:0/3028050483 learned_addr learned my addr 172.21.15.1:0/3028050483 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72e9000640 1 -- 172.21.15.1:0/3028050483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 msgr2=0x55f94b3c61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72e9000640 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 0x55f94b3c61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72e9000640 1 -- 172.21.15.1:0/3028050483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3c2fb0 msgr2=0x55f94b448100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72e9000640 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3c2fb0 0x55f94b448100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72e9000640 1 -- 172.21.15.1:0/3028050483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f94b0fb620 con 0x55f94b3b86f0 2024-04-26T21:47:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72e9000640 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b86f0 0x55f94b3c66e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f72d009c800 tx=0x7f72d009d100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8d3f9c9661b1b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.974+0000 7f72cb400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:47:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.975+0000 7f72cb400640 1 -- 172.21.15.1:0/3028050483 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0098da0 con 0x55f94b3b86f0 2024-04-26T21:47:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.975+0000 7f72cb400640 1 -- 172.21.15.1:0/3028050483 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7f72d00d48e0 con 0x55f94b3b86f0 2024-04-26T21:47:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.975+0000 7f72ef174740 1 -- 172.21.15.1:0/3028050483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72dc036050 con 0x55f94b3b86f0 2024-04-26T21:47:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.975+0000 7f72cb400640 1 -- 172.21.15.1:0/3028050483 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f72d009c460 con 0x55f94b3b86f0 2024-04-26T21:47:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.975+0000 7f72ef174740 10 client.26837 fetch_fsmap learned FSMap version 53 2024-04-26T21:47:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.975+0000 7f72ef174740 1 -- 172.21.15.1:0/3028050483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x55f94b2d7cb0 con 0x55f94b3b86f0 2024-04-26T21:47:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.976+0000 7f72cb400640 1 -- 172.21.15.1:0/3028050483 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72dc036050 con 0x55f94b3b86f0 2024-04-26T21:47:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.976+0000 7f72cb400640 1 -- 172.21.15.1:0/3028050483 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f72d00a5370 con 0x55f94b3b86f0 2024-04-26T21:47:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.977+0000 7f72ef174740 10 client.26837 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T21:47:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.977+0000 7f72ef174740 10 client.26837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:47:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.977+0000 7f72ef174740 20 client.26837 populate_metadata read hostname 'smithi001' 2024-04-26T21:47:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.977+0000 7f72ef174740 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f94b449930 0x55f94b44bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.977+0000 7f72ef174740 4 client.26837 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:47:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.977+0000 7f72ef174740 1 -- 172.21.15.1:0/3028050483 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f94b2a04b0 con 0x55f94b449930 2024-04-26T21:47:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.977+0000 7f72e9a00640 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f94b449930 0x55f94b44bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.977+0000 7f72e9a00640 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f94b449930 0x55f94b44bd20 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7483a35f0ae4565 server_cookie=72c431c3923031a in_seq=0 out_seq=0 2024-04-26T21:47:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:32.977+0000 7f72cb400640 10 client.26837 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:33.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 94052 23.41 MB/sec execute 90 sec latency 147.789 ms 2024-04-26T21:47:33.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.477+0000 7f72cb400640 1 -- 172.21.15.1:0/3028050483 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f94b2a04b0 con 0x55f94b449930 2024-04-26T21:47:33.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.477+0000 7f72cb400640 10 client.26837 handle_command_reply: tid=0 2024-04-26T21:47:33.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.477+0000 7f72ef174740 1 client.26837 shutdown 2024-04-26T21:47:33.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.477+0000 7f72ef174740 1 -- 172.21.15.1:0/3028050483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b86f0 msgr2=0x55f94b3c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:33.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.477+0000 7f72ef174740 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b86f0 0x55f94b3c66e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f72d009c800 tx=0x7f72d009d100 comp rx=0 tx=0).stop 2024-04-26T21:47:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.477+0000 7f72ef174740 1 -- 172.21.15.1:0/3028050483 shutdown_connections 2024-04-26T21:47:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.477+0000 7f72ef174740 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55f94b449930 0x55f94b44bd20 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.478+0000 7f72cb400640 0 client.26837 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.477+0000 7f72ef174740 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55f94b3c2fb0 0x55f94b448100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.478+0000 7f72ef174740 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f94b3b86f0 0x55f94b3c66e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.478+0000 7f72ef174740 1 --2- 172.21.15.1:0/3028050483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f94b3b7dd0 0x55f94b3c61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.478+0000 7f72ef174740 1 -- 172.21.15.1:0/3028050483 >> 172.21.15.1:0/3028050483 conn(0x55f94b3b42e0 msgr2=0x55f94b3bfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.478+0000 7f72ef174740 1 -- 172.21.15.1:0/3028050483 shutdown_connections 2024-04-26T21:47:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.478+0000 7f72ef174740 1 -- 172.21.15.1:0/3028050483 wait complete. 2024-04-26T21:47:33.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.478+0000 7f72ef174740 20 client.26837 trim_cache size 0 max 16384 2024-04-26T21:47:33.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:47:33.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:47:33.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:47:33.480 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.479+0000 7f72ec400640 1 -- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f72cc0a3d90 msgr2=0x7f72cc0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.479+0000 7f72ec400640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f72cc0a3d90 0x7f72cc0a6250 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f72dc02add0 tx=0x7f72dc009360 comp rx=0 tx=0).stop 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.479+0000 7f72ec400640 1 -- 172.21.15.1:0/3690151909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e4103300 msgr2=0x7f72e4109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.479+0000 7f72ec400640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e4103300 0x7f72e4109db0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f72d000e9f0 tx=0x7f72d000eeb0 comp rx=0 tx=0).stop 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.480+0000 7f72ec400640 1 -- 172.21.15.1:0/3690151909 shutdown_connections 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.480+0000 7f72ec400640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f72cc0a3d90 0x7f72cc0a6250 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.480+0000 7f72ec400640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e4103300 0x7f72e4109db0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.480+0000 7f72ec400640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f72e4108950 0x7f72e410a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.480+0000 7f72ec400640 1 --2- 172.21.15.1:0/3690151909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72e4102940 0x7f72e4109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.480+0000 7f72ec400640 1 -- 172.21.15.1:0/3690151909 >> 172.21.15.1:0/3690151909 conn(0x7f72e40fe3f0 msgr2=0x7f72e410dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.480+0000 7f72ec400640 1 -- 172.21.15.1:0/3690151909 shutdown_connections 2024-04-26T21:47:33.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:33.480+0000 7f72ec400640 1 -- 172.21.15.1:0/3690151909 wait complete. 2024-04-26T21:47:34.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 94451 23.35 MB/sec execute 91 sec latency 327.633 ms 2024-04-26T21:47:34.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:34 smithi186 ceph-mon[32991]: pgmap v544: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 9.6 MiB/s wr, 482 op/s 2024-04-26T21:47:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:34 smithi001 ceph-mon[28054]: pgmap v544: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 9.6 MiB/s wr, 482 op/s 2024-04-26T21:47:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:34 smithi038 ceph-mon[32724]: pgmap v544: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 9.6 MiB/s wr, 482 op/s 2024-04-26T21:47:35.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 94983 23.46 MB/sec execute 92 sec latency 73.274 ms 2024-04-26T21:47:36.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 95603 23.55 MB/sec execute 93 sec latency 91.598 ms 2024-04-26T21:47:36.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:36 smithi186 ceph-mon[32991]: pgmap v545: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 704 op/s 2024-04-26T21:47:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:36 smithi001 ceph-mon[28054]: pgmap v545: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 704 op/s 2024-04-26T21:47:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:36 smithi038 ceph-mon[32724]: pgmap v545: 129 pgs: 129 active+clean; 1.1 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 704 op/s 2024-04-26T21:47:37.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 96166 23.64 MB/sec execute 94 sec latency 75.896 ms 2024-04-26T21:47:38.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 96733 23.71 MB/sec execute 95 sec latency 112.546 ms 2024-04-26T21:47:38.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:38 smithi186 ceph-mon[32991]: pgmap v546: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 7.1 MiB/s wr, 507 op/s 2024-04-26T21:47:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:38 smithi001 ceph-mon[28054]: pgmap v546: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 7.1 MiB/s wr, 507 op/s 2024-04-26T21:47:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:38 smithi038 ceph-mon[32724]: pgmap v546: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 7.1 MiB/s wr, 507 op/s 2024-04-26T21:47:39.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 97301 23.74 MB/sec execute 96 sec latency 49.625 ms 2024-04-26T21:47:40.152 INFO:tasks.workunit.client.0.smithi001.stdout: 10 97686 23.72 MB/sec execute 97 sec latency 207.724 ms 2024-04-26T21:47:40.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:40 smithi186 ceph-mon[32991]: pgmap v547: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 5.6 MiB/s wr, 443 op/s 2024-04-26T21:47:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:40 smithi001 ceph-mon[28054]: pgmap v547: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 5.6 MiB/s wr, 443 op/s 2024-04-26T21:47:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:40 smithi038 ceph-mon[32724]: pgmap v547: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 5.6 MiB/s wr, 443 op/s 2024-04-26T21:47:41.153 INFO:tasks.workunit.client.0.smithi001.stdout: 10 98321 23.86 MB/sec execute 98 sec latency 61.385 ms 2024-04-26T21:47:42.153 INFO:tasks.workunit.client.0.smithi001.stdout: 10 98945 23.97 MB/sec execute 99 sec latency 43.168 ms 2024-04-26T21:47:42.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:42 smithi186 ceph-mon[32991]: pgmap v548: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.0 MiB/s wr, 604 op/s 2024-04-26T21:47:42.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:42 smithi001 ceph-mon[28054]: pgmap v548: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.0 MiB/s wr, 604 op/s 2024-04-26T21:47:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:42 smithi038 ceph-mon[32724]: pgmap v548: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.0 MiB/s wr, 604 op/s 2024-04-26T21:47:43.153 INFO:tasks.workunit.client.0.smithi001.stdout: 10 99442 23.97 MB/sec execute 100 sec latency 79.040 ms 2024-04-26T21:47:44.156 INFO:tasks.workunit.client.0.smithi001.stdout: 10 99962 24.03 MB/sec execute 101 sec latency 66.544 ms 2024-04-26T21:47:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:44 smithi001 ceph-mon[28054]: pgmap v549: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.8 MiB/s wr, 431 op/s 2024-04-26T21:47:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:44 smithi038 ceph-mon[32724]: pgmap v549: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.8 MiB/s wr, 431 op/s 2024-04-26T21:47:44.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:44 smithi186 ceph-mon[32991]: pgmap v549: 129 pgs: 129 active+clean; 1.1 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.8 MiB/s wr, 431 op/s 2024-04-26T21:47:45.156 INFO:tasks.workunit.client.0.smithi001.stdout: 10 100381 24.01 MB/sec execute 102 sec latency 373.119 ms 2024-04-26T21:47:46.156 INFO:tasks.workunit.client.0.smithi001.stdout: 10 100937 24.03 MB/sec execute 103 sec latency 87.028 ms 2024-04-26T21:47:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:46 smithi001 ceph-mon[28054]: pgmap v550: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 536 op/s 2024-04-26T21:47:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:46 smithi038 ceph-mon[32724]: pgmap v550: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 536 op/s 2024-04-26T21:47:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:46 smithi186 ceph-mon[32991]: pgmap v550: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.3 MiB/s wr, 536 op/s 2024-04-26T21:47:47.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 101598 24.20 MB/sec execute 104 sec latency 77.052 ms 2024-04-26T21:47:48.157 INFO:tasks.workunit.client.0.smithi001.stdout: 10 101671 23.99 MB/sec execute 105 sec latency 599.264 ms 2024-04-26T21:47:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:48 smithi038 ceph-mon[32724]: pgmap v551: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 4.8 MiB/s wr, 354 op/s 2024-04-26T21:47:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:48 smithi186 ceph-mon[32991]: pgmap v551: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 4.8 MiB/s wr, 354 op/s 2024-04-26T21:47:49.158 INFO:tasks.workunit.client.0.smithi001.stdout: 10 101697 23.78 MB/sec execute 106 sec latency 1388.046 ms 2024-04-26T21:47:49.559 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:48 smithi001 ceph-mon[28054]: pgmap v551: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 4.8 MiB/s wr, 354 op/s 2024-04-26T21:47:50.158 INFO:tasks.workunit.client.0.smithi001.stdout: 10 101745 23.57 MB/sec execute 107 sec latency 2270.020 ms 2024-04-26T21:47:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:50 smithi001 ceph-mon[28054]: pgmap v552: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.5 MiB/s wr, 305 op/s 2024-04-26T21:47:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:50 smithi038 ceph-mon[32724]: pgmap v552: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.5 MiB/s wr, 305 op/s 2024-04-26T21:47:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:50 smithi186 ceph-mon[32991]: pgmap v552: 129 pgs: 129 active+clean; 1.1 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.5 MiB/s wr, 305 op/s 2024-04-26T21:47:50.902 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.901+0000 7f6bb4600640 1 -- 172.21.15.1:0/1600933618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac103bc0 msgr2=0x7f6bac103fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.901+0000 7f6bb4600640 1 --2- 172.21.15.1:0/1600933618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac103bc0 0x7f6bac103fa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0009900 tx=0x7f6ba002a480 comp rx=0 tx=0).stop 2024-04-26T21:47:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.901+0000 7f6bb4600640 1 -- 172.21.15.1:0/1600933618 shutdown_connections 2024-04-26T21:47:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.901+0000 7f6bb4600640 1 --2- 172.21.15.1:0/1600933618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bac10cf90 0x7f6bac10f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.901+0000 7f6bb4600640 1 --2- 172.21.15.1:0/1600933618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bac1044e0 0x7f6bac10c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.901+0000 7f6bb4600640 1 --2- 172.21.15.1:0/1600933618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac103bc0 0x7f6bac103fa0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.901+0000 7f6bb4600640 1 -- 172.21.15.1:0/1600933618 >> 172.21.15.1:0/1600933618 conn(0x7f6bac0789b0 msgr2=0x7f6bac078dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:50.903 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.902+0000 7f6bb4600640 1 -- 172.21.15.1:0/1600933618 shutdown_connections 2024-04-26T21:47:50.904 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.902+0000 7f6bb4600640 1 -- 172.21.15.1:0/1600933618 wait complete. 2024-04-26T21:47:50.905 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.904+0000 7f6bb4600640 1 Processor -- start 2024-04-26T21:47:50.906 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.905+0000 7f6bb4600640 1 -- start start 2024-04-26T21:47:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.906+0000 7f6bb4600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac103bc0 0x7f6bac101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.906+0000 7f6bb4600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bac1044e0 0x7f6bac101d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.907+0000 7f6bb4600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bac10cf90 0x7f6bac102290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.907+0000 7f6bb4600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac112c80 con 0x7f6bac1044e0 2024-04-26T21:47:50.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.907+0000 7f6bb4600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac112b00 con 0x7f6bac10cf90 2024-04-26T21:47:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.907+0000 7f6bb4600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac112e00 con 0x7f6bac103bc0 2024-04-26T21:47:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.907+0000 7f6bb1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac103bc0 0x7f6bac101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.907+0000 7f6bb1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac103bc0 0x7f6bac101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:48104/0 (socket says 172.21.15.1:48104) 2024-04-26T21:47:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.907+0000 7f6bb1c00640 1 -- 172.21.15.1:0/1949207371 learned_addr learned my addr 172.21.15.1:0/1949207371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.907+0000 7f6bb1200640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bac1044e0 0x7f6bac101d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:50.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.907+0000 7f6bb2600640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bac10cf90 0x7f6bac102290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.908+0000 7f6bb1c00640 1 -- 172.21.15.1:0/1949207371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bac10cf90 msgr2=0x7f6bac102290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.908+0000 7f6bb1c00640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bac10cf90 0x7f6bac102290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.908+0000 7f6bb1c00640 1 -- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bac1044e0 msgr2=0x7f6bac101d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.908+0000 7f6bb1c00640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bac1044e0 0x7f6bac101d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.908+0000 7f6bb1c00640 1 -- 172.21.15.1:0/1949207371 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bac1a76a0 con 0x7f6bac103bc0 2024-04-26T21:47:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.909+0000 7f6bb1200640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bac1044e0 0x7f6bac101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:47:50.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.909+0000 7f6bb2600640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bac10cf90 0x7f6bac102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:47:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.911+0000 7f6bb1c00640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac103bc0 0x7f6bac101810 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0005ab0 tx=0x7f6ba0003c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.911+0000 7f6baa000640 1 -- 172.21.15.1:0/1949207371 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba003f070 con 0x7f6bac103bc0 2024-04-26T21:47:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.911+0000 7f6baa000640 1 -- 172.21.15.1:0/1949207371 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ba0031d60 con 0x7f6bac103bc0 2024-04-26T21:47:50.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.911+0000 7f6baa000640 1 -- 172.21.15.1:0/1949207371 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0035030 con 0x7f6bac103bc0 2024-04-26T21:47:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.912+0000 7f6bb4600640 1 -- 172.21.15.1:0/1949207371 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bac1a7930 con 0x7f6bac103bc0 2024-04-26T21:47:50.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.912+0000 7f6bb4600640 1 -- 172.21.15.1:0/1949207371 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bac1a7dd0 con 0x7f6bac103bc0 2024-04-26T21:47:50.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.920+0000 7f6baa000640 1 -- 172.21.15.1:0/1949207371 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6ba0042070 con 0x7f6bac103bc0 2024-04-26T21:47:50.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.924+0000 7f6baa000640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f6b940a3dc0 0x7f6b940a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.925+0000 7f6baa000640 1 -- 172.21.15.1:0/1949207371 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7f6ba00f1470 con 0x7f6bac103bc0 2024-04-26T21:47:50.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.925+0000 7f6bb1200640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f6b940a3dc0 0x7f6b940a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.927+0000 7f6bb1200640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f6b940a3dc0 0x7f6b940a6280 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6b98006fd0 tx=0x7f6b98008040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:50.945 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.944+0000 7f6bb759d740 1 Processor -- start 2024-04-26T21:47:50.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.945+0000 7f6bb759d740 1 -- start start 2024-04-26T21:47:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.947+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1155e0 0x562c8c1159c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.948+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 0x562c8c120280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.948+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1207c0 0x562c8c122bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.948+0000 7f6bb759d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562c8c072380 con 0x562c8c115f00 2024-04-26T21:47:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.948+0000 7f6bb759d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562c8c00ec30 con 0x562c8c1155e0 2024-04-26T21:47:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.948+0000 7f6bb759d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562c8c06e340 con 0x562c8c1207c0 2024-04-26T21:47:50.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.948+0000 7f6bb1200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 0x562c8c120280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:50.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.948+0000 7f6bb1200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 0x562c8c120280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37970/0 (socket says 172.21.15.1:37970) 2024-04-26T21:47:50.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.948+0000 7f6bb1200640 1 -- 172.21.15.1:0/1919697407 learned_addr learned my addr 172.21.15.1:0/1919697407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.950+0000 7f6bb2600640 1 --2- 172.21.15.1:0/1919697407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1207c0 0x562c8c122bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.950+0000 7f6bb1c00640 1 --2- 172.21.15.1:0/1919697407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1155e0 0x562c8c1159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.950+0000 7f6bb2600640 1 -- 172.21.15.1:0/1919697407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1155e0 msgr2=0x562c8c1159c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:50.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.950+0000 7f6bb2600640 1 --2- 172.21.15.1:0/1919697407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1155e0 0x562c8c1159c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.950+0000 7f6bb2600640 1 -- 172.21.15.1:0/1919697407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 msgr2=0x562c8c120280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.950+0000 7f6bb2600640 1 --2- 172.21.15.1:0/1919697407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 0x562c8c120280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.950+0000 7f6bb2600640 1 -- 172.21.15.1:0/1919697407 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c8c030940 con 0x562c8c1207c0 2024-04-26T21:47:50.952 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.951+0000 7f6bb2600640 1 --2- 172.21.15.1:0/1919697407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1207c0 0x562c8c122bb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4005fd0 tx=0x7f6ba400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d982ec6f5ed8d655 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:50.955 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.952+0000 7f6b9ea00640 1 -- 172.21.15.1:0/1919697407 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4003b70 con 0x562c8c1207c0 2024-04-26T21:47:50.955 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.952+0000 7f6b9ea00640 1 -- 172.21.15.1:0/1919697407 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ba4003d10 con 0x562c8c1207c0 2024-04-26T21:47:50.955 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.952+0000 7f6b9ea00640 1 -- 172.21.15.1:0/1919697407 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4011900 con 0x562c8c1207c0 2024-04-26T21:47:50.955 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.954+0000 7f6bb759d740 1 -- 172.21.15.1:0/1919697407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1207c0 msgr2=0x562c8c122bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:50.955 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.954+0000 7f6bb759d740 1 --2- 172.21.15.1:0/1919697407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1207c0 0x562c8c122bb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4005fd0 tx=0x7f6ba400f730 comp rx=0 tx=0).stop 2024-04-26T21:47:50.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.954+0000 7f6bb759d740 1 -- 172.21.15.1:0/1919697407 shutdown_connections 2024-04-26T21:47:50.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.954+0000 7f6bb759d740 1 --2- 172.21.15.1:0/1919697407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1207c0 0x562c8c122bb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.954+0000 7f6bb759d740 1 --2- 172.21.15.1:0/1919697407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 0x562c8c120280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.954+0000 7f6bb759d740 1 --2- 172.21.15.1:0/1919697407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1155e0 0x562c8c1159c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.954+0000 7f6bb759d740 1 -- 172.21.15.1:0/1919697407 >> 172.21.15.1:0/1919697407 conn(0x562c8c111af0 msgr2=0x562c8c111ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:50.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.954+0000 7f6bb759d740 1 -- 172.21.15.1:0/1919697407 shutdown_connections 2024-04-26T21:47:50.957 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.955+0000 7f6bb759d740 1 -- 172.21.15.1:0/1919697407 wait complete. 2024-04-26T21:47:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.956+0000 7f6bb759d740 1 -- start start 2024-04-26T21:47:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.957+0000 7f6bb759d740 1 Processor -- start 2024-04-26T21:47:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.958+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1155e0 0x562c8c11d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.958+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 0x562c8c11dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.958+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 0x562c8c11e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.958+0000 7f6bb1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1155e0 0x562c8c11d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.958+0000 7f6bb1200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 0x562c8c11dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.958+0000 7f6bb1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1155e0 0x562c8c11d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:48118/0 (socket says 172.21.15.1:48118) 2024-04-26T21:47:50.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.958+0000 7f6bb1200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 0x562c8c11dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37986/0 (socket says 172.21.15.1:37986) 2024-04-26T21:47:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.959+0000 7f6bb759d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562c8c1233f0 con 0x562c8c115f00 2024-04-26T21:47:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.959+0000 7f6bb1c00640 1 -- 172.21.15.1:0/380430196 learned_addr learned my addr 172.21.15.1:0/380430196 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.960+0000 7f6bb759d740 1 -- 172.21.15.1:0/380430196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562c8bff9c60 con 0x562c8c1207c0 2024-04-26T21:47:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.960+0000 7f6bb759d740 1 -- 172.21.15.1:0/380430196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562c8c123570 con 0x562c8c1155e0 2024-04-26T21:47:50.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.960+0000 7f6bb759d740 10 client.0 _ll_register_callbacks cb 0x562c8c025e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:47:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.961+0000 7f6bb1c00640 1 -- 172.21.15.1:0/380430196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 msgr2=0x562c8c11e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.961+0000 7f6bb1c00640 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 0x562c8c11e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.961+0000 7f6bb1c00640 1 -- 172.21.15.1:0/380430196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 msgr2=0x562c8c11dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.961+0000 7f6bb1c00640 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 0x562c8c11dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.962 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.961+0000 7f6bb1c00640 1 -- 172.21.15.1:0/380430196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c8c008510 con 0x562c8c1155e0 2024-04-26T21:47:50.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.962+0000 7f6bb1c00640 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1155e0 0x562c8c11d700 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0005c40 tx=0x7f6ba00bc910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a846637c2943efa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:50.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.962+0000 7f6b93400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:47:50.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.962+0000 7f6b93400640 1 -- 172.21.15.1:0/380430196 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba00d5070 con 0x562c8c1155e0 2024-04-26T21:47:50.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.962+0000 7f6b93400640 1 -- 172.21.15.1:0/380430196 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7f6ba00b9620 con 0x562c8c1155e0 2024-04-26T21:47:50.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.963+0000 7f6bb759d740 1 -- 172.21.15.1:0/380430196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ba4003d10 con 0x562c8c1155e0 2024-04-26T21:47:50.966 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.965+0000 7f6b93400640 1 -- 172.21.15.1:0/380430196 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f6ba0049040 con 0x562c8c1155e0 2024-04-26T21:47:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.965+0000 7f6bb759d740 10 client.26845 fetch_fsmap learned FSMap version 53 2024-04-26T21:47:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.965+0000 7f6bb759d740 1 -- 172.21.15.1:0/380430196 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x562c8bd84210 con 0x562c8c1155e0 2024-04-26T21:47:50.967 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.966+0000 7f6b93400640 1 -- 172.21.15.1:0/380430196 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ba4003d10 con 0x562c8c1155e0 2024-04-26T21:47:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.967+0000 7f6b93400640 1 -- 172.21.15.1:0/380430196 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f6ba00c5790 con 0x562c8c1155e0 2024-04-26T21:47:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.968+0000 7f6bb759d740 10 client.26845 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T21:47:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.968+0000 7f6bb759d740 10 client.26845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:47:50.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.968+0000 7f6bb759d740 20 client.26845 populate_metadata read hostname 'smithi001' 2024-04-26T21:47:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.968+0000 7f6bb759d740 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562c8c130510 0x562c8c1329a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.968+0000 7f6bb759d740 4 client.26845 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:47:50.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.968+0000 7f6bb759d740 1 -- 172.21.15.1:0/380430196 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c8c07ff90 con 0x562c8c130510 2024-04-26T21:47:50.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.975+0000 7f6bb1200640 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562c8c130510 0x562c8c1329a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:50.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.975+0000 7f6bb1200640 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562c8c130510 0x562c8c1329a0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=281d54dd1ae84b54 server_cookie=c067066f5615feee in_seq=0 out_seq=0 2024-04-26T21:47:50.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.975+0000 7f6b93400640 10 client.26845 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.978+0000 7f6b93400640 1 -- 172.21.15.1:0/380430196 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562c8c07ff90 con 0x562c8c130510 2024-04-26T21:47:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.978+0000 7f6b93400640 10 client.26845 handle_command_reply: tid=0 2024-04-26T21:47:50.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.978+0000 7f6bb759d740 1 client.26845 shutdown 2024-04-26T21:47:50.983 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.982+0000 7f6bb759d740 1 -- 172.21.15.1:0/380430196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1155e0 msgr2=0x562c8c11d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:50.983 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.982+0000 7f6bb759d740 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1155e0 0x562c8c11d700 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0005c40 tx=0x7f6ba00bc910 comp rx=0 tx=0).stop 2024-04-26T21:47:50.983 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.982+0000 7f6bb759d740 1 -- 172.21.15.1:0/380430196 shutdown_connections 2024-04-26T21:47:50.983 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.982+0000 7f6bb759d740 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562c8c130510 0x562c8c1329a0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.982+0000 7f6bb759d740 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 0x562c8c11e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.982+0000 7f6bb759d740 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c115f00 0x562c8c11dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.982+0000 7f6bb759d740 1 --2- 172.21.15.1:0/380430196 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c1155e0 0x562c8c11d700 secure :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0005c40 tx=0x7f6ba00bc910 comp rx=0 tx=0).stop 2024-04-26T21:47:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.982+0000 7f6bb759d740 1 -- 172.21.15.1:0/380430196 >> 172.21.15.1:0/380430196 conn(0x562c8c111af0 msgr2=0x562c8c117870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.983+0000 7f6b93400640 0 client.26845 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.983+0000 7f6bb759d740 1 -- 172.21.15.1:0/380430196 shutdown_connections 2024-04-26T21:47:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.983+0000 7f6bb759d740 1 -- 172.21.15.1:0/380430196 wait complete. 2024-04-26T21:47:50.985 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:50.984+0000 7f6bb759d740 20 client.26845 trim_cache size 0 max 16384 2024-04-26T21:47:51.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.020+0000 7f6bb759d740 1 Processor -- start 2024-04-26T21:47:51.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.023+0000 7f6bb759d740 1 -- start start 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 0x562c8c11fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 0x562c8c12c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c11c860 0x562c8c1382b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb759d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1b0610 con 0x562c8c11c860 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb759d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1adee0 con 0x562c8c1207c0 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb759d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1abd50 con 0x562c8c115f00 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 0x562c8c12c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 0x562c8c12c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59130/0 (socket says 172.21.15.1:59130) 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb1200640 1 -- 172.21.15.1:0/2240411781 learned_addr learned my addr 172.21.15.1:0/2240411781 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb1200640 1 -- 172.21.15.1:0/2240411781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 msgr2=0x562c8c11fe30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb1200640 1 --2- 172.21.15.1:0/2240411781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 0x562c8c11fe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb1200640 1 -- 172.21.15.1:0/2240411781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c11c860 msgr2=0x562c8c1382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb1200640 1 --2- 172.21.15.1:0/2240411781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c11c860 0x562c8c1382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.024+0000 7f6bb1200640 1 -- 172.21.15.1:0/2240411781 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c8be50620 con 0x562c8c1207c0 2024-04-26T21:47:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.025+0000 7f6bb1200640 1 --2- 172.21.15.1:0/2240411781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 0x562c8c12c2b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800a1a0 tx=0x7f6b98012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbdcaa94b9165bf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.025+0000 7f6b9e000640 1 -- 172.21.15.1:0/2240411781 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b980282a0 con 0x562c8c1207c0 2024-04-26T21:47:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.025+0000 7f6b9e000640 1 -- 172.21.15.1:0/2240411781 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b980288f0 con 0x562c8c1207c0 2024-04-26T21:47:51.028 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.025+0000 7f6b9e000640 1 -- 172.21.15.1:0/2240411781 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b98021410 con 0x562c8c1207c0 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.026+0000 7f6bb759d740 1 -- 172.21.15.1:0/2240411781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 msgr2=0x562c8c12c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.026+0000 7f6bb759d740 1 --2- 172.21.15.1:0/2240411781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 0x562c8c12c2b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800a1a0 tx=0x7f6b98012000 comp rx=0 tx=0).stop 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.026+0000 7f6bb759d740 1 -- 172.21.15.1:0/2240411781 shutdown_connections 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.026+0000 7f6bb759d740 1 --2- 172.21.15.1:0/2240411781 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c11c860 0x562c8c1382b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.026+0000 7f6bb759d740 1 --2- 172.21.15.1:0/2240411781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c1207c0 0x562c8c12c2b0 secure :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800a1a0 tx=0x7f6b98012000 comp rx=0 tx=0).stop 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.026+0000 7f6bb759d740 1 --2- 172.21.15.1:0/2240411781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 0x562c8c11fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.026+0000 7f6bb759d740 1 -- 172.21.15.1:0/2240411781 >> 172.21.15.1:0/2240411781 conn(0x562c8c111af0 msgr2=0x562c8c111ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.027+0000 7f6bb759d740 1 -- 172.21.15.1:0/2240411781 shutdown_connections 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.027+0000 7f6bb759d740 1 -- 172.21.15.1:0/2240411781 wait complete. 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.027+0000 7f6bb759d740 1 -- start start 2024-04-26T21:47:51.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.028+0000 7f6bb759d740 1 Processor -- start 2024-04-26T21:47:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.028+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 0x562c8c11a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.028+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c11c860 0x562c8c11aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.028+0000 7f6bb759d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c11af50 0x562c8c195b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.028+0000 7f6bb759d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1ac3a0 con 0x562c8c11c860 2024-04-26T21:47:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.028+0000 7f6bb759d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1ad810 con 0x562c8c11af50 2024-04-26T21:47:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.028+0000 7f6bb759d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6bac1aca70 con 0x562c8c115f00 2024-04-26T21:47:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.028+0000 7f6bb759d740 10 client.0 _ll_register_callbacks cb 0x7f6ba0003f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:47:51.030 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.029+0000 7f6bb1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 0x562c8c11a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.029+0000 7f6bb1c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 0x562c8c11a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:48148/0 (socket says 172.21.15.1:48148) 2024-04-26T21:47:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.029+0000 7f6bb1c00640 1 -- 172.21.15.1:0/2310692416 learned_addr learned my addr 172.21.15.1:0/2310692416 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:47:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.029+0000 7f6bb2600640 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c11af50 0x562c8c195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.029+0000 7f6bb1200640 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c11c860 0x562c8c11aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.030+0000 7f6bb1c00640 1 -- 172.21.15.1:0/2310692416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c11af50 msgr2=0x562c8c195b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.030+0000 7f6bb1c00640 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c11af50 0x562c8c195b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.030+0000 7f6bb1c00640 1 -- 172.21.15.1:0/2310692416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c11c860 msgr2=0x562c8c11aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:47:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.030+0000 7f6bb1c00640 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c11c860 0x562c8c11aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.030+0000 7f6bb1c00640 1 -- 172.21.15.1:0/2310692416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c8c02ccb0 con 0x562c8c115f00 2024-04-26T21:47:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.030+0000 7f6bb1c00640 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 0x562c8c11a4d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00eed60 tx=0x7f6ba0031f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e19124a8f9ba8bb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:47:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.031+0000 7f6b93400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:47:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.031+0000 7f6b93400640 1 -- 172.21.15.1:0/2310692416 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ba00d5040 con 0x562c8c115f00 2024-04-26T21:47:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.031+0000 7f6b93400640 1 -- 172.21.15.1:0/2310692416 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7f6ba00d8cf0 con 0x562c8c115f00 2024-04-26T21:47:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.031+0000 7f6bb759d740 1 -- 172.21.15.1:0/2310692416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b980288f0 con 0x562c8c115f00 2024-04-26T21:47:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.032+0000 7f6b93400640 1 -- 172.21.15.1:0/2310692416 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7f6ba00dd040 con 0x562c8c115f00 2024-04-26T21:47:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.032+0000 7f6bb759d740 10 client.26848 fetch_fsmap learned FSMap version 53 2024-04-26T21:47:51.033 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.032+0000 7f6bb759d740 1 -- 172.21.15.1:0/2310692416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x562c8c02ccb0 con 0x562c8c115f00 2024-04-26T21:47:51.034 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.033+0000 7f6b93400640 1 -- 172.21.15.1:0/2310692416 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b980288f0 con 0x562c8c115f00 2024-04-26T21:47:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.035+0000 7f6b93400640 1 -- 172.21.15.1:0/2310692416 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7f6ba00318b0 con 0x562c8c115f00 2024-04-26T21:47:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.035+0000 7f6bb759d740 10 client.26848 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T21:47:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.035+0000 7f6bb759d740 10 client.26848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:47:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.035+0000 7f6bb759d740 20 client.26848 populate_metadata read hostname 'smithi001' 2024-04-26T21:47:51.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.035+0000 7f6bb759d740 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562c8c1a8520 0x562c8c1aa940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:47:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.035+0000 7f6bb759d740 4 client.26848 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:47:51.038 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.037+0000 7f6bb759d740 1 -- 172.21.15.1:0/2310692416 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c8c079e00 con 0x562c8c1a8520 2024-04-26T21:47:51.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.038+0000 7f6bb1200640 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562c8c1a8520 0x562c8c1aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:47:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.041+0000 7f6bb1200640 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562c8c1a8520 0x562c8c1aa940 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5263e6486fb90591 server_cookie=4ca3a99ae71aad04 in_seq=0 out_seq=0 2024-04-26T21:47:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.041+0000 7f6b93400640 10 client.26848 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:51.158 INFO:tasks.workunit.client.0.smithi001.stdout: 10 102007 23.44 MB/sec execute 108 sec latency 290.344 ms 2024-04-26T21:47:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.309+0000 7f6b93400640 1 -- 172.21.15.1:0/2310692416 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c8c079e00 con 0x562c8c1a8520 2024-04-26T21:47:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.310+0000 7f6b93400640 10 client.26848 handle_command_reply: tid=0 2024-04-26T21:47:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.310+0000 7f6bb759d740 1 client.26848 shutdown 2024-04-26T21:47:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.310+0000 7f6bb759d740 1 -- 172.21.15.1:0/2310692416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 msgr2=0x562c8c11a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.310+0000 7f6bb759d740 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 0x562c8c11a4d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00eed60 tx=0x7f6ba0031f10 comp rx=0 tx=0).stop 2024-04-26T21:47:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.310+0000 7f6bb759d740 1 -- 172.21.15.1:0/2310692416 shutdown_connections 2024-04-26T21:47:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.310+0000 7f6bb759d740 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562c8c1a8520 0x562c8c1aa940 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.312 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.311+0000 7f6b93400640 0 client.26848 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:47:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.311+0000 7f6bb759d740 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562c8c11af50 0x562c8c195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.311+0000 7f6bb759d740 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562c8c11c860 0x562c8c11aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.311+0000 7f6bb759d740 1 --2- 172.21.15.1:0/2310692416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562c8c115f00 0x562c8c11a4d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.311+0000 7f6bb759d740 1 -- 172.21.15.1:0/2310692416 >> 172.21.15.1:0/2310692416 conn(0x562c8c111af0 msgr2=0x562c8c11d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.311+0000 7f6bb759d740 1 -- 172.21.15.1:0/2310692416 shutdown_connections 2024-04-26T21:47:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.311+0000 7f6bb759d740 1 -- 172.21.15.1:0/2310692416 wait complete. 2024-04-26T21:47:51.313 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.311+0000 7f6bb759d740 20 client.26848 trim_cache size 0 max 16384 2024-04-26T21:47:51.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:47:51.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:47:51.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:47:51.314 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:47:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.315+0000 7f6bb4600640 1 -- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f6b940a3dc0 msgr2=0x7f6b940a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.315+0000 7f6bb4600640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f6b940a3dc0 0x7f6b940a6280 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6b98006fd0 tx=0x7f6b98008040 comp rx=0 tx=0).stop 2024-04-26T21:47:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.315+0000 7f6bb4600640 1 -- 172.21.15.1:0/1949207371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac103bc0 msgr2=0x7f6bac101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:47:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.315+0000 7f6bb4600640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac103bc0 0x7f6bac101810 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0005ab0 tx=0x7f6ba0003c80 comp rx=0 tx=0).stop 2024-04-26T21:47:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.315+0000 7f6bb4600640 1 -- 172.21.15.1:0/1949207371 shutdown_connections 2024-04-26T21:47:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.315+0000 7f6bb4600640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f6b940a3dc0 0x7f6b940a6280 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.315+0000 7f6bb4600640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6bac10cf90 0x7f6bac102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.315+0000 7f6bb4600640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bac1044e0 0x7f6bac101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.315+0000 7f6bb4600640 1 --2- 172.21.15.1:0/1949207371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac103bc0 0x7f6bac101810 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:47:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.315+0000 7f6bb4600640 1 -- 172.21.15.1:0/1949207371 >> 172.21.15.1:0/1949207371 conn(0x7f6bac0789b0 msgr2=0x7f6bac0fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:47:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.316+0000 7f6bb4600640 1 -- 172.21.15.1:0/1949207371 shutdown_connections 2024-04-26T21:47:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:47:51.316+0000 7f6bb4600640 1 -- 172.21.15.1:0/1949207371 wait complete. 2024-04-26T21:47:52.158 INFO:tasks.workunit.client.0.smithi001.stdout: 10 102280 23.39 MB/sec execute 109 sec latency 138.238 ms 2024-04-26T21:47:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:52 smithi001 ceph-mon[28054]: pgmap v553: 129 pgs: 129 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 17 MiB/s wr, 583 op/s 2024-04-26T21:47:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:52 smithi038 ceph-mon[32724]: pgmap v553: 129 pgs: 129 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 17 MiB/s wr, 583 op/s 2024-04-26T21:47:52.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:52 smithi186 ceph-mon[32991]: pgmap v553: 129 pgs: 129 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 17 MiB/s wr, 583 op/s 2024-04-26T21:47:53.158 INFO:tasks.workunit.client.0.smithi001.stdout: 10 102569 23.30 MB/sec execute 110 sec latency 92.244 ms 2024-04-26T21:47:54.158 INFO:tasks.workunit.client.0.smithi001.stdout: 10 102947 23.29 MB/sec execute 111 sec latency 153.172 ms 2024-04-26T21:47:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:54 smithi001 ceph-mon[28054]: pgmap v554: 129 pgs: 129 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 15 MiB/s wr, 423 op/s 2024-04-26T21:47:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:54 smithi038 ceph-mon[32724]: pgmap v554: 129 pgs: 129 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 15 MiB/s wr, 423 op/s 2024-04-26T21:47:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:54 smithi186 ceph-mon[32991]: pgmap v554: 129 pgs: 129 active+clean; 1.3 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 15 MiB/s wr, 423 op/s 2024-04-26T21:47:55.158 INFO:tasks.workunit.client.0.smithi001.stdout: 10 103491 23.42 MB/sec execute 112 sec latency 89.018 ms 2024-04-26T21:47:56.158 INFO:tasks.workunit.client.0.smithi001.stdout: 10 104117 23.40 MB/sec execute 113 sec latency 53.958 ms 2024-04-26T21:47:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:56 smithi001 ceph-mon[28054]: pgmap v555: 129 pgs: 129 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 17 MiB/s wr, 711 op/s 2024-04-26T21:47:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:56 smithi038 ceph-mon[32724]: pgmap v555: 129 pgs: 129 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 17 MiB/s wr, 711 op/s 2024-04-26T21:47:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:56 smithi186 ceph-mon[32991]: pgmap v555: 129 pgs: 129 active+clean; 1.2 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 17 MiB/s wr, 711 op/s 2024-04-26T21:47:57.158 INFO:tasks.workunit.client.0.smithi001.stdout: 10 104649 23.51 MB/sec execute 114 sec latency 72.179 ms 2024-04-26T21:47:58.158 INFO:tasks.workunit.client.0.smithi001.stdout: 10 105245 23.61 MB/sec execute 115 sec latency 77.788 ms 2024-04-26T21:47:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:47:58 smithi001 ceph-mon[28054]: pgmap v556: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 16 MiB/s wr, 678 op/s 2024-04-26T21:47:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:47:58 smithi038 ceph-mon[32724]: pgmap v556: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 16 MiB/s wr, 678 op/s 2024-04-26T21:47:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:47:58 smithi186 ceph-mon[32991]: pgmap v556: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 16 MiB/s wr, 678 op/s 2024-04-26T21:47:59.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 105787 23.57 MB/sec execute 116 sec latency 65.743 ms 2024-04-26T21:48:00.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 106220 23.58 MB/sec execute 117 sec latency 346.541 ms 2024-04-26T21:48:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:00 smithi001 ceph-mon[28054]: pgmap v557: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 15 MiB/s wr, 638 op/s 2024-04-26T21:48:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:00 smithi038 ceph-mon[32724]: pgmap v557: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 15 MiB/s wr, 638 op/s 2024-04-26T21:48:00.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:00 smithi186 ceph-mon[32991]: pgmap v557: 129 pgs: 129 active+clean; 1.1 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 15 MiB/s wr, 638 op/s 2024-04-26T21:48:01.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 106719 23.61 MB/sec execute 118 sec latency 344.723 ms 2024-04-26T21:48:02.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 107026 23.54 MB/sec execute 119 sec latency 482.913 ms 2024-04-26T21:48:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:02 smithi001 ceph-mon[28054]: pgmap v558: 129 pgs: 129 active+clean; 1.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 17 MiB/s wr, 790 op/s 2024-04-26T21:48:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:02 smithi038 ceph-mon[32724]: pgmap v558: 129 pgs: 129 active+clean; 1.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 17 MiB/s wr, 790 op/s 2024-04-26T21:48:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:02 smithi186 ceph-mon[32991]: pgmap v558: 129 pgs: 129 active+clean; 1.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 17 MiB/s wr, 790 op/s 2024-04-26T21:48:03.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 107053 23.40 MB/sec execute 120 sec latency 502.068 ms 2024-04-26T21:48:03.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:03 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:48:03.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:03 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:48:03.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:03 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:48:04.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 107165 23.25 MB/sec execute 121 sec latency 1278.858 ms 2024-04-26T21:48:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:04 smithi001 ceph-mon[28054]: pgmap v559: 129 pgs: 129 active+clean; 1.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.5 MiB/s wr, 512 op/s 2024-04-26T21:48:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:04 smithi038 ceph-mon[32724]: pgmap v559: 129 pgs: 129 active+clean; 1.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.5 MiB/s wr, 512 op/s 2024-04-26T21:48:04.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:04 smithi186 ceph-mon[32991]: pgmap v559: 129 pgs: 129 active+clean; 1.2 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.5 MiB/s wr, 512 op/s 2024-04-26T21:48:04.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.840+0000 7fe06fe00640 1 -- 172.21.15.1:0/3913352793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe060098b40 msgr2=0x7fe06009cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:04.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.840+0000 7fe06fe00640 1 --2- 172.21.15.1:0/3913352793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe060098b40 0x7fe06009cb80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe054009900 tx=0x7fe05402a3f0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.842+0000 7fe06fe00640 1 -- 172.21.15.1:0/3913352793 shutdown_connections 2024-04-26T21:48:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.842+0000 7fe06fe00640 1 --2- 172.21.15.1:0/3913352793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe060098b40 0x7fe06009cb80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.842+0000 7fe06fe00640 1 --2- 172.21.15.1:0/3913352793 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe06009d260 0x7fe0600a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.842+0000 7fe06fe00640 1 --2- 172.21.15.1:0/3913352793 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe060098190 0x7fe060098570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.842+0000 7fe06fe00640 1 -- 172.21.15.1:0/3913352793 >> 172.21.15.1:0/3913352793 conn(0x7fe0600939f0 msgr2=0x7fe060095e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.842+0000 7fe06fe00640 1 -- 172.21.15.1:0/3913352793 shutdown_connections 2024-04-26T21:48:04.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.844+0000 7fe06fe00640 1 -- 172.21.15.1:0/3913352793 wait complete. 2024-04-26T21:48:04.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.846+0000 7fe06fe00640 1 Processor -- start 2024-04-26T21:48:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.846+0000 7fe06fe00640 1 -- start start 2024-04-26T21:48:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.847+0000 7fe06fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe060098190 0x7fe0600090a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.847+0000 7fe06fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe06009d260 0x7fe0600095e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.847+0000 7fe06fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe060009b20 0x7fe06014fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.847+0000 7fe06fe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe0600b91e0 con 0x7fe060098190 2024-04-26T21:48:04.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.847+0000 7fe06fe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe0600b9060 con 0x7fe060009b20 2024-04-26T21:48:04.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.847+0000 7fe06fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe0600b9360 con 0x7fe06009d260 2024-04-26T21:48:04.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.847+0000 7fe06de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe060009b20 0x7fe06014fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.848+0000 7fe06de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe060009b20 0x7fe06014fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43352/0 (socket says 172.21.15.1:43352) 2024-04-26T21:48:04.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.848+0000 7fe06de00640 1 -- 172.21.15.1:0/1564143160 learned_addr learned my addr 172.21.15.1:0/1564143160 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:04.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.848+0000 7fe06ca00640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe06009d260 0x7fe0600095e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.848+0000 7fe06d400640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe060098190 0x7fe0600090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.849+0000 7fe06de00640 1 -- 172.21.15.1:0/1564143160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe06009d260 msgr2=0x7fe0600095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:04.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.849+0000 7fe06de00640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe06009d260 0x7fe0600095e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.849+0000 7fe06de00640 1 -- 172.21.15.1:0/1564143160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe060098190 msgr2=0x7fe0600090a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:04.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.849+0000 7fe06de00640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe060098190 0x7fe0600090a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.849+0000 7fe06de00640 1 -- 172.21.15.1:0/1564143160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe06013f350 con 0x7fe060009b20 2024-04-26T21:48:04.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.849+0000 7fe06de00640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe060009b20 0x7fe06014fbb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe0680687b0 tx=0x7fe06807a630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:04.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.849+0000 7fe066000640 1 -- 172.21.15.1:0/1564143160 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06806b110 con 0x7fe060009b20 2024-04-26T21:48:04.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.850+0000 7fe066000640 1 -- 172.21.15.1:0/1564143160 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe06806b2b0 con 0x7fe060009b20 2024-04-26T21:48:04.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.850+0000 7fe066000640 1 -- 172.21.15.1:0/1564143160 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06806b110 con 0x7fe060009b20 2024-04-26T21:48:04.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.850+0000 7fe06fe00640 1 -- 172.21.15.1:0/1564143160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe06013f640 con 0x7fe060009b20 2024-04-26T21:48:04.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.851+0000 7fe06fe00640 1 -- 172.21.15.1:0/1564143160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe06013fb10 con 0x7fe060009b20 2024-04-26T21:48:04.859 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.858+0000 7fe066000640 1 -- 172.21.15.1:0/1564143160 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fe068069470 con 0x7fe060009b20 2024-04-26T21:48:04.862 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.861+0000 7fe066000640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe0500a4020 0x7fe0500a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.863 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.862+0000 7fe06d400640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe0500a4020 0x7fe0500a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.868 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.864+0000 7fe066000640 1 -- 172.21.15.1:0/1564143160 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7fe068078040 con 0x7fe060009b20 2024-04-26T21:48:04.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.869+0000 7fe072bf1740 1 Processor -- start 2024-04-26T21:48:04.870 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.869+0000 7fe072bf1740 1 -- start start 2024-04-26T21:48:04.871 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.869+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c3fc0 0x5647c52c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.870+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52cec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.870+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52cf1a0 0x5647c52d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.870+0000 7fe072bf1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5647c5229380 con 0x5647c52cf1a0 2024-04-26T21:48:04.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.870+0000 7fe072bf1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5647c51c5c30 con 0x5647c52c3fc0 2024-04-26T21:48:04.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.870+0000 7fe072bf1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5647c5225340 con 0x5647c52c48e0 2024-04-26T21:48:04.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.870+0000 7fe06ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.870+0000 7fe06d400640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe0500a4020 0x7fe0500a64e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0095f0 tx=0x7fe05c005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:04.872 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.870+0000 7fe06ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58022/0 (socket says 172.21.15.1:58022) 2024-04-26T21:48:04.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.870+0000 7fe06ca00640 1 -- 172.21.15.1:0/1055686641 learned_addr learned my addr 172.21.15.1:0/1055686641 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:04.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.871+0000 7fe06ca00640 1 -- 172.21.15.1:0/1055686641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c3fc0 msgr2=0x5647c52c43a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:48:04.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.871+0000 7fe06ca00640 1 --2- 172.21.15.1:0/1055686641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c3fc0 0x5647c52c43a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.871+0000 7fe06ca00640 1 -- 172.21.15.1:0/1055686641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52cf1a0 msgr2=0x5647c52d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:04.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.871+0000 7fe06ca00640 1 --2- 172.21.15.1:0/1055686641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52cf1a0 0x5647c52d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.871+0000 7fe06ca00640 1 -- 172.21.15.1:0/1055686641 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647c51e7940 con 0x5647c52c48e0 2024-04-26T21:48:04.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.871+0000 7fe06ca00640 1 --2- 172.21.15.1:0/1055686641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52cec60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe060005b50 tx=0x7fe0540050c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b0ca0f82807c9dc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:04.873 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.872+0000 7fe05a600640 1 -- 172.21.15.1:0/1055686641 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054035040 con 0x5647c52c48e0 2024-04-26T21:48:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.872+0000 7fe05a600640 1 -- 172.21.15.1:0/1055686641 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe054004740 con 0x5647c52c48e0 2024-04-26T21:48:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.872+0000 7fe05a600640 1 -- 172.21.15.1:0/1055686641 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054003850 con 0x5647c52c48e0 2024-04-26T21:48:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.873+0000 7fe072bf1740 1 -- 172.21.15.1:0/1055686641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 msgr2=0x5647c52cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.873+0000 7fe072bf1740 1 --2- 172.21.15.1:0/1055686641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52cec60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe060005b50 tx=0x7fe0540050c0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.873+0000 7fe072bf1740 1 -- 172.21.15.1:0/1055686641 shutdown_connections 2024-04-26T21:48:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.873+0000 7fe072bf1740 1 --2- 172.21.15.1:0/1055686641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52cec60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.873+0000 7fe072bf1740 1 --2- 172.21.15.1:0/1055686641 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52cf1a0 0x5647c52d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.873+0000 7fe072bf1740 1 --2- 172.21.15.1:0/1055686641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c3fc0 0x5647c52c43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.873+0000 7fe072bf1740 1 -- 172.21.15.1:0/1055686641 >> 172.21.15.1:0/1055686641 conn(0x5647c52c04d0 msgr2=0x5647c52c08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.873+0000 7fe072bf1740 1 -- 172.21.15.1:0/1055686641 shutdown_connections 2024-04-26T21:48:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.873+0000 7fe072bf1740 1 -- 172.21.15.1:0/1055686641 wait complete. 2024-04-26T21:48:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.874+0000 7fe072bf1740 1 -- start start 2024-04-26T21:48:04.875 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.874+0000 7fe072bf1740 1 Processor -- start 2024-04-26T21:48:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.876+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c3fc0 0x5647c52cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.876+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c48e0 0x5647c52cc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.877+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.877+0000 7fe072bf1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5647c52d1d70 con 0x5647c52c3fc0 2024-04-26T21:48:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.877+0000 7fe072bf1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5647c51b0c60 con 0x5647c52c48e0 2024-04-26T21:48:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.877+0000 7fe072bf1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5647c52d1ef0 con 0x5647c52cf1a0 2024-04-26T21:48:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.877+0000 7fe072bf1740 10 client.0 _ll_register_callbacks cb 0x5647c51dce50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:48:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.877+0000 7fe06de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.877+0000 7fe06de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52ccc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58024/0 (socket says 172.21.15.1:58024) 2024-04-26T21:48:04.879 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.877+0000 7fe06de00640 1 -- 172.21.15.1:0/3651153311 learned_addr learned my addr 172.21.15.1:0/3651153311 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.877+0000 7fe06ca00640 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c48e0 0x5647c52cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.877+0000 7fe06d400640 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c3fc0 0x5647c52cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.878+0000 7fe06de00640 1 -- 172.21.15.1:0/3651153311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c48e0 msgr2=0x5647c52cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.878+0000 7fe06de00640 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c48e0 0x5647c52cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.878+0000 7fe06de00640 1 -- 172.21.15.1:0/3651153311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c3fc0 msgr2=0x5647c52cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.878+0000 7fe06de00640 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c3fc0 0x5647c52cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.878+0000 7fe06de00640 1 -- 172.21.15.1:0/3651153311 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647c51bf510 con 0x5647c52cf1a0 2024-04-26T21:48:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.878+0000 7fe06de00640 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52ccc40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe068000b60 tx=0x7fe0680ffb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e83699731057da8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.879+0000 7fe04ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:48:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.879+0000 7fe04ea00640 1 -- 172.21.15.1:0/3651153311 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe068119060 con 0x5647c52cf1a0 2024-04-26T21:48:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.879+0000 7fe04ea00640 1 -- 172.21.15.1:0/3651153311 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7fe068114be0 con 0x5647c52cf1a0 2024-04-26T21:48:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.879+0000 7fe072bf1740 1 -- 172.21.15.1:0/3651153311 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe054004740 con 0x5647c52cf1a0 2024-04-26T21:48:04.881 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.880+0000 7fe04ea00640 1 -- 172.21.15.1:0/3651153311 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fe068131d80 con 0x5647c52cf1a0 2024-04-26T21:48:04.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.881+0000 7fe072bf1740 10 client.26860 fetch_fsmap learned FSMap version 53 2024-04-26T21:48:04.882 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.881+0000 7fe072bf1740 1 -- 172.21.15.1:0/3651153311 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5647c4f3b210 con 0x5647c52cf1a0 2024-04-26T21:48:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.881+0000 7fe04ea00640 1 -- 172.21.15.1:0/3651153311 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe054004740 con 0x5647c52cf1a0 2024-04-26T21:48:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.881+0000 7fe04ea00640 1 -- 172.21.15.1:0/3651153311 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7fe0680fbf40 con 0x5647c52cf1a0 2024-04-26T21:48:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.882+0000 7fe072bf1740 10 client.26860 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T21:48:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.882+0000 7fe072bf1740 10 client.26860 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:48:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.882+0000 7fe072bf1740 20 client.26860 populate_metadata read hostname 'smithi001' 2024-04-26T21:48:04.883 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.882+0000 7fe072bf1740 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5647c52def60 0x5647c52e1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.882+0000 7fe072bf1740 4 client.26860 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:48:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.882+0000 7fe072bf1740 1 -- 172.21.15.1:0/3651153311 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647c5236f90 con 0x5647c52def60 2024-04-26T21:48:04.884 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.883+0000 7fe06d400640 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5647c52def60 0x5647c52e1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.886 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.885+0000 7fe04ea00640 10 client.26860 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:04.887 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.886+0000 7fe06d400640 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5647c52def60 0x5647c52e1350 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5a4aec6470de6fc server_cookie=85d3fac12899939b in_seq=0 out_seq=0 2024-04-26T21:48:04.892 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.891+0000 7fe04ea00640 1 -- 172.21.15.1:0/3651153311 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5647c5236f90 con 0x5647c52def60 2024-04-26T21:48:04.892 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.891+0000 7fe04ea00640 10 client.26860 handle_command_reply: tid=0 2024-04-26T21:48:04.892 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.891+0000 7fe072bf1740 1 client.26860 shutdown 2024-04-26T21:48:04.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.892+0000 7fe072bf1740 1 -- 172.21.15.1:0/3651153311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 msgr2=0x5647c52ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:04.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.892+0000 7fe072bf1740 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52ccc40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe068000b60 tx=0x7fe0680ffb50 comp rx=0 tx=0).stop 2024-04-26T21:48:04.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.892+0000 7fe072bf1740 1 -- 172.21.15.1:0/3651153311 shutdown_connections 2024-04-26T21:48:04.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.892+0000 7fe072bf1740 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5647c52def60 0x5647c52e1350 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.892+0000 7fe072bf1740 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52ccc40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.893 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.892+0000 7fe072bf1740 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c48e0 0x5647c52cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.892+0000 7fe072bf1740 1 --2- 172.21.15.1:0/3651153311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c3fc0 0x5647c52cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.892+0000 7fe072bf1740 1 -- 172.21.15.1:0/3651153311 >> 172.21.15.1:0/3651153311 conn(0x5647c52c04d0 msgr2=0x5647c52c6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.893+0000 7fe04ea00640 0 client.26860 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:04.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.894+0000 7fe072bf1740 1 -- 172.21.15.1:0/3651153311 shutdown_connections 2024-04-26T21:48:04.896 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.894+0000 7fe072bf1740 1 -- 172.21.15.1:0/3651153311 wait complete. 2024-04-26T21:48:04.896 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.895+0000 7fe072bf1740 20 client.26860 trim_cache size 0 max 16384 2024-04-26T21:48:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.945+0000 7fe072bf1740 1 Processor -- start 2024-04-26T21:48:04.946 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.945+0000 7fe072bf1740 1 -- start start 2024-04-26T21:48:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.946+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c3fc0 0x5647c52dac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.947 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.946+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c48e0 0x5647c52cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.946+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.948 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.946+0000 7fe072bf1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe060165950 con 0x5647c52c48e0 2024-04-26T21:48:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.946+0000 7fe072bf1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe060163220 con 0x5647c52c3fc0 2024-04-26T21:48:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.946+0000 7fe072bf1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe06000a580 con 0x5647c52cf1a0 2024-04-26T21:48:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.947+0000 7fe06de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.947+0000 7fe06de00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52cdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58028/0 (socket says 172.21.15.1:58028) 2024-04-26T21:48:04.949 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.947+0000 7fe06de00640 1 -- 172.21.15.1:0/3452160552 learned_addr learned my addr 172.21.15.1:0/3452160552 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.948+0000 7fe06d400640 1 --2- 172.21.15.1:0/3452160552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c3fc0 0x5647c52dac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.948+0000 7fe06ca00640 1 --2- 172.21.15.1:0/3452160552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c48e0 0x5647c52cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.948+0000 7fe06de00640 1 -- 172.21.15.1:0/3452160552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c3fc0 msgr2=0x5647c52dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.948+0000 7fe06de00640 1 --2- 172.21.15.1:0/3452160552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c3fc0 0x5647c52dac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.948+0000 7fe06de00640 1 -- 172.21.15.1:0/3452160552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c48e0 msgr2=0x5647c52cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.948+0000 7fe06de00640 1 --2- 172.21.15.1:0/3452160552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c48e0 0x5647c52cca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.948+0000 7fe06de00640 1 -- 172.21.15.1:0/3452160552 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647c5007620 con 0x5647c52cf1a0 2024-04-26T21:48:04.950 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.949+0000 7fe06de00640 1 --2- 172.21.15.1:0/3452160552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52cdff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe068051fc0 tx=0x7fe068104670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29a4a27dc8bddeca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.949+0000 7fe059c00640 1 -- 172.21.15.1:0/3452160552 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0680fffc0 con 0x5647c52cf1a0 2024-04-26T21:48:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.949+0000 7fe059c00640 1 -- 172.21.15.1:0/3452160552 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe068108db0 con 0x5647c52cf1a0 2024-04-26T21:48:04.951 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.950+0000 7fe059c00640 1 -- 172.21.15.1:0/3452160552 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0680ff8a0 con 0x5647c52cf1a0 2024-04-26T21:48:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.952+0000 7fe072bf1740 1 -- 172.21.15.1:0/3452160552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 msgr2=0x5647c52cdff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.952+0000 7fe072bf1740 1 --2- 172.21.15.1:0/3452160552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52cdff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe068051fc0 tx=0x7fe068104670 comp rx=0 tx=0).stop 2024-04-26T21:48:04.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.954+0000 7fe072bf1740 1 -- 172.21.15.1:0/3452160552 shutdown_connections 2024-04-26T21:48:04.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.954+0000 7fe072bf1740 1 --2- 172.21.15.1:0/3452160552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52cf1a0 0x5647c52cdff0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.954+0000 7fe072bf1740 1 --2- 172.21.15.1:0/3452160552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c48e0 0x5647c52cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.954+0000 7fe072bf1740 1 --2- 172.21.15.1:0/3452160552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52c3fc0 0x5647c52dac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.954+0000 7fe072bf1740 1 -- 172.21.15.1:0/3452160552 >> 172.21.15.1:0/3452160552 conn(0x5647c52c04d0 msgr2=0x5647c52cd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:04.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.954+0000 7fe072bf1740 1 -- 172.21.15.1:0/3452160552 shutdown_connections 2024-04-26T21:48:04.956 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.955+0000 7fe072bf1740 1 -- 172.21.15.1:0/3452160552 wait complete. 2024-04-26T21:48:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.955+0000 7fe072bf1740 1 -- start start 2024-04-26T21:48:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.955+0000 7fe072bf1740 1 Processor -- start 2024-04-26T21:48:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.958+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c3fc0 0x5647c52d7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.958+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52d76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.958+0000 7fe072bf1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52cf1a0 0x5647c52d2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.958+0000 7fe072bf1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe060147860 con 0x5647c52c3fc0 2024-04-26T21:48:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.958+0000 7fe072bf1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe060162b50 con 0x5647c52cf1a0 2024-04-26T21:48:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.958+0000 7fe072bf1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe060161db0 con 0x5647c52c48e0 2024-04-26T21:48:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.958+0000 7fe06ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52d76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.959+0000 7fe06de00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52cf1a0 0x5647c52d2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.960 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.959+0000 7fe06ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52d76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:58032/0 (socket says 172.21.15.1:58032) 2024-04-26T21:48:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.959+0000 7fe06ca00640 1 -- 172.21.15.1:0/1828121085 learned_addr learned my addr 172.21.15.1:0/1828121085 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.959+0000 7fe072bf1740 10 client.0 _ll_register_callbacks cb 0x7fe06807af20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:48:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.959+0000 7fe06ca00640 1 -- 172.21.15.1:0/1828121085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52cf1a0 msgr2=0x5647c52d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.959+0000 7fe06ca00640 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52cf1a0 0x5647c52d2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.959+0000 7fe06ca00640 1 -- 172.21.15.1:0/1828121085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c3fc0 msgr2=0x5647c52d7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.959+0000 7fe06ca00640 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c3fc0 0x5647c52d7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.959+0000 7fe06ca00640 1 -- 172.21.15.1:0/1828121085 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647c51e3cb0 con 0x5647c52c48e0 2024-04-26T21:48:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.960+0000 7fe06ca00640 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52d76d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe054009900 tx=0x7fe05402ab70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94b24f523a25901 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.960+0000 7fe04ea00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:48:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.961+0000 7fe04ea00640 1 -- 172.21.15.1:0/1828121085 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe054035040 con 0x5647c52c48e0 2024-04-26T21:48:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.961+0000 7fe072bf1740 1 -- 172.21.15.1:0/1828121085 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe068108db0 con 0x5647c52c48e0 2024-04-26T21:48:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.961+0000 7fe04ea00640 1 -- 172.21.15.1:0/1828121085 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(120..120 src has 1..120) ==== 9642+0+0 (secure 0 0 0) 0x7fe054004a10 con 0x5647c52c48e0 2024-04-26T21:48:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.961+0000 7fe04ea00640 1 -- 172.21.15.1:0/1828121085 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=53) ==== 24+0+0 (secure 0 0 0) 0x7fe054058050 con 0x5647c52c48e0 2024-04-26T21:48:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.961+0000 7fe072bf1740 10 client.26866 fetch_fsmap learned FSMap version 53 2024-04-26T21:48:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.961+0000 7fe072bf1740 1 -- 172.21.15.1:0/1828121085 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=53,monmap=4+}) -- 0x5647c5344f90 con 0x5647c52c48e0 2024-04-26T21:48:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.962+0000 7fe04ea00640 1 -- 172.21.15.1:0/1828121085 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe068108db0 con 0x5647c52c48e0 2024-04-26T21:48:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.962+0000 7fe04ea00640 1 -- 172.21.15.1:0/1828121085 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 53) ==== 8346+0+0 (secure 0 0 0) 0x7fe0540524c0 con 0x5647c52c48e0 2024-04-26T21:48:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.963+0000 7fe072bf1740 10 client.26866 fetch_fsmap finished waiting for FSMap version 53 2024-04-26T21:48:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.963+0000 7fe072bf1740 10 client.26866 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:48:04.964 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.963+0000 7fe072bf1740 20 client.26866 populate_metadata read hostname 'smithi001' 2024-04-26T21:48:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.963+0000 7fe072bf1740 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5647c5345280 0x5647c5347670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.963+0000 7fe072bf1740 4 client.26866 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:48:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.963+0000 7fe072bf1740 1 -- 172.21.15.1:0/1828121085 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5647c5230e00 con 0x5647c5345280 2024-04-26T21:48:04.965 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.964+0000 7fe06d400640 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5647c5345280 0x5647c5347670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.967+0000 7fe06d400640 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5647c5345280 0x5647c5347670 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8413752d22983907 server_cookie=4716d42bbb2ce697 in_seq=0 out_seq=0 2024-04-26T21:48:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:04.967+0000 7fe04ea00640 10 client.26866 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:05.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 107483 23.27 MB/sec execute 122 sec latency 196.995 ms 2024-04-26T21:48:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.249+0000 7fe04ea00640 1 -- 172.21.15.1:0/1828121085 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5647c5230e00 con 0x5647c5345280 2024-04-26T21:48:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.249+0000 7fe04ea00640 10 client.26866 handle_command_reply: tid=0 2024-04-26T21:48:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.251+0000 7fe072bf1740 1 client.26866 shutdown 2024-04-26T21:48:05.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 1 -- 172.21.15.1:0/1828121085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 msgr2=0x5647c52d76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:05.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52d76d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe054009900 tx=0x7fe05402ab70 comp rx=0 tx=0).stop 2024-04-26T21:48:05.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 1 -- 172.21.15.1:0/1828121085 shutdown_connections 2024-04-26T21:48:05.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5647c5345280 0x5647c5347670 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:05.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5647c52c48e0 0x5647c52d76d0 secure :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe054009900 tx=0x7fe05402ab70 comp rx=0 tx=0).stop 2024-04-26T21:48:05.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5647c52cf1a0 0x5647c52d2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:05.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 1 --2- 172.21.15.1:0/1828121085 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5647c52c3fc0 0x5647c52d7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:05.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 1 -- 172.21.15.1:0/1828121085 >> 172.21.15.1:0/1828121085 conn(0x5647c52c04d0 msgr2=0x5647c52c6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:05.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe04ea00640 0 client.26866 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:05.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 1 -- 172.21.15.1:0/1828121085 shutdown_connections 2024-04-26T21:48:05.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 1 -- 172.21.15.1:0/1828121085 wait complete. 2024-04-26T21:48:05.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.253+0000 7fe072bf1740 20 client.26866 trim_cache size 0 max 16384 2024-04-26T21:48:05.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:48:05.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:48:05.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:48:05.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:48:05.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.256+0000 7fe06fe00640 1 -- 172.21.15.1:0/1564143160 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe0500a4020 msgr2=0x7fe0500a64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:05.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.256+0000 7fe06fe00640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe0500a4020 0x7fe0500a64e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0095f0 tx=0x7fe05c005f90 comp rx=0 tx=0).stop 2024-04-26T21:48:05.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.256+0000 7fe06fe00640 1 -- 172.21.15.1:0/1564143160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe060009b20 msgr2=0x7fe06014fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:05.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.256+0000 7fe06fe00640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe060009b20 0x7fe06014fbb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe0680687b0 tx=0x7fe06807a630 comp rx=0 tx=0).stop 2024-04-26T21:48:05.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.256+0000 7fe06fe00640 1 -- 172.21.15.1:0/1564143160 shutdown_connections 2024-04-26T21:48:05.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.256+0000 7fe06fe00640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe0500a4020 0x7fe0500a64e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:05.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.256+0000 7fe06fe00640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe06009d260 0x7fe0600095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:05.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.256+0000 7fe06fe00640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe060009b20 0x7fe06014fbb0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:05.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.256+0000 7fe06fe00640 1 --2- 172.21.15.1:0/1564143160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe060098190 0x7fe0600090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:05.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.256+0000 7fe06fe00640 1 -- 172.21.15.1:0/1564143160 >> 172.21.15.1:0/1564143160 conn(0x7fe0600939f0 msgr2=0x7fe060094410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:05.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.257+0000 7fe06fe00640 1 -- 172.21.15.1:0/1564143160 shutdown_connections 2024-04-26T21:48:05.258 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:05.257+0000 7fe06fe00640 1 -- 172.21.15.1:0/1564143160 wait complete. 2024-04-26T21:48:06.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 107925 23.16 MB/sec execute 123 sec latency 101.287 ms 2024-04-26T21:48:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:06 smithi001 ceph-mon[28054]: pgmap v560: 129 pgs: 129 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.9 MiB/s wr, 736 op/s 2024-04-26T21:48:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:06 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:48:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:06 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:48:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:06 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:48:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:06 smithi038 ceph-mon[32724]: pgmap v560: 129 pgs: 129 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.9 MiB/s wr, 736 op/s 2024-04-26T21:48:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:06 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:48:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:06 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:48:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:06 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:48:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:06 smithi186 ceph-mon[32991]: pgmap v560: 129 pgs: 129 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.9 MiB/s wr, 736 op/s 2024-04-26T21:48:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:06 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:48:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:06 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:48:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:06 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:48:07.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 108376 23.21 MB/sec execute 124 sec latency 161.773 ms 2024-04-26T21:48:08.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 108682 23.18 MB/sec execute 125 sec latency 1161.835 ms 2024-04-26T21:48:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:08 smithi001 ceph-mon[28054]: pgmap v561: 129 pgs: 129 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 9.1 MiB/s wr, 517 op/s 2024-04-26T21:48:08.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:08 smithi001 ceph-mon[28054]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T21:48:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:08 smithi038 ceph-mon[32724]: pgmap v561: 129 pgs: 129 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 9.1 MiB/s wr, 517 op/s 2024-04-26T21:48:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:08 smithi038 ceph-mon[32724]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T21:48:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:08 smithi186 ceph-mon[32991]: pgmap v561: 129 pgs: 129 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 9.1 MiB/s wr, 517 op/s 2024-04-26T21:48:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:08 smithi186 ceph-mon[32991]: osdmap e121: 12 total, 12 up, 12 in 2024-04-26T21:48:09.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 108840 23.06 MB/sec execute 126 sec latency 2161.899 ms 2024-04-26T21:48:10.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 108992 22.96 MB/sec execute 127 sec latency 3161.950 ms 2024-04-26T21:48:10.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:10 smithi038 ceph-mon[32724]: pgmap v563: 129 pgs: 129 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 11 MiB/s wr, 534 op/s 2024-04-26T21:48:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:10 smithi186 ceph-mon[32991]: pgmap v563: 129 pgs: 129 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 11 MiB/s wr, 534 op/s 2024-04-26T21:48:10.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:10 smithi001 ceph-mon[28054]: pgmap v563: 129 pgs: 129 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 11 MiB/s wr, 534 op/s 2024-04-26T21:48:11.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 109141 22.87 MB/sec execute 128 sec latency 4162.006 ms 2024-04-26T21:48:12.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 109404 22.74 MB/sec execute 129 sec latency 5162.073 ms 2024-04-26T21:48:12.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:12 smithi038 ceph-mon[32724]: pgmap v564: 129 pgs: 19 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 69 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.9 MiB/s wr, 550 op/s 2024-04-26T21:48:12.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:12 smithi001 ceph-mon[28054]: pgmap v564: 129 pgs: 19 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 69 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.9 MiB/s wr, 550 op/s 2024-04-26T21:48:12.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:12 smithi186 ceph-mon[32991]: pgmap v564: 129 pgs: 19 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 69 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.9 MiB/s wr, 550 op/s 2024-04-26T21:48:13.159 INFO:tasks.workunit.client.0.smithi001.stdout: 10 109698 22.70 MB/sec execute 130 sec latency 6162.131 ms 2024-04-26T21:48:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:13 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:48:13.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:13 smithi186 ceph-mon[32991]: pgmap v565: 129 pgs: 19 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 69 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.9 MiB/s wr, 550 op/s 2024-04-26T21:48:13.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:13 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:48:13.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:13 smithi001 ceph-mon[28054]: pgmap v565: 129 pgs: 19 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 69 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.9 MiB/s wr, 550 op/s 2024-04-26T21:48:13.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:13 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:48:13.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:13 smithi038 ceph-mon[32724]: pgmap v565: 129 pgs: 19 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 69 active+clean; 1.2 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 9.9 MiB/s wr, 550 op/s 2024-04-26T21:48:14.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 110196 22.71 MB/sec execute 131 sec latency 7162.185 ms 2024-04-26T21:48:15.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 110693 22.75 MB/sec execute 132 sec latency 8162.241 ms 2024-04-26T21:48:16.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 111123 22.81 MB/sec execute 133 sec latency 9162.302 ms 2024-04-26T21:48:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:16 smithi186 ceph-mon[32991]: pgmap v566: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 4.4 MiB/s wr, 356 op/s 2024-04-26T21:48:16.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:16 smithi001 ceph-mon[28054]: pgmap v566: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 4.4 MiB/s wr, 356 op/s 2024-04-26T21:48:16.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:16 smithi038 ceph-mon[32724]: pgmap v566: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 4.4 MiB/s wr, 356 op/s 2024-04-26T21:48:17.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 111447 22.72 MB/sec execute 134 sec latency 9509.095 ms 2024-04-26T21:48:18.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 111840 22.72 MB/sec execute 135 sec latency 9076.156 ms 2024-04-26T21:48:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:18 smithi186 ceph-mon[32991]: pgmap v567: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 4.0 MiB/s wr, 347 op/s 2024-04-26T21:48:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:18 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:48:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.479+0000 7fd51ec00640 1 -- 172.21.15.1:0/115584886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102920 msgr2=0x7fd518102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.479+0000 7fd51ec00640 1 --2- 172.21.15.1:0/115584886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102920 0x7fd518102da0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd504009900 tx=0x7fd50402e800 comp rx=0 tx=0).stop 2024-04-26T21:48:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.479+0000 7fd51ec00640 1 -- 172.21.15.1:0/115584886 shutdown_connections 2024-04-26T21:48:18.481 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.479+0000 7fd51ec00640 1 --2- 172.21.15.1:0/115584886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd5181032e0 0x7fd51810f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.479+0000 7fd51ec00640 1 --2- 172.21.15.1:0/115584886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518102920 0x7fd518102da0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.479+0000 7fd51ec00640 1 --2- 172.21.15.1:0/115584886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd518108930 0x7fd518108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.479+0000 7fd51ec00640 1 -- 172.21.15.1:0/115584886 >> 172.21.15.1:0/115584886 conn(0x7fd5180fe3f0 msgr2=0x7fd518100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.479+0000 7fd51ec00640 1 -- 172.21.15.1:0/115584886 shutdown_connections 2024-04-26T21:48:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.480+0000 7fd51ec00640 1 -- 172.21.15.1:0/115584886 wait complete. 2024-04-26T21:48:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.481+0000 7fd51ec00640 1 Processor -- start 2024-04-26T21:48:18.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.481+0000 7fd51ec00640 1 -- start start 2024-04-26T21:48:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.482+0000 7fd51ec00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd518102920 0x7fd51819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.482+0000 7fd51ec00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5181032e0 0x7fd518195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.482+0000 7fd51ec00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518108930 0x7fd5181964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.482+0000 7fd51ec00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd518110a30 con 0x7fd518102920 2024-04-26T21:48:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.482+0000 7fd51ec00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181108b0 con 0x7fd518108930 2024-04-26T21:48:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.482+0000 7fd51ec00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd518110bb0 con 0x7fd5181032e0 2024-04-26T21:48:18.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.482+0000 7fd517400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5181032e0 0x7fd518195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.482+0000 7fd517400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5181032e0 0x7fd518195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:49188/0 (socket says 172.21.15.1:49188) 2024-04-26T21:48:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.482+0000 7fd517400640 1 -- 172.21.15.1:0/4011613868 learned_addr learned my addr 172.21.15.1:0/4011613868 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.483+0000 7fd51cc00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518108930 0x7fd5181964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.483+0000 7fd517e00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd518102920 0x7fd51819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.483+0000 7fd517400640 1 -- 172.21.15.1:0/4011613868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518108930 msgr2=0x7fd5181964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.483+0000 7fd517400640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518108930 0x7fd5181964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.483+0000 7fd517400640 1 -- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd518102920 msgr2=0x7fd51819dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.483+0000 7fd517400640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd518102920 0x7fd51819dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.483+0000 7fd517400640 1 -- 172.21.15.1:0/4011613868 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd518196cd0 con 0x7fd5181032e0 2024-04-26T21:48:18.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.483+0000 7fd517e00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd518102920 0x7fd51819dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:48:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.483+0000 7fd517400640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5181032e0 0x7fd518195f70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd50402edf0 tx=0x7fd504003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.484+0000 7fd514c00640 1 -- 172.21.15.1:0/4011613868 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50403a030 con 0x7fd5181032e0 2024-04-26T21:48:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.484+0000 7fd514c00640 1 -- 172.21.15.1:0/4011613868 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd504036ad0 con 0x7fd5181032e0 2024-04-26T21:48:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.484+0000 7fd514c00640 1 -- 172.21.15.1:0/4011613868 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50403e9d0 con 0x7fd5181032e0 2024-04-26T21:48:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.484+0000 7fd51ec00640 1 -- 172.21.15.1:0/4011613868 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5181b44c0 con 0x7fd5181032e0 2024-04-26T21:48:18.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.484+0000 7fd51ec00640 1 -- 172.21.15.1:0/4011613868 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5181b4960 con 0x7fd5181032e0 2024-04-26T21:48:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.488+0000 7fd5219df740 1 Processor -- start 2024-04-26T21:48:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.489+0000 7fd5219df740 1 -- start start 2024-04-26T21:48:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.489+0000 7fd5219df740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34be81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.489+0000 7fd5219df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34bf2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.489+0000 7fd5219df740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 0x55be34bf53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.489+0000 7fd5219df740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55be34b4d380 con 0x55be34bf2fb0 2024-04-26T21:48:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.489+0000 7fd5219df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55be34ae9c30 con 0x55be34be86f0 2024-04-26T21:48:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.489+0000 7fd5219df740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55be34b49340 con 0x55be34be7dd0 2024-04-26T21:48:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.489+0000 7fd514c00640 1 -- 172.21.15.1:0/4011613868 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd504036620 con 0x7fd5181032e0 2024-04-26T21:48:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.490+0000 7fd51cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34be81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.490+0000 7fd51cc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34be81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:49200/0 (socket says 172.21.15.1:49200) 2024-04-26T21:48:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.490+0000 7fd517400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 0x55be34bf53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.490+0000 7fd51cc00640 1 -- 172.21.15.1:0/1371845811 learned_addr learned my addr 172.21.15.1:0/1371845811 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.490+0000 7fd517e00640 1 --2- 172.21.15.1:0/1371845811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34bf2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.491+0000 7fd51cc00640 1 -- 172.21.15.1:0/1371845811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 msgr2=0x55be34bf2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.491+0000 7fd51cc00640 1 --2- 172.21.15.1:0/1371845811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34bf2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.491+0000 7fd51cc00640 1 -- 172.21.15.1:0/1371845811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 msgr2=0x55be34bf53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.491+0000 7fd51cc00640 1 --2- 172.21.15.1:0/1371845811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 0x55be34bf53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.491+0000 7fd51cc00640 1 -- 172.21.15.1:0/1371845811 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be34b0b940 con 0x55be34be7dd0 2024-04-26T21:48:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.491+0000 7fd517400640 1 --2- 172.21.15.1:0/1371845811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 0x55be34bf53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:48:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.492+0000 7fd51cc00640 1 --2- 172.21.15.1:0/1371845811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34be81b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd510009ee0 tx=0x7fd510010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ce38bffca6ad749 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.492+0000 7fd514c00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4fc0a3d90 0x7fd4fc0a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.492+0000 7fd50d200640 1 -- 172.21.15.1:0/1371845811 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd510016070 con 0x55be34be7dd0 2024-04-26T21:48:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.492+0000 7fd514c00640 1 -- 172.21.15.1:0/4011613868 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 9730+0+0 (secure 0 0 0) 0x7fd5040f1170 con 0x7fd5181032e0 2024-04-26T21:48:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.492+0000 7fd50d200640 1 -- 172.21.15.1:0/1371845811 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd510004d20 con 0x55be34be7dd0 2024-04-26T21:48:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.492+0000 7fd50d200640 1 -- 172.21.15.1:0/1371845811 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5100126b0 con 0x55be34be7dd0 2024-04-26T21:48:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.492+0000 7fd517e00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4fc0a3d90 0x7fd4fc0a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.493+0000 7fd5219df740 1 -- 172.21.15.1:0/1371845811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 msgr2=0x55be34be81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.493+0000 7fd5219df740 1 --2- 172.21.15.1:0/1371845811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34be81b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd510009ee0 tx=0x7fd510010740 comp rx=0 tx=0).stop 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.493+0000 7fd5219df740 1 -- 172.21.15.1:0/1371845811 shutdown_connections 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.493+0000 7fd5219df740 1 --2- 172.21.15.1:0/1371845811 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 0x55be34bf53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.493+0000 7fd5219df740 1 --2- 172.21.15.1:0/1371845811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34bf2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.493+0000 7fd517e00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4fc0a3d90 0x7fd4fc0a6250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x55be34be9370 tx=0x7fd508005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.493+0000 7fd5219df740 1 --2- 172.21.15.1:0/1371845811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34be81b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.493+0000 7fd5219df740 1 -- 172.21.15.1:0/1371845811 >> 172.21.15.1:0/1371845811 conn(0x55be34be42e0 msgr2=0x55be34be46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.494+0000 7fd5219df740 1 -- 172.21.15.1:0/1371845811 shutdown_connections 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.494+0000 7fd5219df740 1 -- 172.21.15.1:0/1371845811 wait complete. 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.495+0000 7fd5219df740 1 -- start start 2024-04-26T21:48:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.495+0000 7fd5219df740 1 Processor -- start 2024-04-26T21:48:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd5219df740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be7dd0 0x55be34befe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd5219df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34bf03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd5219df740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34bf2fb0 0x55be34bf08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd5219df740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55be34bf5be0 con 0x55be34be7dd0 2024-04-26T21:48:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd5219df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55be34ad4c60 con 0x55be34be86f0 2024-04-26T21:48:18.498 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd5219df740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55be34bf5d60 con 0x55be34bf2fb0 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd5219df740 10 client.0 _ll_register_callbacks cb 0x55be34b00e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd517400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34bf03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd517400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34bf03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:33598/0 (socket says 172.21.15.1:33598) 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd517400640 1 -- 172.21.15.1:0/3337736021 learned_addr learned my addr 172.21.15.1:0/3337736021 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd517e00640 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be7dd0 0x55be34befe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd51cc00640 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34bf2fb0 0x55be34bf08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd517400640 1 -- 172.21.15.1:0/3337736021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34bf2fb0 msgr2=0x55be34bf08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd517400640 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34bf2fb0 0x55be34bf08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd517400640 1 -- 172.21.15.1:0/3337736021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be7dd0 msgr2=0x55be34befe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd517400640 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be7dd0 0x55be34befe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.496+0000 7fd517400640 1 -- 172.21.15.1:0/3337736021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be34ae3510 con 0x55be34be86f0 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.497+0000 7fd51cc00640 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34bf2fb0 0x55be34bf08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.497+0000 7fd517e00640 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be7dd0 0x55be34befe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:48:18.499 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.497+0000 7fd517400640 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34bf03b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd504005ab0 tx=0x7fd50403e500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2084ee0b6c778cb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.497+0000 7fd501600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.497+0000 7fd501600640 1 -- 172.21.15.1:0/3337736021 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5040bc7c0 con 0x55be34be86f0 2024-04-26T21:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.497+0000 7fd501600640 1 -- 172.21.15.1:0/3337736021 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9730+0+0 (secure 0 0 0) 0x7fd5040dd0e0 con 0x55be34be86f0 2024-04-26T21:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.497+0000 7fd5219df740 1 -- 172.21.15.1:0/3337736021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd510004d20 con 0x55be34be86f0 2024-04-26T21:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.498+0000 7fd501600640 1 -- 172.21.15.1:0/3337736021 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fd5040bc5e0 con 0x55be34be86f0 2024-04-26T21:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.498+0000 7fd5219df740 10 client.26870 fetch_fsmap learned FSMap version 55 2024-04-26T21:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.498+0000 7fd5219df740 1 -- 172.21.15.1:0/3337736021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55be3485f210 con 0x55be34be86f0 2024-04-26T21:48:18.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.498+0000 7fd501600640 1 -- 172.21.15.1:0/3337736021 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd510004d20 con 0x55be34be86f0 2024-04-26T21:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.498+0000 7fd501600640 1 -- 172.21.15.1:0/3337736021 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 55) ==== 8354+0+0 (secure 0 0 0) 0x7fd504035070 con 0x55be34be86f0 2024-04-26T21:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.498+0000 7fd5219df740 10 client.26870 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T21:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.499+0000 7fd5219df740 10 client.26870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.499+0000 7fd5219df740 20 client.26870 populate_metadata read hostname 'smithi001' 2024-04-26T21:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.499+0000 7fd5219df740 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55be34c02a50 0x55be34c04ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.499+0000 7fd5219df740 4 client.26870 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.499+0000 7fd5219df740 1 -- 172.21.15.1:0/3337736021 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be34b5af90 con 0x55be34c02a50 2024-04-26T21:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.499+0000 7fd517e00640 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55be34c02a50 0x55be34c04ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.500+0000 7fd517e00640 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55be34c02a50 0x55be34c04ee0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e21343c13541d7d server_cookie=4b260e54325673f6 in_seq=0 out_seq=0 2024-04-26T21:48:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.500+0000 7fd501600640 10 client.26870 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.503+0000 7fd501600640 1 -- 172.21.15.1:0/3337736021 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55be34b5af90 con 0x55be34c02a50 2024-04-26T21:48:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.503+0000 7fd501600640 10 client.26870 handle_command_reply: tid=0 2024-04-26T21:48:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.503+0000 7fd5219df740 1 client.26870 shutdown 2024-04-26T21:48:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.504+0000 7fd5219df740 1 -- 172.21.15.1:0/3337736021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 msgr2=0x55be34bf03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.504+0000 7fd5219df740 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34bf03b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd504005ab0 tx=0x7fd50403e500 comp rx=0 tx=0).stop 2024-04-26T21:48:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.504+0000 7fd5219df740 1 -- 172.21.15.1:0/3337736021 shutdown_connections 2024-04-26T21:48:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.504+0000 7fd5219df740 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55be34c02a50 0x55be34c04ee0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.504+0000 7fd5219df740 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34bf2fb0 0x55be34bf08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.504+0000 7fd5219df740 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34bf03b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.504+0000 7fd5219df740 1 --2- 172.21.15.1:0/3337736021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be7dd0 0x55be34befe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.504+0000 7fd5219df740 1 -- 172.21.15.1:0/3337736021 >> 172.21.15.1:0/3337736021 conn(0x55be34be42e0 msgr2=0x55be34bea010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.504+0000 7fd501600640 0 client.26870 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.505+0000 7fd5219df740 1 -- 172.21.15.1:0/3337736021 shutdown_connections 2024-04-26T21:48:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.505+0000 7fd5219df740 1 -- 172.21.15.1:0/3337736021 wait complete. 2024-04-26T21:48:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.506+0000 7fd5219df740 20 client.26870 trim_cache size 0 max 16384 2024-04-26T21:48:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.529+0000 7fd5219df740 1 Processor -- start 2024-04-26T21:48:18.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.529+0000 7fd5219df740 1 -- start start 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.529+0000 7fd5219df740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34bfeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.529+0000 7fd5219df740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be86f0 0x55be34bea790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.529+0000 7fd5219df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34bf2fb0 0x55be34c0a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.529+0000 7fd5219df740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181c17f0 con 0x55be34be86f0 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.529+0000 7fd5219df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181bf0c0 con 0x55be34bf2fb0 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.529+0000 7fd5219df740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181bcf30 con 0x55be34be7dd0 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd517e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34bfeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd517e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34bfeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:49218/0 (socket says 172.21.15.1:49218) 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd517e00640 1 -- 172.21.15.1:0/3266224542 learned_addr learned my addr 172.21.15.1:0/3266224542 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd51cc00640 1 --2- 172.21.15.1:0/3266224542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34bf2fb0 0x55be34c0a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd517400640 1 --2- 172.21.15.1:0/3266224542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be86f0 0x55be34bea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd517e00640 1 -- 172.21.15.1:0/3266224542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34bf2fb0 msgr2=0x55be34c0a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd517e00640 1 --2- 172.21.15.1:0/3266224542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34bf2fb0 0x55be34c0a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd517e00640 1 -- 172.21.15.1:0/3266224542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be86f0 msgr2=0x55be34bea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd517e00640 1 --2- 172.21.15.1:0/3266224542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be86f0 0x55be34bea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd517e00640 1 -- 172.21.15.1:0/3266224542 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be3492b620 con 0x55be34be7dd0 2024-04-26T21:48:18.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.530+0000 7fd517e00640 1 --2- 172.21.15.1:0/3266224542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34bfeaf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd508005aa0 tx=0x7fd508015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2aef5964e3ef3589 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.531+0000 7fd503e00640 1 -- 172.21.15.1:0/3266224542 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50802b070 con 0x55be34be7dd0 2024-04-26T21:48:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.531+0000 7fd503e00640 1 -- 172.21.15.1:0/3266224542 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd508026390 con 0x55be34be7dd0 2024-04-26T21:48:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.531+0000 7fd503e00640 1 -- 172.21.15.1:0/3266224542 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508003b90 con 0x55be34be7dd0 2024-04-26T21:48:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.532+0000 7fd5219df740 1 -- 172.21.15.1:0/3266224542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 msgr2=0x55be34bfeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.532+0000 7fd5219df740 1 --2- 172.21.15.1:0/3266224542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34bfeaf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd508005aa0 tx=0x7fd508015000 comp rx=0 tx=0).stop 2024-04-26T21:48:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.532+0000 7fd5219df740 1 -- 172.21.15.1:0/3266224542 shutdown_connections 2024-04-26T21:48:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.532+0000 7fd5219df740 1 --2- 172.21.15.1:0/3266224542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34bf2fb0 0x55be34c0a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.532+0000 7fd5219df740 1 --2- 172.21.15.1:0/3266224542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34be86f0 0x55be34bea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.532+0000 7fd5219df740 1 --2- 172.21.15.1:0/3266224542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34bfeaf0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.532+0000 7fd5219df740 1 -- 172.21.15.1:0/3266224542 >> 172.21.15.1:0/3266224542 conn(0x55be34be42e0 msgr2=0x55be34beb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.532+0000 7fd5219df740 1 -- 172.21.15.1:0/3266224542 shutdown_connections 2024-04-26T21:48:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.532+0000 7fd5219df740 1 -- 172.21.15.1:0/3266224542 wait complete. 2024-04-26T21:48:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.532+0000 7fd5219df740 1 -- start start 2024-04-26T21:48:18.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.533+0000 7fd5219df740 1 Processor -- start 2024-04-26T21:48:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.533+0000 7fd5219df740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34c61480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.533+0000 7fd5219df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34c619c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.533+0000 7fd5219df740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 0x55be34bf63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.533+0000 7fd5219df740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181bd580 con 0x55be34bf2fb0 2024-04-26T21:48:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.533+0000 7fd5219df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181be9f0 con 0x55be34be86f0 2024-04-26T21:48:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.533+0000 7fd5219df740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd5181bdc50 con 0x55be34be7dd0 2024-04-26T21:48:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.533+0000 7fd5219df740 10 client.0 _ll_register_callbacks cb 0x7fd5040f4910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:48:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.534+0000 7fd517e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34c61480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.534+0000 7fd517400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34c619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.534+0000 7fd517400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34c619c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:33616/0 (socket says 172.21.15.1:33616) 2024-04-26T21:48:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.534+0000 7fd517400640 1 -- 172.21.15.1:0/3953002714 learned_addr learned my addr 172.21.15.1:0/3953002714 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.534+0000 7fd51cc00640 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 0x55be34bf63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.534+0000 7fd517400640 1 -- 172.21.15.1:0/3953002714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 msgr2=0x55be34c61480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.534+0000 7fd517400640 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34c61480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.534+0000 7fd517400640 1 -- 172.21.15.1:0/3953002714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 msgr2=0x55be34bf63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.534+0000 7fd517400640 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 0x55be34bf63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.534+0000 7fd517400640 1 -- 172.21.15.1:0/3953002714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be34b07cb0 con 0x55be34be86f0 2024-04-26T21:48:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.535+0000 7fd517400640 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34c619c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd5040b90e0 tx=0x7fd5040c59b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20104705990cfc47 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:18.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.535+0000 7fd501600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:48:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.536+0000 7fd501600640 1 -- 172.21.15.1:0/3953002714 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5040bc7c0 con 0x55be34be86f0 2024-04-26T21:48:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.536+0000 7fd501600640 1 -- 172.21.15.1:0/3953002714 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(121..121 src has 1..121) ==== 9730+0+0 (secure 0 0 0) 0x7fd5040c55d0 con 0x55be34be86f0 2024-04-26T21:48:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.537+0000 7fd5219df740 1 -- 172.21.15.1:0/3953002714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd508026390 con 0x55be34be86f0 2024-04-26T21:48:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.537+0000 7fd501600640 1 -- 172.21.15.1:0/3953002714 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7fd504036910 con 0x55be34be86f0 2024-04-26T21:48:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.537+0000 7fd5219df740 10 client.26876 fetch_fsmap learned FSMap version 55 2024-04-26T21:48:18.537 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.537+0000 7fd5219df740 1 -- 172.21.15.1:0/3953002714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55be34c689c0 con 0x55be34be86f0 2024-04-26T21:48:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.537+0000 7fd501600640 1 -- 172.21.15.1:0/3953002714 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd508026390 con 0x55be34be86f0 2024-04-26T21:48:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.537+0000 7fd501600640 1 -- 172.21.15.1:0/3953002714 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 55) ==== 8354+0+0 (secure 0 0 0) 0x7fd504035070 con 0x55be34be86f0 2024-04-26T21:48:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.538+0000 7fd5219df740 10 client.26876 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T21:48:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.538+0000 7fd5219df740 10 client.26876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:48:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.538+0000 7fd5219df740 20 client.26876 populate_metadata read hostname 'smithi001' 2024-04-26T21:48:18.538 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.538+0000 7fd5219df740 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55be34c68fb0 0x55be34c6b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.538+0000 7fd5219df740 4 client.26876 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:48:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.538+0000 7fd5219df740 1 -- 172.21.15.1:0/3953002714 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55be34b54e00 con 0x55be34c68fb0 2024-04-26T21:48:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.538+0000 7fd517e00640 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55be34c68fb0 0x55be34c6b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.539+0000 7fd517e00640 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55be34c68fb0 0x55be34c6b3d0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6e95bd22974bfb2 server_cookie=3d264f8847c698fb in_seq=0 out_seq=0 2024-04-26T21:48:18.539 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.539+0000 7fd501600640 10 client.26876 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:18 smithi001 ceph-mon[28054]: pgmap v567: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 4.0 MiB/s wr, 347 op/s 2024-04-26T21:48:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:18 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:48:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:18 smithi038 ceph-mon[32724]: pgmap v567: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 4.0 MiB/s wr, 347 op/s 2024-04-26T21:48:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:18 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:48:18.891 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.891+0000 7fd501600640 1 -- 172.21.15.1:0/3953002714 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55be34b54e00 con 0x55be34c68fb0 2024-04-26T21:48:18.892 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.891+0000 7fd501600640 10 client.26876 handle_command_reply: tid=0 2024-04-26T21:48:18.892 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.891+0000 7fd5219df740 1 client.26876 shutdown 2024-04-26T21:48:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.895+0000 7fd5219df740 1 -- 172.21.15.1:0/3953002714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 msgr2=0x55be34c619c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:18.895 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.895+0000 7fd5219df740 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34c619c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd5040b90e0 tx=0x7fd5040c59b0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.897+0000 7fd5219df740 1 -- 172.21.15.1:0/3953002714 shutdown_connections 2024-04-26T21:48:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.897+0000 7fd5219df740 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55be34c68fb0 0x55be34c6b3d0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.897+0000 7fd5219df740 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55be34bf2fb0 0x55be34bf63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.897+0000 7fd5219df740 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be34be86f0 0x55be34c619c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.897+0000 7fd5219df740 1 --2- 172.21.15.1:0/3953002714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be34be7dd0 0x55be34c61480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.897+0000 7fd5219df740 1 -- 172.21.15.1:0/3953002714 >> 172.21.15.1:0/3953002714 conn(0x55be34be42e0 msgr2=0x55be34befe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.897+0000 7fd501600640 0 client.26876 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.898+0000 7fd5219df740 1 -- 172.21.15.1:0/3953002714 shutdown_connections 2024-04-26T21:48:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.898+0000 7fd5219df740 1 -- 172.21.15.1:0/3953002714 wait complete. 2024-04-26T21:48:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.898+0000 7fd5219df740 20 client.26876 trim_cache size 0 max 16384 2024-04-26T21:48:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:48:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:48:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:48:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:48:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.899+0000 7fd51ec00640 1 -- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4fc0a3d90 msgr2=0x7fd4fc0a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.899+0000 7fd51ec00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4fc0a3d90 0x7fd4fc0a6250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x55be34be9370 tx=0x7fd508005f90 comp rx=0 tx=0).stop 2024-04-26T21:48:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.899+0000 7fd51ec00640 1 -- 172.21.15.1:0/4011613868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5181032e0 msgr2=0x7fd518195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.899+0000 7fd51ec00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5181032e0 0x7fd518195f70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd50402edf0 tx=0x7fd504003c10 comp rx=0 tx=0).stop 2024-04-26T21:48:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.900+0000 7fd51ec00640 1 -- 172.21.15.1:0/4011613868 shutdown_connections 2024-04-26T21:48:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.900+0000 7fd51ec00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4fc0a3d90 0x7fd4fc0a6250 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.900+0000 7fd51ec00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5181032e0 0x7fd518195f70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.900+0000 7fd51ec00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd518108930 0x7fd5181964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.900+0000 7fd51ec00640 1 --2- 172.21.15.1:0/4011613868 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd518102920 0x7fd51819dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.900+0000 7fd51ec00640 1 -- 172.21.15.1:0/4011613868 >> 172.21.15.1:0/4011613868 conn(0x7fd5180fe3f0 msgr2=0x7fd5180febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.900+0000 7fd51ec00640 1 -- 172.21.15.1:0/4011613868 shutdown_connections 2024-04-26T21:48:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:18.901+0000 7fd51ec00640 1 -- 172.21.15.1:0/4011613868 wait complete. 2024-04-26T21:48:19.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 112111 22.64 MB/sec execute 136 sec latency 10076.215 ms 2024-04-26T21:48:20.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 112193 22.49 MB/sec execute 137 sec latency 11076.283 ms 2024-04-26T21:48:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:20 smithi186 ceph-mon[32991]: pgmap v568: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.4 MiB/s wr, 293 op/s 2024-04-26T21:48:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:20 smithi186 ceph-mon[32991]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T21:48:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:20 smithi001 ceph-mon[28054]: pgmap v568: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.4 MiB/s wr, 293 op/s 2024-04-26T21:48:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:20 smithi001 ceph-mon[28054]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T21:48:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:20 smithi038 ceph-mon[32724]: pgmap v568: 129 pgs: 129 active+clean; 1.2 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.4 MiB/s wr, 293 op/s 2024-04-26T21:48:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:20 smithi038 ceph-mon[32724]: osdmap e122: 12 total, 12 up, 12 in 2024-04-26T21:48:21.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 112383 22.37 MB/sec execute 138 sec latency 11177.639 ms 2024-04-26T21:48:22.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 112957 22.42 MB/sec execute 139 sec latency 170.980 ms 2024-04-26T21:48:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:22 smithi186 ceph-mon[32991]: pgmap v570: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.1 MiB/s wr, 480 op/s 2024-04-26T21:48:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:22 smithi001 ceph-mon[28054]: pgmap v570: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.1 MiB/s wr, 480 op/s 2024-04-26T21:48:22.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:22 smithi038 ceph-mon[32724]: pgmap v570: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.1 MiB/s wr, 480 op/s 2024-04-26T21:48:23.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 113682 22.59 MB/sec execute 140 sec latency 71.510 ms 2024-04-26T21:48:24.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 114229 22.71 MB/sec execute 141 sec latency 103.626 ms 2024-04-26T21:48:24.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:24 smithi186 ceph-mon[32991]: pgmap v571: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.1 MiB/s wr, 480 op/s 2024-04-26T21:48:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:24 smithi001 ceph-mon[28054]: pgmap v571: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.1 MiB/s wr, 480 op/s 2024-04-26T21:48:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:24 smithi038 ceph-mon[32724]: pgmap v571: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 9.1 MiB/s wr, 480 op/s 2024-04-26T21:48:25.160 INFO:tasks.workunit.client.0.smithi001.stdout: 10 114545 22.68 MB/sec execute 142 sec latency 287.750 ms 2024-04-26T21:48:26.161 INFO:tasks.workunit.client.0.smithi001.stdout: 10 114731 22.57 MB/sec execute 143 sec latency 419.786 ms 2024-04-26T21:48:26.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:26 smithi186 ceph-mon[32991]: pgmap v572: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 11 MiB/s wr, 602 op/s 2024-04-26T21:48:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:26 smithi001 ceph-mon[28054]: pgmap v572: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 11 MiB/s wr, 602 op/s 2024-04-26T21:48:26.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:26 smithi038 ceph-mon[32724]: pgmap v572: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 11 MiB/s wr, 602 op/s 2024-04-26T21:48:27.161 INFO:tasks.workunit.client.0.smithi001.stdout: 10 115078 22.51 MB/sec execute 144 sec latency 273.580 ms 2024-04-26T21:48:28.161 INFO:tasks.workunit.client.0.smithi001.stdout: 10 115449 22.47 MB/sec execute 145 sec latency 143.472 ms 2024-04-26T21:48:28.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:28 smithi186 ceph-mon[32991]: pgmap v573: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 11 MiB/s wr, 588 op/s 2024-04-26T21:48:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:28 smithi001 ceph-mon[28054]: pgmap v573: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 11 MiB/s wr, 588 op/s 2024-04-26T21:48:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:28 smithi038 ceph-mon[32724]: pgmap v573: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 11 MiB/s wr, 588 op/s 2024-04-26T21:48:29.161 INFO:tasks.workunit.client.0.smithi001.stdout: 10 115869 22.45 MB/sec execute 146 sec latency 172.409 ms 2024-04-26T21:48:30.161 INFO:tasks.workunit.client.0.smithi001.stdout: 10 116462 22.52 MB/sec execute 147 sec latency 123.854 ms 2024-04-26T21:48:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:30 smithi001 ceph-mon[28054]: pgmap v574: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 11 MiB/s wr, 588 op/s 2024-04-26T21:48:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:30 smithi038 ceph-mon[32724]: pgmap v574: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 11 MiB/s wr, 588 op/s 2024-04-26T21:48:30.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:30 smithi186 ceph-mon[32991]: pgmap v574: 129 pgs: 129 active+clean; 1.2 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 11 MiB/s wr, 588 op/s 2024-04-26T21:48:31.162 INFO:tasks.workunit.client.0.smithi001.stdout: 10 117119 22.61 MB/sec execute 148 sec latency 113.060 ms 2024-04-26T21:48:32.162 INFO:tasks.workunit.client.0.smithi001.stdout: 10 117665 22.70 MB/sec execute 149 sec latency 115.344 ms 2024-04-26T21:48:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.201+0000 7f0e8a600640 1 -- 172.21.15.1:0/685863180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e841044c0 msgr2=0x7f0e8410c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.201+0000 7f0e8a600640 1 --2- 172.21.15.1:0/685863180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e841044c0 0x7f0e8410c9a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009900 tx=0x7f0e7002a480 comp rx=0 tx=0).stop 2024-04-26T21:48:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.201+0000 7f0e8a600640 1 -- 172.21.15.1:0/685863180 shutdown_connections 2024-04-26T21:48:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.201+0000 7f0e8a600640 1 --2- 172.21.15.1:0/685863180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e8410cf70 0x7f0e8410f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.201+0000 7f0e8a600640 1 --2- 172.21.15.1:0/685863180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e841044c0 0x7f0e8410c9a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.201+0000 7f0e8a600640 1 --2- 172.21.15.1:0/685863180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e84103ba0 0x7f0e84103f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.201+0000 7f0e8a600640 1 -- 172.21.15.1:0/685863180 >> 172.21.15.1:0/685863180 conn(0x7f0e84078950 msgr2=0x7f0e84078d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.201+0000 7f0e8a600640 1 -- 172.21.15.1:0/685863180 shutdown_connections 2024-04-26T21:48:32.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.201+0000 7f0e8a600640 1 -- 172.21.15.1:0/685863180 wait complete. 2024-04-26T21:48:32.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.202+0000 7f0e8a600640 1 Processor -- start 2024-04-26T21:48:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.203+0000 7f0e8a600640 1 -- start start 2024-04-26T21:48:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.203+0000 7f0e8a600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e84103ba0 0x7f0e841017f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.203+0000 7f0e8a600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e841044c0 0x7f0e84101d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.203+0000 7f0e8a600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e8410cf70 0x7f0e84102270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.203+0000 7f0e8a600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0e84112c00 con 0x7f0e84103ba0 2024-04-26T21:48:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.203+0000 7f0e8a600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e84112a80 con 0x7f0e8410cf70 2024-04-26T21:48:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.203+0000 7f0e8a600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e84112d80 con 0x7f0e841044c0 2024-04-26T21:48:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.203+0000 7f0e82a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e841044c0 0x7f0e84101d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.204+0000 7f0e82a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e841044c0 0x7f0e84101d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:48312/0 (socket says 172.21.15.1:48312) 2024-04-26T21:48:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.204+0000 7f0e82a00640 1 -- 172.21.15.1:0/655330783 learned_addr learned my addr 172.21.15.1:0/655330783 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.204+0000 7f0e83e00640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e8410cf70 0x7f0e84102270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.204+0000 7f0e83400640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e84103ba0 0x7f0e841017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.204+0000 7f0e83e00640 1 -- 172.21.15.1:0/655330783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e841044c0 msgr2=0x7f0e84101d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.204+0000 7f0e83e00640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e841044c0 0x7f0e84101d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.204+0000 7f0e83e00640 1 -- 172.21.15.1:0/655330783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e84103ba0 msgr2=0x7f0e841017f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.204+0000 7f0e83e00640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e84103ba0 0x7f0e841017f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.204+0000 7f0e83e00640 1 -- 172.21.15.1:0/655330783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8419d280 con 0x7f0e8410cf70 2024-04-26T21:48:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.205+0000 7f0e83e00640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e8410cf70 0x7f0e84102270 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c005fd0 tx=0x7f0e7c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.205+0000 7f0e7be00640 1 -- 172.21.15.1:0/655330783 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c00cdf0 con 0x7f0e8410cf70 2024-04-26T21:48:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.205+0000 7f0e7be00640 1 -- 172.21.15.1:0/655330783 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e7c005020 con 0x7f0e8410cf70 2024-04-26T21:48:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.205+0000 7f0e7be00640 1 -- 172.21.15.1:0/655330783 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c010690 con 0x7f0e8410cf70 2024-04-26T21:48:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.205+0000 7f0e8a600640 1 -- 172.21.15.1:0/655330783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e8419d570 con 0x7f0e8410cf70 2024-04-26T21:48:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.205+0000 7f0e8a600640 1 -- 172.21.15.1:0/655330783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e8419da90 con 0x7f0e8410cf70 2024-04-26T21:48:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.209+0000 7f0e8d44f740 1 Processor -- start 2024-04-26T21:48:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.210+0000 7f0e7be00640 1 -- 172.21.15.1:0/655330783 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f0e7c004a40 con 0x7f0e8410cf70 2024-04-26T21:48:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.210+0000 7f0e8d44f740 1 -- start start 2024-04-26T21:48:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.211+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.211+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b406f0 0x55a097b4aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.211+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b4afb0 0x55a097b4d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.211+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a097aa5380 con 0x55a097b406f0 2024-04-26T21:48:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.211+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a097a41c30 con 0x55a097b3fdd0 2024-04-26T21:48:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.211+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a097aa1340 con 0x55a097b4afb0 2024-04-26T21:48:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.211+0000 7f0e82a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.211+0000 7f0e82a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40474/0 (socket says 172.21.15.1:40474) 2024-04-26T21:48:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.211+0000 7f0e82a00640 1 -- 172.21.15.1:0/301591407 learned_addr learned my addr 172.21.15.1:0/301591407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.211+0000 7f0e82a00640 1 --2- 172.21.15.1:0/301591407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b4afb0 0x55a097b4d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.212+0000 7f0e82a00640 1 -- 172.21.15.1:0/301591407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b4afb0 msgr2=0x55a097b4d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.212+0000 7f0e82a00640 1 --2- 172.21.15.1:0/301591407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b4afb0 0x55a097b4d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.212+0000 7f0e82a00640 1 -- 172.21.15.1:0/301591407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b406f0 msgr2=0x55a097b4aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.212+0000 7f0e82a00640 1 --2- 172.21.15.1:0/301591407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b406f0 0x55a097b4aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.212+0000 7f0e82a00640 1 -- 172.21.15.1:0/301591407 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a097a63940 con 0x55a097b3fdd0 2024-04-26T21:48:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.212+0000 7f0e7be00640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e680a3e60 0x7f0e680a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.214 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.212+0000 7f0e7be00640 1 -- 172.21.15.1:0/655330783 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 9642+0+0 (secure 0 0 0) 0x7f0e7c0d0ff0 con 0x7f0e8410cf70 2024-04-26T21:48:32.214 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.213+0000 7f0e82a00640 1 --2- 172.21.15.1:0/301591407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b401b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0e70005ab0 tx=0x7f0e70005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47216dc42ad65637 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:32.214 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.213+0000 7f0e83400640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e680a3e60 0x7f0e680a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.214 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.214+0000 7f0e83400640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e680a3e60 0x7f0e680a6320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x55a097b41370 tx=0x7f0e74009510 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.219+0000 7f0e78c00640 1 -- 172.21.15.1:0/301591407 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70035040 con 0x55a097b3fdd0 2024-04-26T21:48:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.219+0000 7f0e78c00640 1 -- 172.21.15.1:0/301591407 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e70005020 con 0x55a097b3fdd0 2024-04-26T21:48:32.219 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.219+0000 7f0e78c00640 1 -- 172.21.15.1:0/301591407 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e70004a00 con 0x55a097b3fdd0 2024-04-26T21:48:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.220+0000 7f0e8d44f740 1 -- 172.21.15.1:0/301591407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 msgr2=0x55a097b401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.220+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/301591407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b401b0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0e70005ab0 tx=0x7f0e70005a20 comp rx=0 tx=0).stop 2024-04-26T21:48:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.220+0000 7f0e8d44f740 1 -- 172.21.15.1:0/301591407 shutdown_connections 2024-04-26T21:48:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.220+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/301591407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b4afb0 0x55a097b4d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.220+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/301591407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b406f0 0x55a097b4aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.220+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/301591407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b401b0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.220+0000 7f0e8d44f740 1 -- 172.21.15.1:0/301591407 >> 172.21.15.1:0/301591407 conn(0x55a097b3c2e0 msgr2=0x55a097b3c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.220+0000 7f0e8d44f740 1 -- 172.21.15.1:0/301591407 shutdown_connections 2024-04-26T21:48:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.220+0000 7f0e8d44f740 1 -- 172.21.15.1:0/301591407 wait complete. 2024-04-26T21:48:32.223 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.221+0000 7f0e8d44f740 1 -- start start 2024-04-26T21:48:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.221+0000 7f0e8d44f740 1 Processor -- start 2024-04-26T21:48:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b47e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b406f0 0x55a097b483b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 0x55a097b488f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a097b4db80 con 0x55a097b4afb0 2024-04-26T21:48:32.224 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a097a2cc60 con 0x55a097b3fdd0 2024-04-26T21:48:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a097b4dd00 con 0x55a097b406f0 2024-04-26T21:48:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e8d44f740 10 client.0 _ll_register_callbacks cb 0x55a097a58e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:48:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e83400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b47e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e82a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b406f0 0x55a097b483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e83400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b47e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40488/0 (socket says 172.21.15.1:40488) 2024-04-26T21:48:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e83400640 1 -- 172.21.15.1:0/1077785703 learned_addr learned my addr 172.21.15.1:0/1077785703 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.222+0000 7f0e83e00640 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 0x55a097b488f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.223+0000 7f0e82a00640 1 -- 172.21.15.1:0/1077785703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 msgr2=0x55a097b47e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.223+0000 7f0e82a00640 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b47e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.223+0000 7f0e82a00640 1 -- 172.21.15.1:0/1077785703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 msgr2=0x55a097b488f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.223+0000 7f0e82a00640 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 0x55a097b488f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.223+0000 7f0e82a00640 1 -- 172.21.15.1:0/1077785703 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a097a3b510 con 0x55a097b406f0 2024-04-26T21:48:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.223+0000 7f0e83e00640 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 0x55a097b488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:48:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.223+0000 7f0e82a00640 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b406f0 0x55a097b483b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0e70005ab0 tx=0x7f0e70003670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a66e9d8c369ae660 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.223+0000 7f0e6d600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:48:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.224+0000 7f0e6d600640 1 -- 172.21.15.1:0/1077785703 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7006f070 con 0x55a097b406f0 2024-04-26T21:48:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.224+0000 7f0e6d600640 1 -- 172.21.15.1:0/1077785703 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9642+0+0 (secure 0 0 0) 0x7f0e700038c0 con 0x55a097b406f0 2024-04-26T21:48:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.224+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1077785703 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e70005020 con 0x55a097b406f0 2024-04-26T21:48:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.224+0000 7f0e6d600640 1 -- 172.21.15.1:0/1077785703 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f0e7007b050 con 0x55a097b406f0 2024-04-26T21:48:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.224+0000 7f0e8d44f740 10 client.26893 fetch_fsmap learned FSMap version 55 2024-04-26T21:48:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.224+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1077785703 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55a0977b7210 con 0x55a097b406f0 2024-04-26T21:48:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.225+0000 7f0e6d600640 1 -- 172.21.15.1:0/1077785703 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e70005020 con 0x55a097b406f0 2024-04-26T21:48:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.225+0000 7f0e6d600640 1 -- 172.21.15.1:0/1077785703 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 55) ==== 8354+0+0 (secure 0 0 0) 0x7f0e700314a0 con 0x55a097b406f0 2024-04-26T21:48:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.225+0000 7f0e8d44f740 10 client.26893 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T21:48:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.226+0000 7f0e8d44f740 10 client.26893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:48:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.226+0000 7f0e8d44f740 20 client.26893 populate_metadata read hostname 'smithi001' 2024-04-26T21:48:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.226+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a097b581d0 0x55a097b5a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.226+0000 7f0e8d44f740 4 client.26893 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:48:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.226+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1077785703 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a097ab2f90 con 0x55a097b581d0 2024-04-26T21:48:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.226+0000 7f0e83400640 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a097b581d0 0x55a097b5a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.228+0000 7f0e6d600640 10 client.26893 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.228+0000 7f0e83400640 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a097b581d0 0x55a097b5a610 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b536e928fb3227ec server_cookie=ac68c88874ea8eff in_seq=0 out_seq=0 2024-04-26T21:48:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.231+0000 7f0e6d600640 1 -- 172.21.15.1:0/1077785703 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a097ab2f90 con 0x55a097b581d0 2024-04-26T21:48:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.231+0000 7f0e6d600640 10 client.26893 handle_command_reply: tid=0 2024-04-26T21:48:32.231 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.231+0000 7f0e8d44f740 1 client.26893 shutdown 2024-04-26T21:48:32.232 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1077785703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b406f0 msgr2=0x55a097b483b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b406f0 0x55a097b483b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0e70005ab0 tx=0x7f0e70003670 comp rx=0 tx=0).stop 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1077785703 shutdown_connections 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a097b581d0 0x55a097b5a610 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b406f0 0x55a097b483b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 0x55a097b488f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/1077785703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b3fdd0 0x55a097b47e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1077785703 >> 172.21.15.1:0/1077785703 conn(0x55a097b3c2e0 msgr2=0x55a097b42060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e6d600640 0 client.26893 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1077785703 shutdown_connections 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1077785703 wait complete. 2024-04-26T21:48:32.233 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.232+0000 7f0e8d44f740 20 client.26893 trim_cache size 0 max 16384 2024-04-26T21:48:32.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e8d44f740 1 Processor -- start 2024-04-26T21:48:32.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e8d44f740 1 -- start start 2024-04-26T21:48:32.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b3fdd0 0x55a097b46ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.253 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b4afb0 0x55a097b37170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b49ca0 0x55a097b45110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0e64012a20 con 0x55a097b3fdd0 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e640102f0 con 0x55a097b4afb0 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6400e160 con 0x55a097b49ca0 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e82a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b4afb0 0x55a097b37170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e83400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b3fdd0 0x55a097b46ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e82a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b4afb0 0x55a097b37170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40504/0 (socket says 172.21.15.1:40504) 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e82a00640 1 -- 172.21.15.1:0/1428089292 learned_addr learned my addr 172.21.15.1:0/1428089292 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.252+0000 7f0e83e00640 1 --2- 172.21.15.1:0/1428089292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b49ca0 0x55a097b45110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.253+0000 7f0e83400640 1 -- 172.21.15.1:0/1428089292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b49ca0 msgr2=0x55a097b45110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.253+0000 7f0e83400640 1 --2- 172.21.15.1:0/1428089292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b49ca0 0x55a097b45110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.253+0000 7f0e83400640 1 -- 172.21.15.1:0/1428089292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b4afb0 msgr2=0x55a097b37170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.253+0000 7f0e83400640 1 --2- 172.21.15.1:0/1428089292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b4afb0 0x55a097b37170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.253+0000 7f0e83400640 1 -- 172.21.15.1:0/1428089292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a097b37940 con 0x55a097b3fdd0 2024-04-26T21:48:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.253+0000 7f0e82a00640 1 --2- 172.21.15.1:0/1428089292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b4afb0 0x55a097b37170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:48:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.253+0000 7f0e83400640 1 --2- 172.21.15.1:0/1428089292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b3fdd0 0x55a097b46ff0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0e74004700 tx=0x7f0e74014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f416b2be2c9467 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.254+0000 7f0e6fe00640 1 -- 172.21.15.1:0/1428089292 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e74028660 con 0x55a097b3fdd0 2024-04-26T21:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.254+0000 7f0e6fe00640 1 -- 172.21.15.1:0/1428089292 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e7402e070 con 0x55a097b3fdd0 2024-04-26T21:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.254+0000 7f0e6fe00640 1 -- 172.21.15.1:0/1428089292 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7401f690 con 0x55a097b3fdd0 2024-04-26T21:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.254+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1428089292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b3fdd0 msgr2=0x55a097b46ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.254+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/1428089292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b3fdd0 0x55a097b46ff0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0e74004700 tx=0x7f0e74014000 comp rx=0 tx=0).stop 2024-04-26T21:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.254+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1428089292 shutdown_connections 2024-04-26T21:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.254+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/1428089292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b49ca0 0x55a097b45110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.254+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/1428089292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b4afb0 0x55a097b37170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.254+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/1428089292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b3fdd0 0x55a097b46ff0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.254+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1428089292 >> 172.21.15.1:0/1428089292 conn(0x55a097b3c2e0 msgr2=0x55a097b49590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.255+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1428089292 shutdown_connections 2024-04-26T21:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.255+0000 7f0e8d44f740 1 -- 172.21.15.1:0/1428089292 wait complete. 2024-04-26T21:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.255+0000 7f0e8d44f740 1 -- start start 2024-04-26T21:48:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e8d44f740 1 Processor -- start 2024-04-26T21:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b49ca0 0x55a097b5d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 0x55a097b5a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e8d44f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b5a720 0x55a097b5aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6400e7b0 con 0x55a097b4afb0 2024-04-26T21:48:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6400fc20 con 0x55a097b49ca0 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e8d44f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6400ee80 con 0x55a097b5a720 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e8d44f740 10 client.0 _ll_register_callbacks cb 0x7f0e70009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e83400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b49ca0 0x55a097b5d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e83400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b49ca0 0x55a097b5d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:40508/0 (socket says 172.21.15.1:40508) 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.256+0000 7f0e83400640 1 -- 172.21.15.1:0/2300609876 learned_addr learned my addr 172.21.15.1:0/2300609876 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.257+0000 7f0e82a00640 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 0x55a097b5a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.257+0000 7f0e83e00640 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b5a720 0x55a097b5aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.257+0000 7f0e83400640 1 -- 172.21.15.1:0/2300609876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b5a720 msgr2=0x55a097b5aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.257+0000 7f0e83400640 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b5a720 0x55a097b5aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.257+0000 7f0e83400640 1 -- 172.21.15.1:0/2300609876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 msgr2=0x55a097b5a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.257+0000 7f0e83400640 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 0x55a097b5a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.257+0000 7f0e83400640 1 -- 172.21.15.1:0/2300609876 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a097b40520 con 0x55a097b49ca0 2024-04-26T21:48:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.257+0000 7f0e82a00640 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 0x55a097b5a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.257+0000 7f0e83400640 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b49ca0 0x55a097b5d400 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0e74011ca0 tx=0x7f0e74009970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d332d9f81262afc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.257+0000 7f0e6d600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.258+0000 7f0e6d600640 1 -- 172.21.15.1:0/2300609876 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e740039b0 con 0x55a097b49ca0 2024-04-26T21:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.258+0000 7f0e6d600640 1 -- 172.21.15.1:0/2300609876 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(122..122 src has 1..122) ==== 9642+0+0 (secure 0 0 0) 0x7f0e74028460 con 0x55a097b49ca0 2024-04-26T21:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.258+0000 7f0e8d44f740 1 -- 172.21.15.1:0/2300609876 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e7402e070 con 0x55a097b49ca0 2024-04-26T21:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.259+0000 7f0e6d600640 1 -- 172.21.15.1:0/2300609876 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) ==== 24+0+0 (secure 0 0 0) 0x7f0e740286e0 con 0x55a097b49ca0 2024-04-26T21:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.259+0000 7f0e8d44f740 10 client.26894 fetch_fsmap learned FSMap version 55 2024-04-26T21:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.259+0000 7f0e8d44f740 1 -- 172.21.15.1:0/2300609876 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) -- 0x55a097883620 con 0x55a097b49ca0 2024-04-26T21:48:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.259+0000 7f0e6d600640 1 -- 172.21.15.1:0/2300609876 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e7402e070 con 0x55a097b49ca0 2024-04-26T21:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.259+0000 7f0e6d600640 1 -- 172.21.15.1:0/2300609876 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 55) ==== 8354+0+0 (secure 0 0 0) 0x7f0e740047c0 con 0x55a097b49ca0 2024-04-26T21:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.259+0000 7f0e8d44f740 10 client.26894 fetch_fsmap finished waiting for FSMap version 55 2024-04-26T21:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.259+0000 7f0e8d44f740 10 client.26894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.260+0000 7f0e8d44f740 20 client.26894 populate_metadata read hostname 'smithi001' 2024-04-26T21:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.260+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a097bc1f20 0x55a097bc4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.260+0000 7f0e8d44f740 4 client.26894 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:48:32.262 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.260+0000 7f0e8d44f740 1 -- 172.21.15.1:0/2300609876 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a097aace00 con 0x55a097bc1f20 2024-04-26T21:48:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.260+0000 7f0e82a00640 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a097bc1f20 0x55a097bc4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.261+0000 7f0e82a00640 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a097bc1f20 0x55a097bc4340 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e68d49ccf1b2369 server_cookie=5a45e21fdaeb2072 in_seq=0 out_seq=0 2024-04-26T21:48:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.261+0000 7f0e6d600640 10 client.26894 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:32.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:32 smithi001 ceph-mon[28054]: pgmap v575: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-26T21:48:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:32 smithi038 ceph-mon[32724]: pgmap v575: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-26T21:48:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.629+0000 7f0e6d600640 1 -- 172.21.15.1:0/2300609876 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a097aace00 con 0x55a097bc1f20 2024-04-26T21:48:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.629+0000 7f0e6d600640 10 client.26894 handle_command_reply: tid=0 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 client.26894 shutdown 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 -- 172.21.15.1:0/2300609876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b49ca0 msgr2=0x55a097b5d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b49ca0 0x55a097b5d400 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0e74011ca0 tx=0x7f0e74009970 comp rx=0 tx=0).stop 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 -- 172.21.15.1:0/2300609876 shutdown_connections 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a097bc1f20 0x55a097bc4340 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a097b5a720 0x55a097b5aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a097b4afb0 0x55a097b5a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 --2- 172.21.15.1:0/2300609876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a097b49ca0 0x55a097b5d400 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e6d600640 0 client.26894 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 -- 172.21.15.1:0/2300609876 >> 172.21.15.1:0/2300609876 conn(0x55a097b3c2e0 msgr2=0x55a097b3abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 -- 172.21.15.1:0/2300609876 shutdown_connections 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 1 -- 172.21.15.1:0/2300609876 wait complete. 2024-04-26T21:48:32.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.630+0000 7f0e8d44f740 20 client.26894 trim_cache size 0 max 16384 2024-04-26T21:48:32.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:48:32.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:48:32.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:48:32.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:48:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.632+0000 7f0e8a600640 1 -- 172.21.15.1:0/655330783 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e680a3e60 msgr2=0x7f0e680a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.632+0000 7f0e8a600640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e680a3e60 0x7f0e680a6320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x55a097b41370 tx=0x7f0e74009510 comp rx=0 tx=0).stop 2024-04-26T21:48:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.632+0000 7f0e8a600640 1 -- 172.21.15.1:0/655330783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e8410cf70 msgr2=0x7f0e84102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.632+0000 7f0e8a600640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e8410cf70 0x7f0e84102270 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c005fd0 tx=0x7f0e7c00ef10 comp rx=0 tx=0).stop 2024-04-26T21:48:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.632+0000 7f0e8a600640 1 -- 172.21.15.1:0/655330783 shutdown_connections 2024-04-26T21:48:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.632+0000 7f0e8a600640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f0e680a3e60 0x7f0e680a6320 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.632+0000 7f0e8a600640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e841044c0 0x7f0e84101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.632+0000 7f0e8a600640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0e8410cf70 0x7f0e84102270 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.632+0000 7f0e8a600640 1 --2- 172.21.15.1:0/655330783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e84103ba0 0x7f0e841017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:32.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.632+0000 7f0e8a600640 1 -- 172.21.15.1:0/655330783 >> 172.21.15.1:0/655330783 conn(0x7f0e84078950 msgr2=0x7f0e840fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.633+0000 7f0e8a600640 1 -- 172.21.15.1:0/655330783 shutdown_connections 2024-04-26T21:48:32.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:32.633+0000 7f0e8a600640 1 -- 172.21.15.1:0/655330783 wait complete. 2024-04-26T21:48:32.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:32 smithi186 ceph-mon[32991]: pgmap v575: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 682 op/s 2024-04-26T21:48:33.162 INFO:tasks.workunit.client.0.smithi001.stdout: 10 118047 22.71 MB/sec execute 150 sec latency 250.034 ms 2024-04-26T21:48:33.204 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:48:32 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: ignoring --setuser ceph since I am not root 2024-04-26T21:48:33.204 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:48:32 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: ignoring --setgroup ceph since I am not root 2024-04-26T21:48:33.204 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:48:32 smithi186 ceph-mds[63876]: -- 172.21.15.186:0/1207113254 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d5e633a40 con 0x555d5e606800 2024-04-26T21:48:33.204 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:48:32 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: starting mds.i at 2024-04-26T21:48:33.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:33 smithi001 ceph-mon[28054]: daemon mds.i restarted 2024-04-26T21:48:33.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:33 smithi038 ceph-mon[32724]: daemon mds.i restarted 2024-04-26T21:48:33.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:33 smithi186 ceph-mon[32991]: daemon mds.i restarted 2024-04-26T21:48:34.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 118444 22.66 MB/sec execute 151 sec latency 98.676 ms 2024-04-26T21:48:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:34 smithi001 ceph-mon[28054]: pgmap v576: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 6.9 MiB/s wr, 403 op/s 2024-04-26T21:48:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:34 smithi001 ceph-mon[28054]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T21:48:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:34 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.186:6838/3084942661,v1:172.21.15.186:6839/3084942661] up:boot 2024-04-26T21:48:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:34 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T21:48:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:34 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:48:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:34 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:48:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:34 smithi038 ceph-mon[32724]: pgmap v576: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 6.9 MiB/s wr, 403 op/s 2024-04-26T21:48:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:34 smithi038 ceph-mon[32724]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T21:48:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:34 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.186:6838/3084942661,v1:172.21.15.186:6839/3084942661] up:boot 2024-04-26T21:48:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:34 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T21:48:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:34 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:48:34.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:34 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:48:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:34 smithi186 ceph-mon[32991]: pgmap v576: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 6.9 MiB/s wr, 403 op/s 2024-04-26T21:48:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:34 smithi186 ceph-mon[32991]: osdmap e123: 12 total, 12 up, 12 in 2024-04-26T21:48:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:34 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.186:6838/3084942661,v1:172.21.15.186:6839/3084942661] up:boot 2024-04-26T21:48:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:34 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-04-26T21:48:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:34 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:48:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:34 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:48:35.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 118894 22.67 MB/sec execute 152 sec latency 87.788 ms 2024-04-26T21:48:36.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 119423 22.69 MB/sec execute 153 sec latency 180.131 ms 2024-04-26T21:48:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:36 smithi001 ceph-mon[28054]: pgmap v578: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 7.8 MiB/s wr, 468 op/s 2024-04-26T21:48:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:36 smithi038 ceph-mon[32724]: pgmap v578: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 7.8 MiB/s wr, 468 op/s 2024-04-26T21:48:36.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:36 smithi186 ceph-mon[32991]: pgmap v578: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 7.8 MiB/s wr, 468 op/s 2024-04-26T21:48:37.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 120039 22.75 MB/sec execute 154 sec latency 98.813 ms 2024-04-26T21:48:38.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 120632 22.86 MB/sec execute 155 sec latency 81.716 ms 2024-04-26T21:48:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:38 smithi001 ceph-mon[28054]: pgmap v579: 129 pgs: 129 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.2 MiB/s wr, 491 op/s 2024-04-26T21:48:38.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:38 smithi038 ceph-mon[32724]: pgmap v579: 129 pgs: 129 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.2 MiB/s wr, 491 op/s 2024-04-26T21:48:38.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:38 smithi186 ceph-mon[32991]: pgmap v579: 129 pgs: 129 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.2 MiB/s wr, 491 op/s 2024-04-26T21:48:39.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 120768 22.75 MB/sec execute 156 sec latency 695.962 ms 2024-04-26T21:48:40.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 120983 22.68 MB/sec execute 157 sec latency 1127.659 ms 2024-04-26T21:48:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:40 smithi001 ceph-mon[28054]: pgmap v580: 129 pgs: 129 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.2 MiB/s wr, 491 op/s 2024-04-26T21:48:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:40 smithi038 ceph-mon[32724]: pgmap v580: 129 pgs: 129 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.2 MiB/s wr, 491 op/s 2024-04-26T21:48:40.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:40 smithi186 ceph-mon[32991]: pgmap v580: 129 pgs: 129 active+clean; 1.3 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 7.2 MiB/s wr, 491 op/s 2024-04-26T21:48:41.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 121221 22.61 MB/sec execute 158 sec latency 196.980 ms 2024-04-26T21:48:42.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 121478 22.54 MB/sec execute 159 sec latency 228.785 ms 2024-04-26T21:48:42.505 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:42 smithi186 ceph-mon[32991]: pgmap v581: 129 pgs: 129 active+clean; 1.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 525 op/s 2024-04-26T21:48:42.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:42 smithi001 ceph-mon[28054]: pgmap v581: 129 pgs: 129 active+clean; 1.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 525 op/s 2024-04-26T21:48:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:42 smithi038 ceph-mon[32724]: pgmap v581: 129 pgs: 129 active+clean; 1.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 525 op/s 2024-04-26T21:48:43.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 121988 22.59 MB/sec execute 160 sec latency 98.329 ms 2024-04-26T21:48:44.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 122408 22.59 MB/sec execute 161 sec latency 76.811 ms 2024-04-26T21:48:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:44 smithi001 ceph-mon[28054]: pgmap v582: 129 pgs: 129 active+clean; 1.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 525 op/s 2024-04-26T21:48:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:44 smithi038 ceph-mon[32724]: pgmap v582: 129 pgs: 129 active+clean; 1.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 525 op/s 2024-04-26T21:48:44.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:44 smithi186 ceph-mon[32991]: pgmap v582: 129 pgs: 129 active+clean; 1.3 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 MiB/s wr, 525 op/s 2024-04-26T21:48:45.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 122936 22.59 MB/sec execute 162 sec latency 80.825 ms 2024-04-26T21:48:46.163 INFO:tasks.workunit.client.0.smithi001.stdout: 10 123382 22.58 MB/sec execute 163 sec latency 188.338 ms 2024-04-26T21:48:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:46 smithi001 ceph-mon[28054]: pgmap v583: 129 pgs: 129 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 10 MiB/s wr, 579 op/s 2024-04-26T21:48:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:46 smithi038 ceph-mon[32724]: pgmap v583: 129 pgs: 129 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 10 MiB/s wr, 579 op/s 2024-04-26T21:48:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:46 smithi186 ceph-mon[32991]: pgmap v583: 129 pgs: 129 active+clean; 1.3 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 10 MiB/s wr, 579 op/s 2024-04-26T21:48:46.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.777+0000 7ff7de200640 1 -- 172.21.15.1:0/3052922079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d8107650 msgr2=0x7ff7d8107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:46.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.777+0000 7ff7de200640 1 --2- 172.21.15.1:0/3052922079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d8107650 0x7ff7d8107a30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009890 tx=0x7ff7c002a360 comp rx=0 tx=0).stop 2024-04-26T21:48:46.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.778+0000 7ff7de200640 1 -- 172.21.15.1:0/3052922079 shutdown_connections 2024-04-26T21:48:46.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.778+0000 7ff7de200640 1 --2- 172.21.15.1:0/3052922079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d8102000 0x7ff7d810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.778+0000 7ff7de200640 1 --2- 172.21.15.1:0/3052922079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7d8101640 0x7ff7d8101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.778+0000 7ff7de200640 1 --2- 172.21.15.1:0/3052922079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d8107650 0x7ff7d8107a30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.778+0000 7ff7de200640 1 -- 172.21.15.1:0/3052922079 >> 172.21.15.1:0/3052922079 conn(0x7ff7d80fd110 msgr2=0x7ff7d80ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.778+0000 7ff7de200640 1 -- 172.21.15.1:0/3052922079 shutdown_connections 2024-04-26T21:48:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.778+0000 7ff7de200640 1 -- 172.21.15.1:0/3052922079 wait complete. 2024-04-26T21:48:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.779+0000 7ff7de200640 1 Processor -- start 2024-04-26T21:48:46.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.779+0000 7ff7de200640 1 -- start start 2024-04-26T21:48:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.780+0000 7ff7de200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7d8101640 0x7ff7d8197490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.780+0000 7ff7de200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d8102000 0x7ff7d81979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.780+0000 7ff7de200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d8107650 0x7ff7d8073460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.780+0000 7ff7de200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d806da50 con 0x7ff7d8107650 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.780+0000 7ff7de200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d806d8d0 con 0x7ff7d8101640 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.780+0000 7ff7de200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d806dbd0 con 0x7ff7d8102000 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.780+0000 7ff7dce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7d8101640 0x7ff7d8197490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.781+0000 7ff7d7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d8102000 0x7ff7d81979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.781+0000 7ff7d7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d8102000 0x7ff7d81979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50800/0 (socket says 172.21.15.1:50800) 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.781+0000 7ff7d7e00640 1 -- 172.21.15.1:0/3972605155 learned_addr learned my addr 172.21.15.1:0/3972605155 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.781+0000 7ff7dd800640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d8107650 0x7ff7d8073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.781+0000 7ff7d7e00640 1 -- 172.21.15.1:0/3972605155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7d8101640 msgr2=0x7ff7d8197490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.781+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7d8101640 0x7ff7d8197490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.781+0000 7ff7d7e00640 1 -- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d8107650 msgr2=0x7ff7d8073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.781+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d8107650 0x7ff7d8073460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.781+0000 7ff7d7e00640 1 -- 172.21.15.1:0/3972605155 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d8073be0 con 0x7ff7d8102000 2024-04-26T21:48:46.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.781+0000 7ff7dd800640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d8107650 0x7ff7d8073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:48:46.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.782+0000 7ff7dce00640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7d8101640 0x7ff7d8197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:48:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.782+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d8102000 0x7ff7d81979d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff7c801fbc0 tx=0x7ff7c801d750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.782+0000 7ff7d5600640 1 -- 172.21.15.1:0/3972605155 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c802c070 con 0x7ff7d8102000 2024-04-26T21:48:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.782+0000 7ff7d5600640 1 -- 172.21.15.1:0/3972605155 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c801ca30 con 0x7ff7d8102000 2024-04-26T21:48:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.782+0000 7ff7de200640 1 -- 172.21.15.1:0/3972605155 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d8073ed0 con 0x7ff7d8102000 2024-04-26T21:48:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.782+0000 7ff7d5600640 1 -- 172.21.15.1:0/3972605155 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8027ba0 con 0x7ff7d8102000 2024-04-26T21:48:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.782+0000 7ff7de200640 1 -- 172.21.15.1:0/3972605155 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d819f4a0 con 0x7ff7d8102000 2024-04-26T21:48:46.787 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.787+0000 7ff7d5600640 1 -- 172.21.15.1:0/3972605155 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7ff7c802f070 con 0x7ff7d8102000 2024-04-26T21:48:46.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.790+0000 7ff7e25c1740 1 Processor -- start 2024-04-26T21:48:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.790+0000 7ff7d5600640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff7bc0a3e00 0x7ff7bc0a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.791+0000 7ff7dd800640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff7bc0a3e00 0x7ff7bc0a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.791+0000 7ff7d5600640 1 -- 172.21.15.1:0/3972605155 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9554+0+0 (secure 0 0 0) 0x7ff7c80e1d30 con 0x7ff7d8102000 2024-04-26T21:48:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.792+0000 7ff7e25c1740 1 -- start start 2024-04-26T21:48:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.793+0000 7ff7dd800640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff7bc0a3e00 0x7ff7bc0a62c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0009ad0 tx=0x7ff7d0005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.793+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24ba8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.793+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24ba9110 0x55ce24bb3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.794+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24bb39d0 0x55ce24bb5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.794+0000 7ff7dce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24ba8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.794+0000 7ff7dce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24ba8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60452/0 (socket says 172.21.15.1:60452) 2024-04-26T21:48:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.794+0000 7ff7dce00640 1 -- 172.21.15.1:0/1193029092 learned_addr learned my addr 172.21.15.1:0/1193029092 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.794+0000 7ff7dd800640 1 --2- 172.21.15.1:0/1193029092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24bb39d0 0x55ce24bb5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.794+0000 7ff7e25c1740 1 -- 172.21.15.1:0/1193029092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ce24b05380 con 0x55ce24ba9110 2024-04-26T21:48:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.794+0000 7ff7e25c1740 1 -- 172.21.15.1:0/1193029092 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ce24aa1c30 con 0x55ce24ba87f0 2024-04-26T21:48:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.794+0000 7ff7e25c1740 1 -- 172.21.15.1:0/1193029092 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ce24b01340 con 0x55ce24bb39d0 2024-04-26T21:48:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.794+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/1193029092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24ba9110 0x55ce24bb3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.795+0000 7ff7d7e00640 1 -- 172.21.15.1:0/1193029092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24bb39d0 msgr2=0x55ce24bb5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.795+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/1193029092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24bb39d0 0x55ce24bb5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.795+0000 7ff7d7e00640 1 -- 172.21.15.1:0/1193029092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 msgr2=0x55ce24ba8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.795+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/1193029092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24ba8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.795+0000 7ff7d7e00640 1 -- 172.21.15.1:0/1193029092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce24ac3940 con 0x55ce24ba9110 2024-04-26T21:48:46.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.796+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/1193029092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24ba9110 0x55ce24bb3490 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80df010 tx=0x7ff7c80adf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1365a9e0a41dd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:46.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.796+0000 7ff7ce000640 1 -- 172.21.15.1:0/1193029092 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80b64d0 con 0x55ce24ba9110 2024-04-26T21:48:46.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.796+0000 7ff7ce000640 1 -- 172.21.15.1:0/1193029092 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c801cbe0 con 0x55ce24ba9110 2024-04-26T21:48:46.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.796+0000 7ff7ce000640 1 -- 172.21.15.1:0/1193029092 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c80aeb90 con 0x55ce24ba9110 2024-04-26T21:48:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.797+0000 7ff7e25c1740 1 -- 172.21.15.1:0/1193029092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24ba9110 msgr2=0x55ce24bb3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.797+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/1193029092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24ba9110 0x55ce24bb3490 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff7c80df010 tx=0x7ff7c80adf60 comp rx=0 tx=0).stop 2024-04-26T21:48:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.797+0000 7ff7e25c1740 1 -- 172.21.15.1:0/1193029092 shutdown_connections 2024-04-26T21:48:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.797+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/1193029092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24bb39d0 0x55ce24bb5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.797+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/1193029092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24ba9110 0x55ce24bb3490 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.797+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/1193029092 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24ba8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.797+0000 7ff7e25c1740 1 -- 172.21.15.1:0/1193029092 >> 172.21.15.1:0/1193029092 conn(0x55ce24ba4d00 msgr2=0x55ce24ba50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.798+0000 7ff7e25c1740 1 -- 172.21.15.1:0/1193029092 shutdown_connections 2024-04-26T21:48:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.798+0000 7ff7e25c1740 1 -- 172.21.15.1:0/1193029092 wait complete. 2024-04-26T21:48:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.798+0000 7ff7e25c1740 1 -- start start 2024-04-26T21:48:46.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7e25c1740 1 Processor -- start 2024-04-26T21:48:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bb0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba9110 0x55ce24bb0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 0x55ce24bb1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7e25c1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ce24bb65a0 con 0x55ce24bb39d0 2024-04-26T21:48:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7e25c1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ce24a8cc60 con 0x55ce24ba87f0 2024-04-26T21:48:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7e25c1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ce24bb6720 con 0x55ce24ba9110 2024-04-26T21:48:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7e25c1740 10 client.0 _ll_register_callbacks cb 0x55ce24ab8e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:48:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7dce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bb0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7dce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bb0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60460/0 (socket says 172.21.15.1:60460) 2024-04-26T21:48:46.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.799+0000 7ff7dce00640 1 -- 172.21.15.1:0/4016295709 learned_addr learned my addr 172.21.15.1:0/4016295709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:46.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.800+0000 7ff7dce00640 1 -- 172.21.15.1:0/4016295709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba9110 msgr2=0x55ce24bb0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:46.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.800+0000 7ff7dd800640 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 0x55ce24bb1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.800+0000 7ff7dce00640 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba9110 0x55ce24bb0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.800+0000 7ff7dce00640 1 -- 172.21.15.1:0/4016295709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 msgr2=0x55ce24bb1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.800+0000 7ff7dce00640 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 0x55ce24bb1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.800+0000 7ff7dce00640 1 -- 172.21.15.1:0/4016295709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce24a9b510 con 0x55ce24ba87f0 2024-04-26T21:48:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.800+0000 7ff7dd800640 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 0x55ce24bb1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:48:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.800+0000 7ff7dce00640 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bb0a00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0035040 tx=0x7ff7c0030ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23a4ba4218ee44f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.800+0000 7ff7c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:48:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.801+0000 7ff7c6a00640 1 -- 172.21.15.1:0/4016295709 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0037040 con 0x55ce24ba87f0 2024-04-26T21:48:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.801+0000 7ff7c6a00640 1 -- 172.21.15.1:0/4016295709 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9554+0+0 (secure 0 0 0) 0x7ff7c0004450 con 0x55ce24ba87f0 2024-04-26T21:48:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.801+0000 7ff7e25c1740 1 -- 172.21.15.1:0/4016295709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c801cbe0 con 0x55ce24ba87f0 2024-04-26T21:48:46.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.802+0000 7ff7c6a00640 1 -- 172.21.15.1:0/4016295709 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7ff7c0030340 con 0x55ce24ba87f0 2024-04-26T21:48:46.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.803+0000 7ff7e25c1740 10 client.26912 fetch_fsmap learned FSMap version 57 2024-04-26T21:48:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.803+0000 7ff7e25c1740 1 -- 172.21.15.1:0/4016295709 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55ce24817210 con 0x55ce24ba87f0 2024-04-26T21:48:46.804 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.804+0000 7ff7c6a00640 1 -- 172.21.15.1:0/4016295709 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c801cbe0 con 0x55ce24ba87f0 2024-04-26T21:48:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.804+0000 7ff7c6a00640 1 -- 172.21.15.1:0/4016295709 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7ff7c00046c0 con 0x55ce24ba87f0 2024-04-26T21:48:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.804+0000 7ff7e25c1740 10 client.26912 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:48:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.805+0000 7ff7e25c1740 10 client.26912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:48:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.805+0000 7ff7e25c1740 20 client.26912 populate_metadata read hostname 'smithi001' 2024-04-26T21:48:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.805+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ce24bc37b0 0x55ce24bc5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.805+0000 7ff7e25c1740 4 client.26912 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:48:46.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.805+0000 7ff7e25c1740 1 -- 172.21.15.1:0/4016295709 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce24b12f90 con 0x55ce24bc37b0 2024-04-26T21:48:46.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.805+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ce24bc37b0 0x55ce24bc5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.806+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ce24bc37b0 0x55ce24bc5be0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc6821793f3eaeaa server_cookie=23af7a3b1c462df5 in_seq=0 out_seq=0 2024-04-26T21:48:46.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.806+0000 7ff7c6a00640 10 client.26912 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.809+0000 7ff7c6a00640 1 -- 172.21.15.1:0/4016295709 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce24b12f90 con 0x55ce24bc37b0 2024-04-26T21:48:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.809+0000 7ff7c6a00640 10 client.26912 handle_command_reply: tid=0 2024-04-26T21:48:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.809+0000 7ff7e25c1740 1 client.26912 shutdown 2024-04-26T21:48:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7e25c1740 1 -- 172.21.15.1:0/4016295709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 msgr2=0x55ce24bb0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bb0a00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0035040 tx=0x7ff7c0030ce0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7e25c1740 1 -- 172.21.15.1:0/4016295709 shutdown_connections 2024-04-26T21:48:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ce24bc37b0 0x55ce24bc5be0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba9110 0x55ce24bb0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 0x55ce24bb1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/4016295709 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bb0a00 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7e25c1740 1 -- 172.21.15.1:0/4016295709 >> 172.21.15.1:0/4016295709 conn(0x55ce24ba4d00 msgr2=0x55ce24b9b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7c6a00640 0 client.26912 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7e25c1740 1 -- 172.21.15.1:0/4016295709 shutdown_connections 2024-04-26T21:48:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.810+0000 7ff7e25c1740 1 -- 172.21.15.1:0/4016295709 wait complete. 2024-04-26T21:48:46.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.811+0000 7ff7e25c1740 20 client.26912 trim_cache size 0 max 16384 2024-04-26T21:48:46.837 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.837+0000 7ff7e25c1740 1 Processor -- start 2024-04-26T21:48:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.838+0000 7ff7e25c1740 1 -- start start 2024-04-26T21:48:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.839+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bbf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.839+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba9110 0x55ce24bab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.839+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 0x55ce24b9b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.839+0000 7ff7e25c1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d81a7b20 con 0x55ce24bb39d0 2024-04-26T21:48:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.839+0000 7ff7e25c1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d81a53f0 con 0x55ce24ba87f0 2024-04-26T21:48:46.839 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.839+0000 7ff7e25c1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d81a3280 con 0x55ce24ba9110 2024-04-26T21:48:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.839+0000 7ff7dce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bbf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.839+0000 7ff7dce00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bbf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:60474/0 (socket says 172.21.15.1:60474) 2024-04-26T21:48:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.839+0000 7ff7dce00640 1 -- 172.21.15.1:0/2543977493 learned_addr learned my addr 172.21.15.1:0/2543977493 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.839+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/2543977493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba9110 0x55ce24bab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7dd800640 1 --2- 172.21.15.1:0/2543977493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 0x55ce24b9b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7dce00640 1 -- 172.21.15.1:0/2543977493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba9110 msgr2=0x55ce24bab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7dce00640 1 --2- 172.21.15.1:0/2543977493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba9110 0x55ce24bab1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7dce00640 1 -- 172.21.15.1:0/2543977493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 msgr2=0x55ce24b9b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7dce00640 1 --2- 172.21.15.1:0/2543977493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 0x55ce24b9b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7dce00640 1 -- 172.21.15.1:0/2543977493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce248e3620 con 0x55ce24ba87f0 2024-04-26T21:48:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/2543977493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba9110 0x55ce24bab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:48:46.841 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7dce00640 1 --2- 172.21.15.1:0/2543977493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bbf460 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff7c002d020 tx=0x7ff7c0030fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98ae83b6efdb200e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7cd600640 1 -- 172.21.15.1:0/2543977493 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0054070 con 0x55ce24ba87f0 2024-04-26T21:48:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7cd600640 1 -- 172.21.15.1:0/2543977493 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c0030d50 con 0x55ce24ba87f0 2024-04-26T21:48:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.840+0000 7ff7cd600640 1 -- 172.21.15.1:0/2543977493 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0037030 con 0x55ce24ba87f0 2024-04-26T21:48:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.841+0000 7ff7e25c1740 1 -- 172.21.15.1:0/2543977493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 msgr2=0x55ce24bbf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:46.842 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.841+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/2543977493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bbf460 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff7c002d020 tx=0x7ff7c0030fc0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.842+0000 7ff7e25c1740 1 -- 172.21.15.1:0/2543977493 shutdown_connections 2024-04-26T21:48:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.842+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/2543977493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba9110 0x55ce24bab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.842+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/2543977493 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24bb39d0 0x55ce24b9b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.842+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/2543977493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24ba87f0 0x55ce24bbf460 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.844+0000 7ff7e25c1740 1 -- 172.21.15.1:0/2543977493 >> 172.21.15.1:0/2543977493 conn(0x55ce24ba4d00 msgr2=0x55ce24b9adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:46.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.844+0000 7ff7e25c1740 1 -- 172.21.15.1:0/2543977493 shutdown_connections 2024-04-26T21:48:46.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.844+0000 7ff7e25c1740 1 -- 172.21.15.1:0/2543977493 wait complete. 2024-04-26T21:48:46.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.845+0000 7ff7e25c1740 1 -- start start 2024-04-26T21:48:46.845 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.845+0000 7ff7e25c1740 1 Processor -- start 2024-04-26T21:48:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.846+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba87f0 0x55ce24bad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.846+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24ba9110 0x55ce24badf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.846+0000 7ff7e25c1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24bb39d0 0x55ce24c28d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.846 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.846+0000 7ff7e25c1740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d81a38d0 con 0x55ce24ba9110 2024-04-26T21:48:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.846+0000 7ff7e25c1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d81a4d20 con 0x55ce24bb39d0 2024-04-26T21:48:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.846+0000 7ff7e25c1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d81a3f80 con 0x55ce24ba87f0 2024-04-26T21:48:46.847 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.846+0000 7ff7e25c1740 10 client.0 _ll_register_callbacks cb 0x7ff7c0035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:48:46.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.846+0000 7ff7dd800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24bb39d0 0x55ce24c28d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.846+0000 7ff7dce00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba87f0 0x55ce24bad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.847+0000 7ff7dce00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba87f0 0x55ce24bad9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:50828/0 (socket says 172.21.15.1:50828) 2024-04-26T21:48:46.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.847+0000 7ff7dce00640 1 -- 172.21.15.1:0/3745141143 learned_addr learned my addr 172.21.15.1:0/3745141143 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:48:46.848 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.847+0000 7ff7dce00640 1 -- 172.21.15.1:0/3745141143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24bb39d0 msgr2=0x55ce24c28d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:46.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.847+0000 7ff7dce00640 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24bb39d0 0x55ce24c28d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.847+0000 7ff7dce00640 1 -- 172.21.15.1:0/3745141143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24ba9110 msgr2=0x55ce24badf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:48:46.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.847+0000 7ff7dce00640 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24ba9110 0x55ce24badf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:46.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.847+0000 7ff7dce00640 1 -- 172.21.15.1:0/3745141143 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce24abfcb0 con 0x55ce24ba87f0 2024-04-26T21:48:46.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.848+0000 7ff7dce00640 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba87f0 0x55ce24bad9f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0004ca0 tx=0x7ff7c0004500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=825114a6a9037cd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:48:46.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.848+0000 7ff7c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:48:46.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.848+0000 7ff7c6a00640 1 -- 172.21.15.1:0/3745141143 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0054040 con 0x55ce24ba87f0 2024-04-26T21:48:46.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.848+0000 7ff7c6a00640 1 -- 172.21.15.1:0/3745141143 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9554+0+0 (secure 0 0 0) 0x7ff7c00307a0 con 0x55ce24ba87f0 2024-04-26T21:48:46.849 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.848+0000 7ff7e25c1740 1 -- 172.21.15.1:0/3745141143 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c0030d50 con 0x55ce24ba87f0 2024-04-26T21:48:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.849+0000 7ff7c6a00640 1 -- 172.21.15.1:0/3745141143 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7ff7c0062050 con 0x55ce24ba87f0 2024-04-26T21:48:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.849+0000 7ff7e25c1740 10 client.26920 fetch_fsmap learned FSMap version 57 2024-04-26T21:48:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.849+0000 7ff7e25c1740 1 -- 172.21.15.1:0/3745141143 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55ce24c2a1a0 con 0x55ce24ba87f0 2024-04-26T21:48:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.849+0000 7ff7c6a00640 1 -- 172.21.15.1:0/3745141143 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c0030d50 con 0x55ce24ba87f0 2024-04-26T21:48:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.850+0000 7ff7c6a00640 1 -- 172.21.15.1:0/3745141143 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7ff7c00050d0 con 0x55ce24ba87f0 2024-04-26T21:48:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.850+0000 7ff7e25c1740 10 client.26920 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:48:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.850+0000 7ff7e25c1740 10 client.26920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:48:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.850+0000 7ff7e25c1740 20 client.26920 populate_metadata read hostname 'smithi001' 2024-04-26T21:48:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.850+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ce24c2a800 0x55ce24c2cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:48:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.850+0000 7ff7e25c1740 4 client.26920 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:48:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.850+0000 7ff7e25c1740 1 -- 172.21.15.1:0/3745141143 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ce24b0ce00 con 0x55ce24c2a800 2024-04-26T21:48:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.851+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ce24c2a800 0x55ce24c2cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:48:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.851+0000 7ff7d7e00640 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ce24c2a800 0x55ce24c2cc10 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21e9cd9471ca58d server_cookie=21b9bd994bb6bf1e in_seq=0 out_seq=0 2024-04-26T21:48:46.852 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:46.851+0000 7ff7c6a00640 10 client.26920 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:47.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 123800 22.62 MB/sec execute 164 sec latency 215.848 ms 2024-04-26T21:48:47.201 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.201+0000 7ff7c6a00640 1 -- 172.21.15.1:0/3745141143 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ce24b0ce00 con 0x55ce24c2a800 2024-04-26T21:48:47.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.201+0000 7ff7c6a00640 10 client.26920 handle_command_reply: tid=0 2024-04-26T21:48:47.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.201+0000 7ff7e25c1740 1 client.26920 shutdown 2024-04-26T21:48:47.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.202+0000 7ff7e25c1740 1 -- 172.21.15.1:0/3745141143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba87f0 msgr2=0x55ce24bad9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:47.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.202+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba87f0 0x55ce24bad9f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0004ca0 tx=0x7ff7c0004500 comp rx=0 tx=0).stop 2024-04-26T21:48:47.202 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.202+0000 7ff7e25c1740 1 -- 172.21.15.1:0/3745141143 shutdown_connections 2024-04-26T21:48:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.202+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ce24c2a800 0x55ce24c2cc10 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.202+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ce24bb39d0 0x55ce24c28d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.202+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ce24ba9110 0x55ce24badf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.202+0000 7ff7e25c1740 1 --2- 172.21.15.1:0/3745141143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ce24ba87f0 0x55ce24bad9f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.202+0000 7ff7e25c1740 1 -- 172.21.15.1:0/3745141143 >> 172.21.15.1:0/3745141143 conn(0x55ce24ba4d00 msgr2=0x55ce24b9b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.202+0000 7ff7c6a00640 0 client.26920 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:48:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.203+0000 7ff7e25c1740 1 -- 172.21.15.1:0/3745141143 shutdown_connections 2024-04-26T21:48:47.203 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.203+0000 7ff7e25c1740 1 -- 172.21.15.1:0/3745141143 wait complete. 2024-04-26T21:48:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.203+0000 7ff7e25c1740 20 client.26920 trim_cache size 0 max 16384 2024-04-26T21:48:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:48:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:48:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:48:47.204 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:48:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.204+0000 7ff7de200640 1 -- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff7bc0a3e00 msgr2=0x7ff7bc0a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.204+0000 7ff7de200640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff7bc0a3e00 0x7ff7bc0a62c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0009ad0 tx=0x7ff7d0005f90 comp rx=0 tx=0).stop 2024-04-26T21:48:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.204+0000 7ff7de200640 1 -- 172.21.15.1:0/3972605155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d8102000 msgr2=0x7ff7d81979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:48:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.204+0000 7ff7de200640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d8102000 0x7ff7d81979d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff7c801fbc0 tx=0x7ff7c801d750 comp rx=0 tx=0).stop 2024-04-26T21:48:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.205+0000 7ff7de200640 1 -- 172.21.15.1:0/3972605155 shutdown_connections 2024-04-26T21:48:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.205+0000 7ff7de200640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7ff7bc0a3e00 0x7ff7bc0a62c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.205+0000 7ff7de200640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d8102000 0x7ff7d81979d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.205+0000 7ff7de200640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d8107650 0x7ff7d8073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.205+0000 7ff7de200640 1 --2- 172.21.15.1:0/3972605155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7d8101640 0x7ff7d8197490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:48:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.205+0000 7ff7de200640 1 -- 172.21.15.1:0/3972605155 >> 172.21.15.1:0/3972605155 conn(0x7ff7d80fd110 msgr2=0x7ff7d810c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:48:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.205+0000 7ff7de200640 1 -- 172.21.15.1:0/3972605155 shutdown_connections 2024-04-26T21:48:47.206 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:48:47.205+0000 7ff7de200640 1 -- 172.21.15.1:0/3972605155 wait complete. 2024-04-26T21:48:48.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 124384 22.70 MB/sec execute 165 sec latency 214.777 ms 2024-04-26T21:48:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:48 smithi001 ceph-mon[28054]: pgmap v584: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.6 MiB/s wr, 499 op/s 2024-04-26T21:48:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:48 smithi038 ceph-mon[32724]: pgmap v584: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.6 MiB/s wr, 499 op/s 2024-04-26T21:48:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:48 smithi186 ceph-mon[32991]: pgmap v584: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.6 MiB/s wr, 499 op/s 2024-04-26T21:48:49.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 124841 22.69 MB/sec execute 166 sec latency 94.834 ms 2024-04-26T21:48:50.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 125252 22.67 MB/sec execute 167 sec latency 161.287 ms 2024-04-26T21:48:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:50 smithi001 ceph-mon[28054]: pgmap v585: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.0 MiB/s wr, 430 op/s 2024-04-26T21:48:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:50 smithi038 ceph-mon[32724]: pgmap v585: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.0 MiB/s wr, 430 op/s 2024-04-26T21:48:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:50 smithi186 ceph-mon[32991]: pgmap v585: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 9.0 MiB/s wr, 430 op/s 2024-04-26T21:48:51.166 INFO:tasks.workunit.client.0.smithi001.stdout: 10 125768 22.70 MB/sec execute 168 sec latency 83.497 ms 2024-04-26T21:48:52.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 126367 22.77 MB/sec execute 169 sec latency 89.547 ms 2024-04-26T21:48:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:52 smithi001 ceph-mon[28054]: pgmap v586: 129 pgs: 129 active+clean; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 645 op/s 2024-04-26T21:48:52.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:52 smithi038 ceph-mon[32724]: pgmap v586: 129 pgs: 129 active+clean; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 645 op/s 2024-04-26T21:48:52.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:52 smithi186 ceph-mon[32991]: pgmap v586: 129 pgs: 129 active+clean; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 11 MiB/s wr, 645 op/s 2024-04-26T21:48:53.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 126903 22.81 MB/sec execute 170 sec latency 87.391 ms 2024-04-26T21:48:54.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 127323 22.82 MB/sec execute 171 sec latency 172.554 ms 2024-04-26T21:48:54.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:54 smithi001 ceph-mon[28054]: pgmap v587: 129 pgs: 129 active+clean; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.1 MiB/s wr, 428 op/s 2024-04-26T21:48:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:54 smithi038 ceph-mon[32724]: pgmap v587: 129 pgs: 129 active+clean; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.1 MiB/s wr, 428 op/s 2024-04-26T21:48:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:54 smithi186 ceph-mon[32991]: pgmap v587: 129 pgs: 129 active+clean; 1.1 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.1 MiB/s wr, 428 op/s 2024-04-26T21:48:55.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 127825 22.84 MB/sec execute 172 sec latency 121.565 ms 2024-04-26T21:48:56.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 128149 22.79 MB/sec execute 173 sec latency 233.185 ms 2024-04-26T21:48:56.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:56 smithi001 ceph-mon[28054]: pgmap v588: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 573 op/s 2024-04-26T21:48:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:56 smithi038 ceph-mon[32724]: pgmap v588: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 573 op/s 2024-04-26T21:48:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:56 smithi186 ceph-mon[32991]: pgmap v588: 129 pgs: 129 active+clean; 1.2 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.9 MiB/s wr, 573 op/s 2024-04-26T21:48:57.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 128579 22.79 MB/sec execute 174 sec latency 233.891 ms 2024-04-26T21:48:58.165 INFO:tasks.workunit.client.0.smithi001.stdout: 10 129218 22.86 MB/sec execute 175 sec latency 95.908 ms 2024-04-26T21:48:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:48:58 smithi001 ceph-mon[28054]: pgmap v589: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 5.2 MiB/s wr, 485 op/s 2024-04-26T21:48:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:48:58 smithi038 ceph-mon[32724]: pgmap v589: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 5.2 MiB/s wr, 485 op/s 2024-04-26T21:48:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:48:58 smithi186 ceph-mon[32991]: pgmap v589: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 5.2 MiB/s wr, 485 op/s 2024-04-26T21:48:59.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 129827 22.94 MB/sec execute 176 sec latency 74.287 ms 2024-04-26T21:49:00.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 130324 22.96 MB/sec execute 177 sec latency 92.362 ms 2024-04-26T21:49:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:00 smithi001 ceph-mon[28054]: pgmap v590: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.4 MiB/s wr, 410 op/s 2024-04-26T21:49:00.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:00 smithi038 ceph-mon[32724]: pgmap v590: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.4 MiB/s wr, 410 op/s 2024-04-26T21:49:00.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:00 smithi186 ceph-mon[32991]: pgmap v590: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.4 MiB/s wr, 410 op/s 2024-04-26T21:49:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.099+0000 7fa215c00640 1 -- 172.21.15.1:0/2293610517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa210105510 msgr2=0x7fa2101119d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.099+0000 7fa215c00640 1 --2- 172.21.15.1:0/2293610517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa210105510 0x7fa2101119d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa208009c10 tx=0x7fa20802a6e0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.099+0000 7fa215c00640 1 -- 172.21.15.1:0/2293610517 shutdown_connections 2024-04-26T21:49:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.099+0000 7fa215c00640 1 --2- 172.21.15.1:0/2293610517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa210104b50 0x7fa210104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.099+0000 7fa215c00640 1 --2- 172.21.15.1:0/2293610517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa210105510 0x7fa2101119d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.099+0000 7fa215c00640 1 --2- 172.21.15.1:0/2293610517 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21010ac10 0x7fa21010aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.099+0000 7fa215c00640 1 -- 172.21.15.1:0/2293610517 >> 172.21.15.1:0/2293610517 conn(0x7fa210100600 msgr2=0x7fa210102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.099+0000 7fa215c00640 1 -- 172.21.15.1:0/2293610517 shutdown_connections 2024-04-26T21:49:01.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.099+0000 7fa215c00640 1 -- 172.21.15.1:0/2293610517 wait complete. 2024-04-26T21:49:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.100+0000 7fa215c00640 1 Processor -- start 2024-04-26T21:49:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.100+0000 7fa215c00640 1 -- start start 2024-04-26T21:49:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa215c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa210104b50 0x7fa21019b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa215c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa210105510 0x7fa21019b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa215c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21010ac10 0x7fa21019fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa215c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa210114dc0 con 0x7fa21010ac10 2024-04-26T21:49:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa215c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa210114c40 con 0x7fa210105510 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa215c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa210114f40 con 0x7fa210104b50 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa20ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa210104b50 0x7fa21019b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa20ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa210104b50 0x7fa21019b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:49460/0 (socket says 172.21.15.1:49460) 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa20ea00640 1 -- 172.21.15.1:0/2526495392 learned_addr learned my addr 172.21.15.1:0/2526495392 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa20f400640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21010ac10 0x7fa21019fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.101+0000 7fa207e00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa210105510 0x7fa21019b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.102+0000 7fa20ea00640 1 -- 172.21.15.1:0/2526495392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa210105510 msgr2=0x7fa21019b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.102+0000 7fa20ea00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa210105510 0x7fa21019b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.102+0000 7fa20ea00640 1 -- 172.21.15.1:0/2526495392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21010ac10 msgr2=0x7fa21019fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.102+0000 7fa20ea00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21010ac10 0x7fa21019fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.102+0000 7fa20ea00640 1 -- 172.21.15.1:0/2526495392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2101a0240 con 0x7fa210104b50 2024-04-26T21:49:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.102+0000 7fa20ea00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa210104b50 0x7fa21019b2a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa20000e9c0 tx=0x7fa20000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.102+0000 7fa207400640 1 -- 172.21.15.1:0/2526495392 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20000cde0 con 0x7fa210104b50 2024-04-26T21:49:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.103+0000 7fa215c00640 1 -- 172.21.15.1:0/2526495392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2101a0530 con 0x7fa210104b50 2024-04-26T21:49:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.103+0000 7fa207400640 1 -- 172.21.15.1:0/2526495392 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa2000035f0 con 0x7fa210104b50 2024-04-26T21:49:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.103+0000 7fa207400640 1 -- 172.21.15.1:0/2526495392 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200010690 con 0x7fa210104b50 2024-04-26T21:49:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.103+0000 7fa215c00640 1 -- 172.21.15.1:0/2526495392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2101a7dc0 con 0x7fa210104b50 2024-04-26T21:49:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.106+0000 7fa218a51740 1 Processor -- start 2024-04-26T21:49:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.106+0000 7fa207400640 1 -- 172.21.15.1:0/2526495392 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fa200004000 con 0x7fa210104b50 2024-04-26T21:49:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.106+0000 7fa218a51740 1 -- start start 2024-04-26T21:49:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa218a51740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 0x5645768061b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa218a51740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5645768066f0 0x564576810a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa218a51740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576810fb0 0x5645768133a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa218a51740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x56457676b380 con 0x5645768066f0 2024-04-26T21:49:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa218a51740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564576707c30 con 0x564576805dd0 2024-04-26T21:49:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa218a51740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564576767340 con 0x564576810fb0 2024-04-26T21:49:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa20ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5645768066f0 0x564576810a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa20f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 0x5645768061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa20ea00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5645768066f0 0x564576810a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42034/0 (socket says 172.21.15.1:42034) 2024-04-26T21:49:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa207e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576810fb0 0x5645768133a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.107+0000 7fa20ea00640 1 -- 172.21.15.1:0/3490945622 learned_addr learned my addr 172.21.15.1:0/3490945622 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.108+0000 7fa20f400640 1 -- 172.21.15.1:0/3490945622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576810fb0 msgr2=0x5645768133a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.108+0000 7fa20f400640 1 --2- 172.21.15.1:0/3490945622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576810fb0 0x5645768133a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.108+0000 7fa20f400640 1 -- 172.21.15.1:0/3490945622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5645768066f0 msgr2=0x564576810a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.108+0000 7fa20f400640 1 --2- 172.21.15.1:0/3490945622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5645768066f0 0x564576810a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.108+0000 7fa20f400640 1 -- 172.21.15.1:0/3490945622 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564576729940 con 0x564576805dd0 2024-04-26T21:49:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.108+0000 7fa207400640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa1f40a3d90 0x7fa1f40a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.108+0000 7fa20f400640 1 --2- 172.21.15.1:0/3490945622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 0x5645768061b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa208034040 tx=0x7fa208030430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0cb242f8e408d5c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.108+0000 7fa207400640 1 -- 172.21.15.1:0/2526495392 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 9554+0+0 (secure 0 0 0) 0x7fa2000d0f10 con 0x7fa210104b50 2024-04-26T21:49:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.108+0000 7fa20ea00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa1f40a3d90 0x7fa1f40a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.109+0000 7fa1ffe00640 1 -- 172.21.15.1:0/3490945622 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208041030 con 0x564576805dd0 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.109+0000 7fa1ffe00640 1 -- 172.21.15.1:0/3490945622 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa208003e20 con 0x564576805dd0 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.109+0000 7fa1ffe00640 1 -- 172.21.15.1:0/3490945622 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208036040 con 0x564576805dd0 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.109+0000 7fa218a51740 1 -- 172.21.15.1:0/3490945622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 msgr2=0x5645768061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.109+0000 7fa218a51740 1 --2- 172.21.15.1:0/3490945622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 0x5645768061b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fa208034040 tx=0x7fa208030430 comp rx=0 tx=0).stop 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.109+0000 7fa20ea00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa1f40a3d90 0x7fa1f40a6250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa200000c00 tx=0x7fa2000d49c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.110+0000 7fa218a51740 1 -- 172.21.15.1:0/3490945622 shutdown_connections 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.110+0000 7fa218a51740 1 --2- 172.21.15.1:0/3490945622 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576810fb0 0x5645768133a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.110+0000 7fa218a51740 1 --2- 172.21.15.1:0/3490945622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5645768066f0 0x564576810a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.110+0000 7fa218a51740 1 --2- 172.21.15.1:0/3490945622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 0x5645768061b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.110+0000 7fa218a51740 1 -- 172.21.15.1:0/3490945622 >> 172.21.15.1:0/3490945622 conn(0x5645768022e0 msgr2=0x5645768026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.110+0000 7fa218a51740 1 -- 172.21.15.1:0/3490945622 shutdown_connections 2024-04-26T21:49:01.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.110+0000 7fa218a51740 1 -- 172.21.15.1:0/3490945622 wait complete. 2024-04-26T21:49:01.114 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.110+0000 7fa218a51740 1 -- start start 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.111+0000 7fa218a51740 1 Processor -- start 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.111+0000 7fa218a51740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 0x56457680df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.111+0000 7fa218a51740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 0x56457680e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.111+0000 7fa218a51740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 0x56457680ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.111+0000 7fa218a51740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564576813b80 con 0x564576810fb0 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.111+0000 7fa218a51740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5645766f2c60 con 0x564576805dd0 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.111+0000 7fa218a51740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564576813d00 con 0x5645768066f0 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.111+0000 7fa218a51740 10 client.0 _ll_register_callbacks cb 0x56457671ee50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa20f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 0x56457680ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa20f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 0x56457680df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa207e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 0x56457680e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.115 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa20f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 0x56457680ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42046/0 (socket says 172.21.15.1:42046) 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa20f400640 1 -- 172.21.15.1:0/306168553 learned_addr learned my addr 172.21.15.1:0/306168553 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa20f400640 1 -- 172.21.15.1:0/306168553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 msgr2=0x56457680e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa20f400640 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 0x56457680e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa20f400640 1 -- 172.21.15.1:0/306168553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 msgr2=0x56457680df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa20f400640 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 0x56457680df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa20f400640 1 -- 172.21.15.1:0/306168553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564576701510 con 0x564576810fb0 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa207e00640 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 0x56457680e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa20f400640 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 0x56457680ea00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa208030a00 tx=0x7fa208005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b12bbe1c9b90f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.112+0000 7fa1fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.113+0000 7fa1fcc00640 1 -- 172.21.15.1:0/306168553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208036070 con 0x564576810fb0 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.113+0000 7fa1fcc00640 1 -- 172.21.15.1:0/306168553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9554+0+0 (secure 0 0 0) 0x7fa20804b030 con 0x564576810fb0 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.113+0000 7fa218a51740 1 -- 172.21.15.1:0/306168553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa208003e20 con 0x564576810fb0 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.113+0000 7fa1fcc00640 1 -- 172.21.15.1:0/306168553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fa208058050 con 0x564576810fb0 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.113+0000 7fa218a51740 10 client.16635 fetch_fsmap learned FSMap version 57 2024-04-26T21:49:01.116 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.113+0000 7fa218a51740 1 -- 172.21.15.1:0/306168553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x56457647d210 con 0x564576810fb0 2024-04-26T21:49:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.113+0000 7fa1fcc00640 1 -- 172.21.15.1:0/306168553 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa208003e20 con 0x564576810fb0 2024-04-26T21:49:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.114+0000 7fa1fcc00640 1 -- 172.21.15.1:0/306168553 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fa2080045a0 con 0x564576810fb0 2024-04-26T21:49:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.114+0000 7fa218a51740 10 client.16635 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:49:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.114+0000 7fa218a51740 10 client.16635 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:49:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.114+0000 7fa218a51740 20 client.16635 populate_metadata read hostname 'smithi001' 2024-04-26T21:49:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.114+0000 7fa218a51740 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564576820b70 0x564576822f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.114+0000 7fa218a51740 4 client.16635 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:49:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.114+0000 7fa218a51740 1 -- 172.21.15.1:0/306168553 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564576778f90 con 0x564576820b70 2024-04-26T21:49:01.117 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.114+0000 7fa207e00640 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564576820b70 0x564576822f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.115+0000 7fa207e00640 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564576820b70 0x564576822f60 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c873df7f3d03dca server_cookie=4d7b91b521491365 in_seq=0 out_seq=0 2024-04-26T21:49:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.115+0000 7fa1fcc00640 10 client.16635 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa1fcc00640 1 -- 172.21.15.1:0/306168553 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564576778f90 con 0x564576820b70 2024-04-26T21:49:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa1fcc00640 10 client.16635 handle_command_reply: tid=0 2024-04-26T21:49:01.118 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 client.16635 shutdown 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 -- 172.21.15.1:0/306168553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 msgr2=0x56457680ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 0x56457680ea00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa208030a00 tx=0x7fa208005fd0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 -- 172.21.15.1:0/306168553 shutdown_connections 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564576820b70 0x564576822f60 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 0x56457680e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 0x56457680ea00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 --2- 172.21.15.1:0/306168553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576805dd0 0x56457680df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa1fcc00640 0 client.16635 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 -- 172.21.15.1:0/306168553 >> 172.21.15.1:0/306168553 conn(0x5645768022e0 msgr2=0x564576808060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 -- 172.21.15.1:0/306168553 shutdown_connections 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.116+0000 7fa218a51740 1 -- 172.21.15.1:0/306168553 wait complete. 2024-04-26T21:49:01.119 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.117+0000 7fa218a51740 20 client.16635 trim_cache size 0 max 16384 2024-04-26T21:49:01.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.133+0000 7fa218a51740 1 Processor -- start 2024-04-26T21:49:01.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.133+0000 7fa218a51740 1 -- start start 2024-04-26T21:49:01.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.133+0000 7fa218a51740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576805dd0 0x5645768106b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.133+0000 7fa218a51740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645768066f0 0x56457681ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.134 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.133+0000 7fa218a51740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 0x564576828610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.133+0000 7fa218a51740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa2101b4800 con 0x564576810fb0 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.133+0000 7fa218a51740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2101b20d0 con 0x5645768066f0 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.133+0000 7fa218a51740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2101aff20 con 0x564576805dd0 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa20f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576805dd0 0x5645768106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa20f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 0x564576828610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa20f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576805dd0 0x5645768106b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:49494/0 (socket says 172.21.15.1:49494) 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa20f400640 1 -- 172.21.15.1:0/3488246731 learned_addr learned my addr 172.21.15.1:0/3488246731 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa207e00640 1 --2- 172.21.15.1:0/3488246731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645768066f0 0x56457681ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa20f400640 1 -- 172.21.15.1:0/3488246731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645768066f0 msgr2=0x56457681ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa20f400640 1 --2- 172.21.15.1:0/3488246731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645768066f0 0x56457681ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa20f400640 1 -- 172.21.15.1:0/3488246731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 msgr2=0x564576828610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa20f400640 1 --2- 172.21.15.1:0/3488246731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 0x564576828610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.135 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa20f400640 1 -- 172.21.15.1:0/3488246731 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564576549620 con 0x564576805dd0 2024-04-26T21:49:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.134+0000 7fa20f400640 1 --2- 172.21.15.1:0/3488246731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576805dd0 0x5645768106b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x56457681dc70 tx=0x7fa208004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd795beddf2332ef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa1ff400640 1 -- 172.21.15.1:0/3488246731 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208036040 con 0x564576805dd0 2024-04-26T21:49:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa1ff400640 1 -- 172.21.15.1:0/3488246731 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa20804a020 con 0x564576805dd0 2024-04-26T21:49:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa1ff400640 1 -- 172.21.15.1:0/3488246731 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208041040 con 0x564576805dd0 2024-04-26T21:49:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa218a51740 1 -- 172.21.15.1:0/3488246731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576805dd0 msgr2=0x5645768106b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa218a51740 1 --2- 172.21.15.1:0/3488246731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576805dd0 0x5645768106b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x56457681dc70 tx=0x7fa208004a40 comp rx=0 tx=0).stop 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa218a51740 1 -- 172.21.15.1:0/3488246731 shutdown_connections 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa218a51740 1 --2- 172.21.15.1:0/3488246731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576810fb0 0x564576828610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa218a51740 1 --2- 172.21.15.1:0/3488246731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5645768066f0 0x56457681ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa218a51740 1 --2- 172.21.15.1:0/3488246731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564576805dd0 0x5645768106b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa218a51740 1 -- 172.21.15.1:0/3488246731 >> 172.21.15.1:0/3488246731 conn(0x5645768022e0 msgr2=0x56457680e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa218a51740 1 -- 172.21.15.1:0/3488246731 shutdown_connections 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.135+0000 7fa218a51740 1 -- 172.21.15.1:0/3488246731 wait complete. 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.136+0000 7fa218a51740 1 -- start start 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.136+0000 7fa218a51740 1 Processor -- start 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.136+0000 7fa218a51740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576805dd0 0x564576814a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa218a51740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 0x564576814fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa218a51740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576810fb0 0x5645768154e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa218a51740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa2101b0590 con 0x564576805dd0 2024-04-26T21:49:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa218a51740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fa2101b1a00 con 0x564576810fb0 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa218a51740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2101b0c60 con 0x5645768066f0 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa218a51740 10 client.0 _ll_register_callbacks cb 0x7fa2000d4890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa20f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576805dd0 0x564576814a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa20f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576805dd0 0x564576814a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42066/0 (socket says 172.21.15.1:42066) 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa20f400640 1 -- 172.21.15.1:0/3878807370 learned_addr learned my addr 172.21.15.1:0/3878807370 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa207e00640 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 0x564576814fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa20f400640 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576810fb0 0x5645768154e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa20f400640 1 -- 172.21.15.1:0/3878807370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 msgr2=0x564576814fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa20f400640 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 0x564576814fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa20f400640 1 -- 172.21.15.1:0/3878807370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576810fb0 msgr2=0x5645768154e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa20f400640 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576810fb0 0x5645768154e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.137+0000 7fa20f400640 1 -- 172.21.15.1:0/3878807370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564576725cb0 con 0x564576805dd0 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.138+0000 7fa20f400640 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576805dd0 0x564576814a60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa208030910 tx=0x7fa208005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20525047f6e06f9e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.138+0000 7fa1fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.138+0000 7fa1fcc00640 1 -- 172.21.15.1:0/3878807370 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208041070 con 0x564576805dd0 2024-04-26T21:49:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.138+0000 7fa1fcc00640 1 -- 172.21.15.1:0/3878807370 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(123..123 src has 1..123) ==== 9554+0+0 (secure 0 0 0) 0x7fa20804b040 con 0x564576805dd0 2024-04-26T21:49:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.138+0000 7fa218a51740 1 -- 172.21.15.1:0/3878807370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa20804a020 con 0x564576805dd0 2024-04-26T21:49:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.138+0000 7fa1fcc00640 1 -- 172.21.15.1:0/3878807370 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fa208066050 con 0x564576805dd0 2024-04-26T21:49:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.138+0000 7fa218a51740 10 client.16641 fetch_fsmap learned FSMap version 57 2024-04-26T21:49:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.138+0000 7fa218a51740 1 -- 172.21.15.1:0/3878807370 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5645768870a0 con 0x564576805dd0 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.138+0000 7fa1fcc00640 1 -- 172.21.15.1:0/3878807370 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa20804a020 con 0x564576805dd0 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.139+0000 7fa1fcc00640 1 -- 172.21.15.1:0/3878807370 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fa20803dcc0 con 0x564576805dd0 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.139+0000 7fa218a51740 10 client.16641 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.139+0000 7fa218a51740 10 client.16641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.139+0000 7fa218a51740 20 client.16641 populate_metadata read hostname 'smithi001' 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.139+0000 7fa218a51740 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564576887660 0x564576889a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.139+0000 7fa218a51740 4 client.16641 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.139+0000 7fa218a51740 1 -- 172.21.15.1:0/3878807370 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564576772e00 con 0x564576887660 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.139+0000 7fa207e00640 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564576887660 0x564576889a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.140+0000 7fa207e00640 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564576887660 0x564576889a50 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8d5257d1fcf2ba0 server_cookie=97388db5a6df7f45 in_seq=0 out_seq=0 2024-04-26T21:49:01.142 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:01.140+0000 7fa1fcc00640 10 client.16641 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:01.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 130393 22.86 MB/sec execute 178 sec latency 925.307 ms 2024-04-26T21:49:02.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 130420 22.73 MB/sec execute 179 sec latency 1226.759 ms 2024-04-26T21:49:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:02 smithi001 ceph-mon[28054]: pgmap v591: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 566 op/s 2024-04-26T21:49:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:02 smithi038 ceph-mon[32724]: pgmap v591: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 566 op/s 2024-04-26T21:49:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:02 smithi186 ceph-mon[32991]: pgmap v591: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 566 op/s 2024-04-26T21:49:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.710+0000 7fa1fcc00640 1 -- 172.21.15.1:0/3878807370 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564576772e00 con 0x564576887660 2024-04-26T21:49:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.710+0000 7fa1fcc00640 10 client.16641 handle_command_reply: tid=0 2024-04-26T21:49:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.710+0000 7fa218a51740 1 client.16641 shutdown 2024-04-26T21:49:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.710+0000 7fa218a51740 1 -- 172.21.15.1:0/3878807370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576805dd0 msgr2=0x564576814a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.710+0000 7fa218a51740 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576805dd0 0x564576814a60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa208030910 tx=0x7fa208005d10 comp rx=0 tx=0).stop 2024-04-26T21:49:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.711+0000 7fa218a51740 1 -- 172.21.15.1:0/3878807370 shutdown_connections 2024-04-26T21:49:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.711+0000 7fa218a51740 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564576887660 0x564576889a50 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.711+0000 7fa218a51740 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5645768066f0 0x564576814fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.711+0000 7fa218a51740 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564576810fb0 0x5645768154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.711+0000 7fa218a51740 1 --2- 172.21.15.1:0/3878807370 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564576805dd0 0x564576814a60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.711+0000 7fa218a51740 1 -- 172.21.15.1:0/3878807370 >> 172.21.15.1:0/3878807370 conn(0x5645768022e0 msgr2=0x56457680df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.711+0000 7fa1fcc00640 0 client.16641 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.711+0000 7fa218a51740 1 -- 172.21.15.1:0/3878807370 shutdown_connections 2024-04-26T21:49:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.711+0000 7fa218a51740 1 -- 172.21.15.1:0/3878807370 wait complete. 2024-04-26T21:49:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.711+0000 7fa218a51740 20 client.16641 trim_cache size 0 max 16384 2024-04-26T21:49:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:49:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:49:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:49:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:49:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.712+0000 7fa215c00640 1 -- 172.21.15.1:0/2526495392 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa1f40a3d90 msgr2=0x7fa1f40a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.712+0000 7fa215c00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa1f40a3d90 0x7fa1f40a6250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa200000c00 tx=0x7fa2000d49c0 comp rx=0 tx=0).stop 2024-04-26T21:49:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.713+0000 7fa215c00640 1 -- 172.21.15.1:0/2526495392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa210104b50 msgr2=0x7fa21019b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.713+0000 7fa215c00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa210104b50 0x7fa21019b2a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa20000e9c0 tx=0x7fa20000ee80 comp rx=0 tx=0).stop 2024-04-26T21:49:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.713+0000 7fa215c00640 1 -- 172.21.15.1:0/2526495392 shutdown_connections 2024-04-26T21:49:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.713+0000 7fa215c00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fa1f40a3d90 0x7fa1f40a6250 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.713+0000 7fa215c00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa21010ac10 0x7fa21019fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.713+0000 7fa215c00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa210105510 0x7fa21019b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.713+0000 7fa215c00640 1 --2- 172.21.15.1:0/2526495392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa210104b50 0x7fa21019b2a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.713+0000 7fa215c00640 1 -- 172.21.15.1:0/2526495392 >> 172.21.15.1:0/2526495392 conn(0x7fa210100600 msgr2=0x7fa210101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.714+0000 7fa215c00640 1 -- 172.21.15.1:0/2526495392 shutdown_connections 2024-04-26T21:49:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:02.714+0000 7fa215c00640 1 -- 172.21.15.1:0/2526495392 wait complete. 2024-04-26T21:49:03.164 INFO:tasks.workunit.client.0.smithi001.stdout: 10 130491 22.63 MB/sec execute 180 sec latency 1058.612 ms 2024-04-26T21:49:04.165 INFO:tasks.workunit.client.0.smithi001.stdout: 10 130564 22.51 MB/sec execute 181 sec latency 1138.756 ms 2024-04-26T21:49:04.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:04 smithi001 ceph-mon[28054]: pgmap v592: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.5 MiB/s wr, 350 op/s 2024-04-26T21:49:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:04 smithi038 ceph-mon[32724]: pgmap v592: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.5 MiB/s wr, 350 op/s 2024-04-26T21:49:04.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:04 smithi186 ceph-mon[32991]: pgmap v592: 129 pgs: 129 active+clean; 1.2 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 4.5 MiB/s wr, 350 op/s 2024-04-26T21:49:05.165 INFO:tasks.workunit.client.0.smithi001.stdout: 10 130598 22.40 MB/sec execute 182 sec latency 1041.813 ms 2024-04-26T21:49:06.165 INFO:tasks.workunit.client.0.smithi001.stdout: 10 130850 22.34 MB/sec execute 183 sec latency 151.349 ms 2024-04-26T21:49:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:06 smithi001 ceph-mon[28054]: pgmap v593: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 11 MiB/s wr, 618 op/s 2024-04-26T21:49:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:06 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:49:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:06 smithi038 ceph-mon[32724]: pgmap v593: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 11 MiB/s wr, 618 op/s 2024-04-26T21:49:06.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:06 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:49:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:06 smithi186 ceph-mon[32991]: pgmap v593: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 11 MiB/s wr, 618 op/s 2024-04-26T21:49:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:06 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:49:07.165 INFO:tasks.workunit.client.0.smithi001.stdout: 10 131217 22.34 MB/sec execute 184 sec latency 125.118 ms 2024-04-26T21:49:08.165 INFO:tasks.workunit.client.0.smithi001.stdout: 10 131365 22.27 MB/sec execute 185 sec latency 1029.346 ms 2024-04-26T21:49:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:08 smithi001 ceph-mon[28054]: pgmap v594: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 603 op/s 2024-04-26T21:49:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:08 smithi001 ceph-mon[28054]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T21:49:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:08 smithi038 ceph-mon[32724]: pgmap v594: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 603 op/s 2024-04-26T21:49:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:08 smithi038 ceph-mon[32724]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T21:49:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:08 smithi186 ceph-mon[32991]: pgmap v594: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 603 op/s 2024-04-26T21:49:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:08 smithi186 ceph-mon[32991]: osdmap e124: 12 total, 12 up, 12 in 2024-04-26T21:49:09.165 INFO:tasks.workunit.client.0.smithi001.stdout: 10 131524 22.19 MB/sec execute 186 sec latency 1135.179 ms 2024-04-26T21:49:09.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:09 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:49:09.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:09 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:49:09.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:09 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:49:09.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:09 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:49:09.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:09 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:49:09.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:09 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:49:09.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:09 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:49:09.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:09 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:49:09.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:09 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:49:10.165 INFO:tasks.workunit.client.0.smithi001.stdout: 10 131738 22.10 MB/sec execute 187 sec latency 1315.696 ms 2024-04-26T21:49:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:10 smithi186 ceph-mon[32991]: pgmap v596: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 664 op/s 2024-04-26T21:49:10.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:10 smithi001 ceph-mon[28054]: pgmap v596: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 664 op/s 2024-04-26T21:49:10.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:10 smithi038 ceph-mon[32724]: pgmap v596: 129 pgs: 129 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 664 op/s 2024-04-26T21:49:11.165 INFO:tasks.workunit.client.0.smithi001.stdout: 10 131962 22.05 MB/sec execute 188 sec latency 836.727 ms 2024-04-26T21:49:12.166 INFO:tasks.workunit.client.0.smithi001.stdout: 10 132302 22.01 MB/sec execute 189 sec latency 1561.576 ms 2024-04-26T21:49:12.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:12 smithi001 ceph-mon[28054]: pgmap v597: 129 pgs: 21 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 55 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 669 op/s 2024-04-26T21:49:12.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:12 smithi186 ceph-mon[32991]: pgmap v597: 129 pgs: 21 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 55 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 669 op/s 2024-04-26T21:49:12.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:12 smithi038 ceph-mon[32724]: pgmap v597: 129 pgs: 21 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 55 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 669 op/s 2024-04-26T21:49:13.166 INFO:tasks.workunit.client.0.smithi001.stdout: 10 132657 22.04 MB/sec execute 190 sec latency 863.084 ms 2024-04-26T21:49:14.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 132946 22.03 MB/sec execute 191 sec latency 623.086 ms 2024-04-26T21:49:14.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:14 smithi186 ceph-mon[32991]: pgmap v598: 129 pgs: 21 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 55 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 669 op/s 2024-04-26T21:49:14.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:14 smithi001 ceph-mon[28054]: pgmap v598: 129 pgs: 21 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 55 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 669 op/s 2024-04-26T21:49:14.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:14 smithi038 ceph-mon[32724]: pgmap v598: 129 pgs: 21 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 55 active+clean; 1.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 669 op/s 2024-04-26T21:49:15.166 INFO:tasks.workunit.client.0.smithi001.stdout: 10 133269 21.98 MB/sec execute 192 sec latency 511.684 ms 2024-04-26T21:49:15.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.906+0000 7fdc67400640 1 -- 172.21.15.1:0/3728291452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc601032e0 msgr2=0x7fdc6010f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:15.908 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.906+0000 7fdc67400640 1 --2- 172.21.15.1:0/3728291452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc601032e0 0x7fdc6010f7f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009ca0 tx=0x7fdc5802a8c0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.907+0000 7fdc67400640 1 -- 172.21.15.1:0/3728291452 shutdown_connections 2024-04-26T21:49:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.907+0000 7fdc67400640 1 --2- 172.21.15.1:0/3728291452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc60102920 0x7fdc60102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.907+0000 7fdc67400640 1 --2- 172.21.15.1:0/3728291452 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc601032e0 0x7fdc6010f7f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.907+0000 7fdc67400640 1 --2- 172.21.15.1:0/3728291452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc60108930 0x7fdc60108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.907+0000 7fdc67400640 1 -- 172.21.15.1:0/3728291452 >> 172.21.15.1:0/3728291452 conn(0x7fdc600fe3f0 msgr2=0x7fdc60100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.908+0000 7fdc67400640 1 -- 172.21.15.1:0/3728291452 shutdown_connections 2024-04-26T21:49:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.908+0000 7fdc67400640 1 -- 172.21.15.1:0/3728291452 wait complete. 2024-04-26T21:49:15.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.910+0000 7fdc67400640 1 Processor -- start 2024-04-26T21:49:15.912 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.910+0000 7fdc67400640 1 -- start start 2024-04-26T21:49:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.911+0000 7fdc67400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc60102920 0x7fdc601a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.911+0000 7fdc67400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601032e0 0x7fdc601a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc67400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc60108930 0x7fdc6019b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc67400640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdc600757b0 con 0x7fdc601032e0 2024-04-26T21:49:15.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc67400640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdc60075630 con 0x7fdc60102920 2024-04-26T21:49:15.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc67400640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc60075930 con 0x7fdc60108930 2024-04-26T21:49:15.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc64a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc60102920 0x7fdc601a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc64a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc60102920 0x7fdc601a0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43192/0 (socket says 172.21.15.1:43192) 2024-04-26T21:49:15.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc64a00640 1 -- 172.21.15.1:0/3202689312 learned_addr learned my addr 172.21.15.1:0/3202689312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc64a00640 1 -- 172.21.15.1:0/3202689312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc60108930 msgr2=0x7fdc6019b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc64a00640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc60108930 0x7fdc6019b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc64a00640 1 -- 172.21.15.1:0/3202689312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601032e0 msgr2=0x7fdc601a1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc64a00640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601032e0 0x7fdc601a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.912+0000 7fdc64a00640 1 -- 172.21.15.1:0/3202689312 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6019b9b0 con 0x7fdc60102920 2024-04-26T21:49:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.913+0000 7fdc64a00640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc60102920 0x7fdc601a0fe0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fdc5400e9f0 tx=0x7fdc5400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:15.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.913+0000 7fdc5d600640 1 -- 172.21.15.1:0/3202689312 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5400cd80 con 0x7fdc60102920 2024-04-26T21:49:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.914+0000 7fdc5d600640 1 -- 172.21.15.1:0/3202689312 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc540108e0 con 0x7fdc60102920 2024-04-26T21:49:15.916 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.914+0000 7fdc5d600640 1 -- 172.21.15.1:0/3202689312 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc54018860 con 0x7fdc60102920 2024-04-26T21:49:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.914+0000 7fdc67400640 1 -- 172.21.15.1:0/3202689312 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc6019bca0 con 0x7fdc60102920 2024-04-26T21:49:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.914+0000 7fdc67400640 1 -- 172.21.15.1:0/3202689312 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc601a7f60 con 0x7fdc60102920 2024-04-26T21:49:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.919+0000 7fdc5d600640 1 -- 172.21.15.1:0/3202689312 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fdc54010430 con 0x7fdc60102920 2024-04-26T21:49:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.920+0000 7fdc6a202740 1 Processor -- start 2024-04-26T21:49:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.920+0000 7fdc6a202740 1 -- start start 2024-04-26T21:49:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.921+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5919dd0 0x5572e591a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.922 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.921+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e5924a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.921+0000 7fdc65400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5919dd0 0x5572e591a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.921+0000 7fdc65400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5919dd0 0x5572e591a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:47030/0 (socket says 172.21.15.1:47030) 2024-04-26T21:49:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.922+0000 7fdc64a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e5924a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.922+0000 7fdc64a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e5924a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43194/0 (socket says 172.21.15.1:43194) 2024-04-26T21:49:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.922+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5924fb0 0x5572e59273a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.922+0000 7fdc65400640 1 -- 172.21.15.1:0/33545368 learned_addr learned my addr 172.21.15.1:0/33545368 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.922+0000 7fdc6a202740 1 -- 172.21.15.1:0/33545368 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5572e587f380 con 0x5572e5924fb0 2024-04-26T21:49:15.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.922+0000 7fdc6a202740 1 -- 172.21.15.1:0/33545368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5572e581bc30 con 0x5572e591a6f0 2024-04-26T21:49:15.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.922+0000 7fdc6a202740 1 -- 172.21.15.1:0/33545368 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5572e587b340 con 0x5572e5919dd0 2024-04-26T21:49:15.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.922+0000 7fdc5d600640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdc480a3e10 0x7fdc480a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.923+0000 7fdc5d600640 1 -- 172.21.15.1:0/3202689312 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 9730+0+0 (secure 0 0 0) 0x7fdc540d1050 con 0x7fdc60102920 2024-04-26T21:49:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.923+0000 7fdc64a00640 1 -- 172.21.15.1:0/33545368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5919dd0 msgr2=0x5572e591a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.923+0000 7fdc64a00640 1 --2- 172.21.15.1:0/33545368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5919dd0 0x5572e591a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.923+0000 7fdc64a00640 1 -- 172.21.15.1:0/33545368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5924fb0 msgr2=0x5572e59273a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.923+0000 7fdc64a00640 1 --2- 172.21.15.1:0/33545368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5924fb0 0x5572e59273a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.923+0000 7fdc64a00640 1 -- 172.21.15.1:0/33545368 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572e583d940 con 0x5572e591a6f0 2024-04-26T21:49:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.923+0000 7fdc65400640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdc480a3e10 0x7fdc480a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.923+0000 7fdc64a00640 1 --2- 172.21.15.1:0/33545368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e5924a70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fdc540ce280 tx=0x7fdc54009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f0a42f2593a2526 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.924+0000 7fdc51c00640 1 -- 172.21.15.1:0/33545368 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5409c430 con 0x5572e591a6f0 2024-04-26T21:49:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.924+0000 7fdc51c00640 1 -- 172.21.15.1:0/33545368 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc54010720 con 0x5572e591a6f0 2024-04-26T21:49:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.924+0000 7fdc65400640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdc480a3e10 0x7fdc480a62d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009430 tx=0x7fdc580095c0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.924+0000 7fdc51c00640 1 -- 172.21.15.1:0/33545368 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc54010a90 con 0x5572e591a6f0 2024-04-26T21:49:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.925+0000 7fdc6a202740 1 -- 172.21.15.1:0/33545368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 msgr2=0x5572e5924a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.925+0000 7fdc6a202740 1 --2- 172.21.15.1:0/33545368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e5924a70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fdc540ce280 tx=0x7fdc54009700 comp rx=0 tx=0).stop 2024-04-26T21:49:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.925+0000 7fdc6a202740 1 -- 172.21.15.1:0/33545368 shutdown_connections 2024-04-26T21:49:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.925+0000 7fdc6a202740 1 --2- 172.21.15.1:0/33545368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5924fb0 0x5572e59273a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.925+0000 7fdc6a202740 1 --2- 172.21.15.1:0/33545368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e5924a70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.925+0000 7fdc6a202740 1 --2- 172.21.15.1:0/33545368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5919dd0 0x5572e591a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.925+0000 7fdc6a202740 1 -- 172.21.15.1:0/33545368 >> 172.21.15.1:0/33545368 conn(0x5572e59162e0 msgr2=0x5572e59166c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.925+0000 7fdc6a202740 1 -- 172.21.15.1:0/33545368 shutdown_connections 2024-04-26T21:49:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.925+0000 7fdc6a202740 1 -- 172.21.15.1:0/33545368 wait complete. 2024-04-26T21:49:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.926+0000 7fdc6a202740 1 -- start start 2024-04-26T21:49:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.926+0000 7fdc6a202740 1 Processor -- start 2024-04-26T21:49:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.927+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e5919dd0 0x5572e5921ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.927+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e591a6f0 0x5572e59223e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.927+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5924fb0 0x5572e5922920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.927+0000 7fdc6a202740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5572e5927b80 con 0x5572e5924fb0 2024-04-26T21:49:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.927+0000 7fdc6a202740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5572e5806c60 con 0x5572e5919dd0 2024-04-26T21:49:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.927+0000 7fdc6a202740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5572e5927d00 con 0x5572e591a6f0 2024-04-26T21:49:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.927+0000 7fdc6a202740 10 client.0 _ll_register_callbacks cb 0x5572e5832e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:49:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.928+0000 7fdc64a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e5919dd0 0x5572e5921ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.928+0000 7fdc64a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e5919dd0 0x5572e5921ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43204/0 (socket says 172.21.15.1:43204) 2024-04-26T21:49:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.928+0000 7fdc64a00640 1 -- 172.21.15.1:0/2329118489 learned_addr learned my addr 172.21.15.1:0/2329118489 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.929+0000 7fdc5fe00640 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e591a6f0 0x5572e59223e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.929+0000 7fdc65400640 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5924fb0 0x5572e5922920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.929+0000 7fdc64a00640 1 -- 172.21.15.1:0/2329118489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e591a6f0 msgr2=0x5572e59223e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.929+0000 7fdc64a00640 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e591a6f0 0x5572e59223e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.929+0000 7fdc64a00640 1 -- 172.21.15.1:0/2329118489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5924fb0 msgr2=0x5572e5922920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.929+0000 7fdc64a00640 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5924fb0 0x5572e5922920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.929+0000 7fdc64a00640 1 -- 172.21.15.1:0/2329118489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572e5815510 con 0x5572e5919dd0 2024-04-26T21:49:15.932 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.929+0000 7fdc5fe00640 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e591a6f0 0x5572e59223e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:49:15.933 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.932+0000 7fdc64a00640 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e5919dd0 0x5572e5921ea0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdc540095d0 tx=0x7fdc5409c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b096cbfe36175f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.932+0000 7fdc46000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:49:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.932+0000 7fdc46000640 1 -- 172.21.15.1:0/2329118489 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5409cad0 con 0x5572e5919dd0 2024-04-26T21:49:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.932+0000 7fdc46000640 1 -- 172.21.15.1:0/2329118489 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9730+0+0 (secure 0 0 0) 0x7fdc54099140 con 0x5572e5919dd0 2024-04-26T21:49:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.932+0000 7fdc6a202740 1 -- 172.21.15.1:0/2329118489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5572e588cf90 con 0x5572e5919dd0 2024-04-26T21:49:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.933+0000 7fdc46000640 1 -- 172.21.15.1:0/2329118489 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fdc5409d1d0 con 0x5572e5919dd0 2024-04-26T21:49:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.934+0000 7fdc6a202740 10 client.26948 fetch_fsmap learned FSMap version 57 2024-04-26T21:49:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.934+0000 7fdc6a202740 1 -- 172.21.15.1:0/2329118489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5572e5591210 con 0x5572e5919dd0 2024-04-26T21:49:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.935+0000 7fdc46000640 1 -- 172.21.15.1:0/2329118489 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5572e588cf90 con 0x5572e5919dd0 2024-04-26T21:49:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.935+0000 7fdc46000640 1 -- 172.21.15.1:0/2329118489 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fdc540b8020 con 0x5572e5919dd0 2024-04-26T21:49:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.935+0000 7fdc6a202740 10 client.26948 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:49:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.935+0000 7fdc6a202740 10 client.26948 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:49:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.935+0000 7fdc6a202740 20 client.26948 populate_metadata read hostname 'smithi001' 2024-04-26T21:49:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.935+0000 7fdc6a202740 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5572e594e310 0x5572e5950700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.935+0000 7fdc6a202740 4 client.26948 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:49:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.935+0000 7fdc6a202740 1 -- 172.21.15.1:0/2329118489 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572e5886e00 con 0x5572e594e310 2024-04-26T21:49:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.936+0000 7fdc5fe00640 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5572e594e310 0x5572e5950700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.937+0000 7fdc46000640 10 client.26948 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.937+0000 7fdc5fe00640 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5572e594e310 0x5572e5950700 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f65f1e01e3fe962 server_cookie=4006ad966cc337a1 in_seq=0 out_seq=0 2024-04-26T21:49:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.939+0000 7fdc46000640 1 -- 172.21.15.1:0/2329118489 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5572e5886e00 con 0x5572e594e310 2024-04-26T21:49:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.939+0000 7fdc46000640 10 client.26948 handle_command_reply: tid=0 2024-04-26T21:49:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.939+0000 7fdc6a202740 1 client.26948 shutdown 2024-04-26T21:49:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.939+0000 7fdc6a202740 1 -- 172.21.15.1:0/2329118489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e5919dd0 msgr2=0x5572e5921ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.939+0000 7fdc6a202740 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e5919dd0 0x5572e5921ea0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdc540095d0 tx=0x7fdc5409c4a0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.939+0000 7fdc6a202740 1 -- 172.21.15.1:0/2329118489 shutdown_connections 2024-04-26T21:49:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.939+0000 7fdc6a202740 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5572e594e310 0x5572e5950700 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.940+0000 7fdc6a202740 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e591a6f0 0x5572e59223e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.940+0000 7fdc6a202740 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5924fb0 0x5572e5922920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.940+0000 7fdc6a202740 1 --2- 172.21.15.1:0/2329118489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e5919dd0 0x5572e5921ea0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.940+0000 7fdc6a202740 1 -- 172.21.15.1:0/2329118489 >> 172.21.15.1:0/2329118489 conn(0x5572e59162e0 msgr2=0x5572e591c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.940+0000 7fdc46000640 0 client.26948 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.940+0000 7fdc6a202740 1 -- 172.21.15.1:0/2329118489 shutdown_connections 2024-04-26T21:49:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.940+0000 7fdc6a202740 1 -- 172.21.15.1:0/2329118489 wait complete. 2024-04-26T21:49:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.940+0000 7fdc6a202740 20 client.26948 trim_cache size 0 max 16384 2024-04-26T21:49:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.965+0000 7fdc6a202740 1 Processor -- start 2024-04-26T21:49:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.965+0000 7fdc6a202740 1 -- start start 2024-04-26T21:49:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.966+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5919dd0 0x5572e591c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.966+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e5923cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.966+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e5955dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.966+0000 7fdc6a202740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdc601a8130 con 0x5572e5919dd0 2024-04-26T21:49:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.966+0000 7fdc6a202740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdc60072330 con 0x5572e591a6f0 2024-04-26T21:49:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.966+0000 7fdc6a202740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc600701e0 con 0x5572e5924fb0 2024-04-26T21:49:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.966+0000 7fdc65400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e5955dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.966+0000 7fdc64a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5919dd0 0x5572e591c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.967+0000 7fdc65400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e5955dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:47048/0 (socket says 172.21.15.1:47048) 2024-04-26T21:49:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.967+0000 7fdc65400640 1 -- 172.21.15.1:0/2242913499 learned_addr learned my addr 172.21.15.1:0/2242913499 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.967+0000 7fdc5fe00640 1 --2- 172.21.15.1:0/2242913499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e5923cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.967+0000 7fdc65400640 1 -- 172.21.15.1:0/2242913499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 msgr2=0x5572e5923cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.967+0000 7fdc65400640 1 --2- 172.21.15.1:0/2242913499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e5923cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.967+0000 7fdc65400640 1 -- 172.21.15.1:0/2242913499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5919dd0 msgr2=0x5572e591c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.967+0000 7fdc65400640 1 --2- 172.21.15.1:0/2242913499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5919dd0 0x5572e591c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.967+0000 7fdc65400640 1 -- 172.21.15.1:0/2242913499 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572e565d620 con 0x5572e5924fb0 2024-04-26T21:49:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.968+0000 7fdc65400640 1 --2- 172.21.15.1:0/2242913499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e5955dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009650 tx=0x7fdc58043000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d652239c7bfacd11 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.968+0000 7fdc51200640 1 -- 172.21.15.1:0/2242913499 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58044070 con 0x5572e5924fb0 2024-04-26T21:49:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.968+0000 7fdc51200640 1 -- 172.21.15.1:0/2242913499 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc5804f030 con 0x5572e5924fb0 2024-04-26T21:49:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.968+0000 7fdc51200640 1 -- 172.21.15.1:0/2242913499 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58044070 con 0x5572e5924fb0 2024-04-26T21:49:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.969+0000 7fdc6a202740 1 -- 172.21.15.1:0/2242913499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 msgr2=0x5572e5955dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.969+0000 7fdc6a202740 1 --2- 172.21.15.1:0/2242913499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e5955dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009650 tx=0x7fdc58043000 comp rx=0 tx=0).stop 2024-04-26T21:49:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.969+0000 7fdc6a202740 1 -- 172.21.15.1:0/2242913499 shutdown_connections 2024-04-26T21:49:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.970+0000 7fdc6a202740 1 --2- 172.21.15.1:0/2242913499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e5955dc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.970+0000 7fdc6a202740 1 --2- 172.21.15.1:0/2242913499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e5923cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.970+0000 7fdc6a202740 1 --2- 172.21.15.1:0/2242913499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5919dd0 0x5572e591c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.970+0000 7fdc6a202740 1 -- 172.21.15.1:0/2242913499 >> 172.21.15.1:0/2242913499 conn(0x5572e59162e0 msgr2=0x5572e59235c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.970+0000 7fdc6a202740 1 -- 172.21.15.1:0/2242913499 shutdown_connections 2024-04-26T21:49:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.970+0000 7fdc6a202740 1 -- 172.21.15.1:0/2242913499 wait complete. 2024-04-26T21:49:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.970+0000 7fdc6a202740 1 -- start start 2024-04-26T21:49:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.971+0000 7fdc6a202740 1 Processor -- start 2024-04-26T21:49:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.971+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5919dd0 0x5572e591ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.971+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e591ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.971+0000 7fdc6a202740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e591f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.971+0000 7fdc6a202740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdc60070830 con 0x5572e5919dd0 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.971+0000 7fdc6a202740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fdc60071c60 con 0x5572e591a6f0 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.971+0000 7fdc6a202740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc60070ec0 con 0x5572e5924fb0 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.972+0000 7fdc6a202740 10 client.0 _ll_register_callbacks cb 0x7fdc54005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.972+0000 7fdc65400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e591f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.972+0000 7fdc5fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e591ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.972+0000 7fdc5fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e591ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43214/0 (socket says 172.21.15.1:43214) 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.972+0000 7fdc5fe00640 1 -- 172.21.15.1:0/3342072306 learned_addr learned my addr 172.21.15.1:0/3342072306 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.972+0000 7fdc64a00640 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5919dd0 0x5572e591ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.973+0000 7fdc65400640 1 -- 172.21.15.1:0/3342072306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 msgr2=0x5572e591ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:15.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.973+0000 7fdc65400640 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e591ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.973+0000 7fdc65400640 1 -- 172.21.15.1:0/3342072306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5919dd0 msgr2=0x5572e591ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:15.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.973+0000 7fdc65400640 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5919dd0 0x5572e591ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:15.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.973+0000 7fdc65400640 1 -- 172.21.15.1:0/3342072306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572e5839cb0 con 0x5572e5924fb0 2024-04-26T21:49:15.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.973+0000 7fdc5fe00640 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e591ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:49:15.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.973+0000 7fdc65400640 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e591f4c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdc58005e30 tx=0x7fdc58004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fab63755b700d42a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.973+0000 7fdc46000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:49:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.974+0000 7fdc46000640 1 -- 172.21.15.1:0/3342072306 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58044040 con 0x5572e5924fb0 2024-04-26T21:49:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.974+0000 7fdc46000640 1 -- 172.21.15.1:0/3342072306 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(124..124 src has 1..124) ==== 9730+0+0 (secure 0 0 0) 0x7fdc58004b40 con 0x5572e5924fb0 2024-04-26T21:49:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.974+0000 7fdc6a202740 1 -- 172.21.15.1:0/3342072306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc5804f030 con 0x5572e5924fb0 2024-04-26T21:49:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.974+0000 7fdc46000640 1 -- 172.21.15.1:0/3342072306 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fdc58004d40 con 0x5572e5924fb0 2024-04-26T21:49:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.974+0000 7fdc6a202740 10 client.26953 fetch_fsmap learned FSMap version 57 2024-04-26T21:49:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.974+0000 7fdc6a202740 1 -- 172.21.15.1:0/3342072306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5572e59b4630 con 0x5572e5924fb0 2024-04-26T21:49:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.974+0000 7fdc46000640 1 -- 172.21.15.1:0/3342072306 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc5804f030 con 0x5572e5924fb0 2024-04-26T21:49:15.978 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.975+0000 7fdc46000640 1 -- 172.21.15.1:0/3342072306 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fdc58047030 con 0x5572e5924fb0 2024-04-26T21:49:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.975+0000 7fdc6a202740 10 client.26953 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:49:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.975+0000 7fdc6a202740 10 client.26953 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:49:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.975+0000 7fdc6a202740 20 client.26953 populate_metadata read hostname 'smithi001' 2024-04-26T21:49:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.975+0000 7fdc6a202740 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5572e59b4cd0 0x5572e59b70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.975+0000 7fdc6a202740 4 client.26953 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:49:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.975+0000 7fdc6a202740 1 -- 172.21.15.1:0/3342072306 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5572e58024b0 con 0x5572e59b4cd0 2024-04-26T21:49:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.975+0000 7fdc64a00640 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5572e59b4cd0 0x5572e59b70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.976+0000 7fdc64a00640 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5572e59b4cd0 0x5572e59b70f0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4b2ec17aa8afb6c server_cookie=ca5428058cd1c21e in_seq=0 out_seq=0 2024-04-26T21:49:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:15.977+0000 7fdc46000640 10 client.26953 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:16.166 INFO:tasks.workunit.client.0.smithi001.stdout: 10 133567 21.95 MB/sec execute 193 sec latency 678.424 ms 2024-04-26T21:49:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.262+0000 7fdc46000640 1 -- 172.21.15.1:0/3342072306 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5572e58024b0 con 0x5572e59b4cd0 2024-04-26T21:49:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.262+0000 7fdc46000640 10 client.26953 handle_command_reply: tid=0 2024-04-26T21:49:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.262+0000 7fdc6a202740 1 client.26953 shutdown 2024-04-26T21:49:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.263+0000 7fdc6a202740 1 -- 172.21.15.1:0/3342072306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 msgr2=0x5572e591f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.263+0000 7fdc6a202740 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e591f4c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdc58005e30 tx=0x7fdc58004a40 comp rx=0 tx=0).stop 2024-04-26T21:49:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.263+0000 7fdc6a202740 1 -- 172.21.15.1:0/3342072306 shutdown_connections 2024-04-26T21:49:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.263+0000 7fdc6a202740 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5572e59b4cd0 0x5572e59b70f0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.263+0000 7fdc46000640 0 client.26953 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.263+0000 7fdc6a202740 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5572e5924fb0 0x5572e591f4c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.263+0000 7fdc6a202740 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5572e591a6f0 0x5572e591ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.263+0000 7fdc6a202740 1 --2- 172.21.15.1:0/3342072306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5572e5919dd0 0x5572e591ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.263+0000 7fdc6a202740 1 -- 172.21.15.1:0/3342072306 >> 172.21.15.1:0/3342072306 conn(0x5572e59162e0 msgr2=0x5572e59226f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.263+0000 7fdc6a202740 1 -- 172.21.15.1:0/3342072306 shutdown_connections 2024-04-26T21:49:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.264+0000 7fdc6a202740 1 -- 172.21.15.1:0/3342072306 wait complete. 2024-04-26T21:49:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.264+0000 7fdc6a202740 20 client.26953 trim_cache size 0 max 16384 2024-04-26T21:49:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:49:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:49:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:49:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:49:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 -- 172.21.15.1:0/3202689312 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdc480a3e10 msgr2=0x7fdc480a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdc480a3e10 0x7fdc480a62d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009430 tx=0x7fdc580095c0 comp rx=0 tx=0).stop 2024-04-26T21:49:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 -- 172.21.15.1:0/3202689312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc60102920 msgr2=0x7fdc601a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc60102920 0x7fdc601a0fe0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fdc5400e9f0 tx=0x7fdc5400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:49:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 -- 172.21.15.1:0/3202689312 shutdown_connections 2024-04-26T21:49:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fdc480a3e10 0x7fdc480a62d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc60108930 0x7fdc6019b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc601032e0 0x7fdc601a1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 --2- 172.21.15.1:0/3202689312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdc60102920 0x7fdc601a0fe0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 -- 172.21.15.1:0/3202689312 >> 172.21.15.1:0/3202689312 conn(0x7fdc600fe3f0 msgr2=0x7fdc60106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 -- 172.21.15.1:0/3202689312 shutdown_connections 2024-04-26T21:49:16.269 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:16.265+0000 7fdc67400640 1 -- 172.21.15.1:0/3202689312 wait complete. 2024-04-26T21:49:16.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:16 smithi186 ceph-mon[32991]: pgmap v599: 129 pgs: 10 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 89 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 459 op/s 2024-04-26T21:49:16.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:16 smithi001 ceph-mon[28054]: pgmap v599: 129 pgs: 10 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 89 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 459 op/s 2024-04-26T21:49:16.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:16 smithi038 ceph-mon[32724]: pgmap v599: 129 pgs: 10 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 89 active+clean; 1.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 459 op/s 2024-04-26T21:49:17.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 134112 22.02 MB/sec execute 194 sec latency 183.047 ms 2024-04-26T21:49:18.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 134560 21.99 MB/sec execute 195 sec latency 115.522 ms 2024-04-26T21:49:18.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:18 smithi186 ceph-mon[32991]: pgmap v600: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.6 MiB/s wr, 346 op/s 2024-04-26T21:49:18.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:18 smithi001 ceph-mon[28054]: pgmap v600: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.6 MiB/s wr, 346 op/s 2024-04-26T21:49:18.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:18 smithi038 ceph-mon[32724]: pgmap v600: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 3.6 MiB/s wr, 346 op/s 2024-04-26T21:49:19.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 135074 22.02 MB/sec execute 196 sec latency 112.871 ms 2024-04-26T21:49:20.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 135569 22.05 MB/sec execute 197 sec latency 121.787 ms 2024-04-26T21:49:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:20 smithi186 ceph-mon[32991]: pgmap v601: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.0 MiB/s wr, 292 op/s 2024-04-26T21:49:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:20 smithi186 ceph-mon[32991]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T21:49:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:20 smithi001 ceph-mon[28054]: pgmap v601: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.0 MiB/s wr, 292 op/s 2024-04-26T21:49:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:20 smithi001 ceph-mon[28054]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T21:49:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:20 smithi038 ceph-mon[32724]: pgmap v601: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.2 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.0 MiB/s wr, 292 op/s 2024-04-26T21:49:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:20 smithi038 ceph-mon[32724]: osdmap e125: 12 total, 12 up, 12 in 2024-04-26T21:49:21.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 136049 22.08 MB/sec execute 198 sec latency 144.257 ms 2024-04-26T21:49:22.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 136600 22.10 MB/sec execute 199 sec latency 69.646 ms 2024-04-26T21:49:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:22 smithi186 ceph-mon[32991]: pgmap v603: 129 pgs: 129 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.0 MiB/s wr, 411 op/s 2024-04-26T21:49:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:22 smithi001 ceph-mon[28054]: pgmap v603: 129 pgs: 129 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.0 MiB/s wr, 411 op/s 2024-04-26T21:49:22.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:22 smithi038 ceph-mon[32724]: pgmap v603: 129 pgs: 129 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.0 MiB/s wr, 411 op/s 2024-04-26T21:49:23.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 137164 22.14 MB/sec execute 200 sec latency 66.306 ms 2024-04-26T21:49:24.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 137679 22.18 MB/sec execute 201 sec latency 81.694 ms 2024-04-26T21:49:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:24 smithi001 ceph-mon[28054]: pgmap v604: 129 pgs: 129 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.0 MiB/s wr, 411 op/s 2024-04-26T21:49:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:24 smithi001 ceph-mon[28054]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T21:49:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:24 smithi038 ceph-mon[32724]: pgmap v604: 129 pgs: 129 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.0 MiB/s wr, 411 op/s 2024-04-26T21:49:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:24 smithi038 ceph-mon[32724]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T21:49:24.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:24 smithi186 ceph-mon[32991]: pgmap v604: 129 pgs: 129 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.0 MiB/s wr, 411 op/s 2024-04-26T21:49:24.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:24 smithi186 ceph-mon[32991]: osdmap e126: 12 total, 12 up, 12 in 2024-04-26T21:49:25.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 138175 22.25 MB/sec execute 202 sec latency 101.012 ms 2024-04-26T21:49:26.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 138519 22.21 MB/sec execute 203 sec latency 143.103 ms 2024-04-26T21:49:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:26 smithi001 ceph-mon[28054]: pgmap v606: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.8 MiB/s wr, 492 op/s 2024-04-26T21:49:26.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:26 smithi038 ceph-mon[32724]: pgmap v606: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.8 MiB/s wr, 492 op/s 2024-04-26T21:49:26.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:26 smithi186 ceph-mon[32991]: pgmap v606: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.8 MiB/s wr, 492 op/s 2024-04-26T21:49:27.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 138790 22.14 MB/sec execute 204 sec latency 461.843 ms 2024-04-26T21:49:28.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 139217 22.14 MB/sec execute 205 sec latency 116.038 ms 2024-04-26T21:49:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:28 smithi001 ceph-mon[28054]: pgmap v607: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 531 op/s 2024-04-26T21:49:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:28 smithi038 ceph-mon[32724]: pgmap v607: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 531 op/s 2024-04-26T21:49:28.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:28 smithi186 ceph-mon[32991]: pgmap v607: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.3 MiB/s wr, 531 op/s 2024-04-26T21:49:29.167 INFO:tasks.workunit.client.0.smithi001.stdout: 10 139307 22.05 MB/sec execute 206 sec latency 624.292 ms 2024-04-26T21:49:29.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.500+0000 7fbc2f000640 1 -- 172.21.15.1:0/1927847684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc28115a80 msgr2=0x7fbc28117e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:29.500 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.500+0000 7fbc2f000640 1 --2- 172.21.15.1:0/1927847684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc28115a80 0x7fbc28117e70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbc20009be0 tx=0x7fbc2002a750 comp rx=0 tx=0).stop 2024-04-26T21:49:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.501+0000 7fbc2f000640 1 -- 172.21.15.1:0/1927847684 shutdown_connections 2024-04-26T21:49:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.501+0000 7fbc2f000640 1 --2- 172.21.15.1:0/1927847684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc28115a80 0x7fbc28117e70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.501+0000 7fbc2f000640 1 --2- 172.21.15.1:0/1927847684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc2810a460 0x7fbc28115540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.501 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.501+0000 7fbc2f000640 1 --2- 172.21.15.1:0/1927847684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc28109b40 0x7fbc28109f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.502 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.501+0000 7fbc2f000640 1 -- 172.21.15.1:0/1927847684 >> 172.21.15.1:0/1927847684 conn(0x7fbc2806da10 msgr2=0x7fbc2806de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:29.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.502+0000 7fbc2f000640 1 -- 172.21.15.1:0/1927847684 shutdown_connections 2024-04-26T21:49:29.503 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.503+0000 7fbc2f000640 1 -- 172.21.15.1:0/1927847684 wait complete. 2024-04-26T21:49:29.504 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.504+0000 7fbc2f000640 1 Processor -- start 2024-04-26T21:49:29.505 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.505+0000 7fbc2f000640 1 -- start start 2024-04-26T21:49:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc2f000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc28109b40 0x7fbc2810b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc2f000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc2810a460 0x7fbc2810b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc2f000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc28115a80 0x7fbc2810be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc2f000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2811b470 con 0x7fbc2810a460 2024-04-26T21:49:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc2f000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2811b2f0 con 0x7fbc28109b40 2024-04-26T21:49:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc2f000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2811b5f0 con 0x7fbc28115a80 2024-04-26T21:49:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc2d000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc28115a80 0x7fbc2810be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc2d000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc28115a80 0x7fbc2810be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51654/0 (socket says 172.21.15.1:51654) 2024-04-26T21:49:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc2d000640 1 -- 172.21.15.1:0/2410617728 learned_addr learned my addr 172.21.15.1:0/2410617728 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc27e00640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc28109b40 0x7fbc2810b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.506+0000 7fbc27400640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc2810a460 0x7fbc2810b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.508+0000 7fbc27e00640 1 -- 172.21.15.1:0/2410617728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc28115a80 msgr2=0x7fbc2810be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.508+0000 7fbc27e00640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc28115a80 0x7fbc2810be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.508+0000 7fbc27e00640 1 -- 172.21.15.1:0/2410617728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc2810a460 msgr2=0x7fbc2810b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:29.509 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.508+0000 7fbc27e00640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc2810a460 0x7fbc2810b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.509+0000 7fbc27e00640 1 -- 172.21.15.1:0/2410617728 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc281aff30 con 0x7fbc28109b40 2024-04-26T21:49:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.509+0000 7fbc2d000640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc28115a80 0x7fbc2810be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:49:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.509+0000 7fbc27e00640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc28109b40 0x7fbc2810b3a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800b3f0 tx=0x7fbc1800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.510+0000 7fbc24c00640 1 -- 172.21.15.1:0/2410617728 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18015020 con 0x7fbc28109b40 2024-04-26T21:49:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.510+0000 7fbc24c00640 1 -- 172.21.15.1:0/2410617728 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc18011ce0 con 0x7fbc28109b40 2024-04-26T21:49:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.510+0000 7fbc24c00640 1 -- 172.21.15.1:0/2410617728 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18010d80 con 0x7fbc28109b40 2024-04-26T21:49:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.510+0000 7fbc2f000640 1 -- 172.21.15.1:0/2410617728 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc281b0220 con 0x7fbc28109b40 2024-04-26T21:49:29.511 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.510+0000 7fbc2f000640 1 -- 172.21.15.1:0/2410617728 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc281b0720 con 0x7fbc28109b40 2024-04-26T21:49:29.518 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.517+0000 7fbc31fd6740 1 Processor -- start 2024-04-26T21:49:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.521+0000 7fbc24c00640 1 -- 172.21.15.1:0/2410617728 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fbc18018020 con 0x7fbc28109b40 2024-04-26T21:49:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.521+0000 7fbc31fd6740 1 -- start start 2024-04-26T21:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.523+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4855e0 0x55a02d4859c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.523+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d490280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.523+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d492bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.523+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a02d3e2380 con 0x55a02d4907c0 2024-04-26T21:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.523+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a02d37ec30 con 0x55a02d485f00 2024-04-26T21:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.523+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a02d3de340 con 0x55a02d4855e0 2024-04-26T21:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.524+0000 7fbc27400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d492bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.524+0000 7fbc27e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d490280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.524+0000 7fbc27e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d490280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54430/0 (socket says 172.21.15.1:54430) 2024-04-26T21:49:29.524 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.524+0000 7fbc27e00640 1 -- 172.21.15.1:0/327696699 learned_addr learned my addr 172.21.15.1:0/327696699 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.525+0000 7fbc24c00640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbc0c0a3ea0 0x7fbc0c0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.525+0000 7fbc2d000640 1 --2- 172.21.15.1:0/327696699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4855e0 0x55a02d4859c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.525 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.525+0000 7fbc24c00640 1 -- 172.21.15.1:0/2410617728 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9626+0+0 (secure 0 0 0) 0x7fbc180d0290 con 0x7fbc28109b40 2024-04-26T21:49:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.526+0000 7fbc2d000640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbc0c0a3ea0 0x7fbc0c0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.526+0000 7fbc27e00640 1 -- 172.21.15.1:0/327696699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4855e0 msgr2=0x55a02d4859c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.526+0000 7fbc27e00640 1 --2- 172.21.15.1:0/327696699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4855e0 0x55a02d4859c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.526+0000 7fbc27e00640 1 -- 172.21.15.1:0/327696699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 msgr2=0x55a02d492bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:29.526 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.526+0000 7fbc27e00640 1 --2- 172.21.15.1:0/327696699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d492bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.526+0000 7fbc27e00640 1 -- 172.21.15.1:0/327696699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a02d3a0940 con 0x55a02d485f00 2024-04-26T21:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.526+0000 7fbc27e00640 1 --2- 172.21.15.1:0/327696699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d490280 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbc18010790 tx=0x7fbc180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b4fdf90447ffc20 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.527+0000 7fbc1d600640 1 -- 172.21.15.1:0/327696699 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1809c5f0 con 0x55a02d485f00 2024-04-26T21:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.527+0000 7fbc1d600640 1 -- 172.21.15.1:0/327696699 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc1809bef0 con 0x55a02d485f00 2024-04-26T21:49:29.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.527+0000 7fbc1d600640 1 -- 172.21.15.1:0/327696699 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18098560 con 0x55a02d485f00 2024-04-26T21:49:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.528+0000 7fbc31fd6740 1 -- 172.21.15.1:0/327696699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 msgr2=0x55a02d490280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:29.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.528+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/327696699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d490280 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fbc18010790 tx=0x7fbc180046c0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.529+0000 7fbc31fd6740 1 -- 172.21.15.1:0/327696699 shutdown_connections 2024-04-26T21:49:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.529+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/327696699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d492bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.529+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/327696699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d490280 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.529+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/327696699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4855e0 0x55a02d4859c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.529+0000 7fbc31fd6740 1 -- 172.21.15.1:0/327696699 >> 172.21.15.1:0/327696699 conn(0x55a02d481af0 msgr2=0x55a02d481ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.529+0000 7fbc31fd6740 1 -- 172.21.15.1:0/327696699 shutdown_connections 2024-04-26T21:49:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.530+0000 7fbc2d000640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbc0c0a3ea0 0x7fbc0c0a6360 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbc20006010 tx=0x7fbc2002a750 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.530+0000 7fbc31fd6740 1 -- 172.21.15.1:0/327696699 wait complete. 2024-04-26T21:49:29.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.530+0000 7fbc31fd6740 1 -- start start 2024-04-26T21:49:29.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.531+0000 7fbc31fd6740 1 Processor -- start 2024-04-26T21:49:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.533+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4855e0 0x55a02d48d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.533+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d48dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.533+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d48e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.534+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55a02d493390 con 0x55a02d4907c0 2024-04-26T21:49:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.534+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55a02d369c60 con 0x55a02d485f00 2024-04-26T21:49:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.534+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a02d493510 con 0x55a02d4855e0 2024-04-26T21:49:29.535 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.534+0000 7fbc31fd6740 10 client.0 _ll_register_callbacks cb 0x55a02d395e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:49:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.534+0000 7fbc27400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d48dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.534+0000 7fbc27400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d48dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54432/0 (socket says 172.21.15.1:54432) 2024-04-26T21:49:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.534+0000 7fbc27400640 1 -- 172.21.15.1:0/2163105815 learned_addr learned my addr 172.21.15.1:0/2163105815 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.534+0000 7fbc2d000640 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d48e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.536 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.535+0000 7fbc27e00640 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4855e0 0x55a02d48d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.536+0000 7fbc27400640 1 -- 172.21.15.1:0/2163105815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4855e0 msgr2=0x55a02d48d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:29.543 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.536+0000 7fbc27400640 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4855e0 0x55a02d48d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.536+0000 7fbc27400640 1 -- 172.21.15.1:0/2163105815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 msgr2=0x55a02d48e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.536+0000 7fbc27400640 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d48e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.537+0000 7fbc27400640 1 -- 172.21.15.1:0/2163105815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a02d378510 con 0x55a02d485f00 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.539+0000 7fbc2d000640 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d48e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.539+0000 7fbc27400640 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d48dd30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400b910 tx=0x7fbc1400bdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb24d26070886fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.539+0000 7fbc12000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.540+0000 7fbc12000640 1 -- 172.21.15.1:0/2163105815 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14003d60 con 0x55a02d485f00 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.540+0000 7fbc12000640 1 -- 172.21.15.1:0/2163105815 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9626+0+0 (secure 0 0 0) 0x7fbc14005020 con 0x55a02d485f00 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.540+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2163105815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc1809bef0 con 0x55a02d485f00 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.541+0000 7fbc12000640 1 -- 172.21.15.1:0/2163105815 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fbc14005220 con 0x55a02d485f00 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.541+0000 7fbc31fd6740 10 client.26966 fetch_fsmap learned FSMap version 57 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.541+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2163105815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55a02d0f4210 con 0x55a02d485f00 2024-04-26T21:49:29.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.541+0000 7fbc12000640 1 -- 172.21.15.1:0/2163105815 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc1809bef0 con 0x55a02d485f00 2024-04-26T21:49:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.542+0000 7fbc12000640 1 -- 172.21.15.1:0/2163105815 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fbc1401b470 con 0x55a02d485f00 2024-04-26T21:49:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.542+0000 7fbc31fd6740 10 client.26966 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:49:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.542+0000 7fbc31fd6740 10 client.26966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:49:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.543+0000 7fbc31fd6740 20 client.26966 populate_metadata read hostname 'smithi001' 2024-04-26T21:49:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.543+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a02d4a05b0 0x55a02d4a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.543+0000 7fbc31fd6740 4 client.26966 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:49:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.543+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2163105815 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a02d3eff90 con 0x55a02d4a05b0 2024-04-26T21:49:29.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.543+0000 7fbc27e00640 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a02d4a05b0 0x55a02d4a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.544+0000 7fbc27e00640 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a02d4a05b0 0x55a02d4a29a0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa447db8c4247cac server_cookie=349ff2778f4ff652 in_seq=0 out_seq=0 2024-04-26T21:49:29.546 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.544+0000 7fbc12000640 10 client.26966 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.548+0000 7fbc12000640 1 -- 172.21.15.1:0/2163105815 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a02d3eff90 con 0x55a02d4a05b0 2024-04-26T21:49:29.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.548+0000 7fbc12000640 10 client.26966 handle_command_reply: tid=0 2024-04-26T21:49:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.548+0000 7fbc31fd6740 1 client.26966 shutdown 2024-04-26T21:49:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.549+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2163105815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 msgr2=0x55a02d48dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.549+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d48dd30 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400b910 tx=0x7fbc1400bdd0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.550+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2163105815 shutdown_connections 2024-04-26T21:49:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.550+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a02d4a05b0 0x55a02d4a29a0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.550+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d48e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.550+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d485f00 0x55a02d48dd30 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.550+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2163105815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4855e0 0x55a02d48d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.550+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2163105815 >> 172.21.15.1:0/2163105815 conn(0x55a02d481af0 msgr2=0x55a02d478440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.550+0000 7fbc12000640 0 client.26966 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.550+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2163105815 shutdown_connections 2024-04-26T21:49:29.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.551+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2163105815 wait complete. 2024-04-26T21:49:29.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.551+0000 7fbc31fd6740 20 client.26966 trim_cache size 0 max 16384 2024-04-26T21:49:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.586+0000 7fbc31fd6740 1 Processor -- start 2024-04-26T21:49:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.586+0000 7fbc31fd6740 1 -- start start 2024-04-26T21:49:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.586+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 0x55a02d49c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.586+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d485f00 0x55a02d4a8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.587+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4907c0 0x55a02d4a87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.587+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281b8f80 con 0x55a02d485f00 2024-04-26T21:49:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.587+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281b6850 con 0x55a02d4855e0 2024-04-26T21:49:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.587+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281b46c0 con 0x55a02d4907c0 2024-04-26T21:49:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.587+0000 7fbc2d000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4907c0 0x55a02d4a87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.587+0000 7fbc2d000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4907c0 0x55a02d4a87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51684/0 (socket says 172.21.15.1:51684) 2024-04-26T21:49:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.587+0000 7fbc2d000640 1 -- 172.21.15.1:0/1565454841 learned_addr learned my addr 172.21.15.1:0/1565454841 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.587+0000 7fbc27400640 1 --2- 172.21.15.1:0/1565454841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d485f00 0x55a02d4a8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.588+0000 7fbc2d000640 1 -- 172.21.15.1:0/1565454841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 msgr2=0x55a02d49c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.588+0000 7fbc2d000640 1 --2- 172.21.15.1:0/1565454841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 0x55a02d49c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.588+0000 7fbc2d000640 1 -- 172.21.15.1:0/1565454841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d485f00 msgr2=0x55a02d4a8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.588+0000 7fbc2d000640 1 --2- 172.21.15.1:0/1565454841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d485f00 0x55a02d4a8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.588+0000 7fbc2d000640 1 -- 172.21.15.1:0/1565454841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a02d1c0620 con 0x55a02d4907c0 2024-04-26T21:49:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.592+0000 7fbc2d000640 1 --2- 172.21.15.1:0/1565454841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4907c0 0x55a02d4a87d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbc20003fd0 tx=0x7fbc20040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c3816ae1bd3a19b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.592+0000 7fbc1cc00640 1 -- 172.21.15.1:0/1565454841 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20036040 con 0x55a02d4907c0 2024-04-26T21:49:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.592+0000 7fbc1cc00640 1 -- 172.21.15.1:0/1565454841 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc20044030 con 0x55a02d4907c0 2024-04-26T21:49:29.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.592+0000 7fbc1cc00640 1 -- 172.21.15.1:0/1565454841 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc20031c60 con 0x55a02d4907c0 2024-04-26T21:49:29.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.594+0000 7fbc31fd6740 1 -- 172.21.15.1:0/1565454841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4907c0 msgr2=0x55a02d4a87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:29.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.594+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/1565454841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4907c0 0x55a02d4a87d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbc20003fd0 tx=0x7fbc20040000 comp rx=0 tx=0).stop 2024-04-26T21:49:29.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.594+0000 7fbc31fd6740 1 -- 172.21.15.1:0/1565454841 shutdown_connections 2024-04-26T21:49:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.594+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/1565454841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d4907c0 0x55a02d4a87d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.594+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/1565454841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d485f00 0x55a02d4a8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.594+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/1565454841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 0x55a02d49c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.594+0000 7fbc31fd6740 1 -- 172.21.15.1:0/1565454841 >> 172.21.15.1:0/1565454841 conn(0x55a02d481af0 msgr2=0x55a02d488b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.594+0000 7fbc31fd6740 1 -- 172.21.15.1:0/1565454841 shutdown_connections 2024-04-26T21:49:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.594+0000 7fbc31fd6740 1 -- 172.21.15.1:0/1565454841 wait complete. 2024-04-26T21:49:29.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.595+0000 7fbc31fd6740 1 -- start start 2024-04-26T21:49:29.596 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.595+0000 7fbc31fd6740 1 Processor -- start 2024-04-26T21:49:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.597+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 0x55a02d493a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.597 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.597+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d485f00 0x55a02d493f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.597+0000 7fbc31fd6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d4feb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.597+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281b4d10 con 0x55a02d4907c0 2024-04-26T21:49:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.597+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281b6180 con 0x55a02d4855e0 2024-04-26T21:49:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.597+0000 7fbc31fd6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbc281b53e0 con 0x55a02d485f00 2024-04-26T21:49:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.598+0000 7fbc31fd6740 10 client.0 _ll_register_callbacks cb 0x7fbc14005d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:49:29.598 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.598+0000 7fbc27400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d485f00 0x55a02d493f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.598+0000 7fbc27400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d485f00 0x55a02d493f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51700/0 (socket says 172.21.15.1:51700) 2024-04-26T21:49:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.598+0000 7fbc27e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 0x55a02d493a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.598+0000 7fbc27e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 0x55a02d493a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54446/0 (socket says 172.21.15.1:54446) 2024-04-26T21:49:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.598+0000 7fbc27e00640 1 -- 172.21.15.1:0/2156865720 learned_addr learned my addr 172.21.15.1:0/2156865720 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.599+0000 7fbc27e00640 1 -- 172.21.15.1:0/2156865720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d485f00 msgr2=0x55a02d493f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.599+0000 7fbc27e00640 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d485f00 0x55a02d493f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.599+0000 7fbc27e00640 1 -- 172.21.15.1:0/2156865720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 msgr2=0x55a02d4feb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.599+0000 7fbc27e00640 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d4feb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:29.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.599+0000 7fbc27e00640 1 -- 172.21.15.1:0/2156865720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a02d39ccb0 con 0x55a02d4855e0 2024-04-26T21:49:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.600+0000 7fbc27e00640 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 0x55a02d493a50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800bdb0 tx=0x7fbc1800bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce47d9fb65541948 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.600+0000 7fbc12000640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:49:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.600+0000 7fbc12000640 1 -- 172.21.15.1:0/2156865720 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc180b4020 con 0x55a02d4855e0 2024-04-26T21:49:29.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.600+0000 7fbc12000640 1 -- 172.21.15.1:0/2156865720 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9626+0+0 (secure 0 0 0) 0x7fbc180cd810 con 0x55a02d4855e0 2024-04-26T21:49:29.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.600+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2156865720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc20044030 con 0x55a02d4855e0 2024-04-26T21:49:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.604+0000 7fbc12000640 1 -- 172.21.15.1:0/2156865720 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fbc1809c5f0 con 0x55a02d4855e0 2024-04-26T21:49:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.604+0000 7fbc31fd6740 10 client.26969 fetch_fsmap learned FSMap version 57 2024-04-26T21:49:29.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.604+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2156865720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55a02d506310 con 0x55a02d4855e0 2024-04-26T21:49:29.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.605+0000 7fbc12000640 1 -- 172.21.15.1:0/2156865720 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc20044030 con 0x55a02d4855e0 2024-04-26T21:49:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.606+0000 7fbc12000640 1 -- 172.21.15.1:0/2156865720 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fbc180cdce0 con 0x55a02d4855e0 2024-04-26T21:49:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.606+0000 7fbc31fd6740 10 client.26969 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:49:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.606+0000 7fbc31fd6740 10 client.26969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:49:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.606+0000 7fbc31fd6740 20 client.26969 populate_metadata read hostname 'smithi001' 2024-04-26T21:49:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.606+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a02d5065a0 0x55a02d508990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.607+0000 7fbc31fd6740 4 client.26969 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:49:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.607+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2156865720 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a02d3e9e00 con 0x55a02d5065a0 2024-04-26T21:49:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.607+0000 7fbc27400640 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a02d5065a0 0x55a02d508990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.608+0000 7fbc27400640 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a02d5065a0 0x55a02d508990 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7f2dfb6f4b219c6 server_cookie=3e6477ba724fceb5 in_seq=0 out_seq=0 2024-04-26T21:49:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:29.608+0000 7fbc12000640 10 client.26969 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:30.169 INFO:tasks.workunit.client.0.smithi001.stdout: 10 139426 21.97 MB/sec execute 207 sec latency 654.225 ms 2024-04-26T21:49:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:30 smithi001 ceph-mon[28054]: pgmap v608: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 426 op/s 2024-04-26T21:49:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:30 smithi038 ceph-mon[32724]: pgmap v608: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 426 op/s 2024-04-26T21:49:30.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:30 smithi186 ceph-mon[32991]: pgmap v608: 129 pgs: 129 active+clean; 1.3 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.8 MiB/s wr, 426 op/s 2024-04-26T21:49:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.805+0000 7fbc12000640 1 -- 172.21.15.1:0/2156865720 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a02d3e9e00 con 0x55a02d5065a0 2024-04-26T21:49:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.805+0000 7fbc12000640 10 client.26969 handle_command_reply: tid=0 2024-04-26T21:49:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.806+0000 7fbc31fd6740 1 client.26969 shutdown 2024-04-26T21:49:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.806+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2156865720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 msgr2=0x55a02d493a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.806+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 0x55a02d493a50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800bdb0 tx=0x7fbc1800bee0 comp rx=0 tx=0).stop 2024-04-26T21:49:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.806+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2156865720 shutdown_connections 2024-04-26T21:49:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.807+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55a02d5065a0 0x55a02d508990 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.807+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a02d485f00 0x55a02d493f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.807+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55a02d4907c0 0x55a02d4feb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.807+0000 7fbc31fd6740 1 --2- 172.21.15.1:0/2156865720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a02d4855e0 0x55a02d493a50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.807+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2156865720 >> 172.21.15.1:0/2156865720 conn(0x55a02d481af0 msgr2=0x55a02d48c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.807+0000 7fbc12000640 0 client.26969 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.807+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2156865720 shutdown_connections 2024-04-26T21:49:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.808+0000 7fbc31fd6740 1 -- 172.21.15.1:0/2156865720 wait complete. 2024-04-26T21:49:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.808+0000 7fbc31fd6740 20 client.26969 trim_cache size 0 max 16384 2024-04-26T21:49:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:49:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:49:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:49:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:49:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 -- 172.21.15.1:0/2410617728 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbc0c0a3ea0 msgr2=0x7fbc0c0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbc0c0a3ea0 0x7fbc0c0a6360 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbc20006010 tx=0x7fbc2002a750 comp rx=0 tx=0).stop 2024-04-26T21:49:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 -- 172.21.15.1:0/2410617728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc28109b40 msgr2=0x7fbc2810b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc28109b40 0x7fbc2810b3a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800b3f0 tx=0x7fbc1800b8b0 comp rx=0 tx=0).stop 2024-04-26T21:49:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 -- 172.21.15.1:0/2410617728 shutdown_connections 2024-04-26T21:49:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fbc0c0a3ea0 0x7fbc0c0a6360 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc28115a80 0x7fbc2810be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbc2810a460 0x7fbc2810b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 --2- 172.21.15.1:0/2410617728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc28109b40 0x7fbc2810b3a0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 -- 172.21.15.1:0/2410617728 >> 172.21.15.1:0/2410617728 conn(0x7fbc2806da10 msgr2=0x7fbc28072ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 -- 172.21.15.1:0/2410617728 shutdown_connections 2024-04-26T21:49:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:30.810+0000 7fbc2f000640 1 -- 172.21.15.1:0/2410617728 wait complete. 2024-04-26T21:49:31.170 INFO:tasks.workunit.client.0.smithi001.stdout: 10 139662 21.91 MB/sec execute 208 sec latency 1315.681 ms 2024-04-26T21:49:32.170 INFO:tasks.workunit.client.0.smithi001.stdout: 10 140214 21.97 MB/sec execute 209 sec latency 143.156 ms 2024-04-26T21:49:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:32 smithi001 ceph-mon[28054]: pgmap v609: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-04-26T21:49:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:32 smithi038 ceph-mon[32724]: pgmap v609: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-04-26T21:49:32.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:32 smithi186 ceph-mon[32991]: pgmap v609: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-04-26T21:49:33.170 INFO:tasks.workunit.client.0.smithi001.stdout: 10 140851 22.08 MB/sec execute 210 sec latency 83.680 ms 2024-04-26T21:49:34.170 INFO:tasks.workunit.client.0.smithi001.stdout: 10 141152 22.05 MB/sec execute 211 sec latency 230.766 ms 2024-04-26T21:49:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:34 smithi001 ceph-mon[28054]: pgmap v610: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-04-26T21:49:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:34 smithi038 ceph-mon[32724]: pgmap v610: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-04-26T21:49:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:34 smithi186 ceph-mon[32991]: pgmap v610: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 10 MiB/s wr, 481 op/s 2024-04-26T21:49:35.170 INFO:tasks.workunit.client.0.smithi001.stdout: 10 141340 21.99 MB/sec execute 212 sec latency 510.151 ms 2024-04-26T21:49:36.171 INFO:tasks.workunit.client.0.smithi001.stdout: 10 141583 21.93 MB/sec execute 213 sec latency 95.428 ms 2024-04-26T21:49:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:36 smithi001 ceph-mon[28054]: pgmap v611: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 654 op/s 2024-04-26T21:49:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:36 smithi038 ceph-mon[32724]: pgmap v611: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 654 op/s 2024-04-26T21:49:36.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:36 smithi186 ceph-mon[32991]: pgmap v611: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 13 MiB/s wr, 654 op/s 2024-04-26T21:49:37.171 INFO:tasks.workunit.client.0.smithi001.stdout: 10 142006 21.92 MB/sec execute 214 sec latency 86.070 ms 2024-04-26T21:49:38.171 INFO:tasks.workunit.client.0.smithi001.stdout: 10 142538 21.97 MB/sec execute 215 sec latency 82.608 ms 2024-04-26T21:49:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:38 smithi001 ceph-mon[28054]: pgmap v612: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 543 op/s 2024-04-26T21:49:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:38 smithi038 ceph-mon[32724]: pgmap v612: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 543 op/s 2024-04-26T21:49:38.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:38 smithi186 ceph-mon[32991]: pgmap v612: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 543 op/s 2024-04-26T21:49:39.171 INFO:tasks.workunit.client.0.smithi001.stdout: 10 143077 21.97 MB/sec execute 216 sec latency 79.859 ms 2024-04-26T21:49:40.171 INFO:tasks.workunit.client.0.smithi001.stdout: 10 143661 21.99 MB/sec execute 217 sec latency 89.621 ms 2024-04-26T21:49:40.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:40 smithi001 ceph-mon[28054]: pgmap v613: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 517 op/s 2024-04-26T21:49:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:40 smithi038 ceph-mon[32724]: pgmap v613: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 517 op/s 2024-04-26T21:49:40.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:40 smithi186 ceph-mon[32991]: pgmap v613: 129 pgs: 129 active+clean; 1.2 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 517 op/s 2024-04-26T21:49:41.172 INFO:tasks.workunit.client.0.smithi001.stdout: 10 144169 22.07 MB/sec execute 218 sec latency 105.795 ms 2024-04-26T21:49:42.172 INFO:tasks.workunit.client.0.smithi001.stdout: 10 144670 22.12 MB/sec execute 219 sec latency 111.531 ms 2024-04-26T21:49:42.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:42 smithi001 ceph-mon[28054]: pgmap v614: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 706 op/s 2024-04-26T21:49:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:42 smithi038 ceph-mon[32724]: pgmap v614: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 706 op/s 2024-04-26T21:49:42.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:42 smithi186 ceph-mon[32991]: pgmap v614: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 13 MiB/s wr, 706 op/s 2024-04-26T21:49:43.172 INFO:tasks.workunit.client.0.smithi001.stdout: 10 145040 22.08 MB/sec execute 220 sec latency 78.225 ms 2024-04-26T21:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.017+0000 7fd1d3c00640 1 -- 172.21.15.1:0/3957486752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1cc108920 msgr2=0x7fd1cc108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.017+0000 7fd1d3c00640 1 --2- 172.21.15.1:0/3957486752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1cc108920 0x7fd1cc108d00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009930 tx=0x7fd1b802a440 comp rx=0 tx=0).stop 2024-04-26T21:49:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.019+0000 7fd1d3c00640 1 -- 172.21.15.1:0/3957486752 shutdown_connections 2024-04-26T21:49:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.020+0000 7fd1d3c00640 1 --2- 172.21.15.1:0/3957486752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc1032d0 0x7fd1cc10f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.020+0000 7fd1d3c00640 1 --2- 172.21.15.1:0/3957486752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1cc102910 0x7fd1cc102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.020+0000 7fd1d3c00640 1 --2- 172.21.15.1:0/3957486752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1cc108920 0x7fd1cc108d00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.020+0000 7fd1d3c00640 1 -- 172.21.15.1:0/3957486752 >> 172.21.15.1:0/3957486752 conn(0x7fd1cc0fe420 msgr2=0x7fd1cc100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.020+0000 7fd1d3c00640 1 -- 172.21.15.1:0/3957486752 shutdown_connections 2024-04-26T21:49:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.020+0000 7fd1d3c00640 1 -- 172.21.15.1:0/3957486752 wait complete. 2024-04-26T21:49:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.022+0000 7fd1d3c00640 1 Processor -- start 2024-04-26T21:49:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.022+0000 7fd1d3c00640 1 -- start start 2024-04-26T21:49:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.023+0000 7fd1d3c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1cc102910 0x7fd1cc19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.023+0000 7fd1d3c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1cc1032d0 0x7fd1cc19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.023+0000 7fd1d3c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc108920 0x7fd1cc19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.023+0000 7fd1d3c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc112c30 con 0x7fd1cc108920 2024-04-26T21:49:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.023+0000 7fd1d3c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc112ab0 con 0x7fd1cc102910 2024-04-26T21:49:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.023+0000 7fd1d3c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1cc112db0 con 0x7fd1cc1032d0 2024-04-26T21:49:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.023+0000 7fd1cbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1cc1032d0 0x7fd1cc19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.023+0000 7fd1cbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1cc1032d0 0x7fd1cc19b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41120/0 (socket says 172.21.15.1:41120) 2024-04-26T21:49:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.023+0000 7fd1cbe00640 1 -- 172.21.15.1:0/2658263497 learned_addr learned my addr 172.21.15.1:0/2658263497 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.024+0000 7fd1cbe00640 1 -- 172.21.15.1:0/2658263497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1cc102910 msgr2=0x7fd1cc19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.024+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1cc102910 0x7fd1cc19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.024+0000 7fd1cbe00640 1 -- 172.21.15.1:0/2658263497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc108920 msgr2=0x7fd1cc19fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.024+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc108920 0x7fd1cc19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.024+0000 7fd1cbe00640 1 -- 172.21.15.1:0/2658263497 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1cc1a0380 con 0x7fd1cc1032d0 2024-04-26T21:49:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.024+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1cc1032d0 0x7fd1cc19b8d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000e9f0 tx=0x7fd1c000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.025+0000 7fd1c9600640 1 -- 172.21.15.1:0/2658263497 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c000cd80 con 0x7fd1cc1032d0 2024-04-26T21:49:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.025+0000 7fd1c9600640 1 -- 172.21.15.1:0/2658263497 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c00054d0 con 0x7fd1cc1032d0 2024-04-26T21:49:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.025+0000 7fd1c9600640 1 -- 172.21.15.1:0/2658263497 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0010690 con 0x7fd1cc1032d0 2024-04-26T21:49:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.025+0000 7fd1d3c00640 1 -- 172.21.15.1:0/2658263497 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1cc1a0670 con 0x7fd1cc1032d0 2024-04-26T21:49:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.025+0000 7fd1d3c00640 1 -- 172.21.15.1:0/2658263497 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1cc1a7f00 con 0x7fd1cc1032d0 2024-04-26T21:49:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.033+0000 7fd1d69ca740 1 Processor -- start 2024-04-26T21:49:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.033+0000 7fd1c9600640 1 -- 172.21.15.1:0/2658263497 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd1c0010830 con 0x7fd1cc1032d0 2024-04-26T21:49:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.034+0000 7fd1d69ca740 1 -- start start 2024-04-26T21:49:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.034+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c78169c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.034+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c7816f00 0x55b6c7821280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.034+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78217c0 0x55b6c7823bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.034+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b6c7773380 con 0x55b6c7816f00 2024-04-26T21:49:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.034+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b6c770fc30 con 0x55b6c78165e0 2024-04-26T21:49:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.034+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b6c776f340 con 0x55b6c78217c0 2024-04-26T21:49:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.035+0000 7fd1d1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c78169c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.035+0000 7fd1d1c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c78169c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47318/0 (socket says 172.21.15.1:47318) 2024-04-26T21:49:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.035+0000 7fd1d1c00640 1 -- 172.21.15.1:0/3033812946 learned_addr learned my addr 172.21.15.1:0/3033812946 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.035+0000 7fd1d1c00640 1 -- 172.21.15.1:0/3033812946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78217c0 msgr2=0x55b6c7823bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:44.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.035+0000 7fd1d1c00640 1 --2- 172.21.15.1:0/3033812946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78217c0 0x55b6c7823bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.035+0000 7fd1d1c00640 1 -- 172.21.15.1:0/3033812946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c7816f00 msgr2=0x55b6c7821280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.035+0000 7fd1d1c00640 1 --2- 172.21.15.1:0/3033812946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c7816f00 0x55b6c7821280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.035+0000 7fd1d1c00640 1 -- 172.21.15.1:0/3033812946 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6c7731940 con 0x55b6c78165e0 2024-04-26T21:49:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.036+0000 7fd1d1c00640 1 --2- 172.21.15.1:0/3033812946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c78169c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009d10 tx=0x7fd1c400e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c42732cec26131e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.036+0000 7fd1be000640 1 -- 172.21.15.1:0/3033812946 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c4016070 con 0x55b6c78165e0 2024-04-26T21:49:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.037+0000 7fd1be000640 1 -- 172.21.15.1:0/3033812946 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c4005020 con 0x55b6c78165e0 2024-04-26T21:49:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.037+0000 7fd1be000640 1 -- 172.21.15.1:0/3033812946 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c40116d0 con 0x55b6c78165e0 2024-04-26T21:49:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.038+0000 7fd1c9600640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1b40a3d10 0x7fd1b40a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.038+0000 7fd1c9600640 1 -- 172.21.15.1:0/2658263497 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9626+0+0 (secure 0 0 0) 0x7fd1c00d1080 con 0x7fd1cc1032d0 2024-04-26T21:49:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.039+0000 7fd1d69ca740 1 -- 172.21.15.1:0/3033812946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 msgr2=0x55b6c78169c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.039+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/3033812946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c78169c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd1c4009d10 tx=0x7fd1c400e6c0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.039+0000 7fd1d1200640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1b40a3d10 0x7fd1b40a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.039+0000 7fd1d69ca740 1 -- 172.21.15.1:0/3033812946 shutdown_connections 2024-04-26T21:49:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.039+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/3033812946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78217c0 0x55b6c7823bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.039+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/3033812946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c7816f00 0x55b6c7821280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.039+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/3033812946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c78169c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.040 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.039+0000 7fd1d69ca740 1 -- 172.21.15.1:0/3033812946 >> 172.21.15.1:0/3033812946 conn(0x55b6c7812af0 msgr2=0x55b6c7812ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.040+0000 7fd1d69ca740 1 -- 172.21.15.1:0/3033812946 shutdown_connections 2024-04-26T21:49:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.040+0000 7fd1d69ca740 1 -- 172.21.15.1:0/3033812946 wait complete. 2024-04-26T21:49:44.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.041+0000 7fd1d1200640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1b40a3d10 0x7fd1b40a61d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80096c0 tx=0x7fd1b8000bc0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.042+0000 7fd1d69ca740 1 -- start start 2024-04-26T21:49:44.042 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.042+0000 7fd1d69ca740 1 Processor -- start 2024-04-26T21:49:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.043+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c781e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.043+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 0x55b6c781ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.043+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c78217c0 0x55b6c781f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.043+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b6c78243f0 con 0x55b6c78217c0 2024-04-26T21:49:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.043+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55b6c76fac60 con 0x55b6c7816f00 2024-04-26T21:49:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.043+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b6c7824570 con 0x55b6c78165e0 2024-04-26T21:49:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.043+0000 7fd1d69ca740 10 client.0 _ll_register_callbacks cb 0x55b6c7726e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:49:44.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.044+0000 7fd1d1200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c781e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.044+0000 7fd1d1200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c781e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41138/0 (socket says 172.21.15.1:41138) 2024-04-26T21:49:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.044+0000 7fd1d1200640 1 -- 172.21.15.1:0/2116859938 learned_addr learned my addr 172.21.15.1:0/2116859938 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.044+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 0x55b6c781ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.044+0000 7fd1d1200640 1 -- 172.21.15.1:0/2116859938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 msgr2=0x55b6c781ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.044+0000 7fd1d1200640 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 0x55b6c781ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.044+0000 7fd1d1200640 1 -- 172.21.15.1:0/2116859938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c78217c0 msgr2=0x55b6c781f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:44.046 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.044+0000 7fd1d1200640 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c78217c0 0x55b6c781f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.044+0000 7fd1d1200640 1 -- 172.21.15.1:0/2116859938 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6c7709510 con 0x55b6c78165e0 2024-04-26T21:49:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.045+0000 7fd1d1200640 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c781e7a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8033000 tx=0x7fd1b8005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5b09b70649b4e2c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.045+0000 7fd1b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:49:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.045+0000 7fd1b2a00640 1 -- 172.21.15.1:0/2116859938 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8042050 con 0x55b6c78165e0 2024-04-26T21:49:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.045+0000 7fd1b2a00640 1 -- 172.21.15.1:0/2116859938 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9626+0+0 (secure 0 0 0) 0x7fd1b8004440 con 0x55b6c78165e0 2024-04-26T21:49:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.045+0000 7fd1d69ca740 1 -- 172.21.15.1:0/2116859938 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1c4005020 con 0x55b6c78165e0 2024-04-26T21:49:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.046+0000 7fd1b2a00640 1 -- 172.21.15.1:0/2116859938 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fd1b8035040 con 0x55b6c78165e0 2024-04-26T21:49:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.046+0000 7fd1d69ca740 10 client.26980 fetch_fsmap learned FSMap version 57 2024-04-26T21:49:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.046+0000 7fd1d69ca740 1 -- 172.21.15.1:0/2116859938 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55b6c7485210 con 0x55b6c78165e0 2024-04-26T21:49:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.046+0000 7fd1b2a00640 1 -- 172.21.15.1:0/2116859938 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1c4005020 con 0x55b6c78165e0 2024-04-26T21:49:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.047+0000 7fd1b2a00640 1 -- 172.21.15.1:0/2116859938 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fd1b804f030 con 0x55b6c78165e0 2024-04-26T21:49:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.047+0000 7fd1d69ca740 10 client.26980 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:49:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.047+0000 7fd1d69ca740 10 client.26980 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:49:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.047+0000 7fd1d69ca740 20 client.26980 populate_metadata read hostname 'smithi001' 2024-04-26T21:49:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.047+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b6c7831690 0x55b6c7833ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.047+0000 7fd1d69ca740 4 client.26980 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:49:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.047+0000 7fd1d69ca740 1 -- 172.21.15.1:0/2116859938 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6c7780f90 con 0x55b6c7831690 2024-04-26T21:49:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.048+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b6c7831690 0x55b6c7833ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.050+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b6c7831690 0x55b6c7833ae0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=341587ec92c80896 server_cookie=79c8d99cfb6c038c in_seq=0 out_seq=0 2024-04-26T21:49:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.050+0000 7fd1b2a00640 10 client.26980 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.051+0000 7fd1b2a00640 1 -- 172.21.15.1:0/2116859938 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b6c7780f90 con 0x55b6c7831690 2024-04-26T21:49:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.051+0000 7fd1b2a00640 10 client.26980 handle_command_reply: tid=0 2024-04-26T21:49:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.051+0000 7fd1d69ca740 1 client.26980 shutdown 2024-04-26T21:49:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.052+0000 7fd1d69ca740 1 -- 172.21.15.1:0/2116859938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 msgr2=0x55b6c781e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.052+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c781e7a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8033000 tx=0x7fd1b8005fd0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.052+0000 7fd1d69ca740 1 -- 172.21.15.1:0/2116859938 shutdown_connections 2024-04-26T21:49:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.052+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b6c7831690 0x55b6c7833ae0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.052+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c78217c0 0x55b6c781f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.052+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 0x55b6c781ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.052+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/2116859938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c781e7a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.052+0000 7fd1b2a00640 0 client.26980 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.052+0000 7fd1d69ca740 1 -- 172.21.15.1:0/2116859938 >> 172.21.15.1:0/2116859938 conn(0x55b6c7812af0 msgr2=0x55b6c7809440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.053+0000 7fd1d69ca740 1 -- 172.21.15.1:0/2116859938 shutdown_connections 2024-04-26T21:49:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.053+0000 7fd1d69ca740 1 -- 172.21.15.1:0/2116859938 wait complete. 2024-04-26T21:49:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.053+0000 7fd1d69ca740 20 client.26980 trim_cache size 0 max 16384 2024-04-26T21:49:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.079+0000 7fd1d69ca740 1 Processor -- start 2024-04-26T21:49:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.079+0000 7fd1d69ca740 1 -- start start 2024-04-26T21:49:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.080+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c7820ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.080+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 0x55b6c782d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.080+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c78217c0 0x55b6c78393f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.080+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac012a10 con 0x55b6c78217c0 2024-04-26T21:49:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.080+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac0102e0 con 0x55b6c7816f00 2024-04-26T21:49:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.080+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac00e150 con 0x55b6c78165e0 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.080+0000 7fd1d1200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c7820ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.080+0000 7fd1d1200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c7820ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:41144/0 (socket says 172.21.15.1:41144) 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.080+0000 7fd1d1200640 1 -- 172.21.15.1:0/3927926798 learned_addr learned my addr 172.21.15.1:0/3927926798 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.080+0000 7fd1d1c00640 1 --2- 172.21.15.1:0/3927926798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c78217c0 0x55b6c78393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.081+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/3927926798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 0x55b6c782d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.081+0000 7fd1d1200640 1 -- 172.21.15.1:0/3927926798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 msgr2=0x55b6c782d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.081+0000 7fd1d1200640 1 --2- 172.21.15.1:0/3927926798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 0x55b6c782d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.081+0000 7fd1d1200640 1 -- 172.21.15.1:0/3927926798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c78217c0 msgr2=0x55b6c78393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.081+0000 7fd1d1200640 1 --2- 172.21.15.1:0/3927926798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c78217c0 0x55b6c78393f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.081+0000 7fd1d1200640 1 -- 172.21.15.1:0/3927926798 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6c7551620 con 0x55b6c78165e0 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.081+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/3927926798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 0x55b6c782d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:49:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.081+0000 7fd1d1200640 1 --2- 172.21.15.1:0/3927926798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c7820ed0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8005c40 tx=0x7fd1b802aa80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98ebba6554883a3e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:44.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.082+0000 7fd1bd600640 1 -- 172.21.15.1:0/3927926798 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b805c020 con 0x55b6c78165e0 2024-04-26T21:49:44.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.082+0000 7fd1bd600640 1 -- 172.21.15.1:0/3927926798 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1b8040070 con 0x55b6c78165e0 2024-04-26T21:49:44.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.082+0000 7fd1bd600640 1 -- 172.21.15.1:0/3927926798 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8042070 con 0x55b6c78165e0 2024-04-26T21:49:44.083 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.082+0000 7fd1d69ca740 1 -- 172.21.15.1:0/3927926798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 msgr2=0x55b6c7820ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.082+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/3927926798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c7820ed0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8005c40 tx=0x7fd1b802aa80 comp rx=0 tx=0).stop 2024-04-26T21:49:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.083+0000 7fd1d69ca740 1 -- 172.21.15.1:0/3927926798 shutdown_connections 2024-04-26T21:49:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.083+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/3927926798 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c78217c0 0x55b6c78393f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.083+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/3927926798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c7816f00 0x55b6c782d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.083+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/3927926798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78165e0 0x55b6c7820ed0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.083+0000 7fd1d69ca740 1 -- 172.21.15.1:0/3927926798 >> 172.21.15.1:0/3927926798 conn(0x55b6c7812af0 msgr2=0x55b6c7809440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.083+0000 7fd1d69ca740 1 -- 172.21.15.1:0/3927926798 shutdown_connections 2024-04-26T21:49:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.083+0000 7fd1d69ca740 1 -- 172.21.15.1:0/3927926798 wait complete. 2024-04-26T21:49:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.083+0000 7fd1d69ca740 1 -- start start 2024-04-26T21:49:44.084 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.084+0000 7fd1d69ca740 1 Processor -- start 2024-04-26T21:49:44.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.084+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c7825160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.084+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c7816f00 0x55b6c78256a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d69ca740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78217c0 0x55b6c7825be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac00e7a0 con 0x55b6c7816f00 2024-04-26T21:49:44.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac00fc10 con 0x55b6c78165e0 2024-04-26T21:49:44.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d69ca740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac00ee70 con 0x55b6c78217c0 2024-04-26T21:49:44.085 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d69ca740 10 client.0 _ll_register_callbacks cb 0x7fd1c0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c7825160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d1200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c7825160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47346/0 (socket says 172.21.15.1:47346) 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d1200640 1 -- 172.21.15.1:0/1442271593 learned_addr learned my addr 172.21.15.1:0/1442271593 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d1c00640 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78217c0 0x55b6c7825be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c7816f00 0x55b6c78256a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d1200640 1 -- 172.21.15.1:0/1442271593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78217c0 msgr2=0x55b6c7825be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d1200640 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78217c0 0x55b6c7825be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d1200640 1 -- 172.21.15.1:0/1442271593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c7816f00 msgr2=0x55b6c78256a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d1200640 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c7816f00 0x55b6c78256a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.085+0000 7fd1d1200640 1 -- 172.21.15.1:0/1442271593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6c772dcb0 con 0x55b6c78165e0 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.086+0000 7fd1d1200640 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c7825160 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8006010 tx=0x7fd1b8005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70d8cd390fb13979 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.086+0000 7fd1b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:49:44.087 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.086+0000 7fd1b2a00640 1 -- 172.21.15.1:0/1442271593 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b805c030 con 0x55b6c78165e0 2024-04-26T21:49:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.086+0000 7fd1d69ca740 1 -- 172.21.15.1:0/1442271593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1b8040070 con 0x55b6c78165e0 2024-04-26T21:49:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.086+0000 7fd1b2a00640 1 -- 172.21.15.1:0/1442271593 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9626+0+0 (secure 0 0 0) 0x7fd1b8003b20 con 0x55b6c78165e0 2024-04-26T21:49:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.086+0000 7fd1b2a00640 1 -- 172.21.15.1:0/1442271593 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fd1b8070050 con 0x55b6c78165e0 2024-04-26T21:49:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.087+0000 7fd1d69ca740 10 client.26981 fetch_fsmap learned FSMap version 57 2024-04-26T21:49:44.088 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.087+0000 7fd1d69ca740 1 -- 172.21.15.1:0/1442271593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55b6c7897c50 con 0x55b6c78165e0 2024-04-26T21:49:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.087+0000 7fd1b2a00640 1 -- 172.21.15.1:0/1442271593 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1b8040070 con 0x55b6c78165e0 2024-04-26T21:49:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.088+0000 7fd1b2a00640 1 -- 172.21.15.1:0/1442271593 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fd1b804f040 con 0x55b6c78165e0 2024-04-26T21:49:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.088+0000 7fd1d69ca740 10 client.26981 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:49:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.088+0000 7fd1d69ca740 10 client.26981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:49:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.088+0000 7fd1d69ca740 20 client.26981 populate_metadata read hostname 'smithi001' 2024-04-26T21:49:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.088+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b6c7898300 0x55b6c789a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.088+0000 7fd1d69ca740 4 client.26981 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:49:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.088+0000 7fd1d69ca740 1 -- 172.21.15.1:0/1442271593 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b6c777ae00 con 0x55b6c7898300 2024-04-26T21:49:44.089 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.088+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b6c7898300 0x55b6c789a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:44.090 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.089+0000 7fd1cbe00640 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b6c7898300 0x55b6c789a6f0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffb52d59a0980272 server_cookie=aa8187021189c8aa in_seq=0 out_seq=0 2024-04-26T21:49:44.090 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.089+0000 7fd1b2a00640 10 client.26981 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:44.172 INFO:tasks.workunit.client.0.smithi001.stdout: 10 145407 22.07 MB/sec execute 221 sec latency 80.173 ms 2024-04-26T21:49:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:44 smithi001 ceph-mon[28054]: pgmap v615: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.7 MiB/s wr, 444 op/s 2024-04-26T21:49:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:44 smithi038 ceph-mon[32724]: pgmap v615: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.7 MiB/s wr, 444 op/s 2024-04-26T21:49:44.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:44 smithi186 ceph-mon[32991]: pgmap v615: 129 pgs: 129 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.7 MiB/s wr, 444 op/s 2024-04-26T21:49:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.729+0000 7fd1b2a00640 1 -- 172.21.15.1:0/1442271593 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b6c777ae00 con 0x55b6c7898300 2024-04-26T21:49:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.729+0000 7fd1b2a00640 10 client.26981 handle_command_reply: tid=0 2024-04-26T21:49:44.730 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.730+0000 7fd1d69ca740 1 client.26981 shutdown 2024-04-26T21:49:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.731+0000 7fd1d69ca740 1 -- 172.21.15.1:0/1442271593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 msgr2=0x55b6c7825160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.731+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c7825160 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8006010 tx=0x7fd1b8005e80 comp rx=0 tx=0).stop 2024-04-26T21:49:44.731 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.731+0000 7fd1d69ca740 1 -- 172.21.15.1:0/1442271593 shutdown_connections 2024-04-26T21:49:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.731+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55b6c7898300 0x55b6c789a6f0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.731+0000 7fd1b2a00640 0 client.26981 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.731+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b6c78217c0 0x55b6c7825be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.731+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b6c7816f00 0x55b6c78256a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.731+0000 7fd1d69ca740 1 --2- 172.21.15.1:0/1442271593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b6c78165e0 0x55b6c7825160 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.731+0000 7fd1d69ca740 1 -- 172.21.15.1:0/1442271593 >> 172.21.15.1:0/1442271593 conn(0x55b6c7812af0 msgr2=0x55b6c7817fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:44.732 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.732+0000 7fd1d69ca740 1 -- 172.21.15.1:0/1442271593 shutdown_connections 2024-04-26T21:49:44.733 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.732+0000 7fd1d69ca740 1 -- 172.21.15.1:0/1442271593 wait complete. 2024-04-26T21:49:44.733 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.733+0000 7fd1d69ca740 20 client.26981 trim_cache size 0 max 16384 2024-04-26T21:49:44.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:49:44.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:49:44.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:49:44.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:49:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.734+0000 7fd1d3c00640 1 -- 172.21.15.1:0/2658263497 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1b40a3d10 msgr2=0x7fd1b40a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.734+0000 7fd1d3c00640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1b40a3d10 0x7fd1b40a61d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80096c0 tx=0x7fd1b8000bc0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.734+0000 7fd1d3c00640 1 -- 172.21.15.1:0/2658263497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1cc1032d0 msgr2=0x7fd1cc19b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.734+0000 7fd1d3c00640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1cc1032d0 0x7fd1cc19b8d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000e9f0 tx=0x7fd1c000edf0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.735+0000 7fd1d3c00640 1 -- 172.21.15.1:0/2658263497 shutdown_connections 2024-04-26T21:49:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.735+0000 7fd1d3c00640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd1b40a3d10 0x7fd1b40a61d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.735+0000 7fd1d3c00640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1cc1032d0 0x7fd1cc19b8d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.735+0000 7fd1d3c00640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1cc108920 0x7fd1cc19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.735+0000 7fd1d3c00640 1 --2- 172.21.15.1:0/2658263497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1cc102910 0x7fd1cc19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.735+0000 7fd1d3c00640 1 -- 172.21.15.1:0/2658263497 >> 172.21.15.1:0/2658263497 conn(0x7fd1cc0fe420 msgr2=0x7fd1cc10e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:44.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.735+0000 7fd1d3c00640 1 -- 172.21.15.1:0/2658263497 shutdown_connections 2024-04-26T21:49:44.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:44.735+0000 7fd1d3c00640 1 -- 172.21.15.1:0/2658263497 wait complete. 2024-04-26T21:49:45.172 INFO:tasks.workunit.client.0.smithi001.stdout: 10 145650 22.01 MB/sec execute 222 sec latency 290.193 ms 2024-04-26T21:49:46.172 INFO:tasks.workunit.client.0.smithi001.stdout: 10 146158 22.03 MB/sec execute 223 sec latency 82.206 ms 2024-04-26T21:49:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:46 smithi001 ceph-mon[28054]: pgmap v616: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 587 op/s 2024-04-26T21:49:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:46 smithi038 ceph-mon[32724]: pgmap v616: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 587 op/s 2024-04-26T21:49:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:46 smithi186 ceph-mon[32991]: pgmap v616: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 8.2 MiB/s wr, 587 op/s 2024-04-26T21:49:47.173 INFO:tasks.workunit.client.0.smithi001.stdout: 10 146670 22.04 MB/sec execute 224 sec latency 86.535 ms 2024-04-26T21:49:48.174 INFO:tasks.workunit.client.0.smithi001.stdout: 10 147334 22.10 MB/sec execute 225 sec latency 117.855 ms 2024-04-26T21:49:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:48 smithi001 ceph-mon[28054]: pgmap v617: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.5 MiB/s wr, 482 op/s 2024-04-26T21:49:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:48 smithi038 ceph-mon[32724]: pgmap v617: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.5 MiB/s wr, 482 op/s 2024-04-26T21:49:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:48 smithi186 ceph-mon[32991]: pgmap v617: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 5.5 MiB/s wr, 482 op/s 2024-04-26T21:49:49.173 INFO:tasks.workunit.client.0.smithi001.stdout: 10 147959 22.21 MB/sec execute 226 sec latency 62.783 ms 2024-04-26T21:49:50.174 INFO:tasks.workunit.client.0.smithi001.stdout: 10 148171 22.15 MB/sec execute 227 sec latency 418.160 ms 2024-04-26T21:49:50.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:50 smithi001 ceph-mon[28054]: pgmap v618: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.8 MiB/s wr, 423 op/s 2024-04-26T21:49:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:50 smithi038 ceph-mon[32724]: pgmap v618: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.8 MiB/s wr, 423 op/s 2024-04-26T21:49:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:50 smithi186 ceph-mon[32991]: pgmap v618: 129 pgs: 129 active+clean; 1.3 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.8 MiB/s wr, 423 op/s 2024-04-26T21:49:51.176 INFO:tasks.workunit.client.0.smithi001.stdout: 10 148309 22.08 MB/sec execute 228 sec latency 1287.464 ms 2024-04-26T21:49:52.176 INFO:tasks.workunit.client.0.smithi001.stdout: 10 148436 22.00 MB/sec execute 229 sec latency 440.255 ms 2024-04-26T21:49:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:52 smithi038 ceph-mon[32724]: pgmap v619: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.0 MiB/s wr, 587 op/s 2024-04-26T21:49:52.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:52 smithi186 ceph-mon[32991]: pgmap v619: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.0 MiB/s wr, 587 op/s 2024-04-26T21:49:52.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:52 smithi001 ceph-mon[28054]: pgmap v619: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.0 MiB/s wr, 587 op/s 2024-04-26T21:49:53.176 INFO:tasks.workunit.client.0.smithi001.stdout: 10 148739 21.99 MB/sec execute 230 sec latency 185.590 ms 2024-04-26T21:49:54.176 INFO:tasks.workunit.client.0.smithi001.stdout: 10 149199 22.00 MB/sec execute 231 sec latency 79.390 ms 2024-04-26T21:49:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:54 smithi186 ceph-mon[32991]: pgmap v620: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 6.5 MiB/s wr, 398 op/s 2024-04-26T21:49:54.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:54 smithi001 ceph-mon[28054]: pgmap v620: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 6.5 MiB/s wr, 398 op/s 2024-04-26T21:49:54.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:54 smithi038 ceph-mon[32724]: pgmap v620: 129 pgs: 129 active+clean; 1.3 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 6.5 MiB/s wr, 398 op/s 2024-04-26T21:49:55.177 INFO:tasks.workunit.client.0.smithi001.stdout: 10 149694 22.00 MB/sec execute 232 sec latency 88.359 ms 2024-04-26T21:49:56.177 INFO:tasks.workunit.client.0.smithi001.stdout: 10 149866 21.94 MB/sec execute 233 sec latency 618.203 ms 2024-04-26T21:49:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:56 smithi186 ceph-mon[32991]: pgmap v621: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 576 op/s 2024-04-26T21:49:56.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:56 smithi001 ceph-mon[28054]: pgmap v621: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 576 op/s 2024-04-26T21:49:56.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:56 smithi038 ceph-mon[32724]: pgmap v621: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 10 MiB/s wr, 576 op/s 2024-04-26T21:49:57.177 INFO:tasks.workunit.client.0.smithi001.stdout: 10 150128 21.91 MB/sec execute 234 sec latency 158.159 ms 2024-04-26T21:49:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.968+0000 7fb1d7200640 1 -- 172.21.15.1:0/2011511672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d010cd50 msgr2=0x7fb1d010f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.968+0000 7fb1d7200640 1 --2- 172.21.15.1:0/2011511672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d010cd50 0x7fb1d010f1f0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8009ca0 tx=0x7fb1c802a950 comp rx=0 tx=0).stop 2024-04-26T21:49:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.969+0000 7fb1d7200640 1 -- 172.21.15.1:0/2011511672 shutdown_connections 2024-04-26T21:49:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.969+0000 7fb1d7200640 1 --2- 172.21.15.1:0/2011511672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d010cd50 0x7fb1d010f1f0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.969+0000 7fb1d7200640 1 --2- 172.21.15.1:0/2011511672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1d0104250 0x7fb1d010c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.969+0000 7fb1d7200640 1 --2- 172.21.15.1:0/2011511672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1d0103930 0x7fb1d0103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:57.970 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.969+0000 7fb1d7200640 1 -- 172.21.15.1:0/2011511672 >> 172.21.15.1:0/2011511672 conn(0x7fb1d00fd130 msgr2=0x7fb1d00ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.969+0000 7fb1d7200640 1 -- 172.21.15.1:0/2011511672 shutdown_connections 2024-04-26T21:49:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.969+0000 7fb1d7200640 1 -- 172.21.15.1:0/2011511672 wait complete. 2024-04-26T21:49:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.971+0000 7fb1d7200640 1 Processor -- start 2024-04-26T21:49:57.973 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.973+0000 7fb1d7200640 1 -- start start 2024-04-26T21:49:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.974+0000 7fb1d7200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d0103930 0x7fb1d0074140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.974+0000 7fb1d7200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1d0104250 0x7fb1d006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.974+0000 7fb1d7200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1d010cd50 0x7fb1d006d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:57.974 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.974+0000 7fb1d7200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d0112ab0 con 0x7fb1d0104250 2024-04-26T21:49:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.974+0000 7fb1d7200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d0112930 con 0x7fb1d010cd50 2024-04-26T21:49:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.974+0000 7fb1d7200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d0112c30 con 0x7fb1d0103930 2024-04-26T21:49:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.974+0000 7fb1d5200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1d010cd50 0x7fb1d006d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.974+0000 7fb1d5200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1d010cd50 0x7fb1d006d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50450/0 (socket says 172.21.15.1:50450) 2024-04-26T21:49:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.974+0000 7fb1d5200640 1 -- 172.21.15.1:0/3110069546 learned_addr learned my addr 172.21.15.1:0/3110069546 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.975+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d0103930 0x7fb1d0074140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.975+0000 7fb1d5200640 1 -- 172.21.15.1:0/3110069546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d0103930 msgr2=0x7fb1d0074140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.975+0000 7fb1d5200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d0103930 0x7fb1d0074140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.975+0000 7fb1d5200640 1 -- 172.21.15.1:0/3110069546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1d0104250 msgr2=0x7fb1d006d1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:49:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.976+0000 7fb1d5200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1d0104250 0x7fb1d006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.976+0000 7fb1d5200640 1 -- 172.21.15.1:0/3110069546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d006dcc0 con 0x7fb1d010cd50 2024-04-26T21:49:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.977+0000 7fb1d5200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1d010cd50 0x7fb1d006d700 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8009dc0 tx=0x7fb1c8003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.982+0000 7fb1cd600640 1 -- 172.21.15.1:0/3110069546 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8034040 con 0x7fb1d010cd50 2024-04-26T21:49:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.982+0000 7fb1cd600640 1 -- 172.21.15.1:0/3110069546 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1c8030850 con 0x7fb1d010cd50 2024-04-26T21:49:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.982+0000 7fb1d7200640 1 -- 172.21.15.1:0/3110069546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d006df50 con 0x7fb1d010cd50 2024-04-26T21:49:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.982+0000 7fb1d7200640 1 -- 172.21.15.1:0/3110069546 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d0071ef0 con 0x7fb1d010cd50 2024-04-26T21:49:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.982+0000 7fb1cd600640 1 -- 172.21.15.1:0/3110069546 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8004680 con 0x7fb1d010cd50 2024-04-26T21:49:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.995+0000 7fb1cd600640 1 -- 172.21.15.1:0/3110069546 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb1c8037020 con 0x7fb1d010cd50 2024-04-26T21:49:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.995+0000 7fb1da059740 1 Processor -- start 2024-04-26T21:49:57.997 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.997+0000 7fb1da059740 1 -- start start 2024-04-26T21:49:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.997+0000 7fb1da059740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e3599f00 0x5584e359a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.998+0000 7fb1da059740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e359a820 0x5584e35a9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.998+0000 7fb1da059740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 0x5584e35ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.998+0000 7fb1da059740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5584e34ff380 con 0x5584e359a820 2024-04-26T21:49:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.998+0000 7fb1da059740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5584e349bc30 con 0x5584e3599f00 2024-04-26T21:49:57.998 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:57.998+0000 7fb1da059740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5584e34fb340 con 0x5584e35a95d0 2024-04-26T21:49:58.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.000+0000 7fb1cd600640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb1b40a3e50 0x7fb1b40a6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.000 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.000+0000 7fb1cd600640 1 -- 172.21.15.1:0/3110069546 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 9626+0+0 (secure 0 0 0) 0x7fb1c80f1a80 con 0x7fb1d010cd50 2024-04-26T21:49:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.001+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e359a820 0x5584e35a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.001+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e359a820 0x5584e35a9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49174/0 (socket says 172.21.15.1:49174) 2024-04-26T21:49:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.001+0000 7fb1cfe00640 1 -- 172.21.15.1:0/484247419 learned_addr learned my addr 172.21.15.1:0/484247419 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.002+0000 7fb1cfe00640 1 -- 172.21.15.1:0/484247419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 msgr2=0x5584e35ab9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:49:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.002+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/484247419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 0x5584e35ab9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.002+0000 7fb1cfe00640 1 -- 172.21.15.1:0/484247419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e3599f00 msgr2=0x5584e359a2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:49:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.002+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/484247419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e3599f00 0x5584e359a2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.002+0000 7fb1cfe00640 1 -- 172.21.15.1:0/484247419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584e34bd940 con 0x5584e359a820 2024-04-26T21:49:58.003 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.002+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/484247419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e359a820 0x5584e35a9090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000b820 tx=0x7fb1c000bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91de5ad8f2c5aee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.005+0000 7fb1c5600640 1 -- 172.21.15.1:0/484247419 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c000d950 con 0x5584e359a820 2024-04-26T21:49:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.005+0000 7fb1c5600640 1 -- 172.21.15.1:0/484247419 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1c0019070 con 0x5584e359a820 2024-04-26T21:49:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.005+0000 7fb1c5600640 1 -- 172.21.15.1:0/484247419 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c00123e0 con 0x5584e359a820 2024-04-26T21:49:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.005+0000 7fb1d5200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb1b40a3e50 0x7fb1b40a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.008 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.007+0000 7fb1d5200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb1b40a3e50 0x7fb1b40a6310 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80be3f0 tx=0x7fb1c803c040 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:58.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.008+0000 7fb1da059740 1 -- 172.21.15.1:0/484247419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e359a820 msgr2=0x5584e35a9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:58.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.008+0000 7fb1da059740 1 --2- 172.21.15.1:0/484247419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e359a820 0x5584e35a9090 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000b820 tx=0x7fb1c000bce0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.008+0000 7fb1da059740 1 -- 172.21.15.1:0/484247419 shutdown_connections 2024-04-26T21:49:58.009 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.008+0000 7fb1da059740 1 --2- 172.21.15.1:0/484247419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 0x5584e35ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.008+0000 7fb1da059740 1 --2- 172.21.15.1:0/484247419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e359a820 0x5584e35a9090 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.008+0000 7fb1da059740 1 --2- 172.21.15.1:0/484247419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e3599f00 0x5584e359a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.008+0000 7fb1da059740 1 -- 172.21.15.1:0/484247419 >> 172.21.15.1:0/484247419 conn(0x5584e35964f0 msgr2=0x5584e35968d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:58.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.009+0000 7fb1da059740 1 -- 172.21.15.1:0/484247419 shutdown_connections 2024-04-26T21:49:58.010 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.009+0000 7fb1da059740 1 -- 172.21.15.1:0/484247419 wait complete. 2024-04-26T21:49:58.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.010+0000 7fb1da059740 1 -- start start 2024-04-26T21:49:58.011 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.010+0000 7fb1da059740 1 Processor -- start 2024-04-26T21:49:58.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.012+0000 7fb1da059740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e35a2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.014 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.012+0000 7fb1da059740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e359a820 0x5584e35a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.012+0000 7fb1da059740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 0x5584e35a7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.012+0000 7fb1da059740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5584e359ce60 con 0x5584e3599f00 2024-04-26T21:49:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.012+0000 7fb1da059740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5584e3486c60 con 0x5584e359a820 2024-04-26T21:49:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.012+0000 7fb1da059740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5584e359cfe0 con 0x5584e35a95d0 2024-04-26T21:49:58.015 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.012+0000 7fb1da059740 10 client.0 _ll_register_callbacks cb 0x5584e34b2e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:49:58.016 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.014+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e359a820 0x5584e35a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.015+0000 7fb1c7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 0x5584e35a7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.015+0000 7fb1c7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 0x5584e35a7010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:34612/0 (socket says 172.21.15.1:34612) 2024-04-26T21:49:58.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.015+0000 7fb1c7e00640 1 -- 172.21.15.1:0/3172120566 learned_addr learned my addr 172.21.15.1:0/3172120566 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:58.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.015+0000 7fb1c7e00640 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e35a2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.016+0000 7fb1c7e00640 1 -- 172.21.15.1:0/3172120566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e359a820 msgr2=0x5584e35a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:58.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.016+0000 7fb1c7e00640 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e359a820 0x5584e35a2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.016+0000 7fb1c7e00640 1 -- 172.21.15.1:0/3172120566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 msgr2=0x5584e35a2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:58.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.016+0000 7fb1c7e00640 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e35a2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.016+0000 7fb1c7e00640 1 -- 172.21.15.1:0/3172120566 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584e3495510 con 0x5584e35a95d0 2024-04-26T21:49:58.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.016+0000 7fb1c7e00640 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 0x5584e35a7010 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8009fb0 tx=0x7fb1b8009690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bafced837ac5b33 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:58.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.016+0000 7fb1be000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:49:58.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.018+0000 7fb1be000640 1 -- 172.21.15.1:0/3172120566 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b80130d0 con 0x5584e35a95d0 2024-04-26T21:49:58.019 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.018+0000 7fb1be000640 1 -- 172.21.15.1:0/3172120566 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9626+0+0 (secure 0 0 0) 0x7fb1b8004050 con 0x5584e35a95d0 2024-04-26T21:49:58.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.018+0000 7fb1da059740 1 -- 172.21.15.1:0/3172120566 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1c0019070 con 0x5584e35a95d0 2024-04-26T21:49:58.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.019+0000 7fb1be000640 1 -- 172.21.15.1:0/3172120566 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fb1b8013270 con 0x5584e35a95d0 2024-04-26T21:49:58.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.020+0000 7fb1da059740 10 client.26995 fetch_fsmap learned FSMap version 57 2024-04-26T21:49:58.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.020+0000 7fb1da059740 1 -- 172.21.15.1:0/3172120566 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5584e3211210 con 0x5584e35a95d0 2024-04-26T21:49:58.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.020+0000 7fb1be000640 1 -- 172.21.15.1:0/3172120566 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1c0019070 con 0x5584e35a95d0 2024-04-26T21:49:58.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.021+0000 7fb1be000640 1 -- 172.21.15.1:0/3172120566 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fb1b8003b30 con 0x5584e35a95d0 2024-04-26T21:49:58.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.021+0000 7fb1da059740 10 client.26995 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:49:58.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.021+0000 7fb1da059740 10 client.26995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:49:58.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.021+0000 7fb1da059740 20 client.26995 populate_metadata read hostname 'smithi001' 2024-04-26T21:49:58.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.021+0000 7fb1da059740 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5584e35b50d0 0x5584e35b74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.021+0000 7fb1da059740 4 client.26995 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:49:58.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.021+0000 7fb1da059740 1 -- 172.21.15.1:0/3172120566 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584e350cf90 con 0x5584e35b50d0 2024-04-26T21:49:58.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.023+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5584e35b50d0 0x5584e35b74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.024+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5584e35b50d0 0x5584e35b74c0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c5ebfedaf9aa38d server_cookie=d4a34cc3077f63f8 in_seq=0 out_seq=0 2024-04-26T21:49:58.024 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.024+0000 7fb1be000640 10 client.26995 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:58.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.025+0000 7fb1be000640 1 -- 172.21.15.1:0/3172120566 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584e350cf90 con 0x5584e35b50d0 2024-04-26T21:49:58.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.025+0000 7fb1be000640 10 client.26995 handle_command_reply: tid=0 2024-04-26T21:49:58.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.025+0000 7fb1da059740 1 client.26995 shutdown 2024-04-26T21:49:58.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.026+0000 7fb1da059740 1 -- 172.21.15.1:0/3172120566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 msgr2=0x5584e35a7010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:58.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.026+0000 7fb1da059740 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 0x5584e35a7010 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb1b8009fb0 tx=0x7fb1b8009690 comp rx=0 tx=0).stop 2024-04-26T21:49:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.026+0000 7fb1da059740 1 -- 172.21.15.1:0/3172120566 shutdown_connections 2024-04-26T21:49:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.026+0000 7fb1da059740 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5584e35b50d0 0x5584e35b74c0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.026+0000 7fb1da059740 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e35a95d0 0x5584e35a7010 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.026+0000 7fb1da059740 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e359a820 0x5584e35a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.026+0000 7fb1da059740 1 --2- 172.21.15.1:0/3172120566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e35a2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.026+0000 7fb1da059740 1 -- 172.21.15.1:0/3172120566 >> 172.21.15.1:0/3172120566 conn(0x5584e35964f0 msgr2=0x5584e359c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.026+0000 7fb1be000640 0 client.26995 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.027+0000 7fb1da059740 1 -- 172.21.15.1:0/3172120566 shutdown_connections 2024-04-26T21:49:58.027 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.027+0000 7fb1da059740 1 -- 172.21.15.1:0/3172120566 wait complete. 2024-04-26T21:49:58.029 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.029+0000 7fb1da059740 20 client.26995 trim_cache size 0 max 16384 2024-04-26T21:49:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.088+0000 7fb1da059740 1 Processor -- start 2024-04-26T21:49:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.090+0000 7fb1da059740 1 -- start start 2024-04-26T21:49:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.091+0000 7fb1da059740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e35bcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.091+0000 7fb1da059740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e35bd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.091+0000 7fb1da059740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e35a95d0 0x5584e35a6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.091+0000 7fb1da059740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac012ad0 con 0x5584e3599f00 2024-04-26T21:49:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.091+0000 7fb1da059740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac0103a0 con 0x5584e35a95d0 2024-04-26T21:49:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.091+0000 7fb1da059740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac00e1d0 con 0x5584e359a820 2024-04-26T21:49:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.092+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e35bd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.092+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e35bd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:34622/0 (socket says 172.21.15.1:34622) 2024-04-26T21:49:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.092+0000 7fb1cfe00640 1 -- 172.21.15.1:0/3823142016 learned_addr learned my addr 172.21.15.1:0/3823142016 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.093+0000 7fb1c7e00640 1 --2- 172.21.15.1:0/3823142016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e35bcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.094+0000 7fb1c7e00640 1 --2- 172.21.15.1:0/3823142016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e35a95d0 0x5584e35a6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.094+0000 7fb1cfe00640 1 -- 172.21.15.1:0/3823142016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e35a95d0 msgr2=0x5584e35a6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.094+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/3823142016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e35a95d0 0x5584e35a6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.094+0000 7fb1cfe00640 1 -- 172.21.15.1:0/3823142016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 msgr2=0x5584e35bcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.094+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/3823142016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e35bcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.094+0000 7fb1cfe00640 1 -- 172.21.15.1:0/3823142016 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584e32dd620 con 0x5584e359a820 2024-04-26T21:49:58.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.095+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/3823142016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e35bd2d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x5584e359cb00 tx=0x7fb1c0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35444a3d57bba837 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.096+0000 7fb1c4c00640 1 -- 172.21.15.1:0/3823142016 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c002a070 con 0x5584e359a820 2024-04-26T21:49:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.096+0000 7fb1c4c00640 1 -- 172.21.15.1:0/3823142016 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1c0019040 con 0x5584e359a820 2024-04-26T21:49:58.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.096+0000 7fb1c4c00640 1 -- 172.21.15.1:0/3823142016 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0025680 con 0x5584e359a820 2024-04-26T21:49:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.097+0000 7fb1da059740 1 -- 172.21.15.1:0/3823142016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 msgr2=0x5584e35bd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:58.097 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.097+0000 7fb1da059740 1 --2- 172.21.15.1:0/3823142016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e35bd2d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x5584e359cb00 tx=0x7fb1c0008040 comp rx=0 tx=0).stop 2024-04-26T21:49:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.100+0000 7fb1da059740 1 -- 172.21.15.1:0/3823142016 shutdown_connections 2024-04-26T21:49:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.100+0000 7fb1da059740 1 --2- 172.21.15.1:0/3823142016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e35bd2d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.100+0000 7fb1da059740 1 --2- 172.21.15.1:0/3823142016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e35a95d0 0x5584e35a6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.100+0000 7fb1da059740 1 --2- 172.21.15.1:0/3823142016 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e35bcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.100+0000 7fb1da059740 1 -- 172.21.15.1:0/3823142016 >> 172.21.15.1:0/3823142016 conn(0x5584e35964f0 msgr2=0x5584e359c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.100+0000 7fb1da059740 1 -- 172.21.15.1:0/3823142016 shutdown_connections 2024-04-26T21:49:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.100+0000 7fb1da059740 1 -- 172.21.15.1:0/3823142016 wait complete. 2024-04-26T21:49:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.101+0000 7fb1da059740 1 -- start start 2024-04-26T21:49:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.101+0000 7fb1da059740 1 Processor -- start 2024-04-26T21:49:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.102+0000 7fb1da059740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e359f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.102+0000 7fb1da059740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e359f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.102+0000 7fb1da059740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e35a95d0 0x5584e35b1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.102+0000 7fb1da059740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac00e860 con 0x5584e3599f00 2024-04-26T21:49:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.102+0000 7fb1da059740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac00fcd0 con 0x5584e35a95d0 2024-04-26T21:49:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.102+0000 7fb1da059740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ac00ef30 con 0x5584e359a820 2024-04-26T21:49:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.102+0000 7fb1da059740 10 client.0 _ll_register_callbacks cb 0x7fb1d006b1b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:49:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.103+0000 7fb1c7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e359f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.103+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e359f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.103+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e359f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:34626/0 (socket says 172.21.15.1:34626) 2024-04-26T21:49:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.103+0000 7fb1cfe00640 1 -- 172.21.15.1:0/1615350731 learned_addr learned my addr 172.21.15.1:0/1615350731 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:49:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.104+0000 7fb1cfe00640 1 -- 172.21.15.1:0/1615350731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e35a95d0 msgr2=0x5584e35b1000 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:49:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.104+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e35a95d0 0x5584e35b1000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.104+0000 7fb1cfe00640 1 -- 172.21.15.1:0/1615350731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 msgr2=0x5584e359f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:49:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.104+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e359f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.104+0000 7fb1cfe00640 1 -- 172.21.15.1:0/1615350731 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584e34b9cb0 con 0x5584e359a820 2024-04-26T21:49:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.105+0000 7fb1c7e00640 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e359f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:49:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.105+0000 7fb1cfe00640 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e359f9e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000bd20 tx=0x7fb1c0014640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23b4044ff03e9566 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:49:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.105+0000 7fb1be000640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:49:58.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.106+0000 7fb1be000640 1 -- 172.21.15.1:0/1615350731 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0033060 con 0x5584e359a820 2024-04-26T21:49:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.106+0000 7fb1be000640 1 -- 172.21.15.1:0/1615350731 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(126..126 src has 1..126) ==== 9626+0+0 (secure 0 0 0) 0x7fb1c000dd60 con 0x5584e359a820 2024-04-26T21:49:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.106+0000 7fb1da059740 1 -- 172.21.15.1:0/1615350731 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1c0019040 con 0x5584e359a820 2024-04-26T21:49:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.107+0000 7fb1be000640 1 -- 172.21.15.1:0/1615350731 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7fb1c003a050 con 0x5584e359a820 2024-04-26T21:49:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.107+0000 7fb1da059740 10 client.27001 fetch_fsmap learned FSMap version 57 2024-04-26T21:49:58.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.107+0000 7fb1da059740 1 -- 172.21.15.1:0/1615350731 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x5584e362bf10 con 0x5584e359a820 2024-04-26T21:49:58.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.108+0000 7fb1be000640 1 -- 172.21.15.1:0/1615350731 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1c0019040 con 0x5584e359a820 2024-04-26T21:49:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.108+0000 7fb1be000640 1 -- 172.21.15.1:0/1615350731 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7fb1c0013070 con 0x5584e359a820 2024-04-26T21:49:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.108+0000 7fb1da059740 10 client.27001 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:49:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.108+0000 7fb1da059740 10 client.27001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:49:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.108+0000 7fb1da059740 20 client.27001 populate_metadata read hostname 'smithi001' 2024-04-26T21:49:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.108+0000 7fb1da059740 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5584e362c350 0x5584e362e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:49:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.108+0000 7fb1da059740 4 client.27001 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:49:58.109 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.108+0000 7fb1da059740 1 -- 172.21.15.1:0/1615350731 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5584e3506e00 con 0x5584e362c350 2024-04-26T21:49:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.109+0000 7fb1c7e00640 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5584e362c350 0x5584e362e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:49:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.109+0000 7fb1c7e00640 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5584e362c350 0x5584e362e790 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d303d320f428537a server_cookie=3a614db62ae59970 in_seq=0 out_seq=0 2024-04-26T21:49:58.110 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.111+0000 7fb1be000640 10 client.27001 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:58.177 INFO:tasks.workunit.client.0.smithi001.stdout: 10 150408 21.87 MB/sec execute 235 sec latency 245.472 ms 2024-04-26T21:49:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:49:58 smithi186 ceph-mon[32991]: pgmap v622: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 9.8 MiB/s wr, 484 op/s 2024-04-26T21:49:58.777 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.777+0000 7fb1be000640 1 -- 172.21.15.1:0/1615350731 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5584e3506e00 con 0x5584e362c350 2024-04-26T21:49:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.777+0000 7fb1be000640 10 client.27001 handle_command_reply: tid=0 2024-04-26T21:49:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.777+0000 7fb1da059740 1 client.27001 shutdown 2024-04-26T21:49:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.777+0000 7fb1da059740 1 -- 172.21.15.1:0/1615350731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 msgr2=0x5584e359f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.777+0000 7fb1da059740 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e359f9e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000bd20 tx=0x7fb1c0014640 comp rx=0 tx=0).stop 2024-04-26T21:49:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.777+0000 7fb1da059740 1 -- 172.21.15.1:0/1615350731 shutdown_connections 2024-04-26T21:49:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.777+0000 7fb1da059740 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5584e362c350 0x5584e362e790 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.778+0000 7fb1be000640 0 client.27001 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:49:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.778+0000 7fb1da059740 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5584e359a820 0x5584e359f9e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.778+0000 7fb1da059740 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5584e35a95d0 0x5584e35b1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.778+0000 7fb1da059740 1 --2- 172.21.15.1:0/1615350731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5584e3599f00 0x5584e359f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.778+0000 7fb1da059740 1 -- 172.21.15.1:0/1615350731 >> 172.21.15.1:0/1615350731 conn(0x5584e35964f0 msgr2=0x5584e3594dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.778+0000 7fb1da059740 1 -- 172.21.15.1:0/1615350731 shutdown_connections 2024-04-26T21:49:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.778+0000 7fb1da059740 1 -- 172.21.15.1:0/1615350731 wait complete. 2024-04-26T21:49:58.779 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.779+0000 7fb1da059740 20 client.27001 trim_cache size 0 max 16384 2024-04-26T21:49:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:49:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:49:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:49:58.780 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:49:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 -- 172.21.15.1:0/3110069546 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb1b40a3e50 msgr2=0x7fb1b40a6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:58.781 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb1b40a3e50 0x7fb1b40a6310 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80be3f0 tx=0x7fb1c803c040 comp rx=0 tx=0).stop 2024-04-26T21:49:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 -- 172.21.15.1:0/3110069546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1d010cd50 msgr2=0x7fb1d006d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:49:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1d010cd50 0x7fb1d006d700 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8009dc0 tx=0x7fb1c8003f10 comp rx=0 tx=0).stop 2024-04-26T21:49:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 -- 172.21.15.1:0/3110069546 shutdown_connections 2024-04-26T21:49:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb1b40a3e50 0x7fb1b40a6310 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1d010cd50 0x7fb1d006d700 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1d0104250 0x7fb1d006d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 --2- 172.21.15.1:0/3110069546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1d0103930 0x7fb1d0074140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:49:58.782 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 -- 172.21.15.1:0/3110069546 >> 172.21.15.1:0/3110069546 conn(0x7fb1d00fd130 msgr2=0x7fb1d00fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:49:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 -- 172.21.15.1:0/3110069546 shutdown_connections 2024-04-26T21:49:58.783 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:49:58.781+0000 7fb1d7200640 1 -- 172.21.15.1:0/3110069546 wait complete. 2024-04-26T21:49:58.819 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:49:58 smithi001 ceph-mon[28054]: pgmap v622: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 9.8 MiB/s wr, 484 op/s 2024-04-26T21:49:58.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:49:58 smithi038 ceph-mon[32724]: pgmap v622: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 9.8 MiB/s wr, 484 op/s 2024-04-26T21:49:59.177 INFO:tasks.workunit.client.0.smithi001.stdout: 10 150665 21.83 MB/sec execute 236 sec latency 406.706 ms 2024-04-26T21:50:00.178 INFO:tasks.workunit.client.0.smithi001.stdout: 10 151190 21.88 MB/sec execute 237 sec latency 109.135 ms 2024-04-26T21:50:00.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:00 smithi186 ceph-mon[32991]: pgmap v623: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.9 MiB/s wr, 392 op/s 2024-04-26T21:50:00.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:00 smithi186 ceph-mon[32991]: overall HEALTH_OK 2024-04-26T21:50:00.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:00 smithi001 ceph-mon[28054]: pgmap v623: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.9 MiB/s wr, 392 op/s 2024-04-26T21:50:00.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:00 smithi001 ceph-mon[28054]: overall HEALTH_OK 2024-04-26T21:50:00.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:00 smithi038 ceph-mon[32724]: pgmap v623: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.9 MiB/s wr, 392 op/s 2024-04-26T21:50:00.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:00 smithi038 ceph-mon[32724]: overall HEALTH_OK 2024-04-26T21:50:01.178 INFO:tasks.workunit.client.0.smithi001.stdout: 10 151575 21.87 MB/sec execute 238 sec latency 186.478 ms 2024-04-26T21:50:02.178 INFO:tasks.workunit.client.0.smithi001.stdout: 10 151772 21.81 MB/sec execute 239 sec latency 408.371 ms 2024-04-26T21:50:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:02 smithi186 ceph-mon[32991]: pgmap v624: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 15 MiB/s wr, 684 op/s 2024-04-26T21:50:02.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:02 smithi001 ceph-mon[28054]: pgmap v624: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 15 MiB/s wr, 684 op/s 2024-04-26T21:50:02.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:02 smithi038 ceph-mon[32724]: pgmap v624: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 15 MiB/s wr, 684 op/s 2024-04-26T21:50:03.178 INFO:tasks.workunit.client.0.smithi001.stdout: 10 151949 21.76 MB/sec execute 240 sec latency 442.789 ms 2024-04-26T21:50:04.178 INFO:tasks.workunit.client.0.smithi001.stdout: 10 152026 21.68 MB/sec execute 241 sec latency 703.385 ms 2024-04-26T21:50:04.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:04 smithi186 ceph-mon[32991]: pgmap v625: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 11 MiB/s wr, 520 op/s 2024-04-26T21:50:04.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:04 smithi001 ceph-mon[28054]: pgmap v625: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 11 MiB/s wr, 520 op/s 2024-04-26T21:50:04.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:04 smithi038 ceph-mon[32724]: pgmap v625: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 11 MiB/s wr, 520 op/s 2024-04-26T21:50:05.178 INFO:tasks.workunit.client.0.smithi001.stdout: 10 152053 21.60 MB/sec execute 242 sec latency 655.672 ms 2024-04-26T21:50:06.178 INFO:tasks.workunit.client.0.smithi001.stdout: 10 152097 21.53 MB/sec execute 243 sec latency 410.850 ms 2024-04-26T21:50:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:06 smithi186 ceph-mon[32991]: pgmap v626: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 12 MiB/s wr, 638 op/s 2024-04-26T21:50:06.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:06 smithi001 ceph-mon[28054]: pgmap v626: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 12 MiB/s wr, 638 op/s 2024-04-26T21:50:06.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:06 smithi038 ceph-mon[32724]: pgmap v626: 129 pgs: 129 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 12 MiB/s wr, 638 op/s 2024-04-26T21:50:07.178 INFO:tasks.workunit.client.0.smithi001.stdout: 10 152358 21.53 MB/sec execute 244 sec latency 398.411 ms 2024-04-26T21:50:08.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 152527 21.49 MB/sec execute 245 sec latency 685.884 ms 2024-04-26T21:50:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:08 smithi186 ceph-mon[32991]: pgmap v627: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.4 MiB/s wr, 541 op/s 2024-04-26T21:50:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:08 smithi186 ceph-mon[32991]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T21:50:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:08 smithi001 ceph-mon[28054]: pgmap v627: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.4 MiB/s wr, 541 op/s 2024-04-26T21:50:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:08 smithi001 ceph-mon[28054]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T21:50:08.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:08 smithi038 ceph-mon[32724]: pgmap v627: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.4 MiB/s wr, 541 op/s 2024-04-26T21:50:08.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:08 smithi038 ceph-mon[32724]: osdmap e127: 12 total, 12 up, 12 in 2024-04-26T21:50:09.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 152615 21.42 MB/sec execute 246 sec latency 1094.242 ms 2024-04-26T21:50:09.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:09 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:50:09.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:09 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:50:09.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:09 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:50:10.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 152728 21.36 MB/sec execute 247 sec latency 1246.051 ms 2024-04-26T21:50:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:10 smithi186 ceph-mon[32991]: pgmap v629: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 9.0 MiB/s wr, 589 op/s 2024-04-26T21:50:10.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:10 smithi001 ceph-mon[28054]: pgmap v629: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 9.0 MiB/s wr, 589 op/s 2024-04-26T21:50:10.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:10 smithi038 ceph-mon[32724]: pgmap v629: 129 pgs: 129 active+clean; 1.3 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 9.0 MiB/s wr, 589 op/s 2024-04-26T21:50:11.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 152873 21.30 MB/sec execute 248 sec latency 973.172 ms 2024-04-26T21:50:11.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:11 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:50:11.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:11 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:50:11.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:11 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:50:11.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:11 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:50:11.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:11 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:50:11.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:11 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:50:11.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:11 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:50:11.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:11 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:50:11.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:11 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:50:12.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.016+0000 7faae2a00640 1 -- 172.21.15.1:0/2208700876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faadc10f1c0 msgr2=0x7faadc1115d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:12.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.016+0000 7faae2a00640 1 --2- 172.21.15.1:0/2208700876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faadc10f1c0 0x7faadc1115d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faad4009ca0 tx=0x7faad402a950 comp rx=0 tx=0).stop 2024-04-26T21:50:12.017 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.017+0000 7faae2a00640 1 -- 172.21.15.1:0/2208700876 shutdown_connections 2024-04-26T21:50:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.017+0000 7faae2a00640 1 --2- 172.21.15.1:0/2208700876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faadc10f1c0 0x7faadc1115d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.017+0000 7faae2a00640 1 --2- 172.21.15.1:0/2208700876 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faadc075620 0x7faadc10ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.017+0000 7faae2a00640 1 --2- 172.21.15.1:0/2208700876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faadc074d00 0x7faadc0750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.017+0000 7faae2a00640 1 -- 172.21.15.1:0/2208700876 >> 172.21.15.1:0/2208700876 conn(0x7faadc0fe410 msgr2=0x7faadc100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.017+0000 7faae2a00640 1 -- 172.21.15.1:0/2208700876 shutdown_connections 2024-04-26T21:50:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.017+0000 7faae2a00640 1 -- 172.21.15.1:0/2208700876 wait complete. 2024-04-26T21:50:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.018+0000 7faae2a00640 1 Processor -- start 2024-04-26T21:50:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.020+0000 7faae2a00640 1 -- start start 2024-04-26T21:50:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.020+0000 7faae2a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faadc074d00 0x7faadc10eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.020+0000 7faae2a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faadc075620 0x7faadc10bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.020+0000 7faae2a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faadc10f1c0 0x7faadc10c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.020+0000 7faae2a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faadc114e20 con 0x7faadc074d00 2024-04-26T21:50:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.020+0000 7faae2a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faadc114ca0 con 0x7faadc10f1c0 2024-04-26T21:50:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.020+0000 7faae2a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faadc114fa0 con 0x7faadc075620 2024-04-26T21:50:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.021+0000 7faadb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faadc075620 0x7faadc10bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.021+0000 7faadb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faadc075620 0x7faadc10bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51156/0 (socket says 172.21.15.1:51156) 2024-04-26T21:50:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.021+0000 7faadb400640 1 -- 172.21.15.1:0/1092640107 learned_addr learned my addr 172.21.15.1:0/1092640107 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.021+0000 7faae0a00640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faadc10f1c0 0x7faadc10c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.021+0000 7faadb400640 1 -- 172.21.15.1:0/1092640107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faadc10f1c0 msgr2=0x7faadc10c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.021+0000 7faadb400640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faadc10f1c0 0x7faadc10c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.021+0000 7faadb400640 1 -- 172.21.15.1:0/1092640107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faadc074d00 msgr2=0x7faadc10eb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:50:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.022+0000 7faadb400640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faadc074d00 0x7faadc10eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.022+0000 7faadb400640 1 -- 172.21.15.1:0/1092640107 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faadc10ca10 con 0x7faadc075620 2024-04-26T21:50:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.022+0000 7faadb400640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faadc075620 0x7faadc10bc10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faacc00d950 tx=0x7faacc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:12.025 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.023+0000 7faad8c00640 1 -- 172.21.15.1:0/1092640107 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc017070 con 0x7faadc075620 2024-04-26T21:50:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.024+0000 7faad8c00640 1 -- 172.21.15.1:0/1092640107 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faacc009c30 con 0x7faadc075620 2024-04-26T21:50:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.024+0000 7faad8c00640 1 -- 172.21.15.1:0/1092640107 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc00f620 con 0x7faadc075620 2024-04-26T21:50:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.025+0000 7faae2a00640 1 -- 172.21.15.1:0/1092640107 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faadc1abc40 con 0x7faadc075620 2024-04-26T21:50:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.025+0000 7faae2a00640 1 -- 172.21.15.1:0/1092640107 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faadc1ac110 con 0x7faadc075620 2024-04-26T21:50:12.034 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.030+0000 7faae5833740 1 Processor -- start 2024-04-26T21:50:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.030+0000 7faae5833740 1 -- start start 2024-04-26T21:50:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.030+0000 7faae5833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae6a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.031+0000 7faae5833740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 0x55c60ae74a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.031+0000 7faae5833740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae74fb0 0x55c60ae773a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.031+0000 7faae5833740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c60adcf380 con 0x55c60ae6a6f0 2024-04-26T21:50:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.031+0000 7faae5833740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c60ad6bc30 con 0x55c60ae74fb0 2024-04-26T21:50:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.031+0000 7faae5833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c60adcb340 con 0x55c60ae69dd0 2024-04-26T21:50:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.031+0000 7faae0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae6a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.031+0000 7faae0a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae6a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51170/0 (socket says 172.21.15.1:51170) 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.031+0000 7faae0a00640 1 -- 172.21.15.1:0/1648662755 learned_addr learned my addr 172.21.15.1:0/1648662755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.031+0000 7faadbe00640 1 --2- 172.21.15.1:0/1648662755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 0x55c60ae74a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.032+0000 7faad8c00640 1 -- 172.21.15.1:0/1092640107 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7faacc010460 con 0x7faadc075620 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.032+0000 7faae0a00640 1 -- 172.21.15.1:0/1648662755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae74fb0 msgr2=0x55c60ae773a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.032+0000 7faae0a00640 1 --2- 172.21.15.1:0/1648662755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae74fb0 0x55c60ae773a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.032+0000 7faae0a00640 1 -- 172.21.15.1:0/1648662755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 msgr2=0x55c60ae74a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.032+0000 7faae0a00640 1 --2- 172.21.15.1:0/1648662755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 0x55c60ae74a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.032+0000 7faae0a00640 1 -- 172.21.15.1:0/1648662755 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c60ad8d940 con 0x55c60ae69dd0 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.033+0000 7faae0a00640 1 --2- 172.21.15.1:0/1648662755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae6a1b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faad4009ca0 tx=0x7faad402f5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abd95a20e5f5ae01 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.034+0000 7faad8c00640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faac00a3e60 0x7faac00a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.034+0000 7faad8c00640 1 -- 172.21.15.1:0/1092640107 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 9730+0+0 (secure 0 0 0) 0x7faacc0d05a0 con 0x7faadc075620 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.034+0000 7faad1200640 1 -- 172.21.15.1:0/1648662755 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad4034040 con 0x55c60ae69dd0 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.034+0000 7faad1200640 1 -- 172.21.15.1:0/1648662755 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faad402fd20 con 0x55c60ae69dd0 2024-04-26T21:50:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.034+0000 7faad1200640 1 -- 172.21.15.1:0/1648662755 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad4004570 con 0x55c60ae69dd0 2024-04-26T21:50:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.036+0000 7faae5833740 1 -- 172.21.15.1:0/1648662755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 msgr2=0x55c60ae6a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.036+0000 7faae5833740 1 --2- 172.21.15.1:0/1648662755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae6a1b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7faad4009ca0 tx=0x7faad402f5e0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.036+0000 7faadbe00640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faac00a3e60 0x7faac00a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.037+0000 7faadbe00640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faac00a3e60 0x7faac00a6320 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55c60ae6b370 tx=0x7faac400a2d0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:12.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.039+0000 7faae5833740 1 -- 172.21.15.1:0/1648662755 shutdown_connections 2024-04-26T21:50:12.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.039+0000 7faae5833740 1 --2- 172.21.15.1:0/1648662755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae74fb0 0x55c60ae773a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.039+0000 7faae5833740 1 --2- 172.21.15.1:0/1648662755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 0x55c60ae74a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.039+0000 7faae5833740 1 --2- 172.21.15.1:0/1648662755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae6a1b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.039+0000 7faae5833740 1 -- 172.21.15.1:0/1648662755 >> 172.21.15.1:0/1648662755 conn(0x55c60ae662e0 msgr2=0x55c60ae666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:12.039 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.039+0000 7faae5833740 1 -- 172.21.15.1:0/1648662755 shutdown_connections 2024-04-26T21:50:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.039+0000 7faae5833740 1 -- 172.21.15.1:0/1648662755 wait complete. 2024-04-26T21:50:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.040+0000 7faae5833740 1 -- start start 2024-04-26T21:50:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.041+0000 7faae5833740 1 Processor -- start 2024-04-26T21:50:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.043+0000 7faae5833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae71f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.043+0000 7faae5833740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae6a6f0 0x55c60ae724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.043+0000 7faae5833740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae74fb0 0x55c60ae729f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.043+0000 7faae5833740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55c60ae77b80 con 0x55c60ae74fb0 2024-04-26T21:50:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.044+0000 7faae5833740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55c60ad56c60 con 0x55c60ae6a6f0 2024-04-26T21:50:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.044+0000 7faae5833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c60ae77d00 con 0x55c60ae69dd0 2024-04-26T21:50:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.044+0000 7faae5833740 10 client.0 _ll_register_callbacks cb 0x55c60ad82e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:50:12.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.044+0000 7faadb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae6a6f0 0x55c60ae724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.044+0000 7faadbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae71f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.044+0000 7faadb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae6a6f0 0x55c60ae724b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54840/0 (socket says 172.21.15.1:54840) 2024-04-26T21:50:12.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.044+0000 7faadb400640 1 -- 172.21.15.1:0/1452376095 learned_addr learned my addr 172.21.15.1:0/1452376095 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:12.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.044+0000 7faadb400640 1 -- 172.21.15.1:0/1452376095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 msgr2=0x55c60ae71f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:12.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.044+0000 7faadb400640 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae71f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.045 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.044+0000 7faadb400640 1 -- 172.21.15.1:0/1452376095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae74fb0 msgr2=0x55c60ae729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:12.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.045+0000 7faadb400640 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae74fb0 0x55c60ae729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.045+0000 7faadb400640 1 -- 172.21.15.1:0/1452376095 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c60ad65510 con 0x55c60ae6a6f0 2024-04-26T21:50:12.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.045+0000 7faadb400640 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae6a6f0 0x55c60ae724b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7faacc09be00 tx=0x7faacc09bb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8afa9dcaeb32d7c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:12.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.045+0000 7faac9600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:50:12.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.047+0000 7faac9600640 1 -- 172.21.15.1:0/1452376095 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc0a04b0 con 0x55c60ae6a6f0 2024-04-26T21:50:12.047 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.047+0000 7faac9600640 1 -- 172.21.15.1:0/1452376095 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9730+0+0 (secure 0 0 0) 0x7faacc0cdab0 con 0x55c60ae6a6f0 2024-04-26T21:50:12.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.047+0000 7faae5833740 1 -- 172.21.15.1:0/1452376095 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faad402fd20 con 0x55c60ae6a6f0 2024-04-26T21:50:12.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.047+0000 7faac9600640 1 -- 172.21.15.1:0/1452376095 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7faacc0a4d70 con 0x55c60ae6a6f0 2024-04-26T21:50:12.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.047+0000 7faae5833740 10 client.26999 fetch_fsmap learned FSMap version 57 2024-04-26T21:50:12.048 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.047+0000 7faae5833740 1 -- 172.21.15.1:0/1452376095 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55c60aae1210 con 0x55c60ae6a6f0 2024-04-26T21:50:12.049 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.048+0000 7faac9600640 1 -- 172.21.15.1:0/1452376095 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faad402fd20 con 0x55c60ae6a6f0 2024-04-26T21:50:12.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.049+0000 7faac9600640 1 -- 172.21.15.1:0/1452376095 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7faacc0b0ba0 con 0x55c60ae6a6f0 2024-04-26T21:50:12.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.049+0000 7faae5833740 10 client.26999 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:50:12.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.049+0000 7faae5833740 10 client.26999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:50:12.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.049+0000 7faae5833740 20 client.26999 populate_metadata read hostname 'smithi001' 2024-04-26T21:50:12.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.049+0000 7faae5833740 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c60ae84ac0 0x55c60ae86eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.049+0000 7faae5833740 4 client.26999 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:50:12.050 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.049+0000 7faae5833740 1 -- 172.21.15.1:0/1452376095 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c60addcf90 con 0x55c60ae84ac0 2024-04-26T21:50:12.051 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.050+0000 7faadbe00640 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c60ae84ac0 0x55c60ae86eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.051 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.050+0000 7faadbe00640 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c60ae84ac0 0x55c60ae86eb0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b6047bf4e8d3098 server_cookie=313181e98dcabfc1 in_seq=0 out_seq=0 2024-04-26T21:50:12.051 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.051+0000 7faac9600640 10 client.26999 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:12.052 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.051+0000 7faac9600640 1 -- 172.21.15.1:0/1452376095 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c60addcf90 con 0x55c60ae84ac0 2024-04-26T21:50:12.052 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.052+0000 7faac9600640 10 client.26999 handle_command_reply: tid=0 2024-04-26T21:50:12.052 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.052+0000 7faae5833740 1 client.26999 shutdown 2024-04-26T21:50:12.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.053+0000 7faae5833740 1 -- 172.21.15.1:0/1452376095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae6a6f0 msgr2=0x55c60ae724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:12.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.053+0000 7faae5833740 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae6a6f0 0x55c60ae724b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7faacc09be00 tx=0x7faacc09bb60 comp rx=0 tx=0).stop 2024-04-26T21:50:12.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.053+0000 7faae5833740 1 -- 172.21.15.1:0/1452376095 shutdown_connections 2024-04-26T21:50:12.053 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.053+0000 7faae5833740 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c60ae84ac0 0x55c60ae86eb0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.053+0000 7faae5833740 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae74fb0 0x55c60ae729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.053+0000 7faae5833740 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae6a6f0 0x55c60ae724b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.053+0000 7faae5833740 1 --2- 172.21.15.1:0/1452376095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae71f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.053+0000 7faae5833740 1 -- 172.21.15.1:0/1452376095 >> 172.21.15.1:0/1452376095 conn(0x55c60ae662e0 msgr2=0x55c60ae6c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:12.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.053+0000 7faac9600640 0 client.26999 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:12.054 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.054+0000 7faae5833740 1 -- 172.21.15.1:0/1452376095 shutdown_connections 2024-04-26T21:50:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.054+0000 7faae5833740 1 -- 172.21.15.1:0/1452376095 wait complete. 2024-04-26T21:50:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.054+0000 7faae5833740 20 client.26999 trim_cache size 0 max 16384 2024-04-26T21:50:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.091+0000 7faae5833740 1 Processor -- start 2024-04-26T21:50:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.091+0000 7faae5833740 1 -- start start 2024-04-26T21:50:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.092+0000 7faae5833740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae69dd0 0x55c60ae746a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.092+0000 7faae5833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae6a6f0 0x55c60ae80a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.092+0000 7faae5833740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae74fb0 0x55c60ae61430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.092+0000 7faae5833740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1d25c0 con 0x55c60ae74fb0 2024-04-26T21:50:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.092+0000 7faae5833740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1cfe90 con 0x55c60ae69dd0 2024-04-26T21:50:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.092+0000 7faae5833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1a24f0 con 0x55c60ae6a6f0 2024-04-26T21:50:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.092+0000 7faadbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae69dd0 0x55c60ae746a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.092+0000 7faadbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae69dd0 0x55c60ae746a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:54852/0 (socket says 172.21.15.1:54852) 2024-04-26T21:50:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.092+0000 7faadbe00640 1 -- 172.21.15.1:0/768547720 learned_addr learned my addr 172.21.15.1:0/768547720 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.092+0000 7faadb400640 1 --2- 172.21.15.1:0/768547720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae6a6f0 0x55c60ae80a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.093+0000 7faadbe00640 1 -- 172.21.15.1:0/768547720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae6a6f0 msgr2=0x55c60ae80a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.093+0000 7faadbe00640 1 --2- 172.21.15.1:0/768547720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae6a6f0 0x55c60ae80a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.093+0000 7faadbe00640 1 -- 172.21.15.1:0/768547720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae74fb0 msgr2=0x55c60ae61430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.093+0000 7faadbe00640 1 --2- 172.21.15.1:0/768547720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae74fb0 0x55c60ae61430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.093+0000 7faadbe00640 1 -- 172.21.15.1:0/768547720 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c60ae70f80 con 0x55c60ae69dd0 2024-04-26T21:50:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.094+0000 7faadbe00640 1 --2- 172.21.15.1:0/768547720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae69dd0 0x55c60ae746a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x55c60ae87740 tx=0x7faac4012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbe7b4daa289ca5d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.094+0000 7faacbe00640 1 -- 172.21.15.1:0/768547720 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4016070 con 0x55c60ae69dd0 2024-04-26T21:50:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.094+0000 7faacbe00640 1 -- 172.21.15.1:0/768547720 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faac401dd40 con 0x55c60ae69dd0 2024-04-26T21:50:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.094+0000 7faacbe00640 1 -- 172.21.15.1:0/768547720 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4010b10 con 0x55c60ae69dd0 2024-04-26T21:50:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.095+0000 7faae5833740 1 -- 172.21.15.1:0/768547720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae69dd0 msgr2=0x55c60ae746a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.095+0000 7faae5833740 1 --2- 172.21.15.1:0/768547720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae69dd0 0x55c60ae746a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x55c60ae87740 tx=0x7faac4012000 comp rx=0 tx=0).stop 2024-04-26T21:50:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.098+0000 7faae5833740 1 -- 172.21.15.1:0/768547720 shutdown_connections 2024-04-26T21:50:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.098+0000 7faae5833740 1 --2- 172.21.15.1:0/768547720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae6a6f0 0x55c60ae80a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.098+0000 7faae5833740 1 --2- 172.21.15.1:0/768547720 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae74fb0 0x55c60ae61430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.098+0000 7faae5833740 1 --2- 172.21.15.1:0/768547720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae69dd0 0x55c60ae746a0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.098+0000 7faae5833740 1 -- 172.21.15.1:0/768547720 >> 172.21.15.1:0/768547720 conn(0x55c60ae662e0 msgr2=0x55c60ae65440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.100+0000 7faae5833740 1 -- 172.21.15.1:0/768547720 shutdown_connections 2024-04-26T21:50:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.100+0000 7faae5833740 1 -- 172.21.15.1:0/768547720 wait complete. 2024-04-26T21:50:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.101+0000 7faae5833740 1 -- start start 2024-04-26T21:50:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.101+0000 7faae5833740 1 Processor -- start 2024-04-26T21:50:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.102+0000 7faae5833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae7cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.102 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.102+0000 7faae5833740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 0x55c60ae7d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.103+0000 7faae5833740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae74fb0 0x55c60ae79f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.103+0000 7faadbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae7cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.103+0000 7faadbe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae7cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:51204/0 (socket says 172.21.15.1:51204) 2024-04-26T21:50:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.103+0000 7faadbe00640 1 -- 172.21.15.1:0/2178372352 learned_addr learned my addr 172.21.15.1:0/2178372352 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.103+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1ce350 con 0x55c60ae6a6f0 2024-04-26T21:50:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.103+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1cf7c0 con 0x55c60ae74fb0 2024-04-26T21:50:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.103+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faadc1cea20 con 0x55c60ae69dd0 2024-04-26T21:50:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.103+0000 7faae5833740 10 client.0 _ll_register_callbacks cb 0x7faacc00df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:50:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.104+0000 7faadb400640 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 0x55c60ae7d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.104+0000 7faadbe00640 1 -- 172.21.15.1:0/2178372352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae74fb0 msgr2=0x55c60ae79f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.104+0000 7faadbe00640 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae74fb0 0x55c60ae79f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.104+0000 7faadbe00640 1 -- 172.21.15.1:0/2178372352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 msgr2=0x55c60ae7d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.104+0000 7faadbe00640 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 0x55c60ae7d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.104+0000 7faadbe00640 1 -- 172.21.15.1:0/2178372352 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c60ae77710 con 0x55c60ae69dd0 2024-04-26T21:50:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.104+0000 7faadb400640 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 0x55c60ae7d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:50:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.106+0000 7faadbe00640 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae7cf50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faac4010010 tx=0x7faac4010660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a7285479f5076b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.106+0000 7faac9600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:50:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.106+0000 7faac9600640 1 -- 172.21.15.1:0/2178372352 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4010ca0 con 0x55c60ae69dd0 2024-04-26T21:50:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.107+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faac401dd40 con 0x55c60ae69dd0 2024-04-26T21:50:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.107+0000 7faac9600640 1 -- 172.21.15.1:0/2178372352 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(127..127 src has 1..127) ==== 9730+0+0 (secure 0 0 0) 0x7faac40252d0 con 0x55c60ae69dd0 2024-04-26T21:50:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.107+0000 7faac9600640 1 -- 172.21.15.1:0/2178372352 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) ==== 24+0+0 (secure 0 0 0) 0x7faac4041050 con 0x55c60ae69dd0 2024-04-26T21:50:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.107+0000 7faae5833740 10 client.27019 fetch_fsmap learned FSMap version 57 2024-04-26T21:50:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.107+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=4+}) -- 0x55c60abad620 con 0x55c60ae69dd0 2024-04-26T21:50:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.107+0000 7faac9600640 1 -- 172.21.15.1:0/2178372352 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faac401dd40 con 0x55c60ae69dd0 2024-04-26T21:50:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.108+0000 7faac9600640 1 -- 172.21.15.1:0/2178372352 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 57) ==== 8354+0+0 (secure 0 0 0) 0x7faac4038940 con 0x55c60ae69dd0 2024-04-26T21:50:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.108+0000 7faae5833740 10 client.27019 fetch_fsmap finished waiting for FSMap version 57 2024-04-26T21:50:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.108+0000 7faae5833740 10 client.27019 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:50:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.108+0000 7faae5833740 20 client.27019 populate_metadata read hostname 'smithi001' 2024-04-26T21:50:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.108+0000 7faae5833740 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c60aeebf90 0x55c60aeee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.108+0000 7faae5833740 4 client.27019 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:50:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.108+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c60add6e00 con 0x55c60aeebf90 2024-04-26T21:50:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.108+0000 7faadb400640 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c60aeebf90 0x55c60aeee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.109+0000 7faadb400640 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c60aeebf90 0x55c60aeee3f0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1817665f7026d078 server_cookie=6a0de66944ee0a61 in_seq=0 out_seq=0 2024-04-26T21:50:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.109+0000 7faac9600640 10 client.27019 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:12.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 152958 21.24 MB/sec execute 249 sec latency 499.089 ms 2024-04-26T21:50:12.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.578+0000 7faac9600640 1 -- 172.21.15.1:0/2178372352 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c60add6e00 con 0x55c60aeebf90 2024-04-26T21:50:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.578+0000 7faac9600640 10 client.27019 handle_command_reply: tid=0 2024-04-26T21:50:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.578+0000 7faae5833740 1 client.27019 shutdown 2024-04-26T21:50:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.579+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 msgr2=0x55c60ae7cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:12.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.579+0000 7faae5833740 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae7cf50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7faac4010010 tx=0x7faac4010660 comp rx=0 tx=0).stop 2024-04-26T21:50:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.579+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 shutdown_connections 2024-04-26T21:50:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.579+0000 7faae5833740 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55c60aeebf90 0x55c60aeee3f0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.579+0000 7faac9600640 0 client.27019 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.579+0000 7faae5833740 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c60ae74fb0 0x55c60ae79f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.579+0000 7faae5833740 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55c60ae6a6f0 0x55c60ae7d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.579+0000 7faae5833740 1 --2- 172.21.15.1:0/2178372352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c60ae69dd0 0x55c60ae7cf50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.579+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 >> 172.21.15.1:0/2178372352 conn(0x55c60ae662e0 msgr2=0x55c60ae6b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:12.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.580+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 shutdown_connections 2024-04-26T21:50:12.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.580+0000 7faae5833740 1 -- 172.21.15.1:0/2178372352 wait complete. 2024-04-26T21:50:12.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.581+0000 7faae5833740 20 client.27019 trim_cache size 0 max 16384 2024-04-26T21:50:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:50:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:50:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:50:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:50:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.582+0000 7faae2a00640 1 -- 172.21.15.1:0/1092640107 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faac00a3e60 msgr2=0x7faac00a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:12.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.582+0000 7faae2a00640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faac00a3e60 0x7faac00a6320 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55c60ae6b370 tx=0x7faac400a2d0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.582+0000 7faae2a00640 1 -- 172.21.15.1:0/1092640107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faadc075620 msgr2=0x7faadc10bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:12.583 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.582+0000 7faae2a00640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faadc075620 0x7faadc10bc10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faacc00d950 tx=0x7faacc00de10 comp rx=0 tx=0).stop 2024-04-26T21:50:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.582+0000 7faae2a00640 1 -- 172.21.15.1:0/1092640107 shutdown_connections 2024-04-26T21:50:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.582+0000 7faae2a00640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7faac00a3e60 0x7faac00a6320 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55c60ae6b370 tx=0x7faac400a2d0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.582+0000 7faae2a00640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faadc075620 0x7faadc10bc10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.582+0000 7faae2a00640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faadc10f1c0 0x7faadc10c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.582+0000 7faae2a00640 1 --2- 172.21.15.1:0/1092640107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faadc074d00 0x7faadc10eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.582+0000 7faae2a00640 1 -- 172.21.15.1:0/1092640107 >> 172.21.15.1:0/1092640107 conn(0x7faadc0fe410 msgr2=0x7faadc0ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.586+0000 7faae2a00640 1 -- 172.21.15.1:0/1092640107 shutdown_connections 2024-04-26T21:50:12.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:12.586+0000 7faae2a00640 1 -- 172.21.15.1:0/1092640107 wait complete. 2024-04-26T21:50:12.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:12 smithi186 ceph-mon[32991]: pgmap v630: 129 pgs: 19 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 61 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 389 op/s 2024-04-26T21:50:12.726 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:12 smithi001 ceph-mon[28054]: pgmap v630: 129 pgs: 19 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 61 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 389 op/s 2024-04-26T21:50:12.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:12 smithi038 ceph-mon[32724]: pgmap v630: 129 pgs: 19 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 61 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 389 op/s 2024-04-26T21:50:13.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 153166 21.20 MB/sec execute 250 sec latency 288.816 ms 2024-04-26T21:50:13.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:13 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:50:13.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:13 smithi186 ceph-mon[32991]: pgmap v631: 129 pgs: 20 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 59 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 390 op/s 2024-04-26T21:50:14.074 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:13 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:50:14.075 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:13 smithi001 ceph-mon[28054]: pgmap v631: 129 pgs: 20 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 59 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 390 op/s 2024-04-26T21:50:14.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:13 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:50:14.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:13 smithi038 ceph-mon[32724]: pgmap v631: 129 pgs: 20 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 59 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.0 MiB/s wr, 390 op/s 2024-04-26T21:50:14.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 153230 21.12 MB/sec execute 251 sec latency 567.734 ms 2024-04-26T21:50:15.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 153405 21.05 MB/sec execute 252 sec latency 317.948 ms 2024-04-26T21:50:16.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 153598 21.00 MB/sec execute 253 sec latency 313.668 ms 2024-04-26T21:50:16.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:16 smithi001 ceph-mon[28054]: pgmap v632: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.6 MiB/s wr, 424 op/s 2024-04-26T21:50:16.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:16 smithi038 ceph-mon[32724]: pgmap v632: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.6 MiB/s wr, 424 op/s 2024-04-26T21:50:16.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:16 smithi186 ceph-mon[32991]: pgmap v632: 129 pgs: 11 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 92 active+clean; 1.3 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.6 MiB/s wr, 424 op/s 2024-04-26T21:50:17.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 153930 20.99 MB/sec execute 254 sec latency 154.309 ms 2024-04-26T21:50:18.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 154403 21.03 MB/sec execute 255 sec latency 110.841 ms 2024-04-26T21:50:18.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:18 smithi001 ceph-mon[28054]: pgmap v633: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.4 MiB/s wr, 392 op/s 2024-04-26T21:50:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:18 smithi038 ceph-mon[32724]: pgmap v633: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.4 MiB/s wr, 392 op/s 2024-04-26T21:50:18.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:18 smithi186 ceph-mon[32991]: pgmap v633: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 6.4 MiB/s wr, 392 op/s 2024-04-26T21:50:19.179 INFO:tasks.workunit.client.0.smithi001.stdout: 10 154971 21.01 MB/sec execute 256 sec latency 136.370 ms 2024-04-26T21:50:20.182 INFO:tasks.workunit.client.0.smithi001.stdout: 10 155445 21.06 MB/sec execute 257 sec latency 111.778 ms 2024-04-26T21:50:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:20 smithi001 ceph-mon[28054]: pgmap v634: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.4 MiB/s wr, 332 op/s 2024-04-26T21:50:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:20 smithi001 ceph-mon[28054]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T21:50:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:20 smithi038 ceph-mon[32724]: pgmap v634: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.4 MiB/s wr, 332 op/s 2024-04-26T21:50:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:20 smithi038 ceph-mon[32724]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T21:50:20.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:20 smithi186 ceph-mon[32991]: pgmap v634: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 5.4 MiB/s wr, 332 op/s 2024-04-26T21:50:20.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:20 smithi186 ceph-mon[32991]: osdmap e128: 12 total, 12 up, 12 in 2024-04-26T21:50:21.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 155990 21.12 MB/sec execute 258 sec latency 60.725 ms 2024-04-26T21:50:22.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 156474 21.16 MB/sec execute 259 sec latency 113.625 ms 2024-04-26T21:50:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:22 smithi001 ceph-mon[28054]: pgmap v636: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.4 MiB/s wr, 401 op/s 2024-04-26T21:50:22.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:22 smithi038 ceph-mon[32724]: pgmap v636: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.4 MiB/s wr, 401 op/s 2024-04-26T21:50:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:22 smithi186 ceph-mon[32991]: pgmap v636: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.4 MiB/s wr, 401 op/s 2024-04-26T21:50:23.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 156829 21.12 MB/sec execute 260 sec latency 90.081 ms 2024-04-26T21:50:24.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 157224 21.10 MB/sec execute 261 sec latency 93.690 ms 2024-04-26T21:50:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:24 smithi001 ceph-mon[28054]: pgmap v637: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 7.7 MiB/s wr, 414 op/s 2024-04-26T21:50:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:24 smithi001 ceph-mon[28054]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T21:50:24.571 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:24 smithi186 ceph-mon[32991]: pgmap v637: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 7.7 MiB/s wr, 414 op/s 2024-04-26T21:50:24.571 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:24 smithi186 ceph-mon[32991]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T21:50:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:24 smithi038 ceph-mon[32724]: pgmap v637: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 7.7 MiB/s wr, 414 op/s 2024-04-26T21:50:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:24 smithi038 ceph-mon[32724]: osdmap e129: 12 total, 12 up, 12 in 2024-04-26T21:50:25.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 157713 21.15 MB/sec execute 262 sec latency 97.698 ms 2024-04-26T21:50:25.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.796+0000 7f698b200640 1 -- 172.21.15.1:0/1228076015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f698c104960 msgr2=0x7f698c104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:25.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.796+0000 7f698b200640 1 --2- 172.21.15.1:0/1228076015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f698c104960 0x7f698c104de0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6978009930 tx=0x7f697802a440 comp rx=0 tx=0).stop 2024-04-26T21:50:25.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.796+0000 7f698b200640 1 -- 172.21.15.1:0/1228076015 shutdown_connections 2024-04-26T21:50:25.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.796+0000 7f698b200640 1 --2- 172.21.15.1:0/1228076015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f698c104960 0x7f698c104de0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.796+0000 7f698b200640 1 --2- 172.21.15.1:0/1228076015 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f698c105320 0x7f698c111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.796+0000 7f698b200640 1 --2- 172.21.15.1:0/1228076015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f698c10a970 0x7f698c10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.796+0000 7f698b200640 1 -- 172.21.15.1:0/1228076015 >> 172.21.15.1:0/1228076015 conn(0x7f698c100430 msgr2=0x7f698c102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:25.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.796+0000 7f698b200640 1 -- 172.21.15.1:0/1228076015 shutdown_connections 2024-04-26T21:50:25.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.796+0000 7f698b200640 1 -- 172.21.15.1:0/1228076015 wait complete. 2024-04-26T21:50:25.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.797+0000 7f698b200640 1 Processor -- start 2024-04-26T21:50:25.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.798+0000 7f698b200640 1 -- start start 2024-04-26T21:50:25.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.798+0000 7f698b200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f698c104960 0x7f698c1117a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.798+0000 7f698b200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f698c105320 0x7f698c10e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.798+0000 7f698b200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f698c10a970 0x7f698c10efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.798+0000 7f698b200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f698c114c80 con 0x7f698c10a970 2024-04-26T21:50:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.798+0000 7f698b200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f698c114b00 con 0x7f698c105320 2024-04-26T21:50:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.798+0000 7f698b200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f698c114e00 con 0x7f698c104960 2024-04-26T21:50:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f6989e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f698c104960 0x7f698c1117a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f6989e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f698c104960 0x7f698c1117a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37552/0 (socket says 172.21.15.1:37552) 2024-04-26T21:50:25.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f6989e00640 1 -- 172.21.15.1:0/1417312647 learned_addr learned my addr 172.21.15.1:0/1417312647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f6989400640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f698c105320 0x7f698c10e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f698a800640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f698c10a970 0x7f698c10efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f6989e00640 1 -- 172.21.15.1:0/1417312647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f698c105320 msgr2=0x7f698c10e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f6989e00640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f698c105320 0x7f698c10e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f6989e00640 1 -- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f698c10a970 msgr2=0x7f698c10efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f6989e00640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f698c10a970 0x7f698c10efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f6989e00640 1 -- 172.21.15.1:0/1417312647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c1ab960 con 0x7f698c104960 2024-04-26T21:50:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f698a800640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f698c10a970 0x7f698c10efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:50:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.799+0000 7f6989e00640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f698c104960 0x7f698c1117a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f697c00ea60 tx=0x7f697c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:25.801 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.800+0000 7f6982a00640 1 -- 172.21.15.1:0/1417312647 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c00cde0 con 0x7f698c104960 2024-04-26T21:50:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.800+0000 7f698b200640 1 -- 172.21.15.1:0/1417312647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698c1abb30 con 0x7f698c104960 2024-04-26T21:50:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.800+0000 7f6982a00640 1 -- 172.21.15.1:0/1417312647 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f697c0035f0 con 0x7f698c104960 2024-04-26T21:50:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.800+0000 7f6982a00640 1 -- 172.21.15.1:0/1417312647 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c010690 con 0x7f698c104960 2024-04-26T21:50:25.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.800+0000 7f698b200640 1 -- 172.21.15.1:0/1417312647 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698c1ac0a0 con 0x7f698c104960 2024-04-26T21:50:25.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.805+0000 7f6982a00640 1 -- 172.21.15.1:0/1417312647 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f697c0037a0 con 0x7f698c104960 2024-04-26T21:50:25.807 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.806+0000 7f6993dc5740 1 Processor -- start 2024-04-26T21:50:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.807+0000 7f6982a00640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f69700a3d80 0x7f69700a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.808+0000 7f6982a00640 1 -- 172.21.15.1:0/1417312647 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9626+0+0 (secure 0 0 0) 0x7f697c0d0f00 con 0x7f698c104960 2024-04-26T21:50:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.808+0000 7f6993dc5740 1 -- start start 2024-04-26T21:50:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.808+0000 7f698a800640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f69700a3d80 0x7f69700a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.808+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2ee23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.809+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee2900 0x55d2c2eecc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.809+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2eed1c0 0x55d2c2eef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.809+0000 7f6993dc5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d2c2e47380 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.809+0000 7f6993dc5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d2c2de3c30 con 0x55d2c2ee2900 2024-04-26T21:50:25.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.809+0000 7f6993dc5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d2c2e43340 con 0x55d2c2eed1c0 2024-04-26T21:50:25.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.809+0000 7f698a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2eed1c0 0x55d2c2eef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.809+0000 7f6989e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2ee23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.809+0000 7f698a800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2eed1c0 0x55d2c2eef5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37566/0 (socket says 172.21.15.1:37566) 2024-04-26T21:50:25.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.809+0000 7f698a800640 1 -- 172.21.15.1:0/949340053 learned_addr learned my addr 172.21.15.1:0/949340053 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:25.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.810+0000 7f6989e00640 1 -- 172.21.15.1:0/949340053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2eed1c0 msgr2=0x55d2c2eef5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:25.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.810+0000 7f6989e00640 1 --2- 172.21.15.1:0/949340053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2eed1c0 0x55d2c2eef5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.810+0000 7f6989e00640 1 -- 172.21.15.1:0/949340053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee2900 msgr2=0x55d2c2eecc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:25.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.810+0000 7f6989e00640 1 --2- 172.21.15.1:0/949340053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee2900 0x55d2c2eecc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.810+0000 7f6989e00640 1 -- 172.21.15.1:0/949340053 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2c2e05940 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.811+0000 7f6989e00640 1 --2- 172.21.15.1:0/949340053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2ee23c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f697c0ce200 tx=0x7f697c09c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4db1c9dc71259e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.811+0000 7f698a800640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f69700a3d80 0x7f69700a6240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6984013910 tx=0x7f698400ae40 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:25.813 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.812+0000 7f6977400640 1 -- 172.21.15.1:0/949340053 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c0bd0d0 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.813+0000 7f6977400640 1 -- 172.21.15.1:0/949340053 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f697c0a0e00 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.814+0000 7f6993dc5740 1 -- 172.21.15.1:0/949340053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 msgr2=0x55d2c2ee23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.814+0000 7f6977400640 1 -- 172.21.15.1:0/949340053 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c0990c0 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.814+0000 7f6993dc5740 1 --2- 172.21.15.1:0/949340053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2ee23c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f697c0ce200 tx=0x7f697c09c3b0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.815+0000 7f6993dc5740 1 -- 172.21.15.1:0/949340053 shutdown_connections 2024-04-26T21:50:25.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.815+0000 7f6993dc5740 1 --2- 172.21.15.1:0/949340053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2eed1c0 0x55d2c2eef5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.815+0000 7f6993dc5740 1 --2- 172.21.15.1:0/949340053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee2900 0x55d2c2eecc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.815+0000 7f6993dc5740 1 --2- 172.21.15.1:0/949340053 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2ee23c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.815+0000 7f6993dc5740 1 -- 172.21.15.1:0/949340053 >> 172.21.15.1:0/949340053 conn(0x55d2c2ede4f0 msgr2=0x55d2c2ede8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.815+0000 7f6993dc5740 1 -- 172.21.15.1:0/949340053 shutdown_connections 2024-04-26T21:50:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.815+0000 7f6993dc5740 1 -- 172.21.15.1:0/949340053 wait complete. 2024-04-26T21:50:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.816+0000 7f6993dc5740 1 -- start start 2024-04-26T21:50:25.817 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.816+0000 7f6993dc5740 1 Processor -- start 2024-04-26T21:50:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.817+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.817+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee2900 0x55d2c2eea610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.817+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 0x55d2c2eeab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.817+0000 7f6993dc5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55d2c2f00da0 con 0x55d2c2eed1c0 2024-04-26T21:50:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.817+0000 7f6993dc5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55d2c2dcec60 con 0x55d2c2ee2900 2024-04-26T21:50:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.817+0000 7f6993dc5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d2c2f00f20 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.817+0000 7f6989e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.817+0000 7f6989e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37580/0 (socket says 172.21.15.1:37580) 2024-04-26T21:50:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.817+0000 7f6989e00640 1 -- 172.21.15.1:0/3858696828 learned_addr learned my addr 172.21.15.1:0/3858696828 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:25.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.817+0000 7f6993dc5740 10 client.0 _ll_register_callbacks cb 0x55d2c2dfae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.818+0000 7f6989400640 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee2900 0x55d2c2eea610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.818+0000 7f6989e00640 1 -- 172.21.15.1:0/3858696828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee2900 msgr2=0x55d2c2eea610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.818+0000 7f6989e00640 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee2900 0x55d2c2eea610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.818+0000 7f698a800640 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 0x55d2c2eeab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.818+0000 7f6989e00640 1 -- 172.21.15.1:0/3858696828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 msgr2=0x55d2c2eeab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.818+0000 7f6989e00640 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 0x55d2c2eeab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.818+0000 7f6989e00640 1 -- 172.21.15.1:0/3858696828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2c2ddd510 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.819+0000 7f6989400640 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee2900 0x55d2c2eea610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.819+0000 7f6989e00640 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f697c09ad00 tx=0x7f697c004280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44a874400c2fa786 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.819+0000 7f696be00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.819+0000 7f696be00640 1 -- 172.21.15.1:0/3858696828 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c0a56c0 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.819+0000 7f696be00640 1 -- 172.21.15.1:0/3858696828 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9626+0+0 (secure 0 0 0) 0x7f697c09ad50 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.819+0000 7f6993dc5740 1 -- 172.21.15.1:0/3858696828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f697c0a0e00 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f696be00640 1 -- 172.21.15.1:0/3858696828 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f697c083050 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f6993dc5740 10 client.27031 fetch_fsmap learned FSMap version 58 2024-04-26T21:50:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f6993dc5740 1 -- 172.21.15.1:0/3858696828 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55d2c2b59210 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f696be00640 1 -- 172.21.15.1:0/3858696828 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f697c0a0e00 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f696be00640 1 -- 172.21.15.1:0/3858696828 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 58) ==== 8350+0+0 (secure 0 0 0) 0x7f697c0870c0 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f6993dc5740 10 client.27031 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T21:50:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f6993dc5740 10 client.27031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:50:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f6993dc5740 20 client.27031 populate_metadata read hostname 'smithi001' 2024-04-26T21:50:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f6993dc5740 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d2c2f0db60 0x55d2c2f0ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f6993dc5740 4 client.27031 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:50:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.820+0000 7f6993dc5740 1 -- 172.21.15.1:0/3858696828 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2c2e54f90 con 0x55d2c2f0db60 2024-04-26T21:50:25.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.821+0000 7f6989400640 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d2c2f0db60 0x55d2c2f0ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.822+0000 7f6989400640 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d2c2f0db60 0x55d2c2f0ff50 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9580bf7d9936f7ac server_cookie=9393a701b726af80 in_seq=0 out_seq=0 2024-04-26T21:50:25.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.822+0000 7f696be00640 10 client.27031 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:25.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.824+0000 7f696be00640 1 -- 172.21.15.1:0/3858696828 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d2c2e54f90 con 0x55d2c2f0db60 2024-04-26T21:50:25.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.824+0000 7f696be00640 10 client.27031 handle_command_reply: tid=0 2024-04-26T21:50:25.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.824+0000 7f6993dc5740 1 client.27031 shutdown 2024-04-26T21:50:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.826+0000 7f6993dc5740 1 -- 172.21.15.1:0/3858696828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee1fe0 msgr2=0x55d2c2eea0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.826+0000 7f6993dc5740 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f697c09ad00 tx=0x7f697c004280 comp rx=0 tx=0).stop 2024-04-26T21:50:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.826+0000 7f6993dc5740 1 -- 172.21.15.1:0/3858696828 shutdown_connections 2024-04-26T21:50:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.826+0000 7f6993dc5740 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d2c2f0db60 0x55d2c2f0ff50 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.826+0000 7f696be00640 0 client.27031 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.826+0000 7f6993dc5740 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 0x55d2c2eeab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.828 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.826+0000 7f6993dc5740 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee2900 0x55d2c2eea610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.826+0000 7f6993dc5740 1 --2- 172.21.15.1:0/3858696828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea0d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.826+0000 7f6993dc5740 1 -- 172.21.15.1:0/3858696828 >> 172.21.15.1:0/3858696828 conn(0x55d2c2ede4f0 msgr2=0x55d2c2ee4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.827+0000 7f6993dc5740 1 -- 172.21.15.1:0/3858696828 shutdown_connections 2024-04-26T21:50:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.827+0000 7f6993dc5740 1 -- 172.21.15.1:0/3858696828 wait complete. 2024-04-26T21:50:25.829 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.828+0000 7f6993dc5740 20 client.27031 trim_cache size 0 max 16384 2024-04-26T21:50:25.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.907+0000 7f6993dc5740 1 Processor -- start 2024-04-26T21:50:25.909 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.908+0000 7f6993dc5740 1 -- start start 2024-04-26T21:50:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.909+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.910 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.909+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2ee8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.909+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f12f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.909+0000 7f6993dc5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1b0540 con 0x55d2c2eed1c0 2024-04-26T21:50:25.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.909+0000 7f6993dc5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1ade10 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.911 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.909+0000 7f6993dc5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f698c074570 con 0x55d2c2ee2900 2024-04-26T21:50:25.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.909+0000 7f6989e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.910+0000 7f6989400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2ee8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.913 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.910+0000 7f6989400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2ee8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:37582/0 (socket says 172.21.15.1:37582) 2024-04-26T21:50:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.910+0000 7f6989e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50604/0 (socket says 172.21.15.1:50604) 2024-04-26T21:50:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.910+0000 7f698a800640 1 --2- 172.21.15.1:0/3665919407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f12f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.911+0000 7f698a800640 1 -- 172.21.15.1:0/3665919407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 msgr2=0x55d2c2ee8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.911+0000 7f698a800640 1 --2- 172.21.15.1:0/3665919407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2ee8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.911+0000 7f698a800640 1 -- 172.21.15.1:0/3665919407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee1fe0 msgr2=0x55d2c2eea970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.911+0000 7f698a800640 1 --2- 172.21.15.1:0/3665919407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.911+0000 7f698a800640 1 -- 172.21.15.1:0/3665919407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2c2ee2730 con 0x55d2c2eed1c0 2024-04-26T21:50:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.912+0000 7f698a800640 1 --2- 172.21.15.1:0/3665919407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f12f70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f698400afd0 tx=0x7f698401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d75ac9cdb58ee50 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.910+0000 7f6989400640 1 -- 172.21.15.1:0/3665919407 learned_addr learned my addr 172.21.15.1:0/3665919407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:25.914 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.912+0000 7f6989400640 1 --2- 172.21.15.1:0/3665919407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2ee8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-26T21:50:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.912+0000 7f6976a00640 1 -- 172.21.15.1:0/3665919407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984011930 con 0x55d2c2eed1c0 2024-04-26T21:50:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.913+0000 7f6976a00640 1 -- 172.21.15.1:0/3665919407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f698400aa30 con 0x55d2c2eed1c0 2024-04-26T21:50:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.913+0000 7f6976a00640 1 -- 172.21.15.1:0/3665919407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984021a60 con 0x55d2c2eed1c0 2024-04-26T21:50:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.913+0000 7f6989e00640 1 --2- 172.21.15.1:0/3665919407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-26T21:50:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.914+0000 7f6993dc5740 1 -- 172.21.15.1:0/3665919407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 msgr2=0x55d2c2f12f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:25.915 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.914+0000 7f6993dc5740 1 --2- 172.21.15.1:0/3665919407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f12f70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f698400afd0 tx=0x7f698401b000 comp rx=0 tx=0).stop 2024-04-26T21:50:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.916+0000 7f6993dc5740 1 -- 172.21.15.1:0/3665919407 shutdown_connections 2024-04-26T21:50:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.916+0000 7f6993dc5740 1 --2- 172.21.15.1:0/3665919407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2ee8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.916+0000 7f6993dc5740 1 --2- 172.21.15.1:0/3665919407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f12f70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.916+0000 7f6993dc5740 1 --2- 172.21.15.1:0/3665919407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2eea970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.917 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.916+0000 7f6993dc5740 1 -- 172.21.15.1:0/3665919407 >> 172.21.15.1:0/3665919407 conn(0x55d2c2ede4f0 msgr2=0x55d2c2ee8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.916+0000 7f6993dc5740 1 -- 172.21.15.1:0/3665919407 shutdown_connections 2024-04-26T21:50:25.918 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.916+0000 7f6993dc5740 1 -- 172.21.15.1:0/3665919407 wait complete. 2024-04-26T21:50:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.917+0000 7f6993dc5740 1 -- start start 2024-04-26T21:50:25.919 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.917+0000 7f6993dc5740 1 Processor -- start 2024-04-26T21:50:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.922+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2f6c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.922+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2f6c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.923 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.922+0000 7f6993dc5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f10300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.922+0000 7f6993dc5740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1ac2f0 con 0x55d2c2ee1fe0 2024-04-26T21:50:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.922+0000 7f6993dc5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1ad740 con 0x55d2c2eed1c0 2024-04-26T21:50:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.922+0000 7f6993dc5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f698c1ac9a0 con 0x55d2c2ee2900 2024-04-26T21:50:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.923+0000 7f6993dc5740 10 client.0 _ll_register_callbacks cb 0x7f697c09c8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:50:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.923+0000 7f698a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f10300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.923+0000 7f698a800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f10300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:50612/0 (socket says 172.21.15.1:50612) 2024-04-26T21:50:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.923+0000 7f698a800640 1 -- 172.21.15.1:0/2338149354 learned_addr learned my addr 172.21.15.1:0/2338149354 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.923+0000 7f6989400640 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2f6c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.923+0000 7f698a800640 1 -- 172.21.15.1:0/2338149354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 msgr2=0x55d2c2f6c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.923+0000 7f698a800640 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2f6c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.923+0000 7f698a800640 1 -- 172.21.15.1:0/2338149354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 msgr2=0x55d2c2f6c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.923+0000 7f6989e00640 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2f6c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.924+0000 7f698a800640 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2f6c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.924+0000 7f698a800640 1 -- 172.21.15.1:0/2338149354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2c2c25620 con 0x55d2c2eed1c0 2024-04-26T21:50:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.925+0000 7f6989e00640 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2f6c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:50:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.925+0000 7f6989400640 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2f6c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:50:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.925+0000 7f698a800640 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f10300 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6984011b60 tx=0x7f6984011b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56959467922f3bd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.925+0000 7f696be00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:50:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.925+0000 7f696be00640 1 -- 172.21.15.1:0/2338149354 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984036060 con 0x55d2c2eed1c0 2024-04-26T21:50:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.926+0000 7f696be00640 1 -- 172.21.15.1:0/2338149354 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9626+0+0 (secure 0 0 0) 0x7f6984004a40 con 0x55d2c2eed1c0 2024-04-26T21:50:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.926+0000 7f6993dc5740 1 -- 172.21.15.1:0/2338149354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f698400aa30 con 0x55d2c2eed1c0 2024-04-26T21:50:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.926+0000 7f696be00640 1 -- 172.21.15.1:0/2338149354 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7f6984004c40 con 0x55d2c2eed1c0 2024-04-26T21:50:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.926+0000 7f6993dc5740 10 client.27011 fetch_fsmap learned FSMap version 58 2024-04-26T21:50:25.928 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.926+0000 7f6993dc5740 1 -- 172.21.15.1:0/2338149354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x55d2c2e01cb0 con 0x55d2c2eed1c0 2024-04-26T21:50:25.929 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.928+0000 7f696be00640 1 -- 172.21.15.1:0/2338149354 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f698400aa30 con 0x55d2c2eed1c0 2024-04-26T21:50:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.928+0000 7f696be00640 1 -- 172.21.15.1:0/2338149354 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 58) ==== 8350+0+0 (secure 0 0 0) 0x7f6984031070 con 0x55d2c2eed1c0 2024-04-26T21:50:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.928+0000 7f6993dc5740 10 client.27011 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T21:50:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.928+0000 7f6993dc5740 10 client.27011 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:50:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.928+0000 7f6993dc5740 20 client.27011 populate_metadata read hostname 'smithi001' 2024-04-26T21:50:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.928+0000 7f6993dc5740 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d2c2f0db60 0x55d2c2f75ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:25.930 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.928+0000 7f6993dc5740 4 client.27011 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:50:25.931 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.928+0000 7f6993dc5740 1 -- 172.21.15.1:0/2338149354 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d2c2e4ee00 con 0x55d2c2f0db60 2024-04-26T21:50:25.932 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.931+0000 7f6989e00640 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d2c2f0db60 0x55d2c2f75ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.935+0000 7f6989e00640 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d2c2f0db60 0x55d2c2f75ae0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aff5c0ab9ea435c1 server_cookie=f0d14748b3fc7d9c in_seq=0 out_seq=0 2024-04-26T21:50:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:25.935+0000 7f696be00640 10 client.27011 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:26.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 158008 21.11 MB/sec execute 263 sec latency 357.383 ms 2024-04-26T21:50:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:26 smithi001 ceph-mon[28054]: pgmap v639: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.8 MiB/s wr, 396 op/s 2024-04-26T21:50:26.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:26 smithi038 ceph-mon[32724]: pgmap v639: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.8 MiB/s wr, 396 op/s 2024-04-26T21:50:26.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:26 smithi186 ceph-mon[32991]: pgmap v639: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.8 MiB/s wr, 396 op/s 2024-04-26T21:50:27.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 158060 21.04 MB/sec execute 264 sec latency 1214.214 ms 2024-04-26T21:50:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.818+0000 7f696be00640 1 -- 172.21.15.1:0/2338149354 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d2c2e4ee00 con 0x55d2c2f0db60 2024-04-26T21:50:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.818+0000 7f696be00640 10 client.27011 handle_command_reply: tid=0 2024-04-26T21:50:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.818+0000 7f6993dc5740 1 client.27011 shutdown 2024-04-26T21:50:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.819+0000 7f6993dc5740 1 -- 172.21.15.1:0/2338149354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2eed1c0 msgr2=0x55d2c2f10300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.819+0000 7f6993dc5740 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f10300 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6984011b60 tx=0x7f6984011b90 comp rx=0 tx=0).stop 2024-04-26T21:50:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.820+0000 7f6993dc5740 1 -- 172.21.15.1:0/2338149354 shutdown_connections 2024-04-26T21:50:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.820+0000 7f6993dc5740 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55d2c2f0db60 0x55d2c2f75ae0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.820+0000 7f6993dc5740 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2c2ee2900 0x55d2c2f6c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.820+0000 7f6993dc5740 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55d2c2eed1c0 0x55d2c2f10300 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.820+0000 7f6993dc5740 1 --2- 172.21.15.1:0/2338149354 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55d2c2ee1fe0 0x55d2c2f6c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.820+0000 7f6993dc5740 1 -- 172.21.15.1:0/2338149354 >> 172.21.15.1:0/2338149354 conn(0x55d2c2ede4f0 msgr2=0x55d2c2ee4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.820+0000 7f696be00640 0 client.27011 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:27.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.821+0000 7f6993dc5740 1 -- 172.21.15.1:0/2338149354 shutdown_connections 2024-04-26T21:50:27.823 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.821+0000 7f6993dc5740 1 -- 172.21.15.1:0/2338149354 wait complete. 2024-04-26T21:50:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.821+0000 7f6993dc5740 20 client.27011 trim_cache size 0 max 16384 2024-04-26T21:50:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:50:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:50:27.824 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:50:27.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:50:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.822+0000 7f698b200640 1 -- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f69700a3d80 msgr2=0x7f69700a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.822+0000 7f698b200640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f69700a3d80 0x7f69700a6240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6984013910 tx=0x7f698400ae40 comp rx=0 tx=0).stop 2024-04-26T21:50:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.823+0000 7f698b200640 1 -- 172.21.15.1:0/1417312647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f698c104960 msgr2=0x7f698c1117a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.823+0000 7f698b200640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f698c104960 0x7f698c1117a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f697c00ea60 tx=0x7f697c00ef20 comp rx=0 tx=0).stop 2024-04-26T21:50:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.823+0000 7f698b200640 1 -- 172.21.15.1:0/1417312647 shutdown_connections 2024-04-26T21:50:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.823+0000 7f698b200640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f69700a3d80 0x7f69700a6240 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:27.826 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.823+0000 7f698b200640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f698c10a970 0x7f698c10efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:27.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.823+0000 7f698b200640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f698c105320 0x7f698c10e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:27.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.823+0000 7f698b200640 1 --2- 172.21.15.1:0/1417312647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f698c104960 0x7f698c1117a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:27.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.823+0000 7f698b200640 1 -- 172.21.15.1:0/1417312647 >> 172.21.15.1:0/1417312647 conn(0x7f698c100430 msgr2=0x7f698c100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:27.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.823+0000 7f698b200640 1 -- 172.21.15.1:0/1417312647 shutdown_connections 2024-04-26T21:50:27.827 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:27.823+0000 7f698b200640 1 -- 172.21.15.1:0/1417312647 wait complete. 2024-04-26T21:50:28.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 158236 20.98 MB/sec execute 265 sec latency 1740.868 ms 2024-04-26T21:50:28.517 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:28 smithi038 ceph-mon[32724]: pgmap v640: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.5 MiB/s wr, 436 op/s 2024-04-26T21:50:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:28 smithi001 ceph-mon[28054]: pgmap v640: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.5 MiB/s wr, 436 op/s 2024-04-26T21:50:28.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:28 smithi186 ceph-mon[32991]: pgmap v640: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 7.5 MiB/s wr, 436 op/s 2024-04-26T21:50:29.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 158566 20.99 MB/sec execute 266 sec latency 199.560 ms 2024-04-26T21:50:30.181 INFO:tasks.workunit.client.0.smithi001.stdout: 10 159097 21.03 MB/sec execute 267 sec latency 166.697 ms 2024-04-26T21:50:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:30 smithi001 ceph-mon[28054]: pgmap v641: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.0 MiB/s wr, 350 op/s 2024-04-26T21:50:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:30 smithi038 ceph-mon[32724]: pgmap v641: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.0 MiB/s wr, 350 op/s 2024-04-26T21:50:30.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:30 smithi186 ceph-mon[32991]: pgmap v641: 129 pgs: 129 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 6.0 MiB/s wr, 350 op/s 2024-04-26T21:50:31.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 159470 21.02 MB/sec execute 268 sec latency 199.565 ms 2024-04-26T21:50:32.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 159975 21.04 MB/sec execute 269 sec latency 65.820 ms 2024-04-26T21:50:32.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:32 smithi001 ceph-mon[28054]: pgmap v642: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 8.5 MiB/s wr, 472 op/s 2024-04-26T21:50:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:32 smithi038 ceph-mon[32724]: pgmap v642: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 8.5 MiB/s wr, 472 op/s 2024-04-26T21:50:32.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:32 smithi186 ceph-mon[32991]: pgmap v642: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 8.5 MiB/s wr, 472 op/s 2024-04-26T21:50:33.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 160505 21.05 MB/sec execute 270 sec latency 104.201 ms 2024-04-26T21:50:34.180 INFO:tasks.workunit.client.0.smithi001.stdout: 10 161004 21.08 MB/sec execute 271 sec latency 88.421 ms 2024-04-26T21:50:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:34 smithi001 ceph-mon[28054]: pgmap v643: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.9 MiB/s wr, 544 op/s 2024-04-26T21:50:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:34 smithi038 ceph-mon[32724]: pgmap v643: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.9 MiB/s wr, 544 op/s 2024-04-26T21:50:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:34 smithi186 ceph-mon[32991]: pgmap v643: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 9.9 MiB/s wr, 544 op/s 2024-04-26T21:50:35.181 INFO:tasks.workunit.client.0.smithi001.stdout: 10 161618 21.15 MB/sec execute 272 sec latency 97.413 ms 2024-04-26T21:50:36.181 INFO:tasks.workunit.client.0.smithi001.stdout: 10 162008 21.16 MB/sec execute 273 sec latency 208.714 ms 2024-04-26T21:50:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:36 smithi001 ceph-mon[28054]: pgmap v644: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 645 op/s 2024-04-26T21:50:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:36 smithi038 ceph-mon[32724]: pgmap v644: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 645 op/s 2024-04-26T21:50:36.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:36 smithi186 ceph-mon[32991]: pgmap v644: 129 pgs: 129 active+clean; 1.3 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 12 MiB/s wr, 645 op/s 2024-04-26T21:50:37.181 INFO:tasks.workunit.client.0.smithi001.stdout: 10 162394 21.15 MB/sec execute 274 sec latency 198.143 ms 2024-04-26T21:50:38.181 INFO:tasks.workunit.client.0.smithi001.stdout: 10 162723 21.12 MB/sec execute 275 sec latency 149.852 ms 2024-04-26T21:50:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:38 smithi001 ceph-mon[28054]: pgmap v645: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.5 MiB/s wr, 490 op/s 2024-04-26T21:50:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:38 smithi038 ceph-mon[32724]: pgmap v645: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.5 MiB/s wr, 490 op/s 2024-04-26T21:50:38.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:38 smithi186 ceph-mon[32991]: pgmap v645: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.5 MiB/s wr, 490 op/s 2024-04-26T21:50:39.181 INFO:tasks.workunit.client.0.smithi001.stdout: 10 163271 21.15 MB/sec execute 276 sec latency 105.933 ms 2024-04-26T21:50:40.181 INFO:tasks.workunit.client.0.smithi001.stdout: 10 163755 21.15 MB/sec execute 277 sec latency 174.247 ms 2024-04-26T21:50:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:40 smithi001 ceph-mon[28054]: pgmap v646: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.0 MiB/s wr, 463 op/s 2024-04-26T21:50:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:40 smithi038 ceph-mon[32724]: pgmap v646: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.0 MiB/s wr, 463 op/s 2024-04-26T21:50:40.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:40 smithi186 ceph-mon[32991]: pgmap v646: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.0 MiB/s wr, 463 op/s 2024-04-26T21:50:41.181 INFO:tasks.workunit.client.0.smithi001.stdout: 10 164355 21.22 MB/sec execute 278 sec latency 87.497 ms 2024-04-26T21:50:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.526+0000 7fe397200640 1 -- 172.21.15.1:0/3315418143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe398104960 msgr2=0x7fe398104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.526+0000 7fe397200640 1 --2- 172.21.15.1:0/3315418143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe398104960 0x7fe398104de0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe384009930 tx=0x7fe38402a420 comp rx=0 tx=0).stop 2024-04-26T21:50:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.526+0000 7fe397200640 1 -- 172.21.15.1:0/3315418143 shutdown_connections 2024-04-26T21:50:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.526+0000 7fe397200640 1 --2- 172.21.15.1:0/3315418143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe398104960 0x7fe398104de0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.526+0000 7fe397200640 1 --2- 172.21.15.1:0/3315418143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe398105320 0x7fe398111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.526+0000 7fe397200640 1 --2- 172.21.15.1:0/3315418143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe39810a970 0x7fe39810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.526+0000 7fe397200640 1 -- 172.21.15.1:0/3315418143 >> 172.21.15.1:0/3315418143 conn(0x7fe398100430 msgr2=0x7fe398102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.526+0000 7fe397200640 1 -- 172.21.15.1:0/3315418143 shutdown_connections 2024-04-26T21:50:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.527+0000 7fe397200640 1 -- 172.21.15.1:0/3315418143 wait complete. 2024-04-26T21:50:41.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.528+0000 7fe397200640 1 Processor -- start 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.528+0000 7fe397200640 1 -- start start 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.529+0000 7fe397200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe398104960 0x7fe398111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.529+0000 7fe397200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe398105320 0x7fe39810f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.529+0000 7fe397200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe39810a970 0x7fe39810fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.529+0000 7fe397200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe398077490 con 0x7fe398104960 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.529+0000 7fe397200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe398077310 con 0x7fe398105320 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.529+0000 7fe397200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe398077610 con 0x7fe39810a970 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.530+0000 7fe396800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe39810a970 0x7fe39810fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.530+0000 7fe396800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe39810a970 0x7fe39810fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33194/0 (socket says 172.21.15.1:33194) 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.530+0000 7fe396800640 1 -- 172.21.15.1:0/392825323 learned_addr learned my addr 172.21.15.1:0/392825323 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.530+0000 7fe395e00640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe398104960 0x7fe398111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.530+0000 7fe396800640 1 -- 172.21.15.1:0/392825323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe398105320 msgr2=0x7fe39810f640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:50:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.530+0000 7fe396800640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe398105320 0x7fe39810f640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.530+0000 7fe396800640 1 -- 172.21.15.1:0/392825323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe398104960 msgr2=0x7fe398111540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.530+0000 7fe396800640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe398104960 0x7fe398111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.530+0000 7fe396800640 1 -- 172.21.15.1:0/392825323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe398110350 con 0x7fe39810a970 2024-04-26T21:50:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.530+0000 7fe396800640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe39810a970 0x7fe39810fb80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe39000eb10 tx=0x7fe39000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.532+0000 7fe38ea00640 1 -- 172.21.15.1:0/392825323 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe390004b20 con 0x7fe39810a970 2024-04-26T21:50:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.532+0000 7fe38ea00640 1 -- 172.21.15.1:0/392825323 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe390004cc0 con 0x7fe39810a970 2024-04-26T21:50:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.532+0000 7fe38ea00640 1 -- 172.21.15.1:0/392825323 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe390010720 con 0x7fe39810a970 2024-04-26T21:50:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.532+0000 7fe397200640 1 -- 172.21.15.1:0/392825323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39806c350 con 0x7fe39810a970 2024-04-26T21:50:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.532+0000 7fe397200640 1 -- 172.21.15.1:0/392825323 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe39806c8a0 con 0x7fe39810a970 2024-04-26T21:50:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.538+0000 7fe39fe3d740 1 Processor -- start 2024-04-26T21:50:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.543+0000 7fe38ea00640 1 -- 172.21.15.1:0/392825323 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fe390005020 con 0x7fe39810a970 2024-04-26T21:50:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.544+0000 7fe39fe3d740 1 -- start start 2024-04-26T21:50:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.544+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562572904fe0 0x5625729053c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.544+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 0x56257290fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.545+0000 7fe395400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562572904fe0 0x5625729053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.545+0000 7fe395400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562572904fe0 0x5625729053c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43496/0 (socket says 172.21.15.1:43496) 2024-04-26T21:50:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.545+0000 7fe395e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 0x56257290fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.545+0000 7fe395e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 0x56257290fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59980/0 (socket says 172.21.15.1:59980) 2024-04-26T21:50:41.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.547+0000 7fe38ea00640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe37c0a3ed0 0x7fe37c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.547 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.547+0000 7fe38ea00640 1 -- 172.21.15.1:0/392825323 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9626+0+0 (secure 0 0 0) 0x7fe3900d12b0 con 0x7fe39810a970 2024-04-26T21:50:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.548+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625729101c0 0x5625729125b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.548+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x56257286a380 con 0x562572905900 2024-04-26T21:50:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.548+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562572806c30 con 0x562572904fe0 2024-04-26T21:50:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.548+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562572866340 con 0x5625729101c0 2024-04-26T21:50:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.549+0000 7fe395e00640 1 -- 172.21.15.1:0/3290237676 learned_addr learned my addr 172.21.15.1:0/3290237676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.549+0000 7fe395400640 1 --2- 172.21.15.1:0/3290237676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625729101c0 0x5625729125b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.549+0000 7fe396800640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe37c0a3ed0 0x7fe37c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.549+0000 7fe395400640 1 -- 172.21.15.1:0/3290237676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625729101c0 msgr2=0x5625729125b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.549+0000 7fe395400640 1 --2- 172.21.15.1:0/3290237676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625729101c0 0x5625729125b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.549+0000 7fe395400640 1 -- 172.21.15.1:0/3290237676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 msgr2=0x56257290fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.551+0000 7fe395400640 1 --2- 172.21.15.1:0/3290237676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 0x56257290fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.551+0000 7fe396800640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe37c0a3ed0 0x7fe37c0a6390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe390099770 tx=0x7fe39000c950 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.551+0000 7fe395400640 1 -- 172.21.15.1:0/3290237676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562572828940 con 0x562572904fe0 2024-04-26T21:50:41.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.551+0000 7fe395e00640 1 --2- 172.21.15.1:0/3290237676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 0x56257290fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:50:41.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.552+0000 7fe395400640 1 --2- 172.21.15.1:0/3290237676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562572904fe0 0x5625729053c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe38402a940 tx=0x7fe384005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be184d7de2b0b09a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.553+0000 7fe383400640 1 -- 172.21.15.1:0/3290237676 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384035040 con 0x562572904fe0 2024-04-26T21:50:41.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.554+0000 7fe383400640 1 -- 172.21.15.1:0/3290237676 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe384004490 con 0x562572904fe0 2024-04-26T21:50:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.555+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3290237676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562572904fe0 msgr2=0x5625729053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:41.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.555+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/3290237676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562572904fe0 0x5625729053c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe38402a940 tx=0x7fe384005d50 comp rx=0 tx=0).stop 2024-04-26T21:50:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.556+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3290237676 shutdown_connections 2024-04-26T21:50:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.556+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/3290237676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625729101c0 0x5625729125b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.556+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/3290237676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 0x56257290fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.556+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/3290237676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562572904fe0 0x5625729053c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.556+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3290237676 >> 172.21.15.1:0/3290237676 conn(0x5625729014f0 msgr2=0x5625729018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.556+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3290237676 shutdown_connections 2024-04-26T21:50:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.556+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3290237676 wait complete. 2024-04-26T21:50:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.557+0000 7fe39fe3d740 1 -- start start 2024-04-26T21:50:41.557 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.557+0000 7fe39fe3d740 1 Processor -- start 2024-04-26T21:50:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.558+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 0x56257290fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.558+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 0x56257290cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.558+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56257290d2c0 0x56257290d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.558+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562572912d80 con 0x562572905900 2024-04-26T21:50:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.558+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x5625727f1c60 con 0x5625729101c0 2024-04-26T21:50:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.558+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562572912f00 con 0x56257290d2c0 2024-04-26T21:50:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.558+0000 7fe39fe3d740 10 client.0 _ll_register_callbacks cb 0x56257281de50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:50:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.559+0000 7fe395e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 0x56257290cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.559+0000 7fe395e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 0x56257290cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43510/0 (socket says 172.21.15.1:43510) 2024-04-26T21:50:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.559+0000 7fe395e00640 1 -- 172.21.15.1:0/3323617735 learned_addr learned my addr 172.21.15.1:0/3323617735 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.560+0000 7fe395e00640 1 -- 172.21.15.1:0/3323617735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56257290d2c0 msgr2=0x56257290d740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:50:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.560+0000 7fe395e00640 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56257290d2c0 0x56257290d740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.560+0000 7fe395e00640 1 -- 172.21.15.1:0/3323617735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 msgr2=0x56257290fa90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:50:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.560+0000 7fe395e00640 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 0x56257290fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.560+0000 7fe395e00640 1 -- 172.21.15.1:0/3323617735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562572800510 con 0x5625729101c0 2024-04-26T21:50:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.562+0000 7fe395e00640 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 0x56257290cd80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe38800f6f0 tx=0x7fe38800fbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf1fdf792e95ef9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.562+0000 7fe377e00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:50:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.565+0000 7fe377e00640 1 -- 172.21.15.1:0/3323617735 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe388016070 con 0x5625729101c0 2024-04-26T21:50:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.565+0000 7fe377e00640 1 -- 172.21.15.1:0/3323617735 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9626+0+0 (secure 0 0 0) 0x7fe38800bb00 con 0x5625729101c0 2024-04-26T21:50:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.565+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3323617735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562572877f90 con 0x5625729101c0 2024-04-26T21:50:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.565+0000 7fe377e00640 1 -- 172.21.15.1:0/3323617735 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fe38800bd00 con 0x5625729101c0 2024-04-26T21:50:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.566+0000 7fe39fe3d740 10 client.27020 fetch_fsmap learned FSMap version 58 2024-04-26T21:50:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.566+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3323617735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x56257257c210 con 0x5625729101c0 2024-04-26T21:50:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.566+0000 7fe377e00640 1 -- 172.21.15.1:0/3323617735 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562572877f90 con 0x5625729101c0 2024-04-26T21:50:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.567+0000 7fe377e00640 1 -- 172.21.15.1:0/3323617735 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 58) ==== 8350+0+0 (secure 0 0 0) 0x7fe38801a4a0 con 0x5625729101c0 2024-04-26T21:50:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.567+0000 7fe39fe3d740 10 client.27020 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T21:50:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.567+0000 7fe39fe3d740 10 client.27020 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:50:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.567+0000 7fe39fe3d740 20 client.27020 populate_metadata read hostname 'smithi001' 2024-04-26T21:50:41.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.568+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562572920110 0x562572922520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.568+0000 7fe39fe3d740 4 client.27020 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:50:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.569+0000 7fe395400640 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562572920110 0x562572922520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.569+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3323617735 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562572871e00 con 0x562572920110 2024-04-26T21:50:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.570+0000 7fe395400640 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562572920110 0x562572922520 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eae8090023b15d17 server_cookie=4279aed8fba6e083 in_seq=0 out_seq=0 2024-04-26T21:50:41.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.570+0000 7fe377e00640 10 client.27020 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.574+0000 7fe377e00640 1 -- 172.21.15.1:0/3323617735 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562572871e00 con 0x562572920110 2024-04-26T21:50:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.574+0000 7fe377e00640 10 client.27020 handle_command_reply: tid=0 2024-04-26T21:50:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.575+0000 7fe39fe3d740 1 client.27020 shutdown 2024-04-26T21:50:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.575+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3323617735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 msgr2=0x56257290cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.575+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 0x56257290cd80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe38800f6f0 tx=0x7fe38800fbb0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.576+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3323617735 shutdown_connections 2024-04-26T21:50:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.576+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562572920110 0x562572922520 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.576+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56257290d2c0 0x56257290d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.576+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 0x56257290cd80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.576+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/3323617735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562572905900 0x56257290fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.576+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3323617735 >> 172.21.15.1:0/3323617735 conn(0x5625729014f0 msgr2=0x5625728ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.576+0000 7fe377e00640 0 client.27020 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.577+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3323617735 shutdown_connections 2024-04-26T21:50:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.577+0000 7fe39fe3d740 1 -- 172.21.15.1:0/3323617735 wait complete. 2024-04-26T21:50:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.577+0000 7fe39fe3d740 20 client.27020 trim_cache size 0 max 16384 2024-04-26T21:50:41.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.619+0000 7fe39fe3d740 1 Processor -- start 2024-04-26T21:50:41.619 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.619+0000 7fe39fe3d740 1 -- start start 2024-04-26T21:50:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.619+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 0x56257291bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.619+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56257290d2c0 0x562572927e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.619+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5625729101c0 0x562572928370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.619+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe3981c1580 con 0x5625729101c0 2024-04-26T21:50:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.619+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe3981bee50 con 0x56257290d2c0 2024-04-26T21:50:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.619+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe3981bccc0 con 0x562572905900 2024-04-26T21:50:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.621+0000 7fe395e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56257290d2c0 0x562572927e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.621+0000 7fe395e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56257290d2c0 0x562572927e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43522/0 (socket says 172.21.15.1:43522) 2024-04-26T21:50:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.621+0000 7fe395e00640 1 -- 172.21.15.1:0/1042346210 learned_addr learned my addr 172.21.15.1:0/1042346210 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.621+0000 7fe395400640 1 --2- 172.21.15.1:0/1042346210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5625729101c0 0x562572928370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.621+0000 7fe395400640 1 --2- 172.21.15.1:0/1042346210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 0x56257291bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.622+0000 7fe395e00640 1 -- 172.21.15.1:0/1042346210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 msgr2=0x56257291bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.622+0000 7fe395e00640 1 --2- 172.21.15.1:0/1042346210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 0x56257291bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.622+0000 7fe395e00640 1 -- 172.21.15.1:0/1042346210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5625729101c0 msgr2=0x562572928370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:41.623 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.622+0000 7fe395e00640 1 --2- 172.21.15.1:0/1042346210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5625729101c0 0x562572928370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.623+0000 7fe395e00640 1 -- 172.21.15.1:0/1042346210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562572648620 con 0x56257290d2c0 2024-04-26T21:50:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.623+0000 7fe395400640 1 --2- 172.21.15.1:0/1042346210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5625729101c0 0x562572928370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:50:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.624+0000 7fe395e00640 1 --2- 172.21.15.1:0/1042346210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56257290d2c0 0x562572927e30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe388001eb0 tx=0x7fe38800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45f93ac4431e9707 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.625+0000 7fe382a00640 1 -- 172.21.15.1:0/1042346210 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38802d030 con 0x56257290d2c0 2024-04-26T21:50:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.626+0000 7fe382a00640 1 -- 172.21.15.1:0/1042346210 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe38801ab00 con 0x56257290d2c0 2024-04-26T21:50:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.626+0000 7fe382a00640 1 -- 172.21.15.1:0/1042346210 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe388016040 con 0x56257290d2c0 2024-04-26T21:50:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.628+0000 7fe39fe3d740 1 -- 172.21.15.1:0/1042346210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56257290d2c0 msgr2=0x562572927e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.628+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/1042346210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56257290d2c0 0x562572927e30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe388001eb0 tx=0x7fe38800bf10 comp rx=0 tx=0).stop 2024-04-26T21:50:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.628+0000 7fe39fe3d740 1 -- 172.21.15.1:0/1042346210 shutdown_connections 2024-04-26T21:50:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.628+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/1042346210 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5625729101c0 0x562572928370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.628+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/1042346210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56257290d2c0 0x562572927e30 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.628+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/1042346210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 0x56257291bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.628+0000 7fe39fe3d740 1 -- 172.21.15.1:0/1042346210 >> 172.21.15.1:0/1042346210 conn(0x5625729014f0 msgr2=0x56257290b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.628+0000 7fe39fe3d740 1 -- 172.21.15.1:0/1042346210 shutdown_connections 2024-04-26T21:50:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.629+0000 7fe39fe3d740 1 -- 172.21.15.1:0/1042346210 wait complete. 2024-04-26T21:50:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.629+0000 7fe39fe3d740 1 -- start start 2024-04-26T21:50:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.629+0000 7fe39fe3d740 1 Processor -- start 2024-04-26T21:50:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.630+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 0x562572918630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.630+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56257290d2c0 0x56257290a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.630+0000 7fe39fe3d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 0x56257290a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.630+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe3981bd310 con 0x56257290d2c0 2024-04-26T21:50:41.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.630+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fe3981be780 con 0x5625729101c0 2024-04-26T21:50:41.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.630+0000 7fe39fe3d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe3981bd9e0 con 0x562572905900 2024-04-26T21:50:41.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.630+0000 7fe39fe3d740 10 client.0 _ll_register_callbacks cb 0x7fe38800d340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:50:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.631+0000 7fe395400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 0x562572918630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.631+0000 7fe395400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 0x562572918630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:33222/0 (socket says 172.21.15.1:33222) 2024-04-26T21:50:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.631+0000 7fe395400640 1 -- 172.21.15.1:0/2005405406 learned_addr learned my addr 172.21.15.1:0/2005405406 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.631+0000 7fe395400640 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 0x56257290a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.632+0000 7fe395400640 1 -- 172.21.15.1:0/2005405406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 msgr2=0x56257290a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.632+0000 7fe395400640 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 0x56257290a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.632+0000 7fe395400640 1 -- 172.21.15.1:0/2005405406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56257290d2c0 msgr2=0x56257290a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.632+0000 7fe395e00640 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56257290d2c0 0x56257290a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.632+0000 7fe395400640 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56257290d2c0 0x56257290a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.632+0000 7fe395400640 1 -- 172.21.15.1:0/2005405406 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562572824cb0 con 0x562572905900 2024-04-26T21:50:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.632+0000 7fe395400640 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 0x562572918630 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe38402a9e0 tx=0x7fe38402a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98b9c817371a457f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.632+0000 7fe377e00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:50:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.633+0000 7fe377e00640 1 -- 172.21.15.1:0/2005405406 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe384035040 con 0x562572905900 2024-04-26T21:50:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.633+0000 7fe377e00640 1 -- 172.21.15.1:0/2005405406 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9626+0+0 (secure 0 0 0) 0x7fe38404b620 con 0x562572905900 2024-04-26T21:50:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.633+0000 7fe39fe3d740 1 -- 172.21.15.1:0/2005405406 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe38801ab00 con 0x562572905900 2024-04-26T21:50:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.634+0000 7fe377e00640 1 -- 172.21.15.1:0/2005405406 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=58) ==== 24+0+0 (secure 0 0 0) 0x7fe38404b820 con 0x562572905900 2024-04-26T21:50:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.635+0000 7fe39fe3d740 10 client.27046 fetch_fsmap learned FSMap version 58 2024-04-26T21:50:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.635+0000 7fe39fe3d740 1 -- 172.21.15.1:0/2005405406 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=58,monmap=4+}) -- 0x5625729859a0 con 0x562572905900 2024-04-26T21:50:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.635+0000 7fe377e00640 1 -- 172.21.15.1:0/2005405406 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe38801ab00 con 0x562572905900 2024-04-26T21:50:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.635+0000 7fe377e00640 1 -- 172.21.15.1:0/2005405406 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 58) ==== 8350+0+0 (secure 0 0 0) 0x7fe38403a420 con 0x562572905900 2024-04-26T21:50:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.635+0000 7fe39fe3d740 10 client.27046 fetch_fsmap finished waiting for FSMap version 58 2024-04-26T21:50:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.635+0000 7fe39fe3d740 10 client.27046 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:50:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.635+0000 7fe39fe3d740 20 client.27046 populate_metadata read hostname 'smithi001' 2024-04-26T21:50:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.635+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562572985c80 0x562572988070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.635+0000 7fe39fe3d740 4 client.27046 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:50:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.635+0000 7fe39fe3d740 1 -- 172.21.15.1:0/2005405406 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5625727ed4b0 con 0x562572985c80 2024-04-26T21:50:41.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.636+0000 7fe395e00640 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562572985c80 0x562572988070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:41.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.637+0000 7fe395e00640 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562572985c80 0x562572988070 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc292eb4f6d0ed84 server_cookie=2b680282c55a310b in_seq=0 out_seq=0 2024-04-26T21:50:41.637 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:41.637+0000 7fe377e00640 10 client.27046 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:42.181 INFO:tasks.workunit.client.0.smithi001.stdout: 10 164451 21.16 MB/sec execute 279 sec latency 480.636 ms 2024-04-26T21:50:42.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:42 smithi001 ceph-mon[28054]: pgmap v647: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.7 MiB/s wr, 570 op/s 2024-04-26T21:50:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:42 smithi038 ceph-mon[32724]: pgmap v647: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.7 MiB/s wr, 570 op/s 2024-04-26T21:50:42.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:42 smithi186 ceph-mon[32991]: pgmap v647: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 9.7 MiB/s wr, 570 op/s 2024-04-26T21:50:43.182 INFO:tasks.workunit.client.0.smithi001.stdout: 10 164501 21.09 MB/sec execute 280 sec latency 1469.599 ms 2024-04-26T21:50:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.182+0000 7fe377e00640 1 -- 172.21.15.1:0/2005405406 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5625727ed4b0 con 0x562572985c80 2024-04-26T21:50:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.182+0000 7fe377e00640 10 client.27046 handle_command_reply: tid=0 2024-04-26T21:50:43.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.182+0000 7fe39fe3d740 1 client.27046 shutdown 2024-04-26T21:50:43.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.182+0000 7fe39fe3d740 1 -- 172.21.15.1:0/2005405406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 msgr2=0x562572918630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:43.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.182+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 0x562572918630 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe38402a9e0 tx=0x7fe38402a420 comp rx=0 tx=0).stop 2024-04-26T21:50:43.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.183+0000 7fe39fe3d740 1 -- 172.21.15.1:0/2005405406 shutdown_connections 2024-04-26T21:50:43.183 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.183+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x562572985c80 0x562572988070 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.183+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5625729101c0 0x56257290a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.183+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x56257290d2c0 0x56257290a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.183+0000 7fe39fe3d740 1 --2- 172.21.15.1:0/2005405406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562572905900 0x562572918630 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.183+0000 7fe39fe3d740 1 -- 172.21.15.1:0/2005405406 >> 172.21.15.1:0/2005405406 conn(0x5625729014f0 msgr2=0x56257290c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.183+0000 7fe377e00640 0 client.27046 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.183+0000 7fe39fe3d740 1 -- 172.21.15.1:0/2005405406 shutdown_connections 2024-04-26T21:50:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.183+0000 7fe39fe3d740 1 -- 172.21.15.1:0/2005405406 wait complete. 2024-04-26T21:50:43.184 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.184+0000 7fe39fe3d740 20 client.27046 trim_cache size 0 max 16384 2024-04-26T21:50:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:50:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:50:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:50:43.185 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 -- 172.21.15.1:0/392825323 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe37c0a3ed0 msgr2=0x7fe37c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe37c0a3ed0 0x7fe37c0a6390 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe390099770 tx=0x7fe39000c950 comp rx=0 tx=0).stop 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 -- 172.21.15.1:0/392825323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe39810a970 msgr2=0x7fe39810fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe39810a970 0x7fe39810fb80 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe39000eb10 tx=0x7fe39000c2d0 comp rx=0 tx=0).stop 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 -- 172.21.15.1:0/392825323 shutdown_connections 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fe37c0a3ed0 0x7fe37c0a6390 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe39810a970 0x7fe39810fb80 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe398105320 0x7fe39810f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 --2- 172.21.15.1:0/392825323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe398104960 0x7fe398111540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 -- 172.21.15.1:0/392825323 >> 172.21.15.1:0/392825323 conn(0x7fe398100430 msgr2=0x7fe398100b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 -- 172.21.15.1:0/392825323 shutdown_connections 2024-04-26T21:50:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:43.185+0000 7fe397200640 1 -- 172.21.15.1:0/392825323 wait complete. 2024-04-26T21:50:43.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:43 smithi001 ceph-mon[28054]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:50:43.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:43 smithi038 ceph-mon[32724]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:50:43.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:43 smithi186 ceph-mon[32991]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-04-26T21:50:44.181 INFO:tasks.workunit.client.0.smithi001.stdout: 10 164626 21.04 MB/sec execute 281 sec latency 1491.510 ms 2024-04-26T21:50:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:44 smithi001 ceph-mon[28054]: pgmap v648: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 375 op/s 2024-04-26T21:50:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:44 smithi038 ceph-mon[32724]: pgmap v648: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 375 op/s 2024-04-26T21:50:44.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:44 smithi186 ceph-mon[32991]: pgmap v648: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 6.6 MiB/s wr, 375 op/s 2024-04-26T21:50:45.183 INFO:tasks.workunit.client.0.smithi001.stdout: 10 164866 21.00 MB/sec execute 282 sec latency 164.700 ms 2024-04-26T21:50:46.183 INFO:tasks.workunit.client.0.smithi001.stdout: 10 165124 20.98 MB/sec execute 283 sec latency 149.482 ms 2024-04-26T21:50:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:46 smithi001 ceph-mon[28054]: pgmap v649: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 562 op/s 2024-04-26T21:50:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:46 smithi038 ceph-mon[32724]: pgmap v649: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 562 op/s 2024-04-26T21:50:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:46 smithi186 ceph-mon[32991]: pgmap v649: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 562 op/s 2024-04-26T21:50:47.184 INFO:tasks.workunit.client.0.smithi001.stdout: 10 165537 21.01 MB/sec execute 284 sec latency 100.434 ms 2024-04-26T21:50:48.184 INFO:tasks.workunit.client.0.smithi001.stdout: 10 165952 20.99 MB/sec execute 285 sec latency 114.546 ms 2024-04-26T21:50:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:48 smithi001 ceph-mon[28054]: pgmap v650: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.9 MiB/s wr, 555 op/s 2024-04-26T21:50:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:48 smithi038 ceph-mon[32724]: pgmap v650: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.9 MiB/s wr, 555 op/s 2024-04-26T21:50:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:48 smithi186 ceph-mon[32991]: pgmap v650: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.9 MiB/s wr, 555 op/s 2024-04-26T21:50:49.184 INFO:tasks.workunit.client.0.smithi001.stdout: 10 166407 21.01 MB/sec execute 286 sec latency 161.205 ms 2024-04-26T21:50:50.184 INFO:tasks.workunit.client.0.smithi001.stdout: 10 166899 21.03 MB/sec execute 287 sec latency 141.044 ms 2024-04-26T21:50:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:50 smithi038 ceph-mon[32724]: pgmap v651: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 9.7 MiB/s wr, 536 op/s 2024-04-26T21:50:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:50 smithi186 ceph-mon[32991]: pgmap v651: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 9.7 MiB/s wr, 536 op/s 2024-04-26T21:50:50.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:50 smithi001 ceph-mon[28054]: pgmap v651: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 9.7 MiB/s wr, 536 op/s 2024-04-26T21:50:51.184 INFO:tasks.workunit.client.0.smithi001.stdout: 10 167449 21.04 MB/sec execute 288 sec latency 100.774 ms 2024-04-26T21:50:52.186 INFO:tasks.workunit.client.0.smithi001.stdout: 10 167949 21.08 MB/sec execute 289 sec latency 85.587 ms 2024-04-26T21:50:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:52 smithi038 ceph-mon[32724]: pgmap v652: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 11 MiB/s wr, 660 op/s 2024-04-26T21:50:52.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:52 smithi186 ceph-mon[32991]: pgmap v652: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 11 MiB/s wr, 660 op/s 2024-04-26T21:50:52.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:52 smithi001 ceph-mon[28054]: pgmap v652: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 11 MiB/s wr, 660 op/s 2024-04-26T21:50:53.186 INFO:tasks.workunit.client.0.smithi001.stdout: 10 168442 21.10 MB/sec execute 290 sec latency 86.604 ms 2024-04-26T21:50:54.186 INFO:tasks.workunit.client.0.smithi001.stdout: 10 168898 21.12 MB/sec execute 291 sec latency 90.376 ms 2024-04-26T21:50:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:54 smithi186 ceph-mon[32991]: pgmap v653: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 10 MiB/s wr, 592 op/s 2024-04-26T21:50:54.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:54 smithi001 ceph-mon[28054]: pgmap v653: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 10 MiB/s wr, 592 op/s 2024-04-26T21:50:54.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:54 smithi038 ceph-mon[32724]: pgmap v653: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 10 MiB/s wr, 592 op/s 2024-04-26T21:50:55.186 INFO:tasks.workunit.client.0.smithi001.stdout: 10 169273 21.10 MB/sec execute 292 sec latency 142.933 ms 2024-04-26T21:50:56.186 INFO:tasks.workunit.client.0.smithi001.stdout: 10 169701 21.09 MB/sec execute 293 sec latency 110.630 ms 2024-04-26T21:50:56.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.384+0000 7feb99000640 1 -- 172.21.15.1:0/3387108342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb94108750 msgr2=0x7feb94108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.384+0000 7feb99000640 1 --2- 172.21.15.1:0/3387108342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb94108750 0x7feb94108b30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7feb84009930 tx=0x7feb8402a440 comp rx=0 tx=0).stop 2024-04-26T21:50:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.384+0000 7feb99000640 1 -- 172.21.15.1:0/3387108342 shutdown_connections 2024-04-26T21:50:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.384+0000 7feb99000640 1 --2- 172.21.15.1:0/3387108342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94103100 0x7feb9410f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.384+0000 7feb99000640 1 --2- 172.21.15.1:0/3387108342 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94102740 0x7feb94102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.384+0000 7feb99000640 1 --2- 172.21.15.1:0/3387108342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb94108750 0x7feb94108b30 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.384+0000 7feb99000640 1 -- 172.21.15.1:0/3387108342 >> 172.21.15.1:0/3387108342 conn(0x7feb940fe270 msgr2=0x7feb94100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.384+0000 7feb99000640 1 -- 172.21.15.1:0/3387108342 shutdown_connections 2024-04-26T21:50:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.384+0000 7feb99000640 1 -- 172.21.15.1:0/3387108342 wait complete. 2024-04-26T21:50:56.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.385+0000 7feb99000640 1 Processor -- start 2024-04-26T21:50:56.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.386+0000 7feb99000640 1 -- start start 2024-04-26T21:50:56.387 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.386+0000 7feb99000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94102740 0x7feb94198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.386+0000 7feb99000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94103100 0x7feb94199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.386+0000 7feb99000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb94108750 0x7feb9419d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.386+0000 7feb99000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7feb94112a20 con 0x7feb94102740 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.386+0000 7feb99000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb941128a0 con 0x7feb94103100 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.386+0000 7feb99000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb94112ba0 con 0x7feb94108750 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.387+0000 7feb92a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94103100 0x7feb94199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.387+0000 7feb93400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94102740 0x7feb94198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.387+0000 7feb92a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94103100 0x7feb94199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:58964/0 (socket says 172.21.15.1:58964) 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.387+0000 7feb93400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94102740 0x7feb94198fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46074/0 (socket says 172.21.15.1:46074) 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.387+0000 7feb92a00640 1 -- 172.21.15.1:0/3195284484 learned_addr learned my addr 172.21.15.1:0/3195284484 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.387+0000 7feb93e00640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb94108750 0x7feb9419d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.388 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.387+0000 7feb93e00640 1 -- 172.21.15.1:0/3195284484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94103100 msgr2=0x7feb94199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.388+0000 7feb93e00640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94103100 0x7feb94199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.388+0000 7feb93e00640 1 -- 172.21.15.1:0/3195284484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94102740 msgr2=0x7feb94198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.388+0000 7feb93e00640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94102740 0x7feb94198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.388+0000 7feb93e00640 1 -- 172.21.15.1:0/3195284484 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9419df50 con 0x7feb94108750 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.388+0000 7feb93400640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94102740 0x7feb94198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.388+0000 7feb92a00640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94103100 0x7feb94199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.388+0000 7feb93e00640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb94108750 0x7feb9419d850 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7feb8c00e940 tx=0x7feb8c00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.389+0000 7feb8be00640 1 -- 172.21.15.1:0/3195284484 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c004d20 con 0x7feb94108750 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.389+0000 7feb8be00640 1 -- 172.21.15.1:0/3195284484 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8c005020 con 0x7feb94108750 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.389+0000 7feb8be00640 1 -- 172.21.15.1:0/3195284484 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c0116d0 con 0x7feb94108750 2024-04-26T21:50:56.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.389+0000 7feb99000640 1 -- 172.21.15.1:0/3195284484 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb9419e240 con 0x7feb94108750 2024-04-26T21:50:56.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.389+0000 7feb99000640 1 -- 172.21.15.1:0/3195284484 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb94074b60 con 0x7feb94108750 2024-04-26T21:50:56.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.393+0000 7feb9d5b0740 1 Processor -- start 2024-04-26T21:50:56.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.394+0000 7feb8be00640 1 -- 172.21.15.1:0/3195284484 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7feb8c005460 con 0x7feb94108750 2024-04-26T21:50:56.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.394+0000 7feb9d5b0740 1 -- start start 2024-04-26T21:50:56.394 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.394+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba01fe0 0x55907ba023c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.394+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba02900 0x55907ba0cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.394+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 0x55907ba0f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.395 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.394+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55907b967380 con 0x55907ba02900 2024-04-26T21:50:56.396 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.394+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55907b903c30 con 0x55907ba01fe0 2024-04-26T21:50:56.396 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.394+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55907b963340 con 0x55907ba0d1c0 2024-04-26T21:50:56.396 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.395+0000 7feb92a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba01fe0 0x55907ba023c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.396 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.395+0000 7feb92a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba01fe0 0x55907ba023c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:58972/0 (socket says 172.21.15.1:58972) 2024-04-26T21:50:56.396 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.395+0000 7feb92a00640 1 -- 172.21.15.1:0/3414461155 learned_addr learned my addr 172.21.15.1:0/3414461155 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:56.396 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.395+0000 7feb92a00640 1 --2- 172.21.15.1:0/3414461155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 0x55907ba0f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.396 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.395+0000 7feb93400640 1 --2- 172.21.15.1:0/3414461155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba02900 0x55907ba0cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.395+0000 7feb8be00640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb780a3db0 0x7feb780a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.396+0000 7feb8be00640 1 -- 172.21.15.1:0/3195284484 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 9626+0+0 (secure 0 0 0) 0x7feb8c0d0bd0 con 0x7feb94108750 2024-04-26T21:50:56.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.396+0000 7feb93400640 1 -- 172.21.15.1:0/3414461155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 msgr2=0x55907ba0f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:56.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.396+0000 7feb93400640 1 --2- 172.21.15.1:0/3414461155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 0x55907ba0f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.396+0000 7feb93400640 1 -- 172.21.15.1:0/3414461155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba01fe0 msgr2=0x55907ba023c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:56.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.396+0000 7feb93400640 1 --2- 172.21.15.1:0/3414461155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba01fe0 0x55907ba023c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.396+0000 7feb93400640 1 -- 172.21.15.1:0/3414461155 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55907b925940 con 0x55907ba02900 2024-04-26T21:50:56.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.396+0000 7feb92a00640 1 --2- 172.21.15.1:0/3414461155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba01fe0 0x55907ba023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:50:56.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.396+0000 7feb93e00640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb780a3db0 0x7feb780a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.397+0000 7feb93400640 1 --2- 172.21.15.1:0/3414461155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba02900 0x55907ba0cc80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7feb8402a960 tx=0x7feb84004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57294fef55da09bc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.397+0000 7feb88c00640 1 -- 172.21.15.1:0/3414461155 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84035040 con 0x55907ba02900 2024-04-26T21:50:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.397+0000 7feb88c00640 1 -- 172.21.15.1:0/3414461155 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb84004c20 con 0x55907ba02900 2024-04-26T21:50:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.397+0000 7feb93e00640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb780a3db0 0x7feb780a6270 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feb8c099140 tx=0x7feb8c005f50 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.397+0000 7feb88c00640 1 -- 172.21.15.1:0/3414461155 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84003a90 con 0x55907ba02900 2024-04-26T21:50:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.398+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3414461155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba02900 msgr2=0x55907ba0cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:56.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.398+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3414461155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba02900 0x55907ba0cc80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7feb8402a960 tx=0x7feb84004560 comp rx=0 tx=0).stop 2024-04-26T21:50:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.398+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3414461155 shutdown_connections 2024-04-26T21:50:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.398+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3414461155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 0x55907ba0f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.398+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3414461155 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba02900 0x55907ba0cc80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.399 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.398+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3414461155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba01fe0 0x55907ba023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.400 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.398+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3414461155 >> 172.21.15.1:0/3414461155 conn(0x55907b9fe4f0 msgr2=0x55907b9fe8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:56.400 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.399+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3414461155 shutdown_connections 2024-04-26T21:50:56.400 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.399+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3414461155 wait complete. 2024-04-26T21:50:56.400 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.399+0000 7feb9d5b0740 1 -- start start 2024-04-26T21:50:56.400 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.399+0000 7feb9d5b0740 1 Processor -- start 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba01fe0 0x55907ba0a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba0a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 0x55907ba0abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55907ba0fd90 con 0x55907ba01fe0 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55907b8eec60 con 0x55907ba02900 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55907ba0ff10 con 0x55907ba0d1c0 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb9d5b0740 10 client.0 _ll_register_callbacks cb 0x55907b91ae50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb93400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba0a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb93400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba0a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:58982/0 (socket says 172.21.15.1:58982) 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb93400640 1 -- 172.21.15.1:0/3987797756 learned_addr learned my addr 172.21.15.1:0/3987797756 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:56.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb92a00640 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba01fe0 0x55907ba0a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb92a00640 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 0x55907ba0abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.400+0000 7feb93400640 1 -- 172.21.15.1:0/3987797756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 msgr2=0x55907ba0abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb93400640 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 0x55907ba0abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb93400640 1 -- 172.21.15.1:0/3987797756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba01fe0 msgr2=0x55907ba0a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb93400640 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba01fe0 0x55907ba0a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb93400640 1 -- 172.21.15.1:0/3987797756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55907b8fd510 con 0x55907ba02900 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb92a00640 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 0x55907ba0abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb93400640 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba0a6b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7feb84005ab0 tx=0x7feb840049a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e318d00c57a6028c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb7d600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb7d600640 1 -- 172.21.15.1:0/3987797756 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84035070 con 0x55907ba02900 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3987797756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb84004c20 con 0x55907ba02900 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb7d600640 1 -- 172.21.15.1:0/3987797756 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9626+0+0 (secure 0 0 0) 0x7feb84005410 con 0x55907ba02900 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb7d600640 1 -- 172.21.15.1:0/3987797756 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7feb84009520 con 0x55907ba02900 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb9d5b0740 10 client.27038 fetch_fsmap learned FSMap version 59 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.401+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3987797756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55907b679210 con 0x55907ba02900 2024-04-26T21:50:56.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.402+0000 7feb7d600640 1 -- 172.21.15.1:0/3987797756 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb84004c20 con 0x55907ba02900 2024-04-26T21:50:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.402+0000 7feb7d600640 1 -- 172.21.15.1:0/3987797756 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7feb8404f570 con 0x55907ba02900 2024-04-26T21:50:56.404 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.402+0000 7feb9d5b0740 10 client.27038 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:50:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.402+0000 7feb9d5b0740 10 client.27038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:50:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.402+0000 7feb9d5b0740 20 client.27038 populate_metadata read hostname 'smithi001' 2024-04-26T21:50:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.403+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55907ba1cda0 0x55907ba1f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.403+0000 7feb9d5b0740 4 client.27038 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:50:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.403+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3987797756 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55907b974f90 con 0x55907ba1cda0 2024-04-26T21:50:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.403+0000 7feb92a00640 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55907ba1cda0 0x55907ba1f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.404+0000 7feb92a00640 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55907ba1cda0 0x55907ba1f190 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1074f5d81b9ceaf8 server_cookie=29a746aa2384192a in_seq=0 out_seq=0 2024-04-26T21:50:56.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.404+0000 7feb7d600640 10 client.27038 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.405+0000 7feb7d600640 1 -- 172.21.15.1:0/3987797756 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55907b974f90 con 0x55907ba1cda0 2024-04-26T21:50:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.405+0000 7feb7d600640 10 client.27038 handle_command_reply: tid=0 2024-04-26T21:50:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 client.27038 shutdown 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3987797756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 msgr2=0x55907ba0a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba0a6b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7feb84005ab0 tx=0x7feb840049a0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3987797756 shutdown_connections 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55907ba1cda0 0x55907ba1f190 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba0d1c0 0x55907ba0abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba0a6b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb7d600640 0 client.27038 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3987797756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba01fe0 0x55907ba0a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3987797756 >> 172.21.15.1:0/3987797756 conn(0x55907b9fe4f0 msgr2=0x55907ba04270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3987797756 shutdown_connections 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3987797756 wait complete. 2024-04-26T21:50:56.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.406+0000 7feb9d5b0740 20 client.27038 trim_cache size 0 max 16384 2024-04-26T21:50:56.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.427+0000 7feb9d5b0740 1 Processor -- start 2024-04-26T21:50:56.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.427+0000 7feb9d5b0740 1 -- start start 2024-04-26T21:50:56.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.427+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba01fe0 0x55907ba221b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.427+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba02900 0x55907ba226f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.427+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba0d1c0 0x55907ba073f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.427+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b8c90 con 0x55907ba01fe0 2024-04-26T21:50:56.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.427+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b6560 con 0x55907ba0d1c0 2024-04-26T21:50:56.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.427+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b43d0 con 0x55907ba02900 2024-04-26T21:50:56.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.427+0000 7feb93400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba02900 0x55907ba226f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.428+0000 7feb92a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba01fe0 0x55907ba221b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.428+0000 7feb93400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba02900 0x55907ba226f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54280/0 (socket says 172.21.15.1:54280) 2024-04-26T21:50:56.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.428+0000 7feb93400640 1 -- 172.21.15.1:0/3210263199 learned_addr learned my addr 172.21.15.1:0/3210263199 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:56.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.428+0000 7feb92a00640 1 --2- 172.21.15.1:0/3210263199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba0d1c0 0x55907ba073f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.428+0000 7feb93400640 1 -- 172.21.15.1:0/3210263199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba0d1c0 msgr2=0x55907ba073f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:56.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.428+0000 7feb93400640 1 --2- 172.21.15.1:0/3210263199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba0d1c0 0x55907ba073f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.428+0000 7feb93400640 1 -- 172.21.15.1:0/3210263199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba01fe0 msgr2=0x55907ba221b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:56.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.428+0000 7feb93400640 1 --2- 172.21.15.1:0/3210263199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba01fe0 0x55907ba221b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.428+0000 7feb93400640 1 -- 172.21.15.1:0/3210263199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55907b745620 con 0x55907ba02900 2024-04-26T21:50:56.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.429+0000 7feb93400640 1 --2- 172.21.15.1:0/3210263199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba02900 0x55907ba226f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7feb8402a960 tx=0x7feb8404fe00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b40b82a5ef65cd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:56.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.429+0000 7feb7fe00640 1 -- 172.21.15.1:0/3210263199 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84035070 con 0x55907ba02900 2024-04-26T21:50:56.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.429+0000 7feb7fe00640 1 -- 172.21.15.1:0/3210263199 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8404a070 con 0x55907ba02900 2024-04-26T21:50:56.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.429+0000 7feb7fe00640 1 -- 172.21.15.1:0/3210263199 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb840035f0 con 0x55907ba02900 2024-04-26T21:50:56.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.430+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3210263199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba02900 msgr2=0x55907ba226f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:56.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.430+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3210263199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba02900 0x55907ba226f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7feb8402a960 tx=0x7feb8404fe00 comp rx=0 tx=0).stop 2024-04-26T21:50:56.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.430+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3210263199 shutdown_connections 2024-04-26T21:50:56.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.430+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3210263199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba02900 0x55907ba226f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.430+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3210263199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba0d1c0 0x55907ba073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.430+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3210263199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba01fe0 0x55907ba221b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.430+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3210263199 >> 172.21.15.1:0/3210263199 conn(0x55907b9fe4f0 msgr2=0x55907ba0c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:56.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.430+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3210263199 shutdown_connections 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.430+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3210263199 wait complete. 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.431+0000 7feb9d5b0740 1 -- start start 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.431+0000 7feb9d5b0740 1 Processor -- start 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.431+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba01fe0 0x55907ba1fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.431+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba200f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.431+0000 7feb9d5b0740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba0d1c0 0x55907ba20630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.431+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b4a20 con 0x55907ba0d1c0 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.431+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b5e90 con 0x55907ba02900 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.432+0000 7feb9d5b0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb941b50f0 con 0x55907ba01fe0 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.432+0000 7feb9d5b0740 10 client.0 _ll_register_callbacks cb 0x7feb800123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.432+0000 7feb93400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba200f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.432+0000 7feb93400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba200f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:59006/0 (socket says 172.21.15.1:59006) 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.432+0000 7feb93400640 1 -- 172.21.15.1:0/3793337650 learned_addr learned my addr 172.21.15.1:0/3793337650 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:50:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.432+0000 7feb92a00640 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba01fe0 0x55907ba1fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.432+0000 7feb93400640 1 -- 172.21.15.1:0/3793337650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba01fe0 msgr2=0x55907ba1fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:56.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.432+0000 7feb93400640 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba01fe0 0x55907ba1fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.432+0000 7feb93400640 1 -- 172.21.15.1:0/3793337650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba0d1c0 msgr2=0x55907ba20630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:50:56.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.432+0000 7feb93400640 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba0d1c0 0x55907ba20630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:56.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.433+0000 7feb93400640 1 -- 172.21.15.1:0/3793337650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55907b921cb0 con 0x55907ba02900 2024-04-26T21:50:56.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.433+0000 7feb93400640 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba200f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7feb8404f910 tx=0x7feb84005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b97066279ec6caf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:50:56.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.433+0000 7feb7d600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:50:56.435 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.433+0000 7feb7d600640 1 -- 172.21.15.1:0/3793337650 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb84035070 con 0x55907ba02900 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.434+0000 7feb7d600640 1 -- 172.21.15.1:0/3793337650 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 9626+0+0 (secure 0 0 0) 0x7feb84005620 con 0x55907ba02900 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.434+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3793337650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb8404a070 con 0x55907ba02900 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.434+0000 7feb7d600640 1 -- 172.21.15.1:0/3793337650 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7feb8406c050 con 0x55907ba02900 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.434+0000 7feb9d5b0740 10 client.27044 fetch_fsmap learned FSMap version 59 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.434+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3793337650 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55907ba83070 con 0x55907ba02900 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.434+0000 7feb7d600640 1 -- 172.21.15.1:0/3793337650 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8404a070 con 0x55907ba02900 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.435+0000 7feb7d600640 1 -- 172.21.15.1:0/3793337650 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7feb84003b30 con 0x55907ba02900 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.435+0000 7feb9d5b0740 10 client.27044 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.435+0000 7feb9d5b0740 10 client.27044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.435+0000 7feb9d5b0740 20 client.27044 populate_metadata read hostname 'smithi001' 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.435+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55907ba83720 0x55907ba85b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.435+0000 7feb9d5b0740 4 client.27044 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.435+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3793337650 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55907b96ee00 con 0x55907ba83720 2024-04-26T21:50:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.435+0000 7feb92a00640 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55907ba83720 0x55907ba85b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:50:56.437 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.436+0000 7feb92a00640 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55907ba83720 0x55907ba85b10 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6967e584a27cda2a server_cookie=b5fc055491968fc9 in_seq=0 out_seq=0 2024-04-26T21:50:56.437 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:56.436+0000 7feb7d600640 10 client.27044 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:56.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:56 smithi038 ceph-mon[32724]: pgmap v654: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 11 MiB/s wr, 649 op/s 2024-04-26T21:50:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:56 smithi186 ceph-mon[32991]: pgmap v654: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 11 MiB/s wr, 649 op/s 2024-04-26T21:50:56.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:56 smithi001 ceph-mon[28054]: pgmap v654: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 11 MiB/s wr, 649 op/s 2024-04-26T21:50:57.186 INFO:tasks.workunit.client.0.smithi001.stdout: 10 169930 21.07 MB/sec execute 294 sec latency 663.580 ms 2024-04-26T21:50:58.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.146+0000 7feb7d600640 1 -- 172.21.15.1:0/3793337650 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55907b96ee00 con 0x55907ba83720 2024-04-26T21:50:58.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.146+0000 7feb7d600640 10 client.27044 handle_command_reply: tid=0 2024-04-26T21:50:58.146 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.146+0000 7feb9d5b0740 1 client.27044 shutdown 2024-04-26T21:50:58.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.146+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3793337650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 msgr2=0x55907ba200f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:58.147 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.146+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba200f0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7feb8404f910 tx=0x7feb84005850 comp rx=0 tx=0).stop 2024-04-26T21:50:58.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.148+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3793337650 shutdown_connections 2024-04-26T21:50:58.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.148+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55907ba83720 0x55907ba85b10 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:58.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.148+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55907ba0d1c0 0x55907ba20630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:58.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.148+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55907ba02900 0x55907ba200f0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:58.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.148+0000 7feb9d5b0740 1 --2- 172.21.15.1:0/3793337650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55907ba01fe0 0x55907ba1fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:58.148 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.148+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3793337650 >> 172.21.15.1:0/3793337650 conn(0x55907b9fe4f0 msgr2=0x55907ba0a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:58.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.148+0000 7feb7d600640 0 client.27044 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:50:58.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.148+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3793337650 shutdown_connections 2024-04-26T21:50:58.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.148+0000 7feb9d5b0740 1 -- 172.21.15.1:0/3793337650 wait complete. 2024-04-26T21:50:58.149 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.149+0000 7feb9d5b0740 20 client.27044 trim_cache size 0 max 16384 2024-04-26T21:50:58.150 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:50:58.150 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:50:58.150 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:50:58.150 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 -- 172.21.15.1:0/3195284484 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb780a3db0 msgr2=0x7feb780a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb780a3db0 0x7feb780a6270 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feb8c099140 tx=0x7feb8c005f50 comp rx=0 tx=0).stop 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 -- 172.21.15.1:0/3195284484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb94108750 msgr2=0x7feb9419d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb94108750 0x7feb9419d850 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7feb8c00e940 tx=0x7feb8c00ee00 comp rx=0 tx=0).stop 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 -- 172.21.15.1:0/3195284484 shutdown_connections 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7feb780a3db0 0x7feb780a6270 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb94108750 0x7feb9419d850 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb94103100 0x7feb94199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 --2- 172.21.15.1:0/3195284484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feb94102740 0x7feb94198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 -- 172.21.15.1:0/3195284484 >> 172.21.15.1:0/3195284484 conn(0x7feb940fe270 msgr2=0x7feb940feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:50:58.151 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 -- 172.21.15.1:0/3195284484 shutdown_connections 2024-04-26T21:50:58.152 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:50:58.150+0000 7feb99000640 1 -- 172.21.15.1:0/3195284484 wait complete. 2024-04-26T21:50:58.186 INFO:tasks.workunit.client.0.smithi001.stdout: 10 170044 21.02 MB/sec execute 295 sec latency 1350.708 ms 2024-04-26T21:50:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:50:58 smithi186 ceph-mon[32991]: pgmap v655: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 6.5 MiB/s wr, 440 op/s 2024-04-26T21:50:58.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:50:58 smithi001 ceph-mon[28054]: pgmap v655: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 6.5 MiB/s wr, 440 op/s 2024-04-26T21:50:58.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:50:58 smithi038 ceph-mon[32724]: pgmap v655: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 6.5 MiB/s wr, 440 op/s 2024-04-26T21:50:59.186 INFO:tasks.workunit.client.0.smithi001.stdout: 10 170285 20.98 MB/sec execute 296 sec latency 1685.697 ms 2024-04-26T21:51:00.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 170641 20.98 MB/sec execute 297 sec latency 168.378 ms 2024-04-26T21:51:00.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:00 smithi186 ceph-mon[32991]: pgmap v656: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.6 MiB/s wr, 310 op/s 2024-04-26T21:51:00.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:00 smithi001 ceph-mon[28054]: pgmap v656: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.6 MiB/s wr, 310 op/s 2024-04-26T21:51:00.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:00 smithi038 ceph-mon[32724]: pgmap v656: 129 pgs: 129 active+clean; 1.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.6 MiB/s wr, 310 op/s 2024-04-26T21:51:01.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 170991 20.98 MB/sec execute 298 sec latency 566.738 ms 2024-04-26T21:51:02.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 171308 20.96 MB/sec execute 299 sec latency 900.277 ms 2024-04-26T21:51:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:02 smithi186 ceph-mon[32991]: pgmap v657: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 579 op/s 2024-04-26T21:51:02.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:02 smithi001 ceph-mon[28054]: pgmap v657: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 579 op/s 2024-04-26T21:51:02.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:02 smithi038 ceph-mon[32724]: pgmap v657: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 9.6 MiB/s wr, 579 op/s 2024-04-26T21:51:03.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 171470 20.92 MB/sec execute 300 sec latency 491.191 ms 2024-04-26T21:51:04.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 171478 20.86 MB/sec execute 301 sec latency 441.790 ms 2024-04-26T21:51:04.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:04 smithi186 ceph-mon[32991]: pgmap v658: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 527 op/s 2024-04-26T21:51:04.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:04 smithi001 ceph-mon[28054]: pgmap v658: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 527 op/s 2024-04-26T21:51:04.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:04 smithi038 ceph-mon[32724]: pgmap v658: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 9.0 MiB/s wr, 527 op/s 2024-04-26T21:51:05.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 171485 20.80 MB/sec execute 302 sec latency 432.231 ms 2024-04-26T21:51:06.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 171500 20.74 MB/sec execute 303 sec latency 410.858 ms 2024-04-26T21:51:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:06 smithi186 ceph-mon[32991]: pgmap v659: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-04-26T21:51:06.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:06 smithi001 ceph-mon[28054]: pgmap v659: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-04-26T21:51:06.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:06 smithi038 ceph-mon[32724]: pgmap v659: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.3 MiB/s wr, 629 op/s 2024-04-26T21:51:07.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 171790 20.73 MB/sec execute 304 sec latency 357.961 ms 2024-04-26T21:51:08.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 172024 20.71 MB/sec execute 305 sec latency 232.965 ms 2024-04-26T21:51:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:08 smithi186 ceph-mon[32991]: pgmap v660: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.8 MiB/s wr, 581 op/s 2024-04-26T21:51:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:08 smithi186 ceph-mon[32991]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T21:51:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:08 smithi001 ceph-mon[28054]: pgmap v660: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.8 MiB/s wr, 581 op/s 2024-04-26T21:51:08.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:08 smithi001 ceph-mon[28054]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T21:51:08.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:08 smithi038 ceph-mon[32724]: pgmap v660: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.8 MiB/s wr, 581 op/s 2024-04-26T21:51:08.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:08 smithi038 ceph-mon[32724]: osdmap e130: 12 total, 12 up, 12 in 2024-04-26T21:51:09.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 172186 20.67 MB/sec execute 306 sec latency 202.367 ms 2024-04-26T21:51:10.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 172347 20.64 MB/sec execute 307 sec latency 262.496 ms 2024-04-26T21:51:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:10 smithi186 ceph-mon[32991]: pgmap v662: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 8.6 MiB/s wr, 637 op/s 2024-04-26T21:51:10.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:10 smithi001 ceph-mon[28054]: pgmap v662: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 8.6 MiB/s wr, 637 op/s 2024-04-26T21:51:10.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:10 smithi038 ceph-mon[32724]: pgmap v662: 129 pgs: 129 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 8.6 MiB/s wr, 637 op/s 2024-04-26T21:51:11.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 172514 20.58 MB/sec execute 308 sec latency 188.887 ms 2024-04-26T21:51:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.378+0000 7f82c7000640 1 -- 172.21.15.1:0/760967825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b8009750 msgr2=0x7f82b8007ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.378+0000 7f82c7000640 1 --2- 172.21.15.1:0/760967825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b8009750 0x7f82b8007ae0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009900 tx=0x7f82b402a420 comp rx=0 tx=0).stop 2024-04-26T21:51:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.378+0000 7f82c7000640 1 -- 172.21.15.1:0/760967825 shutdown_connections 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.378+0000 7f82c7000640 1 --2- 172.21.15.1:0/760967825 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b80a6b80 0x7f82b80a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.378+0000 7f82c7000640 1 --2- 172.21.15.1:0/760967825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82b8008020 0x7f82b80084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.378+0000 7f82c7000640 1 --2- 172.21.15.1:0/760967825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b8009750 0x7f82b8007ae0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.378+0000 7f82c7000640 1 -- 172.21.15.1:0/760967825 >> 172.21.15.1:0/760967825 conn(0x7f82b800b890 msgr2=0x7f82b800bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.379+0000 7f82c7000640 1 -- 172.21.15.1:0/760967825 shutdown_connections 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.379+0000 7f82c7000640 1 -- 172.21.15.1:0/760967825 wait complete. 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.380+0000 7f82c7000640 1 Processor -- start 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.381+0000 7f82c7000640 1 -- start start 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.382+0000 7f82c7000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82b8008020 0x7f82b8138810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.383+0000 7f82c7000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b8009750 0x7f82b8138d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.383+0000 7f82c7000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b80a6b80 0x7f82b81329c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.383+0000 7f82c7000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f82b80ac620 con 0x7f82b8009750 2024-04-26T21:51:11.384 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.383+0000 7f82c7000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f82b80ac4a0 con 0x7f82b8008020 2024-04-26T21:51:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.383+0000 7f82c7000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f82b80ac7a0 con 0x7f82b80a6b80 2024-04-26T21:51:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.383+0000 7f82c6600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b80a6b80 0x7f82b81329c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.383+0000 7f82c6600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b80a6b80 0x7f82b81329c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:38480/0 (socket says 172.21.15.1:38480) 2024-04-26T21:51:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.383+0000 7f82c6600640 1 -- 172.21.15.1:0/119285690 learned_addr learned my addr 172.21.15.1:0/119285690 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.384+0000 7f82c5200640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b8009750 0x7f82b8138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.384+0000 7f82c6600640 1 -- 172.21.15.1:0/119285690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82b8008020 msgr2=0x7f82b8138810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.384+0000 7f82c6600640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82b8008020 0x7f82b8138810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.384+0000 7f82c6600640 1 -- 172.21.15.1:0/119285690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b8009750 msgr2=0x7f82b8138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.384+0000 7f82c6600640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b8009750 0x7f82b8138d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.384+0000 7f82c6600640 1 -- 172.21.15.1:0/119285690 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b8133140 con 0x7f82b80a6b80 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.384+0000 7f82c6600640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b80a6b80 0x7f82b81329c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f82c00680a0 tx=0x7f82c006bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.385+0000 7f82be000640 1 -- 172.21.15.1:0/119285690 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c007f650 con 0x7f82b80a6b80 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.385+0000 7f82be000640 1 -- 172.21.15.1:0/119285690 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82c007fca0 con 0x7f82b80a6b80 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.385+0000 7f82c7000640 1 -- 172.21.15.1:0/119285690 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b8133430 con 0x7f82b80a6b80 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.385+0000 7f82be000640 1 -- 172.21.15.1:0/119285690 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c0080db0 con 0x7f82b80a6b80 2024-04-26T21:51:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.385+0000 7f82c7000640 1 -- 172.21.15.1:0/119285690 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b813f6d0 con 0x7f82b80a6b80 2024-04-26T21:51:11.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.390+0000 7f82cb3e6740 1 Processor -- start 2024-04-26T21:51:11.392 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.392+0000 7f82be000640 1 -- 172.21.15.1:0/119285690 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f82c0087030 con 0x7f82b80a6b80 2024-04-26T21:51:11.397 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.397+0000 7f82be000640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82a80a3da0 0x7f82a80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.397+0000 7f82be000640 1 -- 172.21.15.1:0/119285690 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 9730+0+0 (secure 0 0 0) 0x7f82c013c8f0 con 0x7f82b80a6b80 2024-04-26T21:51:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.398+0000 7f82cb3e6740 1 -- start start 2024-04-26T21:51:11.399 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.398+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.399 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.398+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7c1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.399 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.398+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7c4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.399 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.398+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55dd9f71c380 con 0x55dd9f7c21a0 2024-04-26T21:51:11.400 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.398+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55dd9f6b8c30 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.400 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.398+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55dd9f718340 con 0x55dd9f7b78e0 2024-04-26T21:51:11.400 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.399+0000 7f82c5c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.400 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.399+0000 7f82c5c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:49338/0 (socket says 172.21.15.1:49338) 2024-04-26T21:51:11.400 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.399+0000 7f82c5c00640 1 -- 172.21.15.1:0/1880369841 learned_addr learned my addr 172.21.15.1:0/1880369841 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:11.401 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.399+0000 7f82c6600640 1 --2- 172.21.15.1:0/1880369841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.401 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.399+0000 7f82c5c00640 1 -- 172.21.15.1:0/1880369841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b78e0 msgr2=0x55dd9f7c1c60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:51:11.401 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.399+0000 7f82c5c00640 1 --2- 172.21.15.1:0/1880369841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7c1c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.401 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.399+0000 7f82c5c00640 1 -- 172.21.15.1:0/1880369841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7c21a0 msgr2=0x55dd9f7c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:11.401 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.399+0000 7f82c5200640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82a80a3da0 0x7f82a80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.399+0000 7f82c5c00640 1 --2- 172.21.15.1:0/1880369841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7c4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.400+0000 7f82c5c00640 1 -- 172.21.15.1:0/1880369841 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd9f6da940 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.400+0000 7f82c6600640 1 --2- 172.21.15.1:0/1880369841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7c4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:51:11.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.401+0000 7f82c5c00640 1 --2- 172.21.15.1:0/1880369841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b73a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f82b4005ab0 tx=0x7f82b4005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ddcca28dd473433 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:11.402 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.401+0000 7f82c5200640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82a80a3da0 0x7f82a80a6260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55dd9f7b8560 tx=0x7f82ac005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:11.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.402+0000 7f82b2a00640 1 -- 172.21.15.1:0/1880369841 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b403e070 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.402+0000 7f82b2a00640 1 -- 172.21.15.1:0/1880369841 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82b4004400 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.403 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.402+0000 7f82b2a00640 1 -- 172.21.15.1:0/1880369841 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b4035040 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.403+0000 7f82cb3e6740 1 -- 172.21.15.1:0/1880369841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b6fc0 msgr2=0x55dd9f7b73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.403+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/1880369841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b73a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f82b4005ab0 tx=0x7f82b4005da0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.404+0000 7f82cb3e6740 1 -- 172.21.15.1:0/1880369841 shutdown_connections 2024-04-26T21:51:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.404+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/1880369841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7c1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.404+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/1880369841 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7c4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.404+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/1880369841 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b73a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.404+0000 7f82cb3e6740 1 -- 172.21.15.1:0/1880369841 >> 172.21.15.1:0/1880369841 conn(0x55dd9f7b34d0 msgr2=0x55dd9f7b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:11.405 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.404+0000 7f82cb3e6740 1 -- 172.21.15.1:0/1880369841 shutdown_connections 2024-04-26T21:51:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.404+0000 7f82cb3e6740 1 -- 172.21.15.1:0/1880369841 wait complete. 2024-04-26T21:51:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.405+0000 7f82cb3e6740 1 -- start start 2024-04-26T21:51:11.406 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.405+0000 7f82cb3e6740 1 Processor -- start 2024-04-26T21:51:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.407+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7c1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.407+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7befc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.409 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.407+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7bf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.407+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55dd9f7c4d70 con 0x55dd9f7c21a0 2024-04-26T21:51:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.407+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55dd9f6a3c60 con 0x55dd9f7b78e0 2024-04-26T21:51:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.407+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55dd9f7c4ef0 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.407+0000 7f82cb3e6740 10 client.0 _ll_register_callbacks cb 0x55dd9f6cfe50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:51:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.408+0000 7f82c5c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7c1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.408+0000 7f82c5c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7c1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:38490/0 (socket says 172.21.15.1:38490) 2024-04-26T21:51:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.408+0000 7f82c5c00640 1 -- 172.21.15.1:0/2240722583 learned_addr learned my addr 172.21.15.1:0/2240722583 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.408+0000 7f82c5200640 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7befc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.409+0000 7f82c5c00640 1 -- 172.21.15.1:0/2240722583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 msgr2=0x55dd9f7befc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.409+0000 7f82c5c00640 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7befc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.409+0000 7f82c5c00640 1 -- 172.21.15.1:0/2240722583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7c21a0 msgr2=0x55dd9f7bf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.409+0000 7f82c5c00640 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7bf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.409+0000 7f82c5c00640 1 -- 172.21.15.1:0/2240722583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd9f6b2510 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.409+0000 7f82c5200640 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7befc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:51:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.409+0000 7f82c5c00640 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7c1c30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f82b4000c00 tx=0x7f82b402aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=333b06ae837bf1ca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.409+0000 7f82a7400640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:51:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.412+0000 7f82a7400640 1 -- 172.21.15.1:0/2240722583 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b404b070 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.413 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.412+0000 7f82a7400640 1 -- 172.21.15.1:0/2240722583 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9730+0+0 (secure 0 0 0) 0x7f82b4005250 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.413 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.412+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2240722583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82b4004400 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.413 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.413+0000 7f82a7400640 1 -- 172.21.15.1:0/2240722583 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f82b403e030 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.413 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.413+0000 7f82cb3e6740 10 client.27067 fetch_fsmap learned FSMap version 59 2024-04-26T21:51:11.414 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.413+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2240722583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55dd9f42e210 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.415 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.415+0000 7f82a7400640 1 -- 172.21.15.1:0/2240722583 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82b4004400 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.416+0000 7f82a7400640 1 -- 172.21.15.1:0/2240722583 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7f82b40314a0 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.416 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.416+0000 7f82cb3e6740 10 client.27067 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:51:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.416+0000 7f82cb3e6740 10 client.27067 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:51:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.416+0000 7f82cb3e6740 20 client.27067 populate_metadata read hostname 'smithi001' 2024-04-26T21:51:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.416+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55dd9f7d1bc0 0x55dd9f7d3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.416+0000 7f82cb3e6740 4 client.27067 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:51:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.416+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2240722583 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd9f729f90 con 0x55dd9f7d1bc0 2024-04-26T21:51:11.417 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.416+0000 7f82c5200640 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55dd9f7d1bc0 0x55dd9f7d3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.417+0000 7f82c5200640 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55dd9f7d1bc0 0x55dd9f7d3fb0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e16c907fb75e6e3 server_cookie=21fda6159f4fa8e1 in_seq=0 out_seq=0 2024-04-26T21:51:11.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.417+0000 7f82a7400640 10 client.27067 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:11.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.418+0000 7f82a7400640 1 -- 172.21.15.1:0/2240722583 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd9f729f90 con 0x55dd9f7d1bc0 2024-04-26T21:51:11.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.418+0000 7f82a7400640 10 client.27067 handle_command_reply: tid=0 2024-04-26T21:51:11.418 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.418+0000 7f82cb3e6740 1 client.27067 shutdown 2024-04-26T21:51:11.419 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.419+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2240722583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 msgr2=0x55dd9f7c1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:11.419 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.419+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7c1c30 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f82b4000c00 tx=0x7f82b402aba0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.421 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.421+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2240722583 shutdown_connections 2024-04-26T21:51:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.421+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55dd9f7d1bc0 0x55dd9f7d3fb0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.421+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7bf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.421+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7befc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.421+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2240722583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7c1c30 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.421+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2240722583 >> 172.21.15.1:0/2240722583 conn(0x55dd9f7b34d0 msgr2=0x55dd9f7b9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.421+0000 7f82a7400640 0 client.27067 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.424+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2240722583 shutdown_connections 2024-04-26T21:51:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.424+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2240722583 wait complete. 2024-04-26T21:51:11.424 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.424+0000 7f82cb3e6740 20 client.27067 trim_cache size 0 max 16384 2024-04-26T21:51:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.461+0000 7f82cb3e6740 1 Processor -- start 2024-04-26T21:51:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.462+0000 7f82cb3e6740 1 -- start start 2024-04-26T21:51:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.463+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.463+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7d96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.463+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7d9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.463+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f82b814c1a0 con 0x55dd9f7b78e0 2024-04-26T21:51:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.463+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f82b8149a70 con 0x55dd9f7c21a0 2024-04-26T21:51:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.463+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f82b8147970 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.464+0000 7f82c6600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7d9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.464+0000 7f82c6600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7d9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:49352/0 (socket says 172.21.15.1:49352) 2024-04-26T21:51:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.464+0000 7f82c6600640 1 -- 172.21.15.1:0/2025769120 learned_addr learned my addr 172.21.15.1:0/2025769120 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.464+0000 7f82c5c00640 1 --2- 172.21.15.1:0/2025769120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.464+0000 7f82c5200640 1 --2- 172.21.15.1:0/2025769120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7d96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.465+0000 7f82c5c00640 1 -- 172.21.15.1:0/2025769120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7c21a0 msgr2=0x55dd9f7d9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.465+0000 7f82c5c00640 1 --2- 172.21.15.1:0/2025769120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7d9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.465+0000 7f82c5c00640 1 -- 172.21.15.1:0/2025769120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7b78e0 msgr2=0x55dd9f7d96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.465+0000 7f82c5c00640 1 --2- 172.21.15.1:0/2025769120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7d96d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.466+0000 7f82c5c00640 1 -- 172.21.15.1:0/2025769120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd9f7bfa20 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.467+0000 7f82c5c00640 1 --2- 172.21.15.1:0/2025769120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b9080 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009850 tx=0x7f82b402a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1ac465363cc869c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.468+0000 7f82b2000640 1 -- 172.21.15.1:0/2025769120 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b404b070 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.468+0000 7f82b2000640 1 -- 172.21.15.1:0/2025769120 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82b4004990 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.468+0000 7f82b2000640 1 -- 172.21.15.1:0/2025769120 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b404b070 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.470+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2025769120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 msgr2=0x55dd9f7b9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.470+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2025769120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b9080 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009850 tx=0x7f82b402a420 comp rx=0 tx=0).stop 2024-04-26T21:51:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.470+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2025769120 shutdown_connections 2024-04-26T21:51:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.470+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2025769120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7d9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.470+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2025769120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7d96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.470+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2025769120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7b9080 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.470+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2025769120 >> 172.21.15.1:0/2025769120 conn(0x55dd9f7b34d0 msgr2=0x55dd9f7b8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.470+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2025769120 shutdown_connections 2024-04-26T21:51:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.470+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2025769120 wait complete. 2024-04-26T21:51:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.471+0000 7f82cb3e6740 1 -- start start 2024-04-26T21:51:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.471+0000 7f82cb3e6740 1 Processor -- start 2024-04-26T21:51:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.472+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.472+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.472+0000 7f82cb3e6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7c6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.472+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f82b8147f30 con 0x55dd9f7b6fc0 2024-04-26T21:51:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.472+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f82b81493a0 con 0x55dd9f7b78e0 2024-04-26T21:51:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.472+0000 7f82cb3e6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f82b8148600 con 0x55dd9f7c21a0 2024-04-26T21:51:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.472+0000 7f82cb3e6740 10 client.0 _ll_register_callbacks cb 0x7f82b4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:51:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.472+0000 7f82c5200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.472+0000 7f82c5200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:49364/0 (socket says 172.21.15.1:49364) 2024-04-26T21:51:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.473+0000 7f82c5200640 1 -- 172.21.15.1:0/2740767024 learned_addr learned my addr 172.21.15.1:0/2740767024 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.473+0000 7f82c6600640 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7c6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.473+0000 7f82c5200640 1 -- 172.21.15.1:0/2740767024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7c21a0 msgr2=0x55dd9f7c6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.473+0000 7f82c5200640 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7c6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.473+0000 7f82c5200640 1 -- 172.21.15.1:0/2740767024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7b6fc0 msgr2=0x55dd9f7ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.473+0000 7f82c5200640 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.473+0000 7f82c5200640 1 -- 172.21.15.1:0/2740767024 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd9f4fa620 con 0x55dd9f7b78e0 2024-04-26T21:51:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.474+0000 7f82c5200640 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7c5a70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f82b80a6210 tx=0x7f82ac004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e54c2cd342f8ab54 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.474+0000 7f82a7400640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:51:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.475+0000 7f82a7400640 1 -- 172.21.15.1:0/2740767024 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ac028dc0 con 0x55dd9f7b78e0 2024-04-26T21:51:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.475+0000 7f82a7400640 1 -- 172.21.15.1:0/2740767024 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(130..130 src has 1..130) ==== 9730+0+0 (secure 0 0 0) 0x7f82ac004020 con 0x55dd9f7b78e0 2024-04-26T21:51:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.475+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2740767024 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82b4004990 con 0x55dd9f7b78e0 2024-04-26T21:51:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.476+0000 7f82a7400640 1 -- 172.21.15.1:0/2740767024 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f82ac035050 con 0x55dd9f7b78e0 2024-04-26T21:51:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.476+0000 7f82cb3e6740 10 client.27059 fetch_fsmap learned FSMap version 59 2024-04-26T21:51:11.477 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.476+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2740767024 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55dd9f6d6cb0 con 0x55dd9f7b78e0 2024-04-26T21:51:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.477+0000 7f82a7400640 1 -- 172.21.15.1:0/2740767024 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82b4004990 con 0x55dd9f7b78e0 2024-04-26T21:51:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.477+0000 7f82a7400640 1 -- 172.21.15.1:0/2740767024 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7f82ac00f040 con 0x55dd9f7b78e0 2024-04-26T21:51:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.477+0000 7f82cb3e6740 10 client.27059 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:51:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.477+0000 7f82cb3e6740 10 client.27059 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:51:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.477+0000 7f82cb3e6740 20 client.27059 populate_metadata read hostname 'smithi001' 2024-04-26T21:51:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.477+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55dd9f838590 0x55dd9f83a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.477+0000 7f82cb3e6740 4 client.27059 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:51:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.477+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2740767024 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dd9f723e00 con 0x55dd9f838590 2024-04-26T21:51:11.478 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.478+0000 7f82c5c00640 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55dd9f838590 0x55dd9f83a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.478+0000 7f82c5c00640 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55dd9f838590 0x55dd9f83a9b0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee13a0cb5cf863ab server_cookie=6af2e0fab356eddb in_seq=0 out_seq=0 2024-04-26T21:51:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:11.478+0000 7f82a7400640 10 client.27059 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:11.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:11 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:51:11.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:11 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:51:11.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:11 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:51:12.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 172602 20.52 MB/sec execute 309 sec latency 597.073 ms 2024-04-26T21:51:12.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.353+0000 7f82a7400640 1 -- 172.21.15.1:0/2740767024 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dd9f723e00 con 0x55dd9f838590 2024-04-26T21:51:12.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.354+0000 7f82a7400640 10 client.27059 handle_command_reply: tid=0 2024-04-26T21:51:12.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.354+0000 7f82cb3e6740 1 client.27059 shutdown 2024-04-26T21:51:12.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.355+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2740767024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 msgr2=0x55dd9f7c5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:12.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.355+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7c5a70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f82b80a6210 tx=0x7f82ac004730 comp rx=0 tx=0).stop 2024-04-26T21:51:12.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.356+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2740767024 shutdown_connections 2024-04-26T21:51:12.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.356+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55dd9f838590 0x55dd9f83a9b0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:12.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.356+0000 7f82a7400640 0 client.27059 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:12.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.356+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd9f7c21a0 0x55dd9f7c6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:12.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.356+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dd9f7b78e0 0x55dd9f7c5a70 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:12.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.356+0000 7f82cb3e6740 1 --2- 172.21.15.1:0/2740767024 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55dd9f7b6fc0 0x55dd9f7ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:12.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.356+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2740767024 >> 172.21.15.1:0/2740767024 conn(0x55dd9f7b34d0 msgr2=0x55dd9f7b9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:12.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.356+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2740767024 shutdown_connections 2024-04-26T21:51:12.357 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.357+0000 7f82cb3e6740 1 -- 172.21.15.1:0/2740767024 wait complete. 2024-04-26T21:51:12.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.357+0000 7f82cb3e6740 20 client.27059 trim_cache size 0 max 16384 2024-04-26T21:51:12.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:51:12.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:51:12.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:51:12.358 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:51:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.358+0000 7f82c7000640 1 -- 172.21.15.1:0/119285690 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82a80a3da0 msgr2=0x7f82a80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.358+0000 7f82c7000640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82a80a3da0 0x7f82a80a6260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x55dd9f7b8560 tx=0x7f82ac005f90 comp rx=0 tx=0).stop 2024-04-26T21:51:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.358+0000 7f82c7000640 1 -- 172.21.15.1:0/119285690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b80a6b80 msgr2=0x7f82b81329c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.358+0000 7f82c7000640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b80a6b80 0x7f82b81329c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f82c00680a0 tx=0x7f82c006bd30 comp rx=0 tx=0).stop 2024-04-26T21:51:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.358+0000 7f82c7000640 1 -- 172.21.15.1:0/119285690 shutdown_connections 2024-04-26T21:51:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.359+0000 7f82c7000640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f82a80a3da0 0x7f82a80a6260 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.359+0000 7f82c7000640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82b80a6b80 0x7f82b81329c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.359+0000 7f82c7000640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82b8009750 0x7f82b8138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.359+0000 7f82c7000640 1 --2- 172.21.15.1:0/119285690 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82b8008020 0x7f82b8138810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.359+0000 7f82c7000640 1 -- 172.21.15.1:0/119285690 >> 172.21.15.1:0/119285690 conn(0x7f82b800b890 msgr2=0x7f82b80a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:12.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.359+0000 7f82c7000640 1 -- 172.21.15.1:0/119285690 shutdown_connections 2024-04-26T21:51:12.360 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:12.359+0000 7f82c7000640 1 -- 172.21.15.1:0/119285690 wait complete. 2024-04-26T21:51:12.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:12 smithi186 ceph-mon[32991]: pgmap v663: 129 pgs: 13 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 85 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 435 op/s 2024-04-26T21:51:12.804 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:12 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.804 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:12 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.805 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:12 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.805 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:12 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.805 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:12 smithi001 ceph-mon[28054]: pgmap v663: 129 pgs: 13 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 85 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 435 op/s 2024-04-26T21:51:12.805 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:12 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.805 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:12 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.806 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:12 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.806 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:12 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:12 smithi038 ceph-mon[32724]: pgmap v663: 129 pgs: 13 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 85 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 435 op/s 2024-04-26T21:51:12.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:12 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:12 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:12 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:12.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:12 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:13.187 INFO:tasks.workunit.client.0.smithi001.stdout: 10 172738 20.47 MB/sec execute 310 sec latency 603.313 ms 2024-04-26T21:51:14.188 INFO:tasks.workunit.client.0.smithi001.stdout: 10 172849 20.41 MB/sec execute 311 sec latency 290.176 ms 2024-04-26T21:51:14.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:13 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:14.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:13 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:14.204 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:13 smithi186 ceph-mon[32991]: pgmap v664: 129 pgs: 21 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 51 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 388 op/s 2024-04-26T21:51:14.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:13 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:14.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:13 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:14.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:13 smithi001 ceph-mon[28054]: pgmap v664: 129 pgs: 21 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 51 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 388 op/s 2024-04-26T21:51:14.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:13 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:14.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:13 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:14.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:13 smithi038 ceph-mon[32724]: pgmap v664: 129 pgs: 21 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 51 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.3 MiB/s wr, 388 op/s 2024-04-26T21:51:15.188 INFO:tasks.workunit.client.0.smithi001.stdout: 10 172891 20.36 MB/sec execute 312 sec latency 853.848 ms 2024-04-26T21:51:15.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:15 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:15.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:15 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:15.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:15 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:15 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:15 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.318 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:15 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:15 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:15.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:15 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:15.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:15 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:15 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:15 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:15 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:15 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:15 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:15 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:15 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:15 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:15.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:15 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:16.188 INFO:tasks.workunit.client.0.smithi001.stdout: 10 173058 20.33 MB/sec execute 313 sec latency 898.205 ms 2024-04-26T21:51:16.326 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:16 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T21:51:16.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:16 smithi038 ceph-mon[32724]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:51:16.327 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:16 smithi038 ceph-mon[32724]: pgmap v665: 129 pgs: 17 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 65 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.2 MiB/s wr, 359 op/s 2024-04-26T21:51:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:16 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T21:51:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:16 smithi186 ceph-mon[32991]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:51:16.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:16 smithi186 ceph-mon[32991]: pgmap v665: 129 pgs: 17 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 65 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.2 MiB/s wr, 359 op/s 2024-04-26T21:51:16.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:16 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-26T21:51:16.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:16 smithi001 ceph-mon[28054]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-26T21:51:16.570 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:16 smithi001 ceph-mon[28054]: pgmap v665: 129 pgs: 17 active+clean+snaptrim, 47 active+clean+snaptrim_wait, 65 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.2 MiB/s wr, 359 op/s 2024-04-26T21:51:17.188 INFO:tasks.workunit.client.0.smithi001.stdout: 10 173306 20.31 MB/sec execute 314 sec latency 224.017 ms 2024-04-26T21:51:18.188 INFO:tasks.workunit.client.0.smithi001.stdout: 10 173607 20.30 MB/sec execute 315 sec latency 206.727 ms 2024-04-26T21:51:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:18.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:18 smithi186 ceph-mon[32991]: pgmap v666: 129 pgs: 12 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 92 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 6.2 MiB/s wr, 345 op/s 2024-04-26T21:51:18.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:18 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:18.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:18.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:18.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:18 smithi001 ceph-mon[28054]: pgmap v666: 129 pgs: 12 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 92 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 6.2 MiB/s wr, 345 op/s 2024-04-26T21:51:18.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:18 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:18.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:18.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:18 smithi038 ceph-mon[32724]: pgmap v666: 129 pgs: 12 active+clean+snaptrim, 25 active+clean+snaptrim_wait, 92 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 6.2 MiB/s wr, 345 op/s 2024-04-26T21:51:18.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:18 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:19.188 INFO:tasks.workunit.client.0.smithi001.stdout: 10 174019 20.29 MB/sec execute 316 sec latency 118.443 ms 2024-04-26T21:51:19.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:19 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:19.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:19 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:19.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:19 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:19.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:19 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T21:51:19.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:19 smithi186 ceph-mon[32991]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:51:19.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:19 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:19 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:19 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:19 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T21:51:19.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:19 smithi001 ceph-mon[28054]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:51:19.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:19 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:19.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:19 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:19.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:19 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:19.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:19 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-04-26T21:51:19.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:19 smithi038 ceph-mon[32724]: Unable to set osd_memory_target on smithi038 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-04-26T21:51:20.188 INFO:tasks.workunit.client.0.smithi001.stdout: 10 174196 20.25 MB/sec execute 317 sec latency 597.819 ms 2024-04-26T21:51:20.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:20 smithi186 ceph-mon[32991]: pgmap v667: 129 pgs: 10 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 98 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.4 MiB/s wr, 301 op/s 2024-04-26T21:51:20.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:20 smithi001 ceph-mon[28054]: pgmap v667: 129 pgs: 10 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 98 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.4 MiB/s wr, 301 op/s 2024-04-26T21:51:20.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:20 smithi038 ceph-mon[32724]: pgmap v667: 129 pgs: 10 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 98 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.4 MiB/s wr, 301 op/s 2024-04-26T21:51:21.189 INFO:tasks.workunit.client.0.smithi001.stdout: 10 174640 20.25 MB/sec execute 318 sec latency 131.175 ms 2024-04-26T21:51:21.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:21 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:21.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:21 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:21.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:21 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:21.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:21 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:21.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:21 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:21.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:21 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:22.189 INFO:tasks.workunit.client.0.smithi001.stdout: 10 175225 20.33 MB/sec execute 319 sec latency 79.750 ms 2024-04-26T21:51:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:22 smithi186 ceph-mon[32991]: pgmap v668: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 7.9 MiB/s wr, 426 op/s 2024-04-26T21:51:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:51:22.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:51:22.455 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:22.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:22 smithi038 ceph-mon[32724]: pgmap v668: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 7.9 MiB/s wr, 426 op/s 2024-04-26T21:51:22.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.477 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:51:22.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:51:22.478 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:22 smithi001 ceph-mon[28054]: pgmap v668: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 7.9 MiB/s wr, 426 op/s 2024-04-26T21:51:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-26T21:51:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:51:22.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:51:22.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:51:23.189 INFO:tasks.workunit.client.0.smithi001.stdout: 10 175652 20.35 MB/sec execute 320 sec latency 90.599 ms 2024-04-26T21:51:23.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:23 smithi186 ceph-mon[32991]: Adjusting osd_memory_target on smithi001 to 177.5M 2024-04-26T21:51:23.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:23 smithi186 ceph-mon[32991]: Unable to set osd_memory_target on smithi001 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:51:23.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:23 smithi001 ceph-mon[28054]: Adjusting osd_memory_target on smithi001 to 177.5M 2024-04-26T21:51:23.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:23 smithi001 ceph-mon[28054]: Unable to set osd_memory_target on smithi001 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:51:23.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:23 smithi038 ceph-mon[32724]: Adjusting osd_memory_target on smithi001 to 177.5M 2024-04-26T21:51:23.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:23 smithi038 ceph-mon[32724]: Unable to set osd_memory_target on smithi001 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-26T21:51:24.189 INFO:tasks.workunit.client.0.smithi001.stdout: 10 176104 20.36 MB/sec execute 321 sec latency 71.637 ms 2024-04-26T21:51:24.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:24 smithi001 ceph-mon[28054]: pgmap v669: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.3 MiB/s wr, 343 op/s 2024-04-26T21:51:24.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:24 smithi038 ceph-mon[32724]: pgmap v669: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.3 MiB/s wr, 343 op/s 2024-04-26T21:51:24.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:24 smithi186 ceph-mon[32991]: pgmap v669: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 7.3 MiB/s wr, 343 op/s 2024-04-26T21:51:25.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 176478 20.35 MB/sec execute 322 sec latency 152.739 ms 2024-04-26T21:51:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.565+0000 7f2227800640 1 -- 172.21.15.1:0/1036137872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f222010ac10 msgr2=0x7f222010aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.565+0000 7f2227800640 1 --2- 172.21.15.1:0/1036137872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f222010ac10 0x7f222010aff0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2214009890 tx=0x7f221402a370 comp rx=0 tx=0).stop 2024-04-26T21:51:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.565+0000 7f2227800640 1 -- 172.21.15.1:0/1036137872 shutdown_connections 2024-04-26T21:51:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.565+0000 7f2227800640 1 --2- 172.21.15.1:0/1036137872 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220105510 0x7f22201119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.565+0000 7f2227800640 1 --2- 172.21.15.1:0/1036137872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2220104b50 0x7f2220104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.566 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.566+0000 7f2227800640 1 --2- 172.21.15.1:0/1036137872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f222010ac10 0x7f222010aff0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.566+0000 7f2227800640 1 -- 172.21.15.1:0/1036137872 >> 172.21.15.1:0/1036137872 conn(0x7f2220100620 msgr2=0x7f2220102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:25.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.566+0000 7f2227800640 1 -- 172.21.15.1:0/1036137872 shutdown_connections 2024-04-26T21:51:25.567 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.566+0000 7f2227800640 1 -- 172.21.15.1:0/1036137872 wait complete. 2024-04-26T21:51:25.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:25 smithi001 ceph-mon[28054]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T21:51:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.567+0000 7f2227800640 1 Processor -- start 2024-04-26T21:51:25.568 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.568+0000 7f2227800640 1 -- start start 2024-04-26T21:51:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.568+0000 7f2227800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220104b50 0x7f2220111940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.568+0000 7f2227800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2220105510 0x7f222010ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.568+0000 7f2227800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f222010ac10 0x7f222010f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.568+0000 7f2227800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2220114dc0 con 0x7f222010ac10 2024-04-26T21:51:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.568+0000 7f2227800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2220114c40 con 0x7f2220104b50 2024-04-26T21:51:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.568+0000 7f2227800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2220114f40 con 0x7f2220105510 2024-04-26T21:51:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.568+0000 7f2224e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220104b50 0x7f2220111940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.568+0000 7f2224e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220104b50 0x7f2220111940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:41478/0 (socket says 172.21.15.1:41478) 2024-04-26T21:51:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.568+0000 7f2224e00640 1 -- 172.21.15.1:0/4251432684 learned_addr learned my addr 172.21.15.1:0/4251432684 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.569+0000 7f221fe00640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2220105510 0x7f222010ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.569+0000 7f2224e00640 1 -- 172.21.15.1:0/4251432684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2220105510 msgr2=0x7f222010ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.569+0000 7f2224e00640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2220105510 0x7f222010ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.569+0000 7f2224e00640 1 -- 172.21.15.1:0/4251432684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f222010ac10 msgr2=0x7f222010f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.569+0000 7f2224e00640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f222010ac10 0x7f222010f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.569+0000 7f2224e00640 1 -- 172.21.15.1:0/4251432684 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22201abac0 con 0x7f2220104b50 2024-04-26T21:51:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.569+0000 7f2224e00640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220104b50 0x7f2220111940 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2214030bb0 tx=0x7f2214030be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.570+0000 7f221d600640 1 -- 172.21.15.1:0/4251432684 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2214040020 con 0x7f2220104b50 2024-04-26T21:51:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.570+0000 7f221d600640 1 -- 172.21.15.1:0/4251432684 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2214038040 con 0x7f2220104b50 2024-04-26T21:51:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.570+0000 7f221d600640 1 -- 172.21.15.1:0/4251432684 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2214037070 con 0x7f2220104b50 2024-04-26T21:51:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.570+0000 7f2227800640 1 -- 172.21.15.1:0/4251432684 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22201abc90 con 0x7f2220104b50 2024-04-26T21:51:25.572 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.571+0000 7f2227800640 1 -- 172.21.15.1:0/4251432684 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22201ac190 con 0x7f2220104b50 2024-04-26T21:51:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.575+0000 7f222a7de740 1 Processor -- start 2024-04-26T21:51:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.575+0000 7f222a7de740 1 -- start start 2024-04-26T21:51:25.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:25 smithi038 ceph-mon[32724]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T21:51:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.576+0000 7f222a7de740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 0x564a4ce711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.576+0000 7f222a7de740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 0x564a4ce7ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.576+0000 7f222a7de740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 0x564a4ce7e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.576+0000 7f222a7de740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564a4cdd6380 con 0x564a4ce7bfb0 2024-04-26T21:51:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.576+0000 7f222a7de740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564a4cd72c30 con 0x564a4ce716f0 2024-04-26T21:51:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.576+0000 7f222a7de740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564a4cdd2340 con 0x564a4ce70dd0 2024-04-26T21:51:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.577+0000 7f221d600640 1 -- 172.21.15.1:0/4251432684 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f2214043020 con 0x7f2220104b50 2024-04-26T21:51:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.577+0000 7f2225800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 0x564a4ce711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.577+0000 7f2225800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 0x564a4ce711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:49188/0 (socket says 172.21.15.1:49188) 2024-04-26T21:51:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.577+0000 7f2225800640 1 -- 172.21.15.1:0/2171423496 learned_addr learned my addr 172.21.15.1:0/2171423496 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.577+0000 7f2224e00640 1 --2- 172.21.15.1:0/2171423496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 0x564a4ce7ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.577+0000 7f221fe00640 1 --2- 172.21.15.1:0/2171423496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 0x564a4ce7e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.578+0000 7f2225800640 1 -- 172.21.15.1:0/2171423496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 msgr2=0x564a4ce7ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.578+0000 7f2225800640 1 --2- 172.21.15.1:0/2171423496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 0x564a4ce7ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.578+0000 7f2225800640 1 -- 172.21.15.1:0/2171423496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 msgr2=0x564a4ce7e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.578+0000 7f2225800640 1 --2- 172.21.15.1:0/2171423496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 0x564a4ce7e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.578+0000 7f2225800640 1 -- 172.21.15.1:0/2171423496 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a4cd94940 con 0x564a4ce70dd0 2024-04-26T21:51:25.579 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.578+0000 7f221fe00640 1 --2- 172.21.15.1:0/2171423496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 0x564a4ce7e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:51:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.578+0000 7f2225800640 1 --2- 172.21.15.1:0/2171423496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 0x564a4ce711b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2218009ed0 tx=0x7f221800e7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c0b8ed90570b425 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.579+0000 7f221d600640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22080a3e60 0x7f22080a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.579+0000 7f2212000640 1 -- 172.21.15.1:0/2171423496 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221801b070 con 0x564a4ce70dd0 2024-04-26T21:51:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.579+0000 7f2212000640 1 -- 172.21.15.1:0/2171423496 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f221800cc30 con 0x564a4ce70dd0 2024-04-26T21:51:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.579+0000 7f2212000640 1 -- 172.21.15.1:0/2171423496 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2218016df0 con 0x564a4ce70dd0 2024-04-26T21:51:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.579+0000 7f2224e00640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22080a3e60 0x7f22080a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.580 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.579+0000 7f221d600640 1 -- 172.21.15.1:0/4251432684 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 9654+0+0 (secure 0 0 0) 0x7f22140f02d0 con 0x7f2220104b50 2024-04-26T21:51:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.580+0000 7f222a7de740 1 -- 172.21.15.1:0/2171423496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 msgr2=0x564a4ce711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.580+0000 7f222a7de740 1 --2- 172.21.15.1:0/2171423496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 0x564a4ce711b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2218009ed0 tx=0x7f221800e7c0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.580+0000 7f2224e00640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22080a3e60 0x7f22080a6320 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f22140f3fd0 tx=0x7f22140f3d80 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.581+0000 7f222a7de740 1 -- 172.21.15.1:0/2171423496 shutdown_connections 2024-04-26T21:51:25.581 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.581+0000 7f222a7de740 1 --2- 172.21.15.1:0/2171423496 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 0x564a4ce7e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.581+0000 7f222a7de740 1 --2- 172.21.15.1:0/2171423496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 0x564a4ce7ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.581+0000 7f222a7de740 1 --2- 172.21.15.1:0/2171423496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 0x564a4ce711b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.581+0000 7f222a7de740 1 -- 172.21.15.1:0/2171423496 >> 172.21.15.1:0/2171423496 conn(0x564a4ce6d2e0 msgr2=0x564a4ce6d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.581+0000 7f222a7de740 1 -- 172.21.15.1:0/2171423496 shutdown_connections 2024-04-26T21:51:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.581+0000 7f222a7de740 1 -- 172.21.15.1:0/2171423496 wait complete. 2024-04-26T21:51:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.582+0000 7f222a7de740 1 -- start start 2024-04-26T21:51:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.582+0000 7f222a7de740 1 Processor -- start 2024-04-26T21:51:25.584 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.584+0000 7f222a7de740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce78ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.586 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.584+0000 7f222a7de740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce716f0 0x564a4ce79410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.584+0000 7f2225800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce78ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.584+0000 7f2225800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce78ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:41496/0 (socket says 172.21.15.1:41496) 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.584+0000 7f2225800640 1 -- 172.21.15.1:0/3877785969 learned_addr learned my addr 172.21.15.1:0/3877785969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.584+0000 7f222a7de740 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce7bfb0 0x564a4ce79950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.584+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x564a4ce7ebe0 con 0x564a4ce716f0 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.584+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x564a4cd5dc60 con 0x564a4ce70dd0 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.584+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564a4ce7ed60 con 0x564a4ce7bfb0 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.585+0000 7f2225800640 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce7bfb0 0x564a4ce79950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.585+0000 7f2225800640 1 -- 172.21.15.1:0/3877785969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce7bfb0 msgr2=0x564a4ce79950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.585+0000 7f2225800640 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce7bfb0 0x564a4ce79950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.585+0000 7f2225800640 1 -- 172.21.15.1:0/3877785969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce716f0 msgr2=0x564a4ce79410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.585+0000 7f2225800640 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce716f0 0x564a4ce79410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.585+0000 7f2225800640 1 -- 172.21.15.1:0/3877785969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a4cd6c510 con 0x564a4ce70dd0 2024-04-26T21:51:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.585+0000 7f222a7de740 10 client.0 _ll_register_callbacks cb 0x564a4cd89e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:51:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.586+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f221800cc30 con 0x564a4ce70dd0 2024-04-26T21:51:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.586+0000 7f2225800640 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce78ed0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2218009620 tx=0x7f2218016950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9e9d89ec7757b63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.586+0000 7f2206a00640 10 client.27071 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:51:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.586+0000 7f2206a00640 1 -- 172.21.15.1:0/3877785969 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221802a060 con 0x564a4ce70dd0 2024-04-26T21:51:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.586+0000 7f2206a00640 1 -- 172.21.15.1:0/3877785969 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9654+0+0 (secure 0 0 0) 0x7f2218003970 con 0x564a4ce70dd0 2024-04-26T21:51:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.587+0000 7f2206a00640 1 -- 172.21.15.1:0/3877785969 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f221801b040 con 0x564a4ce70dd0 2024-04-26T21:51:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.587+0000 7f222a7de740 10 client.27071 fetch_fsmap learned FSMap version 59 2024-04-26T21:51:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.587+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({fsmap=59}) -- 0x564a4cae8210 con 0x564a4ce70dd0 2024-04-26T21:51:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.587+0000 7f2206a00640 1 -- 172.21.15.1:0/3877785969 <== mon.2 v2:172.21.15.38:3300/0 4 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7f221800b040 con 0x564a4ce70dd0 2024-04-26T21:51:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.588+0000 7f222a7de740 10 client.27071 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:51:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.588+0000 7f222a7de740 10 client.27071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:51:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.588+0000 7f222a7de740 20 client.27071 populate_metadata read hostname 'smithi001' 2024-04-26T21:51:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.588+0000 7f222a7de740 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564a4ce8b980 0x564a4ce8dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.589 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.588+0000 7f222a7de740 4 client.27071 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:51:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.588+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a4cde3f90 con 0x564a4ce8b980 2024-04-26T21:51:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.588+0000 7f221fe00640 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564a4ce8b980 0x564a4ce8dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.589+0000 7f221fe00640 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564a4ce8b980 0x564a4ce8dd70 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a2f55ad6a8841d1 server_cookie=dcb552d595d0e6be in_seq=0 out_seq=0 2024-04-26T21:51:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.589+0000 7f2206a00640 10 client.27071 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f2206a00640 1 -- 172.21.15.1:0/3877785969 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564a4cde3f90 con 0x564a4ce8b980 2024-04-26T21:51:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f2206a00640 10 client.27071 handle_command_reply: tid=0 2024-04-26T21:51:25.593 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f222a7de740 1 client.27071 shutdown 2024-04-26T21:51:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 msgr2=0x564a4ce78ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f222a7de740 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce78ed0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2218009620 tx=0x7f2218016950 comp rx=0 tx=0).stop 2024-04-26T21:51:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 shutdown_connections 2024-04-26T21:51:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f222a7de740 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564a4ce8b980 0x564a4ce8dd70 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f222a7de740 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce7bfb0 0x564a4ce79950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.594 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f222a7de740 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce716f0 0x564a4ce79410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f222a7de740 1 --2- 172.21.15.1:0/3877785969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce78ed0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f2206a00640 0 client.27071 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.593+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 >> 172.21.15.1:0/3877785969 conn(0x564a4ce6d2e0 msgr2=0x564a4ce73010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.594+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 shutdown_connections 2024-04-26T21:51:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.594+0000 7f222a7de740 1 -- 172.21.15.1:0/3877785969 wait complete. 2024-04-26T21:51:25.595 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.594+0000 7f222a7de740 20 client.27071 trim_cache size 0 max 16384 2024-04-26T21:51:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.628+0000 7f222a7de740 1 Processor -- start 2024-04-26T21:51:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.628+0000 7f222a7de740 1 -- start start 2024-04-26T21:51:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.628+0000 7f222a7de740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 0x564a4ce73790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.629+0000 7f222a7de740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 0x564a4ce79770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.629+0000 7f222a7de740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 0x564a4ce93540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.629+0000 7f222a7de740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22201b8fa0 con 0x564a4ce7bfb0 2024-04-26T21:51:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.629+0000 7f222a7de740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22201b6870 con 0x564a4ce716f0 2024-04-26T21:51:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.629+0000 7f222a7de740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22201b46c0 con 0x564a4ce70dd0 2024-04-26T21:51:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.629+0000 7f221fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 0x564a4ce79770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.629+0000 7f221fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 0x564a4ce79770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:41506/0 (socket says 172.21.15.1:41506) 2024-04-26T21:51:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.629+0000 7f221fe00640 1 -- 172.21.15.1:0/901812090 learned_addr learned my addr 172.21.15.1:0/901812090 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.630+0000 7f221fe00640 1 -- 172.21.15.1:0/901812090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 msgr2=0x564a4ce73790 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:51:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.630+0000 7f221fe00640 1 --2- 172.21.15.1:0/901812090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 0x564a4ce73790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.630+0000 7f221fe00640 1 -- 172.21.15.1:0/901812090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 msgr2=0x564a4ce93540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.630+0000 7f2225800640 1 --2- 172.21.15.1:0/901812090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 0x564a4ce93540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.630+0000 7f221fe00640 1 --2- 172.21.15.1:0/901812090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 0x564a4ce93540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.630+0000 7f221fe00640 1 -- 172.21.15.1:0/901812090 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a4ce68940 con 0x564a4ce716f0 2024-04-26T21:51:25.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.631+0000 7f221fe00640 1 --2- 172.21.15.1:0/901812090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 0x564a4ce79770 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f220c009a90 tx=0x7f220c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d52dbf0608211f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.631+0000 7f2225800640 1 --2- 172.21.15.1:0/901812090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 0x564a4ce93540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:51:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.631+0000 7f2211600640 1 -- 172.21.15.1:0/901812090 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c042030 con 0x564a4ce716f0 2024-04-26T21:51:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.631+0000 7f2211600640 1 -- 172.21.15.1:0/901812090 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f220c014190 con 0x564a4ce716f0 2024-04-26T21:51:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.631+0000 7f2211600640 1 -- 172.21.15.1:0/901812090 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220c005280 con 0x564a4ce716f0 2024-04-26T21:51:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.632+0000 7f222a7de740 1 -- 172.21.15.1:0/901812090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 msgr2=0x564a4ce79770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.632+0000 7f222a7de740 1 --2- 172.21.15.1:0/901812090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 0x564a4ce79770 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f220c009a90 tx=0x7f220c005f90 comp rx=0 tx=0).stop 2024-04-26T21:51:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.633+0000 7f222a7de740 1 -- 172.21.15.1:0/901812090 shutdown_connections 2024-04-26T21:51:25.635 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.633+0000 7f222a7de740 1 --2- 172.21.15.1:0/901812090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce7bfb0 0x564a4ce93540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.633+0000 7f222a7de740 1 --2- 172.21.15.1:0/901812090 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce716f0 0x564a4ce79770 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.633+0000 7f222a7de740 1 --2- 172.21.15.1:0/901812090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce70dd0 0x564a4ce73790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.633+0000 7f222a7de740 1 -- 172.21.15.1:0/901812090 >> 172.21.15.1:0/901812090 conn(0x564a4ce6d2e0 msgr2=0x564a4ce7ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.633+0000 7f222a7de740 1 -- 172.21.15.1:0/901812090 shutdown_connections 2024-04-26T21:51:25.636 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.633+0000 7f222a7de740 1 -- 172.21.15.1:0/901812090 wait complete. 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.633+0000 7f222a7de740 1 -- start start 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.634+0000 7f222a7de740 1 Processor -- start 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.635+0000 7f222a7de740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce84210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.635+0000 7f222a7de740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce716f0 0x564a4ce75760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.635+0000 7f2225800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce84210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.635+0000 7f2225800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce84210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:41508/0 (socket says 172.21.15.1:41508) 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.635+0000 7f2225800640 1 -- 172.21.15.1:0/2197688390 learned_addr learned my addr 172.21.15.1:0/2197688390 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.636+0000 7f222a7de740 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce7bfb0 0x564a4ce75ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.636+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22201b4d30 con 0x564a4ce716f0 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.636+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f22201b61a0 con 0x564a4ce70dd0 2024-04-26T21:51:25.640 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.636+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22201b5400 con 0x564a4ce7bfb0 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.636+0000 7f221fe00640 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce716f0 0x564a4ce75760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.637+0000 7f222a7de740 10 client.0 _ll_register_callbacks cb 0x7f2218019070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.638+0000 7f2225800640 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce7bfb0 0x564a4ce75ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.638+0000 7f2225800640 1 -- 172.21.15.1:0/2197688390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce7bfb0 msgr2=0x564a4ce75ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.638+0000 7f2225800640 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce7bfb0 0x564a4ce75ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.638+0000 7f2225800640 1 -- 172.21.15.1:0/2197688390 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce716f0 msgr2=0x564a4ce75760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.638+0000 7f2225800640 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce716f0 0x564a4ce75760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.638+0000 7f2225800640 1 -- 172.21.15.1:0/2197688390 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a4cbb4620 con 0x564a4ce70dd0 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.639+0000 7f2225800640 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce84210 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f22180092f0 tx=0x7f2218023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d02262e0c28d1e06 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.639+0000 7f2206a00640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.639+0000 7f2206a00640 1 -- 172.21.15.1:0/2197688390 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2218016920 con 0x564a4ce70dd0 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.639+0000 7f2206a00640 1 -- 172.21.15.1:0/2197688390 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(131..131 src has 1..131) ==== 9654+0+0 (secure 0 0 0) 0x7f2218004b20 con 0x564a4ce70dd0 2024-04-26T21:51:25.641 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.640+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f220c014190 con 0x564a4ce70dd0 2024-04-26T21:51:25.642 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.640+0000 7f2206a00640 1 -- 172.21.15.1:0/2197688390 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f221802a020 con 0x564a4ce70dd0 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.640+0000 7f222a7de740 10 client.27077 fetch_fsmap learned FSMap version 59 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.640+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x564a4cd90cb0 con 0x564a4ce70dd0 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.641+0000 7f2206a00640 1 -- 172.21.15.1:0/2197688390 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f220c014190 con 0x564a4ce70dd0 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.641+0000 7f2206a00640 1 -- 172.21.15.1:0/2197688390 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7f221800b070 con 0x564a4ce70dd0 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.641+0000 7f222a7de740 10 client.27077 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.641+0000 7f222a7de740 10 client.27077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.641+0000 7f222a7de740 20 client.27077 populate_metadata read hostname 'smithi001' 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.641+0000 7f222a7de740 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564a4cef1f40 0x564a4cef4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.641+0000 7f222a7de740 4 client.27077 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.641+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a4cddde00 con 0x564a4cef1f40 2024-04-26T21:51:25.643 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.642+0000 7f221fe00640 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564a4cef1f40 0x564a4cef4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.643+0000 7f221fe00640 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564a4cef1f40 0x564a4cef4330 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2342b81b55a9bea server_cookie=eab0130770a61989 in_seq=0 out_seq=0 2024-04-26T21:51:25.644 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:25.643+0000 7f2206a00640 10 client.27077 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:25.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:25 smithi186 ceph-mon[32991]: osdmap e131: 12 total, 12 up, 12 in 2024-04-26T21:51:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.139+0000 7f2206a00640 1 -- 172.21.15.1:0/2197688390 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a4cddde00 con 0x564a4cef1f40 2024-04-26T21:51:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.139+0000 7f2206a00640 10 client.27077 handle_command_reply: tid=0 2024-04-26T21:51:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.139+0000 7f222a7de740 1 client.27077 shutdown 2024-04-26T21:51:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 msgr2=0x564a4ce84210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f222a7de740 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce84210 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f22180092f0 tx=0x7f2218023000 comp rx=0 tx=0).stop 2024-04-26T21:51:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 shutdown_connections 2024-04-26T21:51:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f222a7de740 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x564a4cef1f40 0x564a4cef4330 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f222a7de740 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a4ce7bfb0 0x564a4ce75ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:26.140 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f222a7de740 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x564a4ce716f0 0x564a4ce75760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f222a7de740 1 --2- 172.21.15.1:0/2197688390 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564a4ce70dd0 0x564a4ce84210 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 >> 172.21.15.1:0/2197688390 conn(0x564a4ce6d2e0 msgr2=0x564a4ce7a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f2206a00640 0 client.27077 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 shutdown_connections 2024-04-26T21:51:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.140+0000 7f222a7de740 1 -- 172.21.15.1:0/2197688390 wait complete. 2024-04-26T21:51:26.141 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.141+0000 7f222a7de740 20 client.27077 trim_cache size 0 max 16384 2024-04-26T21:51:26.143 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:51:26.143 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:51:26.143 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:51:26.143 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:51:26.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.143+0000 7f2227800640 1 -- 172.21.15.1:0/4251432684 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22080a3e60 msgr2=0x7f22080a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:26.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.143+0000 7f2227800640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22080a3e60 0x7f22080a6320 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f22140f3fd0 tx=0x7f22140f3d80 comp rx=0 tx=0).stop 2024-04-26T21:51:26.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.143+0000 7f2227800640 1 -- 172.21.15.1:0/4251432684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220104b50 msgr2=0x7f2220111940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:26.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.143+0000 7f2227800640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220104b50 0x7f2220111940 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f2214030bb0 tx=0x7f2214030be0 comp rx=0 tx=0).stop 2024-04-26T21:51:26.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.144+0000 7f2227800640 1 -- 172.21.15.1:0/4251432684 shutdown_connections 2024-04-26T21:51:26.144 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.144+0000 7f2227800640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f22080a3e60 0x7f22080a6320 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.144+0000 7f2227800640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2220105510 0x7f222010ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.144+0000 7f2227800640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f222010ac10 0x7f222010f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.144+0000 7f2227800640 1 --2- 172.21.15.1:0/4251432684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2220104b50 0x7f2220111940 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.144+0000 7f2227800640 1 -- 172.21.15.1:0/4251432684 >> 172.21.15.1:0/4251432684 conn(0x7f2220100620 msgr2=0x7f2220101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.144+0000 7f2227800640 1 -- 172.21.15.1:0/4251432684 shutdown_connections 2024-04-26T21:51:26.145 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:26.144+0000 7f2227800640 1 -- 172.21.15.1:0/4251432684 wait complete. 2024-04-26T21:51:26.189 INFO:tasks.workunit.client.0.smithi001.stdout: 10 176733 20.33 MB/sec execute 323 sec latency 333.080 ms 2024-04-26T21:51:26.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:26 smithi001 ceph-mon[28054]: pgmap v671: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 7.6 MiB/s wr, 365 op/s 2024-04-26T21:51:26.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:26 smithi038 ceph-mon[32724]: pgmap v671: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 7.6 MiB/s wr, 365 op/s 2024-04-26T21:51:26.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:26 smithi186 ceph-mon[32991]: pgmap v671: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 7.6 MiB/s wr, 365 op/s 2024-04-26T21:51:27.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 177196 20.34 MB/sec execute 324 sec latency 107.753 ms 2024-04-26T21:51:28.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 177667 20.33 MB/sec execute 325 sec latency 142.119 ms 2024-04-26T21:51:28.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:28 smithi001 ceph-mon[28054]: pgmap v672: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.3 MiB/s wr, 386 op/s 2024-04-26T21:51:28.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:28 smithi038 ceph-mon[32724]: pgmap v672: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.3 MiB/s wr, 386 op/s 2024-04-26T21:51:28.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:28 smithi186 ceph-mon[32991]: pgmap v672: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.3 MiB/s wr, 386 op/s 2024-04-26T21:51:29.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 178276 20.38 MB/sec execute 326 sec latency 88.283 ms 2024-04-26T21:51:30.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 178863 20.43 MB/sec execute 327 sec latency 114.188 ms 2024-04-26T21:51:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:30 smithi001 ceph-mon[28054]: pgmap v673: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.1 MiB/s wr, 383 op/s 2024-04-26T21:51:30.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:30 smithi001 ceph-mon[28054]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T21:51:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:30 smithi038 ceph-mon[32724]: pgmap v673: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.1 MiB/s wr, 383 op/s 2024-04-26T21:51:30.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:30 smithi038 ceph-mon[32724]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T21:51:30.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:30 smithi186 ceph-mon[32991]: pgmap v673: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.1 MiB/s wr, 383 op/s 2024-04-26T21:51:30.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:30 smithi186 ceph-mon[32991]: osdmap e132: 12 total, 12 up, 12 in 2024-04-26T21:51:31.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 179263 20.44 MB/sec execute 328 sec latency 86.976 ms 2024-04-26T21:51:32.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 179678 20.44 MB/sec execute 329 sec latency 82.863 ms 2024-04-26T21:51:32.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:32 smithi001 ceph-mon[28054]: pgmap v675: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.7 MiB/s wr, 509 op/s 2024-04-26T21:51:32.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:32 smithi038 ceph-mon[32724]: pgmap v675: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.7 MiB/s wr, 509 op/s 2024-04-26T21:51:32.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:32 smithi186 ceph-mon[32991]: pgmap v675: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.7 MiB/s wr, 509 op/s 2024-04-26T21:51:33.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 180040 20.42 MB/sec execute 330 sec latency 160.805 ms 2024-04-26T21:51:34.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 180573 20.45 MB/sec execute 331 sec latency 79.839 ms 2024-04-26T21:51:34.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:34 smithi001 ceph-mon[28054]: pgmap v676: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 484 op/s 2024-04-26T21:51:34.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:34 smithi038 ceph-mon[32724]: pgmap v676: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 484 op/s 2024-04-26T21:51:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:34 smithi186 ceph-mon[32991]: pgmap v676: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.4 MiB/s wr, 484 op/s 2024-04-26T21:51:35.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 181231 20.51 MB/sec execute 332 sec latency 92.032 ms 2024-04-26T21:51:36.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 181708 20.53 MB/sec execute 333 sec latency 150.204 ms 2024-04-26T21:51:36.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:36 smithi001 ceph-mon[28054]: pgmap v677: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 441 op/s 2024-04-26T21:51:36.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:36 smithi038 ceph-mon[32724]: pgmap v677: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 441 op/s 2024-04-26T21:51:36.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:36 smithi186 ceph-mon[32991]: pgmap v677: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 441 op/s 2024-04-26T21:51:37.191 INFO:tasks.workunit.client.0.smithi001.stdout: 10 182066 20.52 MB/sec execute 334 sec latency 168.890 ms 2024-04-26T21:51:38.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 182524 20.54 MB/sec execute 335 sec latency 107.714 ms 2024-04-26T21:51:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:38 smithi001 ceph-mon[28054]: pgmap v678: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.0 MiB/s wr, 404 op/s 2024-04-26T21:51:38.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:38 smithi038 ceph-mon[32724]: pgmap v678: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.0 MiB/s wr, 404 op/s 2024-04-26T21:51:38.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:38 smithi186 ceph-mon[32991]: pgmap v678: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 6.0 MiB/s wr, 404 op/s 2024-04-26T21:51:39.190 INFO:tasks.workunit.client.0.smithi001.stdout: 10 183016 20.55 MB/sec execute 336 sec latency 111.280 ms 2024-04-26T21:51:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.347+0000 7f5b86200640 1 -- 172.21.15.1:0/3078402025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b80076e60 msgr2=0x7f5b80077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.347+0000 7f5b86200640 1 --2- 172.21.15.1:0/3078402025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b80076e60 0x7f5b80077240 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009860 tx=0x7f5b7002a340 comp rx=0 tx=0).stop 2024-04-26T21:51:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.348+0000 7f5b86200640 1 -- 172.21.15.1:0/3078402025 shutdown_connections 2024-04-26T21:51:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.348+0000 7f5b86200640 1 --2- 172.21.15.1:0/3078402025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b801137d0 0x7f5b80115bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.348+0000 7f5b86200640 1 --2- 172.21.15.1:0/3078402025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b80077780 0x7f5b80113290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.348+0000 7f5b86200640 1 --2- 172.21.15.1:0/3078402025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b80076e60 0x7f5b80077240 secure :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009860 tx=0x7f5b7002a340 comp rx=0 tx=0).stop 2024-04-26T21:51:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.348+0000 7f5b86200640 1 -- 172.21.15.1:0/3078402025 >> 172.21.15.1:0/3078402025 conn(0x7f5b80100600 msgr2=0x7f5b80102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.348+0000 7f5b86200640 1 -- 172.21.15.1:0/3078402025 shutdown_connections 2024-04-26T21:51:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.348+0000 7f5b86200640 1 -- 172.21.15.1:0/3078402025 wait complete. 2024-04-26T21:51:39.351 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.350+0000 7f5b86200640 1 Processor -- start 2024-04-26T21:51:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.350+0000 7f5b86200640 1 -- start start 2024-04-26T21:51:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.351+0000 7f5b86200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b80077780 0x7f5b801a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.351+0000 7f5b86200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b801137d0 0x7f5b801a5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.351+0000 7f5b86200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8019f7c0 0x7f5b8019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.351+0000 7f5b86200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5b801191d0 con 0x7f5b801137d0 2024-04-26T21:51:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.351+0000 7f5b86200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5b80119050 con 0x7f5b80077780 2024-04-26T21:51:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.351+0000 7f5b86200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5b80119350 con 0x7f5b8019f7c0 2024-04-26T21:51:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.351+0000 7f5b7f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b80077780 0x7f5b801a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.351+0000 7f5b7f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b80077780 0x7f5b801a56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36084/0 (socket says 172.21.15.1:36084) 2024-04-26T21:51:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.351+0000 7f5b7f400640 1 -- 172.21.15.1:0/837335047 learned_addr learned my addr 172.21.15.1:0/837335047 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.351+0000 7f5b7fe00640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8019f7c0 0x7f5b8019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.352+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b801137d0 0x7f5b801a5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.352+0000 7f5b7fe00640 1 -- 172.21.15.1:0/837335047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b80077780 msgr2=0x7f5b801a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.352+0000 7f5b7fe00640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b80077780 0x7f5b801a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.352+0000 7f5b7fe00640 1 -- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b801137d0 msgr2=0x7f5b801a5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.352+0000 7f5b7fe00640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b801137d0 0x7f5b801a5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.352+0000 7f5b7fe00640 1 -- 172.21.15.1:0/837335047 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b801a03c0 con 0x7f5b8019f7c0 2024-04-26T21:51:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.352+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b801137d0 0x7f5b801a5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:51:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.353+0000 7f5b7fe00640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8019f7c0 0x7f5b8019fc40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5b78005fd0 tx=0x7f5b7800edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.354+0000 7f5b77e00640 1 -- 172.21.15.1:0/837335047 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78004d20 con 0x7f5b8019f7c0 2024-04-26T21:51:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.354+0000 7f5b77e00640 1 -- 172.21.15.1:0/837335047 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b78005020 con 0x7f5b8019f7c0 2024-04-26T21:51:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.354+0000 7f5b77e00640 1 -- 172.21.15.1:0/837335047 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b780116b0 con 0x7f5b8019f7c0 2024-04-26T21:51:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.354+0000 7f5b86200640 1 -- 172.21.15.1:0/837335047 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b801ac450 con 0x7f5b8019f7c0 2024-04-26T21:51:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.354+0000 7f5b86200640 1 -- 172.21.15.1:0/837335047 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b801ac800 con 0x7f5b8019f7c0 2024-04-26T21:51:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.360+0000 7f5b77e00640 1 -- 172.21.15.1:0/837335047 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f5b78011850 con 0x7f5b8019f7c0 2024-04-26T21:51:39.362 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.361+0000 7f5b89012740 1 Processor -- start 2024-04-26T21:51:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.362+0000 7f5b89012740 1 -- start start 2024-04-26T21:51:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.363+0000 7f5b89012740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412bddd0 0x55ec412be1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.363+0000 7f5b89012740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412be6f0 0x55ec412c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.363+0000 7f5b89012740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 0x55ec412cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.364+0000 7f5b89012740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ec41223380 con 0x55ec412c8fb0 2024-04-26T21:51:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.364+0000 7f5b89012740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ec411bfc30 con 0x55ec412be6f0 2024-04-26T21:51:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.364+0000 7f5b89012740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ec4121f340 con 0x55ec412bddd0 2024-04-26T21:51:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.364+0000 7f5b7f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412be6f0 0x55ec412c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.364+0000 7f5b7f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412be6f0 0x55ec412c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36086/0 (socket says 172.21.15.1:36086) 2024-04-26T21:51:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.364+0000 7f5b7f400640 1 -- 172.21.15.1:0/4273244723 learned_addr learned my addr 172.21.15.1:0/4273244723 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.364+0000 7f5b77e00640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f5b640a3d90 0x7f5b640a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.365+0000 7f5b7f400640 1 -- 172.21.15.1:0/4273244723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412bddd0 msgr2=0x55ec412be1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T21:51:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.365+0000 7f5b7f400640 1 --2- 172.21.15.1:0/4273244723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412bddd0 0x55ec412be1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.365+0000 7f5b77e00640 1 -- 172.21.15.1:0/837335047 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9626+0+0 (secure 0 0 0) 0x7f5b780d0b90 con 0x7f5b8019f7c0 2024-04-26T21:51:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.365+0000 7f5b7f400640 1 -- 172.21.15.1:0/4273244723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 msgr2=0x55ec412cb3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:51:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.365+0000 7f5b7fe00640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f5b640a3d90 0x7f5b640a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.366+0000 7f5b7fe00640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f5b640a3d90 0x7f5b640a6250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5b780058e0 tx=0x7f5b78005870 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.366+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/4273244723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 0x55ec412cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.367+0000 7f5b7f400640 1 --2- 172.21.15.1:0/4273244723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 0x55ec412cb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.367+0000 7f5b7f400640 1 -- 172.21.15.1:0/4273244723 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec411e1940 con 0x55ec412be6f0 2024-04-26T21:51:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.367+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/4273244723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 0x55ec412cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:51:39.369 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.368+0000 7f5b7f400640 1 --2- 172.21.15.1:0/4273244723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412be6f0 0x55ec412c8a70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009780 tx=0x7f5b70005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6c5419cc59b170d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.368+0000 7f5b6be00640 1 -- 172.21.15.1:0/4273244723 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70041020 con 0x55ec412be6f0 2024-04-26T21:51:39.371 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.368+0000 7f5b6be00640 1 -- 172.21.15.1:0/4273244723 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b700042c0 con 0x55ec412be6f0 2024-04-26T21:51:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.368+0000 7f5b6be00640 1 -- 172.21.15.1:0/4273244723 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70037040 con 0x55ec412be6f0 2024-04-26T21:51:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.369+0000 7f5b89012740 1 -- 172.21.15.1:0/4273244723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412be6f0 msgr2=0x55ec412c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.369+0000 7f5b89012740 1 --2- 172.21.15.1:0/4273244723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412be6f0 0x55ec412c8a70 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009780 tx=0x7f5b70005da0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.370+0000 7f5b89012740 1 -- 172.21.15.1:0/4273244723 shutdown_connections 2024-04-26T21:51:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.370+0000 7f5b89012740 1 --2- 172.21.15.1:0/4273244723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 0x55ec412cb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.370+0000 7f5b89012740 1 --2- 172.21.15.1:0/4273244723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412be6f0 0x55ec412c8a70 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.370+0000 7f5b89012740 1 --2- 172.21.15.1:0/4273244723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412bddd0 0x55ec412be1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.370+0000 7f5b89012740 1 -- 172.21.15.1:0/4273244723 >> 172.21.15.1:0/4273244723 conn(0x55ec412ba2e0 msgr2=0x55ec412ba6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:39.372 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.370+0000 7f5b89012740 1 -- 172.21.15.1:0/4273244723 shutdown_connections 2024-04-26T21:51:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.370+0000 7f5b89012740 1 -- 172.21.15.1:0/4273244723 wait complete. 2024-04-26T21:51:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.371+0000 7f5b89012740 1 -- start start 2024-04-26T21:51:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.371+0000 7f5b89012740 1 Processor -- start 2024-04-26T21:51:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.373+0000 7f5b89012740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 0x55ec412c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.377 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.373+0000 7f5b89012740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412be6f0 0x55ec412c6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.374+0000 7f5b89012740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412c8fb0 0x55ec412c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.374+0000 7f5b89012740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ec412cbb80 con 0x55ec412be6f0 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.374+0000 7f5b89012740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x55ec411aac60 con 0x55ec412bddd0 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.374+0000 7f5b89012740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ec412cbd00 con 0x55ec412c8fb0 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.374+0000 7f5b89012740 10 client.0 _ll_register_callbacks cb 0x55ec411d6e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.374+0000 7f5b7ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 0x55ec412c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.374+0000 7f5b7ea00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 0x55ec412c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:36098/0 (socket says 172.21.15.1:36098) 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.374+0000 7f5b7ea00640 1 -- 172.21.15.1:0/3020566066 learned_addr learned my addr 172.21.15.1:0/3020566066 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.374+0000 7f5b7f400640 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412be6f0 0x55ec412c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.375+0000 7f5b7f400640 1 -- 172.21.15.1:0/3020566066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412c8fb0 msgr2=0x55ec412c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.375+0000 7f5b7f400640 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412c8fb0 0x55ec412c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.375+0000 7f5b7f400640 1 -- 172.21.15.1:0/3020566066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 msgr2=0x55ec412c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:39.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.375+0000 7f5b7f400640 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 0x55ec412c5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.375+0000 7f5b7f400640 1 -- 172.21.15.1:0/3020566066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec411b9510 con 0x55ec412be6f0 2024-04-26T21:51:39.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.376+0000 7f5b7f400640 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412be6f0 0x55ec412c6490 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5b7002a930 tx=0x7f5b70003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c94fc7a8047a10 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:39.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.376+0000 7f5b68c00640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-04-26T21:51:39.379 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.377+0000 7f5b68c00640 1 -- 172.21.15.1:0/3020566066 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70041030 con 0x55ec412be6f0 2024-04-26T21:51:39.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.377+0000 7f5b89012740 1 -- 172.21.15.1:0/3020566066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b700042c0 con 0x55ec412be6f0 2024-04-26T21:51:39.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.377+0000 7f5b68c00640 1 -- 172.21.15.1:0/3020566066 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9626+0+0 (secure 0 0 0) 0x7f5b70005030 con 0x55ec412be6f0 2024-04-26T21:51:39.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.378+0000 7f5b68c00640 1 -- 172.21.15.1:0/3020566066 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f5b70067050 con 0x55ec412be6f0 2024-04-26T21:51:39.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.378+0000 7f5b89012740 10 client.16734 fetch_fsmap learned FSMap version 59 2024-04-26T21:51:39.380 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.378+0000 7f5b89012740 1 -- 172.21.15.1:0/3020566066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55ec40f35210 con 0x55ec412be6f0 2024-04-26T21:51:39.381 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.380+0000 7f5b68c00640 1 -- 172.21.15.1:0/3020566066 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b700042c0 con 0x55ec412be6f0 2024-04-26T21:51:39.382 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.380+0000 7f5b68c00640 1 -- 172.21.15.1:0/3020566066 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7f5b700325c0 con 0x55ec412be6f0 2024-04-26T21:51:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.384+0000 7f5b89012740 10 client.16734 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:51:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.384+0000 7f5b89012740 10 client.16734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:51:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.384+0000 7f5b89012740 20 client.16734 populate_metadata read hostname 'smithi001' 2024-04-26T21:51:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.384+0000 7f5b89012740 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ec412d8df0 0x55ec412db210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.384+0000 7f5b89012740 4 client.16734 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:51:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.384+0000 7f5b89012740 1 -- 172.21.15.1:0/3020566066 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec41230f90 con 0x55ec412d8df0 2024-04-26T21:51:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.384+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ec412d8df0 0x55ec412db210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.385+0000 7f5b68c00640 10 client.16734 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.385+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ec412d8df0 0x55ec412db210 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e2308a7238f9e14 server_cookie=ebacb4ee80861c52 in_seq=0 out_seq=0 2024-04-26T21:51:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.388+0000 7f5b68c00640 1 -- 172.21.15.1:0/3020566066 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec41230f90 con 0x55ec412d8df0 2024-04-26T21:51:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.388+0000 7f5b68c00640 10 client.16734 handle_command_reply: tid=0 2024-04-26T21:51:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.389+0000 7f5b89012740 1 client.16734 shutdown 2024-04-26T21:51:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.390+0000 7f5b89012740 1 -- 172.21.15.1:0/3020566066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412be6f0 msgr2=0x55ec412c6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.390+0000 7f5b89012740 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412be6f0 0x55ec412c6490 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5b7002a930 tx=0x7f5b70003f90 comp rx=0 tx=0).stop 2024-04-26T21:51:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.390+0000 7f5b89012740 1 -- 172.21.15.1:0/3020566066 shutdown_connections 2024-04-26T21:51:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.390+0000 7f5b89012740 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ec412d8df0 0x55ec412db210 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.390+0000 7f5b89012740 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412c8fb0 0x55ec412c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.390+0000 7f5b68c00640 0 client.16734 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.390+0000 7f5b89012740 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412be6f0 0x55ec412c6490 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.390+0000 7f5b89012740 1 --2- 172.21.15.1:0/3020566066 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 0x55ec412c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.391+0000 7f5b89012740 1 -- 172.21.15.1:0/3020566066 >> 172.21.15.1:0/3020566066 conn(0x55ec412ba2e0 msgr2=0x55ec412c0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:39.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.391+0000 7f5b89012740 1 -- 172.21.15.1:0/3020566066 shutdown_connections 2024-04-26T21:51:39.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.391+0000 7f5b89012740 1 -- 172.21.15.1:0/3020566066 wait complete. 2024-04-26T21:51:39.393 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.391+0000 7f5b89012740 20 client.16734 trim_cache size 0 max 16384 2024-04-26T21:51:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.424+0000 7f5b89012740 1 Processor -- start 2024-04-26T21:51:39.426 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.424+0000 7f5b89012740 1 -- start start 2024-04-26T21:51:39.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.425+0000 7f5b89012740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412bddd0 0x55ec412d4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.427 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.425+0000 7f5b89012740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.425+0000 7f5b89012740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412c8fb0 0x55ec412e0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.425+0000 7f5b89012740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5b60012ae0 con 0x55ec412bddd0 2024-04-26T21:51:39.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.426+0000 7f5b7f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.426+0000 7f5b7f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42096/0 (socket says 172.21.15.1:42096) 2024-04-26T21:51:39.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.426+0000 7f5b7f400640 1 -- 172.21.15.1:0/525055534 learned_addr learned my addr 172.21.15.1:0/525055534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:39.428 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.425+0000 7f5b89012740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5b600103b0 con 0x55ec412c8fb0 2024-04-26T21:51:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.426+0000 7f5b89012740 1 -- 172.21.15.1:0/525055534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6000e220 con 0x55ec412be6f0 2024-04-26T21:51:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.426+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/525055534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412c8fb0 0x55ec412e0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.426+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/525055534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412bddd0 0x55ec412d4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.427+0000 7f5b7f400640 1 -- 172.21.15.1:0/525055534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412c8fb0 msgr2=0x55ec412e0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.427+0000 7f5b7f400640 1 --2- 172.21.15.1:0/525055534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412c8fb0 0x55ec412e0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.427+0000 7f5b7f400640 1 -- 172.21.15.1:0/525055534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412bddd0 msgr2=0x55ec412d4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.427+0000 7f5b7f400640 1 --2- 172.21.15.1:0/525055534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412bddd0 0x55ec412d4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.429 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.427+0000 7f5b7f400640 1 -- 172.21.15.1:0/525055534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec41001620 con 0x55ec412be6f0 2024-04-26T21:51:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.428+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/525055534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412bddd0 0x55ec412d4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:51:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.428+0000 7f5b7f400640 1 --2- 172.21.15.1:0/525055534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c67f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009780 tx=0x7f5b70030340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95cba18580866649 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.428+0000 7f5b6b400640 1 -- 172.21.15.1:0/525055534 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70041070 con 0x55ec412be6f0 2024-04-26T21:51:39.430 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.429+0000 7f5b6b400640 1 -- 172.21.15.1:0/525055534 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b70063020 con 0x55ec412be6f0 2024-04-26T21:51:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.429+0000 7f5b6b400640 1 -- 172.21.15.1:0/525055534 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70037070 con 0x55ec412be6f0 2024-04-26T21:51:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.431+0000 7f5b89012740 1 -- 172.21.15.1:0/525055534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 msgr2=0x55ec412c67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.431+0000 7f5b89012740 1 --2- 172.21.15.1:0/525055534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c67f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5b70009780 tx=0x7f5b70030340 comp rx=0 tx=0).stop 2024-04-26T21:51:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.432+0000 7f5b89012740 1 -- 172.21.15.1:0/525055534 shutdown_connections 2024-04-26T21:51:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.432+0000 7f5b89012740 1 --2- 172.21.15.1:0/525055534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c67f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.432+0000 7f5b89012740 1 --2- 172.21.15.1:0/525055534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412c8fb0 0x55ec412e0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.432+0000 7f5b89012740 1 --2- 172.21.15.1:0/525055534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412bddd0 0x55ec412d4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.432+0000 7f5b89012740 1 -- 172.21.15.1:0/525055534 >> 172.21.15.1:0/525055534 conn(0x55ec412ba2e0 msgr2=0x55ec412b9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.432+0000 7f5b89012740 1 -- 172.21.15.1:0/525055534 shutdown_connections 2024-04-26T21:51:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.432+0000 7f5b89012740 1 -- 172.21.15.1:0/525055534 wait complete. 2024-04-26T21:51:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.433+0000 7f5b89012740 1 -- start start 2024-04-26T21:51:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.433+0000 7f5b89012740 1 Processor -- start 2024-04-26T21:51:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.434+0000 7f5b89012740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 0x55ec412d12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.435+0000 7f5b89012740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.435+0000 7f5b89012740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 0x55ec412c2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.435+0000 7f5b89012740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6000e870 con 0x55ec412c8fb0 2024-04-26T21:51:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.435+0000 7f5b89012740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6000fce0 con 0x55ec412bddd0 2024-04-26T21:51:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.435+0000 7f5b89012740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6000ef40 con 0x55ec412be6f0 2024-04-26T21:51:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.435+0000 7f5b89012740 10 client.0 _ll_register_callbacks cb 0x7f5b700096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:51:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.435+0000 7f5b7f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.435+0000 7f5b7f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:42110/0 (socket says 172.21.15.1:42110) 2024-04-26T21:51:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.435+0000 7f5b7f400640 1 -- 172.21.15.1:0/869136648 learned_addr learned my addr 172.21.15.1:0/869136648 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.436+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 0x55ec412c2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.436+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 0x55ec412d12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.436+0000 7f5b7f400640 1 -- 172.21.15.1:0/869136648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 msgr2=0x55ec412d12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.436+0000 7f5b7f400640 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 0x55ec412d12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.436+0000 7f5b7f400640 1 -- 172.21.15.1:0/869136648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 msgr2=0x55ec412c2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.436+0000 7f5b7f400640 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 0x55ec412c2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.436+0000 7f5b7f400640 1 -- 172.21.15.1:0/869136648 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec411ddcb0 con 0x55ec412be6f0 2024-04-26T21:51:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.437+0000 7f5b7f400640 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c28a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5b7002a930 tx=0x7f5b70005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14a76df225a8f4ca server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.437+0000 7f5b68c00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:51:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.437+0000 7f5b68c00640 1 -- 172.21.15.1:0/869136648 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70037070 con 0x55ec412be6f0 2024-04-26T21:51:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.437+0000 7f5b68c00640 1 -- 172.21.15.1:0/869136648 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9626+0+0 (secure 0 0 0) 0x7f5b700324e0 con 0x55ec412be6f0 2024-04-26T21:51:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.437+0000 7f5b89012740 1 -- 172.21.15.1:0/869136648 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b70063020 con 0x55ec412be6f0 2024-04-26T21:51:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.438+0000 7f5b68c00640 1 -- 172.21.15.1:0/869136648 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f5b70079050 con 0x55ec412be6f0 2024-04-26T21:51:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.438+0000 7f5b89012740 10 client.27100 fetch_fsmap learned FSMap version 59 2024-04-26T21:51:39.441 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.438+0000 7f5b89012740 1 -- 172.21.15.1:0/869136648 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x55ec4133f0a0 con 0x55ec412be6f0 2024-04-26T21:51:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.439+0000 7f5b68c00640 1 -- 172.21.15.1:0/869136648 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b70063020 con 0x55ec412be6f0 2024-04-26T21:51:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.439+0000 7f5b68c00640 1 -- 172.21.15.1:0/869136648 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7f5b7003db30 con 0x55ec412be6f0 2024-04-26T21:51:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.439+0000 7f5b89012740 10 client.27100 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:51:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.439+0000 7f5b89012740 10 client.27100 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:51:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.439+0000 7f5b89012740 20 client.27100 populate_metadata read hostname 'smithi001' 2024-04-26T21:51:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.439+0000 7f5b89012740 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ec412d8df0 0x55ec41341740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.439+0000 7f5b89012740 4 client.27100 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:51:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.439+0000 7f5b89012740 1 -- 172.21.15.1:0/869136648 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ec4122ae00 con 0x55ec412d8df0 2024-04-26T21:51:39.442 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.439+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ec412d8df0 0x55ec41341740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.440+0000 7f5b7ea00640 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ec412d8df0 0x55ec41341740 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61cd03382ded14f server_cookie=9b7909b1e0f54e79 in_seq=0 out_seq=0 2024-04-26T21:51:39.443 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:39.440+0000 7f5b68c00640 10 client.27100 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:40.191 INFO:tasks.workunit.client.0.smithi001.stdout: 10 183223 20.51 MB/sec execute 337 sec latency 393.075 ms 2024-04-26T21:51:40.289 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.288+0000 7f5b68c00640 1 -- 172.21.15.1:0/869136648 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ec4122ae00 con 0x55ec412d8df0 2024-04-26T21:51:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.288+0000 7f5b68c00640 10 client.27100 handle_command_reply: tid=0 2024-04-26T21:51:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.288+0000 7f5b89012740 1 client.27100 shutdown 2024-04-26T21:51:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.289+0000 7f5b89012740 1 -- 172.21.15.1:0/869136648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 msgr2=0x55ec412c28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:40.290 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.289+0000 7f5b89012740 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c28a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5b7002a930 tx=0x7f5b70005f80 comp rx=0 tx=0).stop 2024-04-26T21:51:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.289+0000 7f5b89012740 1 -- 172.21.15.1:0/869136648 shutdown_connections 2024-04-26T21:51:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.289+0000 7f5b89012740 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x55ec412d8df0 0x55ec41341740 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.289+0000 7f5b89012740 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ec412be6f0 0x55ec412c28a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.289+0000 7f5b89012740 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ec412c8fb0 0x55ec412c2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.289+0000 7f5b89012740 1 --2- 172.21.15.1:0/869136648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ec412bddd0 0x55ec412d12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.289+0000 7f5b89012740 1 -- 172.21.15.1:0/869136648 >> 172.21.15.1:0/869136648 conn(0x55ec412ba2e0 msgr2=0x55ec412b9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.289+0000 7f5b68c00640 0 client.27100 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.290+0000 7f5b89012740 1 -- 172.21.15.1:0/869136648 shutdown_connections 2024-04-26T21:51:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.290+0000 7f5b89012740 1 -- 172.21.15.1:0/869136648 wait complete. 2024-04-26T21:51:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.290+0000 7f5b89012740 20 client.27100 trim_cache size 0 max 16384 2024-04-26T21:51:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:51:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:51:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:51:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:51:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.291+0000 7f5b86200640 1 -- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f5b640a3d90 msgr2=0x7f5b640a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.291+0000 7f5b86200640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f5b640a3d90 0x7f5b640a6250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5b780058e0 tx=0x7f5b78005870 comp rx=0 tx=0).stop 2024-04-26T21:51:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.291+0000 7f5b86200640 1 -- 172.21.15.1:0/837335047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8019f7c0 msgr2=0x7f5b8019fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.291+0000 7f5b86200640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8019f7c0 0x7f5b8019fc40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5b78005fd0 tx=0x7f5b7800edd0 comp rx=0 tx=0).stop 2024-04-26T21:51:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.292+0000 7f5b86200640 1 -- 172.21.15.1:0/837335047 shutdown_connections 2024-04-26T21:51:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.292+0000 7f5b86200640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f5b640a3d90 0x7f5b640a6250 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.292+0000 7f5b86200640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b8019f7c0 0x7f5b8019fc40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.292+0000 7f5b86200640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5b801137d0 0x7f5b801a5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.292+0000 7f5b86200640 1 --2- 172.21.15.1:0/837335047 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b80077780 0x7f5b801a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.292+0000 7f5b86200640 1 -- 172.21.15.1:0/837335047 >> 172.21.15.1:0/837335047 conn(0x7f5b80100600 msgr2=0x7f5b80113f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.292+0000 7f5b86200640 1 -- 172.21.15.1:0/837335047 shutdown_connections 2024-04-26T21:51:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:40.292+0000 7f5b86200640 1 -- 172.21.15.1:0/837335047 wait complete. 2024-04-26T21:51:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:40 smithi001 ceph-mon[28054]: pgmap v679: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.4 MiB/s wr, 409 op/s 2024-04-26T21:51:40.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:40 smithi038 ceph-mon[32724]: pgmap v679: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.4 MiB/s wr, 409 op/s 2024-04-26T21:51:40.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:40 smithi186 ceph-mon[32991]: pgmap v679: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 6.4 MiB/s wr, 409 op/s 2024-04-26T21:51:41.192 INFO:tasks.workunit.client.0.smithi001.stdout: 10 183677 20.53 MB/sec execute 338 sec latency 97.770 ms 2024-04-26T21:51:42.192 INFO:tasks.workunit.client.0.smithi001.stdout: 10 184077 20.55 MB/sec execute 339 sec latency 317.987 ms 2024-04-26T21:51:42.505 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:42 smithi186 ceph-mon[32991]: pgmap v680: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.5 MiB/s wr, 539 op/s 2024-04-26T21:51:42.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:42 smithi001 ceph-mon[28054]: pgmap v680: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.5 MiB/s wr, 539 op/s 2024-04-26T21:51:42.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:42 smithi038 ceph-mon[32724]: pgmap v680: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 7.5 MiB/s wr, 539 op/s 2024-04-26T21:51:43.192 INFO:tasks.workunit.client.0.smithi001.stdout: 10 184525 20.54 MB/sec execute 340 sec latency 376.967 ms 2024-04-26T21:51:44.192 INFO:tasks.workunit.client.0.smithi001.stdout: 10 185050 20.56 MB/sec execute 341 sec latency 104.365 ms 2024-04-26T21:51:44.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:44 smithi001 ceph-mon[28054]: pgmap v681: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.4 MiB/s wr, 428 op/s 2024-04-26T21:51:44.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:44 smithi038 ceph-mon[32724]: pgmap v681: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.4 MiB/s wr, 428 op/s 2024-04-26T21:51:44.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:44 smithi186 ceph-mon[32991]: pgmap v681: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.4 MiB/s wr, 428 op/s 2024-04-26T21:51:45.192 INFO:tasks.workunit.client.0.smithi001.stdout: 10 185424 20.56 MB/sec execute 342 sec latency 130.559 ms 2024-04-26T21:51:46.192 INFO:tasks.workunit.client.0.smithi001.stdout: 10 185535 20.51 MB/sec execute 343 sec latency 578.056 ms 2024-04-26T21:51:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:46 smithi001 ceph-mon[28054]: pgmap v682: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 524 op/s 2024-04-26T21:51:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:46 smithi038 ceph-mon[32724]: pgmap v682: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 524 op/s 2024-04-26T21:51:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:46 smithi186 ceph-mon[32991]: pgmap v682: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.1 MiB/s wr, 524 op/s 2024-04-26T21:51:47.193 INFO:tasks.workunit.client.0.smithi001.stdout: 10 185649 20.48 MB/sec execute 344 sec latency 1118.776 ms 2024-04-26T21:51:48.193 INFO:tasks.workunit.client.0.smithi001.stdout: 10 185828 20.44 MB/sec execute 345 sec latency 999.002 ms 2024-04-26T21:51:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:48 smithi001 ceph-mon[28054]: pgmap v683: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.0 MiB/s wr, 435 op/s 2024-04-26T21:51:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:48 smithi038 ceph-mon[32724]: pgmap v683: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.0 MiB/s wr, 435 op/s 2024-04-26T21:51:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:48 smithi186 ceph-mon[32991]: pgmap v683: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.0 MiB/s wr, 435 op/s 2024-04-26T21:51:49.193 INFO:tasks.workunit.client.0.smithi001.stdout: 10 186087 20.42 MB/sec execute 346 sec latency 175.295 ms 2024-04-26T21:51:50.193 INFO:tasks.workunit.client.0.smithi001.stdout: 10 186454 20.42 MB/sec execute 347 sec latency 131.645 ms 2024-04-26T21:51:50.569 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:50 smithi001 ceph-mon[28054]: pgmap v684: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.8 MiB/s wr, 436 op/s 2024-04-26T21:51:50.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:50 smithi038 ceph-mon[32724]: pgmap v684: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.8 MiB/s wr, 436 op/s 2024-04-26T21:51:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:50 smithi186 ceph-mon[32991]: pgmap v684: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 6.8 MiB/s wr, 436 op/s 2024-04-26T21:51:51.193 INFO:tasks.workunit.client.0.smithi001.stdout: 10 186872 20.44 MB/sec execute 348 sec latency 275.969 ms 2024-04-26T21:51:52.194 INFO:tasks.workunit.client.0.smithi001.stdout: 10 187009 20.41 MB/sec execute 349 sec latency 824.391 ms 2024-04-26T21:51:52.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:52 smithi001 ceph-mon[28054]: pgmap v685: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 624 op/s 2024-04-26T21:51:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:52 smithi038 ceph-mon[32724]: pgmap v685: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 624 op/s 2024-04-26T21:51:52.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:52 smithi186 ceph-mon[32991]: pgmap v685: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 MiB/s wr, 624 op/s 2024-04-26T21:51:53.193 INFO:tasks.workunit.client.0.smithi001.stdout: 10 187254 20.38 MB/sec execute 350 sec latency 325.856 ms 2024-04-26T21:51:53.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.519+0000 7fc79fe00640 1 -- 172.21.15.1:0/2394383736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7a0103910 msgr2=0x7fc7a0103cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:53.519 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.519+0000 7fc79fe00640 1 --2- 172.21.15.1:0/2394383736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7a0103910 0x7fc7a0103cf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc788009900 tx=0x7fc78802a4a0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.520+0000 7fc79fe00640 1 -- 172.21.15.1:0/2394383736 shutdown_connections 2024-04-26T21:51:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.520+0000 7fc79fe00640 1 --2- 172.21.15.1:0/2394383736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0104230 0x7fc7a010c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.520+0000 7fc79fe00640 1 --2- 172.21.15.1:0/2394383736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a010cd30 0x7fc7a010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.520+0000 7fc79fe00640 1 --2- 172.21.15.1:0/2394383736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7a0103910 0x7fc7a0103cf0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.520 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.520+0000 7fc79fe00640 1 -- 172.21.15.1:0/2394383736 >> 172.21.15.1:0/2394383736 conn(0x7fc7a00fd110 msgr2=0x7fc7a00ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:53.521 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.520+0000 7fc79fe00640 1 -- 172.21.15.1:0/2394383736 shutdown_connections 2024-04-26T21:51:53.521 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.520+0000 7fc79fe00640 1 -- 172.21.15.1:0/2394383736 wait complete. 2024-04-26T21:51:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.522+0000 7fc79fe00640 1 Processor -- start 2024-04-26T21:51:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.526+0000 7fc79fe00640 1 -- start start 2024-04-26T21:51:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.527+0000 7fc79fe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0103910 0x7fc7a010c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.527 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.527+0000 7fc79fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7a0104230 0x7fc7a010c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.528+0000 7fc79fe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a010cd30 0x7fc7a01062c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.528+0000 7fc79fe00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a0110890 con 0x7fc7a010cd30 2024-04-26T21:51:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.528+0000 7fc79fe00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a0110710 con 0x7fc7a0104230 2024-04-26T21:51:53.528 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.528+0000 7fc79fe00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a0110a10 con 0x7fc7a0103910 2024-04-26T21:51:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.528+0000 7fc79ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0103910 0x7fc7a010c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.528+0000 7fc79ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0103910 0x7fc7a010c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43368/0 (socket says 172.21.15.1:43368) 2024-04-26T21:51:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.528+0000 7fc79ea00640 1 -- 172.21.15.1:0/593577568 learned_addr learned my addr 172.21.15.1:0/593577568 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.528+0000 7fc79e000640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7a0104230 0x7fc7a010c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.528+0000 7fc79f400640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a010cd30 0x7fc7a01062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.529 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.529+0000 7fc79ea00640 1 -- 172.21.15.1:0/593577568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7a0104230 msgr2=0x7fc7a010c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.529+0000 7fc79ea00640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7a0104230 0x7fc7a010c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.529+0000 7fc79ea00640 1 -- 172.21.15.1:0/593577568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a010cd30 msgr2=0x7fc7a01062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.529+0000 7fc79ea00640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a010cd30 0x7fc7a01062c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.530 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.529+0000 7fc79ea00640 1 -- 172.21.15.1:0/593577568 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a0106b80 con 0x7fc7a0103910 2024-04-26T21:51:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.532+0000 7fc79ea00640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0103910 0x7fc7a010c130 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc788005ab0 tx=0x7fc788003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:53.532 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.532+0000 7fc797400640 1 -- 172.21.15.1:0/593577568 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78803f070 con 0x7fc7a0103910 2024-04-26T21:51:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.532+0000 7fc797400640 1 -- 172.21.15.1:0/593577568 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc788031d60 con 0x7fc7a0103910 2024-04-26T21:51:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.532+0000 7fc797400640 1 -- 172.21.15.1:0/593577568 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc788035030 con 0x7fc7a0103910 2024-04-26T21:51:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.533+0000 7fc79fe00640 1 -- 172.21.15.1:0/593577568 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7a0106e10 con 0x7fc7a0103910 2024-04-26T21:51:53.534 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.533+0000 7fc79fe00640 1 -- 172.21.15.1:0/593577568 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7a01b48e0 con 0x7fc7a0103910 2024-04-26T21:51:53.540 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.540+0000 7fc7a8764740 1 Processor -- start 2024-04-26T21:51:53.542 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.542+0000 7fc797400640 1 -- 172.21.15.1:0/593577568 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fc788042070 con 0x7fc7a0103910 2024-04-26T21:51:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.542+0000 7fc7a8764740 1 -- start start 2024-04-26T21:51:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.543+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628893c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.543+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562362889900 0x562362893c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.543 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.543+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5623628941c0 0x5623628965b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.543+0000 7fc7a8764740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x5623627ee380 con 0x5623628941c0 2024-04-26T21:51:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.543+0000 7fc7a8764740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x56236278ac30 con 0x562362889900 2024-04-26T21:51:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.543+0000 7fc7a8764740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5623627ea340 con 0x562362888fe0 2024-04-26T21:51:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.543+0000 7fc79f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628893c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.544+0000 7fc79f400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628893c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43380/0 (socket says 172.21.15.1:43380) 2024-04-26T21:51:53.544 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.544+0000 7fc79f400640 1 -- 172.21.15.1:0/3725071747 learned_addr learned my addr 172.21.15.1:0/3725071747 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.544+0000 7fc79e000640 1 --2- 172.21.15.1:0/3725071747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5623628941c0 0x5623628965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.545 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.544+0000 7fc797400640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc7840a3e10 0x7fc7840a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.549 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.544+0000 7fc797400640 1 -- 172.21.15.1:0/593577568 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 9626+0+0 (secure 0 0 0) 0x7fc7880f1520 con 0x7fc7a0103910 2024-04-26T21:51:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.545+0000 7fc79f400640 1 -- 172.21.15.1:0/3725071747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562362889900 msgr2=0x562362893c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T21:51:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.545+0000 7fc79f400640 1 --2- 172.21.15.1:0/3725071747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562362889900 0x562362893c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.545+0000 7fc79f400640 1 -- 172.21.15.1:0/3725071747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5623628941c0 msgr2=0x5623628965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.545+0000 7fc79f400640 1 --2- 172.21.15.1:0/3725071747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5623628941c0 0x5623628965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.545+0000 7fc79f400640 1 -- 172.21.15.1:0/3725071747 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623627ac940 con 0x562362888fe0 2024-04-26T21:51:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.545+0000 7fc79f400640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc7840a3e10 0x7fc7840a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.545+0000 7fc79f400640 1 --2- 172.21.15.1:0/3725071747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628893c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc79800f480 tx=0x7fc79800f940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2481ca26f7f70d05 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.546+0000 7fc79f400640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc7840a3e10 0x7fc7840a62d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc798020000 tx=0x7fc79801d000 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.546+0000 7fc78fe00640 1 -- 172.21.15.1:0/3725071747 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798026480 con 0x562362888fe0 2024-04-26T21:51:53.551 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.546+0000 7fc78fe00640 1 -- 172.21.15.1:0/3725071747 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc79801c8b0 con 0x562362888fe0 2024-04-26T21:51:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.546+0000 7fc78fe00640 1 -- 172.21.15.1:0/3725071747 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798010700 con 0x562362888fe0 2024-04-26T21:51:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.548+0000 7fc7a8764740 1 -- 172.21.15.1:0/3725071747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 msgr2=0x5623628893c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.548+0000 7fc7a8764740 1 --2- 172.21.15.1:0/3725071747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628893c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc79800f480 tx=0x7fc79800f940 comp rx=0 tx=0).stop 2024-04-26T21:51:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.548+0000 7fc7a8764740 1 -- 172.21.15.1:0/3725071747 shutdown_connections 2024-04-26T21:51:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.548+0000 7fc7a8764740 1 --2- 172.21.15.1:0/3725071747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x5623628941c0 0x5623628965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.548+0000 7fc7a8764740 1 --2- 172.21.15.1:0/3725071747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562362889900 0x562362893c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.548+0000 7fc7a8764740 1 --2- 172.21.15.1:0/3725071747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628893c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.548+0000 7fc7a8764740 1 -- 172.21.15.1:0/3725071747 >> 172.21.15.1:0/3725071747 conn(0x5623628854f0 msgr2=0x5623628858d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.550+0000 7fc7a8764740 1 -- 172.21.15.1:0/3725071747 shutdown_connections 2024-04-26T21:51:53.552 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.550+0000 7fc7a8764740 1 -- 172.21.15.1:0/3725071747 wait complete. 2024-04-26T21:51:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.551+0000 7fc7a8764740 1 -- start start 2024-04-26T21:51:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.551+0000 7fc7a8764740 1 Processor -- start 2024-04-26T21:51:53.553 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.552+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 0x5623628911e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362889900 0x562362891720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x562362891c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc7a8764740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x562362896df0 con 0x562362888fe0 2024-04-26T21:51:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc7a8764740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x562362775c60 con 0x5623628941c0 2024-04-26T21:51:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc7a8764740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562362896f70 con 0x562362889900 2024-04-26T21:51:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc7a8764740 10 client.0 _ll_register_callbacks cb 0x5623627a1e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:51:53.554 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc79f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x562362891c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc79f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x562362891c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43740/0 (socket says 172.21.15.1:43740) 2024-04-26T21:51:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc79f400640 1 -- 172.21.15.1:0/697361680 learned_addr learned my addr 172.21.15.1:0/697361680 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc79e000640 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362889900 0x562362891720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.555 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.553+0000 7fc79ea00640 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 0x5623628911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.554+0000 7fc79e000640 1 -- 172.21.15.1:0/697361680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 msgr2=0x562362891c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.554+0000 7fc79e000640 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x562362891c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.554+0000 7fc79e000640 1 -- 172.21.15.1:0/697361680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 msgr2=0x5623628911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.554+0000 7fc79e000640 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 0x5623628911e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.554+0000 7fc79e000640 1 -- 172.21.15.1:0/697361680 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562362784510 con 0x562362889900 2024-04-26T21:51:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.554+0000 7fc79ea00640 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 0x5623628911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:51:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.554+0000 7fc79e000640 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362889900 0x562362891720 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x562362886680 tx=0x7fc790011630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad83040cb2f45048 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.555+0000 7fc78cc00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:51:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.555+0000 7fc78cc00640 1 -- 172.21.15.1:0/697361680 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc790019070 con 0x562362889900 2024-04-26T21:51:53.556 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.555+0000 7fc78cc00640 1 -- 172.21.15.1:0/697361680 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9626+0+0 (secure 0 0 0) 0x7fc790015660 con 0x562362889900 2024-04-26T21:51:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.555+0000 7fc7a8764740 1 -- 172.21.15.1:0/697361680 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc79801c8b0 con 0x562362889900 2024-04-26T21:51:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.556+0000 7fc78cc00640 1 -- 172.21.15.1:0/697361680 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fc790023050 con 0x562362889900 2024-04-26T21:51:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.557+0000 7fc7a8764740 10 client.27109 fetch_fsmap learned FSMap version 59 2024-04-26T21:51:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.557+0000 7fc7a8764740 1 -- 172.21.15.1:0/697361680 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x562362500210 con 0x562362889900 2024-04-26T21:51:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.557+0000 7fc78cc00640 1 -- 172.21.15.1:0/697361680 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc79801c8b0 con 0x562362889900 2024-04-26T21:51:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.557+0000 7fc78cc00640 1 -- 172.21.15.1:0/697361680 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7fc79001ea50 con 0x562362889900 2024-04-26T21:51:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.557+0000 7fc7a8764740 10 client.27109 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:51:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.557+0000 7fc7a8764740 10 client.27109 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:51:53.558 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.557+0000 7fc7a8764740 20 client.27109 populate_metadata read hostname 'smithi001' 2024-04-26T21:51:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.557+0000 7fc7a8764740 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5623628b4df0 0x5623628b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.557+0000 7fc7a8764740 4 client.27109 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:51:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.557+0000 7fc7a8764740 1 -- 172.21.15.1:0/697361680 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623627fbf90 con 0x5623628b4df0 2024-04-26T21:51:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.558+0000 7fc79ea00640 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5623628b4df0 0x5623628b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.559+0000 7fc79ea00640 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5623628b4df0 0x5623628b71e0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=417fd375d8298a3f server_cookie=52578d1e1a867fbb in_seq=0 out_seq=0 2024-04-26T21:51:53.561 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.561+0000 7fc78cc00640 10 client.27109 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.562+0000 7fc78cc00640 1 -- 172.21.15.1:0/697361680 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623627fbf90 con 0x5623628b4df0 2024-04-26T21:51:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.562+0000 7fc78cc00640 10 client.27109 handle_command_reply: tid=0 2024-04-26T21:51:53.562 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.562+0000 7fc7a8764740 1 client.27109 shutdown 2024-04-26T21:51:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.563+0000 7fc7a8764740 1 -- 172.21.15.1:0/697361680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362889900 msgr2=0x562362891720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.563+0000 7fc7a8764740 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362889900 0x562362891720 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x562362886680 tx=0x7fc790011630 comp rx=0 tx=0).stop 2024-04-26T21:51:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.563+0000 7fc7a8764740 1 -- 172.21.15.1:0/697361680 shutdown_connections 2024-04-26T21:51:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.563+0000 7fc7a8764740 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x5623628b4df0 0x5623628b71e0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.563+0000 7fc7a8764740 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362889900 0x562362891720 secure :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0x562362886680 tx=0x7fc790011630 comp rx=0 tx=0).stop 2024-04-26T21:51:53.563 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.563+0000 7fc7a8764740 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x562362891c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.563+0000 7fc7a8764740 1 --2- 172.21.15.1:0/697361680 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 0x5623628911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.563+0000 7fc7a8764740 1 -- 172.21.15.1:0/697361680 >> 172.21.15.1:0/697361680 conn(0x5623628854f0 msgr2=0x56236288b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.563+0000 7fc78cc00640 0 client.27109 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.564+0000 7fc7a8764740 1 -- 172.21.15.1:0/697361680 shutdown_connections 2024-04-26T21:51:53.564 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.564+0000 7fc7a8764740 1 -- 172.21.15.1:0/697361680 wait complete. 2024-04-26T21:51:53.565 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.564+0000 7fc7a8764740 20 client.27109 trim_cache size 0 max 16384 2024-04-26T21:51:53.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.599+0000 7fc7a8764740 1 Processor -- start 2024-04-26T21:51:53.599 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.599+0000 7fc7a8764740 1 -- start start 2024-04-26T21:51:53.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.599+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.599+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x56236288b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.599+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362884dd0 0x5623628bcaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.599+0000 7fc7a8764740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a01b8e00 con 0x562362884dd0 2024-04-26T21:51:53.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.599+0000 7fc7a8764740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a01b66d0 con 0x5623628941c0 2024-04-26T21:51:53.600 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.599+0000 7fc7a8764740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a0074520 con 0x562362888fe0 2024-04-26T21:51:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.600+0000 7fc79f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362884dd0 0x5623628bcaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.600+0000 7fc79ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.600+0000 7fc79ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:43404/0 (socket says 172.21.15.1:43404) 2024-04-26T21:51:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.600+0000 7fc79ea00640 1 -- 172.21.15.1:0/3875639115 learned_addr learned my addr 172.21.15.1:0/3875639115 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:53.601 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.601+0000 7fc79e000640 1 --2- 172.21.15.1:0/3875639115 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x56236288b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.602+0000 7fc79f400640 1 -- 172.21.15.1:0/3875639115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 msgr2=0x5623628b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.602+0000 7fc79f400640 1 --2- 172.21.15.1:0/3875639115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.602+0000 7fc79f400640 1 -- 172.21.15.1:0/3875639115 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 msgr2=0x56236288b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:53.602 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.602+0000 7fc79f400640 1 --2- 172.21.15.1:0/3875639115 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x56236288b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.602+0000 7fc79f400640 1 -- 172.21.15.1:0/3875639115 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623625cc620 con 0x562362884dd0 2024-04-26T21:51:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.602+0000 7fc79f400640 1 --2- 172.21.15.1:0/3875639115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362884dd0 0x5623628bcaa0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc798012f30 tx=0x7fc798012ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760f64dd0112003a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:53.603 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.603+0000 7fc78f400640 1 -- 172.21.15.1:0/3875639115 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7980049e0 con 0x562362884dd0 2024-04-26T21:51:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.603+0000 7fc78f400640 1 -- 172.21.15.1:0/3875639115 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc79802a040 con 0x562362884dd0 2024-04-26T21:51:53.604 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.603+0000 7fc78f400640 1 -- 172.21.15.1:0/3875639115 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798004280 con 0x562362884dd0 2024-04-26T21:51:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.605+0000 7fc7a8764740 1 -- 172.21.15.1:0/3875639115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362884dd0 msgr2=0x5623628bcaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.605+0000 7fc7a8764740 1 --2- 172.21.15.1:0/3875639115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362884dd0 0x5623628bcaa0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc798012f30 tx=0x7fc798012ec0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.605+0000 7fc7a8764740 1 -- 172.21.15.1:0/3875639115 shutdown_connections 2024-04-26T21:51:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.605+0000 7fc7a8764740 1 --2- 172.21.15.1:0/3875639115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362884dd0 0x5623628bcaa0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.605+0000 7fc7a8764740 1 --2- 172.21.15.1:0/3875639115 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x56236288b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.605 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.605+0000 7fc7a8764740 1 --2- 172.21.15.1:0/3875639115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362888fe0 0x5623628b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.605+0000 7fc7a8764740 1 -- 172.21.15.1:0/3875639115 >> 172.21.15.1:0/3875639115 conn(0x5623628854f0 msgr2=0x562362891a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.605+0000 7fc7a8764740 1 -- 172.21.15.1:0/3875639115 shutdown_connections 2024-04-26T21:51:53.606 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.606+0000 7fc7a8764740 1 -- 172.21.15.1:0/3875639115 wait complete. 2024-04-26T21:51:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.606+0000 7fc7a8764740 1 -- start start 2024-04-26T21:51:53.607 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.607+0000 7fc7a8764740 1 Processor -- start 2024-04-26T21:51:53.609 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.609+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362884dd0 0x5623628a9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.610+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 0x5623628a9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.610 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.610+0000 7fc7a8764740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x56236291a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.610+0000 7fc7a8764740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a01b4bc0 con 0x562362888fe0 2024-04-26T21:51:53.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.610+0000 7fc7a8764740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a01b6000 con 0x5623628941c0 2024-04-26T21:51:53.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.610+0000 7fc7a8764740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a01b5260 con 0x562362884dd0 2024-04-26T21:51:53.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.610+0000 7fc7a8764740 10 client.0 _ll_register_callbacks cb 0x7fc790009430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:51:53.611 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.610+0000 7fc79f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x56236291a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.610+0000 7fc79ea00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362884dd0 0x5623628a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.610+0000 7fc79f400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x56236291a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:43756/0 (socket says 172.21.15.1:43756) 2024-04-26T21:51:53.613 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.610+0000 7fc79f400640 1 -- 172.21.15.1:0/2247026149 learned_addr learned my addr 172.21.15.1:0/2247026149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:51:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.611+0000 7fc79e000640 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 0x5623628a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.611+0000 7fc79ea00640 1 -- 172.21.15.1:0/2247026149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 msgr2=0x56236291a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.611+0000 7fc79ea00640 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x56236291a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.611+0000 7fc79ea00640 1 -- 172.21.15.1:0/2247026149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 msgr2=0x5623628a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:51:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.611+0000 7fc79ea00640 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 0x5623628a9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.611+0000 7fc79ea00640 1 -- 172.21.15.1:0/2247026149 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623627a8cb0 con 0x562362884dd0 2024-04-26T21:51:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.612+0000 7fc79ea00640 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362884dd0 0x5623628a9430 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc788005ec0 tx=0x7fc788031f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=529d07d3bc671a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:51:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.612+0000 7fc78cc00640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:51:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.612+0000 7fc78cc00640 1 -- 172.21.15.1:0/2247026149 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc788049040 con 0x562362884dd0 2024-04-26T21:51:53.614 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.613+0000 7fc78cc00640 1 -- 172.21.15.1:0/2247026149 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(132..132 src has 1..132) ==== 9626+0+0 (secure 0 0 0) 0x7fc7880dd040 con 0x562362884dd0 2024-04-26T21:51:53.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.613+0000 7fc79f400640 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x56236291a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:51:53.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.613+0000 7fc7a8764740 1 -- 172.21.15.1:0/2247026149 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc79802a040 con 0x562362884dd0 2024-04-26T21:51:53.615 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.613+0000 7fc78cc00640 1 -- 172.21.15.1:0/2247026149 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7fc788049040 con 0x562362884dd0 2024-04-26T21:51:53.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.627+0000 7fc7a8764740 10 client.27115 fetch_fsmap learned FSMap version 59 2024-04-26T21:51:53.627 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.627+0000 7fc7a8764740 1 -- 172.21.15.1:0/2247026149 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x56236291bac0 con 0x562362884dd0 2024-04-26T21:51:53.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.628+0000 7fc78cc00640 1 -- 172.21.15.1:0/2247026149 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7880c13c0 con 0x562362884dd0 2024-04-26T21:51:53.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.629+0000 7fc78cc00640 1 -- 172.21.15.1:0/2247026149 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7fc7880f1120 con 0x562362884dd0 2024-04-26T21:51:53.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.630+0000 7fc7a8764740 10 client.27115 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:51:53.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.630+0000 7fc7a8764740 10 client.27115 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:51:53.630 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.630+0000 7fc7a8764740 20 client.27115 populate_metadata read hostname 'smithi001' 2024-04-26T21:51:53.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.630+0000 7fc7a8764740 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56236291bf90 0x56236291e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:51:53.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.630+0000 7fc7a8764740 4 client.27115 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:51:53.631 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.630+0000 7fc7a8764740 1 -- 172.21.15.1:0/2247026149 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fc79802a040 con 0x56236291bf90 2024-04-26T21:51:53.634 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.634+0000 7fc79e000640 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56236291bf90 0x56236291e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:51:53.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.639+0000 7fc79e000640 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56236291bf90 0x56236291e3b0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c322d7cf3c8cdc4 server_cookie=bf1ff24f983dcc5d in_seq=0 out_seq=0 2024-04-26T21:51:53.639 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:53.639+0000 7fc78cc00640 10 client.27115 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:54.194 INFO:tasks.workunit.client.0.smithi001.stdout: 10 187399 20.35 MB/sec execute 351 sec latency 507.147 ms 2024-04-26T21:51:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.482+0000 7fc78cc00640 1 -- 172.21.15.1:0/2247026149 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fc79802a040 con 0x56236291bf90 2024-04-26T21:51:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.482+0000 7fc78cc00640 10 client.27115 handle_command_reply: tid=0 2024-04-26T21:51:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.482+0000 7fc7a8764740 1 client.27115 shutdown 2024-04-26T21:51:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.483+0000 7fc7a8764740 1 -- 172.21.15.1:0/2247026149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362884dd0 msgr2=0x5623628a9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.483+0000 7fc7a8764740 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362884dd0 0x5623628a9430 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc788005ec0 tx=0x7fc788031f10 comp rx=0 tx=0).stop 2024-04-26T21:51:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.483+0000 7fc7a8764740 1 -- 172.21.15.1:0/2247026149 shutdown_connections 2024-04-26T21:51:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.483+0000 7fc7a8764740 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x56236291bf90 0x56236291e3b0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.483+0000 7fc7a8764740 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5623628941c0 0x56236291a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.483+0000 7fc7a8764740 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x562362888fe0 0x5623628a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.483+0000 7fc7a8764740 1 --2- 172.21.15.1:0/2247026149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562362884dd0 0x5623628a9430 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.484+0000 7fc7a8764740 1 -- 172.21.15.1:0/2247026149 >> 172.21.15.1:0/2247026149 conn(0x5623628854f0 msgr2=0x562362890fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.483+0000 7fc78cc00640 0 client.27115 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:51:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.484+0000 7fc7a8764740 1 -- 172.21.15.1:0/2247026149 shutdown_connections 2024-04-26T21:51:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.484+0000 7fc7a8764740 1 -- 172.21.15.1:0/2247026149 wait complete. 2024-04-26T21:51:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.484+0000 7fc7a8764740 20 client.27115 trim_cache size 0 max 16384 2024-04-26T21:51:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:51:54.485 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:51:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:51:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:51:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.486+0000 7fc79fe00640 1 -- 172.21.15.1:0/593577568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc7840a3e10 msgr2=0x7fc7840a62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.486+0000 7fc79fe00640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc7840a3e10 0x7fc7840a62d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc798020000 tx=0x7fc79801d000 comp rx=0 tx=0).stop 2024-04-26T21:51:54.486 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.486+0000 7fc79fe00640 1 -- 172.21.15.1:0/593577568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0103910 msgr2=0x7fc7a010c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:51:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.486+0000 7fc79fe00640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0103910 0x7fc7a010c130 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc788005ab0 tx=0x7fc788003cd0 comp rx=0 tx=0).stop 2024-04-26T21:51:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.486+0000 7fc79fe00640 1 -- 172.21.15.1:0/593577568 shutdown_connections 2024-04-26T21:51:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.486+0000 7fc79fe00640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fc7840a3e10 0x7fc7840a62d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.486+0000 7fc79fe00640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc7a010cd30 0x7fc7a01062c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.486+0000 7fc79fe00640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7a0104230 0x7fc7a010c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.486+0000 7fc79fe00640 1 --2- 172.21.15.1:0/593577568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0103910 0x7fc7a010c130 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:51:54.487 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.486+0000 7fc79fe00640 1 -- 172.21.15.1:0/593577568 >> 172.21.15.1:0/593577568 conn(0x7fc7a00fd110 msgr2=0x7fc7a00fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:51:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.487+0000 7fc79fe00640 1 -- 172.21.15.1:0/593577568 shutdown_connections 2024-04-26T21:51:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:51:54.487+0000 7fc79fe00640 1 -- 172.21.15.1:0/593577568 wait complete. 2024-04-26T21:51:54.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:54 smithi001 ceph-mon[28054]: pgmap v686: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 11 MiB/s wr, 502 op/s 2024-04-26T21:51:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:54 smithi038 ceph-mon[32724]: pgmap v686: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 11 MiB/s wr, 502 op/s 2024-04-26T21:51:54.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:54 smithi186 ceph-mon[32991]: pgmap v686: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 11 MiB/s wr, 502 op/s 2024-04-26T21:51:55.194 INFO:tasks.workunit.client.0.smithi001.stdout: 10 187631 20.31 MB/sec execute 352 sec latency 558.431 ms 2024-04-26T21:51:56.194 INFO:tasks.workunit.client.0.smithi001.stdout: 10 188124 20.34 MB/sec execute 353 sec latency 81.016 ms 2024-04-26T21:51:56.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:56 smithi001 ceph-mon[28054]: pgmap v687: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 15 MiB/s wr, 608 op/s 2024-04-26T21:51:56.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:56 smithi038 ceph-mon[32724]: pgmap v687: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 15 MiB/s wr, 608 op/s 2024-04-26T21:51:56.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:56 smithi186 ceph-mon[32991]: pgmap v687: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 15 MiB/s wr, 608 op/s 2024-04-26T21:51:57.194 INFO:tasks.workunit.client.0.smithi001.stdout: 10 188433 20.34 MB/sec execute 354 sec latency 126.572 ms 2024-04-26T21:51:58.194 INFO:tasks.workunit.client.0.smithi001.stdout: 10 188864 20.34 MB/sec execute 355 sec latency 108.899 ms 2024-04-26T21:51:58.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:51:58 smithi001 ceph-mon[28054]: pgmap v688: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 14 MiB/s wr, 559 op/s 2024-04-26T21:51:58.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:51:58 smithi038 ceph-mon[32724]: pgmap v688: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 14 MiB/s wr, 559 op/s 2024-04-26T21:51:58.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:51:58 smithi186 ceph-mon[32991]: pgmap v688: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 14 MiB/s wr, 559 op/s 2024-04-26T21:51:59.197 INFO:tasks.workunit.client.0.smithi001.stdout: 10 189337 20.32 MB/sec execute 356 sec latency 123.963 ms 2024-04-26T21:52:00.195 INFO:tasks.workunit.client.0.smithi001.stdout: 10 190009 20.37 MB/sec execute 357 sec latency 103.518 ms 2024-04-26T21:52:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:00 smithi001 ceph-mon[28054]: pgmap v689: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 14 MiB/s wr, 577 op/s 2024-04-26T21:52:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:00 smithi038 ceph-mon[32724]: pgmap v689: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 14 MiB/s wr, 577 op/s 2024-04-26T21:52:00.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:00 smithi186 ceph-mon[32991]: pgmap v689: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 14 MiB/s wr, 577 op/s 2024-04-26T21:52:01.195 INFO:tasks.workunit.client.0.smithi001.stdout: 10 190524 20.40 MB/sec execute 358 sec latency 234.708 ms 2024-04-26T21:52:02.195 INFO:tasks.workunit.client.0.smithi001.stdout: 10 190693 20.36 MB/sec execute 359 sec latency 374.817 ms 2024-04-26T21:52:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:02 smithi001 ceph-mon[28054]: pgmap v690: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 15 MiB/s wr, 685 op/s 2024-04-26T21:52:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:02 smithi038 ceph-mon[32724]: pgmap v690: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 15 MiB/s wr, 685 op/s 2024-04-26T21:52:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:02 smithi186 ceph-mon[32991]: pgmap v690: 129 pgs: 129 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 15 MiB/s wr, 685 op/s 2024-04-26T21:52:03.195 INFO:tasks.workunit.client.0.smithi001.stdout: 10 190705 20.31 MB/sec execute 360 sec latency 414.626 ms 2024-04-26T21:52:04.195 INFO:tasks.workunit.client.0.smithi001.stdout: 10 190714 20.26 MB/sec execute 361 sec latency 431.397 ms 2024-04-26T21:52:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:04 smithi001 ceph-mon[28054]: pgmap v691: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.8 MiB/s wr, 545 op/s 2024-04-26T21:52:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:04 smithi038 ceph-mon[32724]: pgmap v691: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.8 MiB/s wr, 545 op/s 2024-04-26T21:52:04.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:04 smithi186 ceph-mon[32991]: pgmap v691: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 9.8 MiB/s wr, 545 op/s 2024-04-26T21:52:05.195 INFO:tasks.workunit.client.0.smithi001.stdout: 10 190729 20.21 MB/sec execute 362 sec latency 531.341 ms 2024-04-26T21:52:06.196 INFO:tasks.workunit.client.0.smithi001.stdout: 10 190891 20.19 MB/sec execute 363 sec latency 535.798 ms 2024-04-26T21:52:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:06 smithi038 ceph-mon[32724]: pgmap v692: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.3 MiB/s wr, 597 op/s 2024-04-26T21:52:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:06 smithi186 ceph-mon[32991]: pgmap v692: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.3 MiB/s wr, 597 op/s 2024-04-26T21:52:06.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:06 smithi001 ceph-mon[28054]: pgmap v692: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 9.3 MiB/s wr, 597 op/s 2024-04-26T21:52:07.196 INFO:tasks.workunit.client.0.smithi001.stdout: 10 191108 20.16 MB/sec execute 364 sec latency 443.699 ms 2024-04-26T21:52:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.711+0000 7f00ed600640 1 -- 172.21.15.1:0/2947894677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8104960 msgr2=0x7f00e8104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.711+0000 7f00ed600640 1 --2- 172.21.15.1:0/2947894677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8104960 0x7f00e8104de0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009930 tx=0x7f00d002a420 comp rx=0 tx=0).stop 2024-04-26T21:52:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.713+0000 7f00ed600640 1 -- 172.21.15.1:0/2947894677 shutdown_connections 2024-04-26T21:52:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.713+0000 7f00ed600640 1 --2- 172.21.15.1:0/2947894677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8104960 0x7f00e8104de0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.713+0000 7f00ed600640 1 --2- 172.21.15.1:0/2947894677 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e8105320 0x7f00e8111830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.713+0000 7f00ed600640 1 --2- 172.21.15.1:0/2947894677 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00e810a970 0x7f00e810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.713+0000 7f00ed600640 1 -- 172.21.15.1:0/2947894677 >> 172.21.15.1:0/2947894677 conn(0x7f00e8100430 msgr2=0x7f00e8102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:07.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.714+0000 7f00ed600640 1 -- 172.21.15.1:0/2947894677 shutdown_connections 2024-04-26T21:52:07.714 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.714+0000 7f00ed600640 1 -- 172.21.15.1:0/2947894677 wait complete. 2024-04-26T21:52:07.715 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.715+0000 7f00ed600640 1 Processor -- start 2024-04-26T21:52:07.716 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.716+0000 7f00ed600640 1 -- start start 2024-04-26T21:52:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.716+0000 7f00ed600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00e8104960 0x7f00e81117b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.716+0000 7f00ed600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8105320 0x7f00e810e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.717+0000 7f00ed600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e810a970 0x7f00e810f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.717+0000 7f00dfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8105320 0x7f00e810e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.717+0000 7f00dfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8105320 0x7f00e810e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54452/0 (socket says 172.21.15.1:54452) 2024-04-26T21:52:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.720+0000 7f00dfe00640 1 -- 172.21.15.1:0/1693478940 learned_addr learned my addr 172.21.15.1:0/1693478940 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.720+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f00e8114c80 con 0x7f00e810a970 2024-04-26T21:52:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.720+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f00e8114b00 con 0x7f00e8104960 2024-04-26T21:52:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.720+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f00e8114e00 con 0x7f00e8105320 2024-04-26T21:52:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.721+0000 7f00ecc00640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e810a970 0x7f00e810f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.721+0000 7f00dfe00640 1 -- 172.21.15.1:0/1693478940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00e8104960 msgr2=0x7f00e81117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.721+0000 7f00dfe00640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00e8104960 0x7f00e81117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.721+0000 7f00dfe00640 1 -- 172.21.15.1:0/1693478940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e810a970 msgr2=0x7f00e810f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.721+0000 7f00dfe00640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e810a970 0x7f00e810f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.721+0000 7f00dfe00640 1 -- 172.21.15.1:0/1693478940 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e81ab960 con 0x7f00e8105320 2024-04-26T21:52:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.722+0000 7f00dfe00640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8105320 0x7f00e810e8f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f00d002f550 tx=0x7f00d0003c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.722+0000 7f00e5600640 1 -- 172.21.15.1:0/1693478940 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d003f070 con 0x7f00e8105320 2024-04-26T21:52:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.722+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e81abb30 con 0x7f00e8105320 2024-04-26T21:52:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.722+0000 7f00e5600640 1 -- 172.21.15.1:0/1693478940 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00d002fc60 con 0x7f00e8105320 2024-04-26T21:52:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.722+0000 7f00e5600640 1 -- 172.21.15.1:0/1693478940 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d0035030 con 0x7f00e8105320 2024-04-26T21:52:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.723+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e81ac0d0 con 0x7f00e8105320 2024-04-26T21:52:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.732+0000 7f00f1b64740 1 Processor -- start 2024-04-26T21:52:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.732+0000 7f00f1b64740 1 -- start start 2024-04-26T21:52:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.733+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f293c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.733 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.733+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f33c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.733+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f365b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.733+0000 7f00f1b64740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558fd7e8e380 con 0x558fd7f29900 2024-04-26T21:52:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.733+0000 7f00f1b64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558fd7e2ac30 con 0x558fd7f28fe0 2024-04-26T21:52:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.733+0000 7f00f1b64740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558fd7e8a340 con 0x558fd7f341c0 2024-04-26T21:52:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.733+0000 7f00dfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.733+0000 7f00dfe00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f365b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54458/0 (socket says 172.21.15.1:54458) 2024-04-26T21:52:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.734+0000 7f00dfe00640 1 -- 172.21.15.1:0/2045778874 learned_addr learned my addr 172.21.15.1:0/2045778874 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:07.734 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.734+0000 7f00e5600640 1 -- 172.21.15.1:0/1693478940 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f00d0042070 con 0x7f00e8105320 2024-04-26T21:52:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.734+0000 7f00ecc00640 1 --2- 172.21.15.1:0/2045778874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f293c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.734+0000 7f00e7e00640 1 --2- 172.21.15.1:0/2045778874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f33c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.735 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.735+0000 7f00dfe00640 1 -- 172.21.15.1:0/2045778874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 msgr2=0x558fd7f293c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:52:07.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.735+0000 7f00dfe00640 1 --2- 172.21.15.1:0/2045778874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f293c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.735+0000 7f00dfe00640 1 -- 172.21.15.1:0/2045778874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 msgr2=0x558fd7f33c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:52:07.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.735+0000 7f00dfe00640 1 --2- 172.21.15.1:0/2045778874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f33c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.735+0000 7f00dfe00640 1 -- 172.21.15.1:0/2045778874 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fd7e4c940 con 0x558fd7f341c0 2024-04-26T21:52:07.736 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.735+0000 7f00ecc00640 1 --2- 172.21.15.1:0/2045778874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f293c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:52:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.737+0000 7f00e5600640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f00cc0a3e60 0x7f00cc0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.736+0000 7f00dfe00640 1 --2- 172.21.15.1:0/2045778874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f365b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f00d00ee8c0 tx=0x7f00d00f7a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3da707724b188123 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.737+0000 7f00e5600640 1 -- 172.21.15.1:0/1693478940 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7f00d00f1750 con 0x7f00e8105320 2024-04-26T21:52:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.740+0000 7f00ecc00640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f00cc0a3e60 0x7f00cc0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.742+0000 7f00dd200640 1 -- 172.21.15.1:0/2045778874 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d0049070 con 0x558fd7f341c0 2024-04-26T21:52:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.742+0000 7f00dd200640 1 -- 172.21.15.1:0/2045778874 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00d002fe10 con 0x558fd7f341c0 2024-04-26T21:52:07.742 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.742+0000 7f00dd200640 1 -- 172.21.15.1:0/2045778874 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d00bcd10 con 0x558fd7f341c0 2024-04-26T21:52:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.743+0000 7f00f1b64740 1 -- 172.21.15.1:0/2045778874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 msgr2=0x558fd7f365b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:07.743 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.743+0000 7f00f1b64740 1 --2- 172.21.15.1:0/2045778874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f365b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f00d00ee8c0 tx=0x7f00d00f7a60 comp rx=0 tx=0).stop 2024-04-26T21:52:07.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.744+0000 7f00f1b64740 1 -- 172.21.15.1:0/2045778874 shutdown_connections 2024-04-26T21:52:07.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.744+0000 7f00f1b64740 1 --2- 172.21.15.1:0/2045778874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f365b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.744+0000 7f00f1b64740 1 --2- 172.21.15.1:0/2045778874 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f33c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.744+0000 7f00f1b64740 1 --2- 172.21.15.1:0/2045778874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f293c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.744 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.744+0000 7f00f1b64740 1 -- 172.21.15.1:0/2045778874 >> 172.21.15.1:0/2045778874 conn(0x558fd7f254f0 msgr2=0x558fd7f258d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:07.745 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.745+0000 7f00ecc00640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f00cc0a3e60 0x7f00cc0a6320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00e0005fd0 tx=0x7f00e0009500 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:07.745 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.745+0000 7f00f1b64740 1 -- 172.21.15.1:0/2045778874 shutdown_connections 2024-04-26T21:52:07.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.745+0000 7f00f1b64740 1 -- 172.21.15.1:0/2045778874 wait complete. 2024-04-26T21:52:07.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.746+0000 7f00f1b64740 1 -- start start 2024-04-26T21:52:07.746 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.746+0000 7f00f1b64740 1 Processor -- start 2024-04-26T21:52:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.747+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f310c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.747+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f31600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.747 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.747+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f31b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00f1b64740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x558fd7f36bb0 con 0x558fd7f29900 2024-04-26T21:52:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00ecc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f31b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00f1b64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x558fd7e15c60 con 0x558fd7f28fe0 2024-04-26T21:52:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00ecc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f31b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54464/0 (socket says 172.21.15.1:54464) 2024-04-26T21:52:07.748 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00ecc00640 1 -- 172.21.15.1:0/4116519450 learned_addr learned my addr 172.21.15.1:0/4116519450 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00f1b64740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558fd7f36d30 con 0x558fd7f341c0 2024-04-26T21:52:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00f1b64740 10 client.0 _ll_register_callbacks cb 0x558fd7e41e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:52:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00ecc00640 1 -- 172.21.15.1:0/4116519450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 msgr2=0x558fd7f310c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:52:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00ecc00640 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f310c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00ecc00640 1 -- 172.21.15.1:0/4116519450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 msgr2=0x558fd7f31600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:52:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00ecc00640 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f31600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.749 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.748+0000 7f00ecc00640 1 -- 172.21.15.1:0/4116519450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fd7e24510 con 0x558fd7f341c0 2024-04-26T21:52:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.749+0000 7f00ecc00640 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f31b40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f00e0010bf0 tx=0x7f00e0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69fd7c2d59d2e29e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.749+0000 7f00d5600640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-26T21:52:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.750+0000 7f00d5600640 1 -- 172.21.15.1:0/4116519450 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e000e7f0 con 0x558fd7f341c0 2024-04-26T21:52:07.750 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.750+0000 7f00f1b64740 1 -- 172.21.15.1:0/4116519450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00d002fe10 con 0x558fd7f341c0 2024-04-26T21:52:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.750+0000 7f00d5600640 1 -- 172.21.15.1:0/4116519450 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7f00e000e5f0 con 0x558fd7f341c0 2024-04-26T21:52:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.751+0000 7f00d5600640 1 -- 172.21.15.1:0/4116519450 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f00e000e7f0 con 0x558fd7f341c0 2024-04-26T21:52:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.751+0000 7f00f1b64740 10 client.27127 fetch_fsmap learned FSMap version 59 2024-04-26T21:52:07.751 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.751+0000 7f00f1b64740 1 -- 172.21.15.1:0/4116519450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x558fd7ba0210 con 0x558fd7f341c0 2024-04-26T21:52:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.753+0000 7f00d5600640 1 -- 172.21.15.1:0/4116519450 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00d002fe10 con 0x558fd7f341c0 2024-04-26T21:52:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.753+0000 7f00d5600640 1 -- 172.21.15.1:0/4116519450 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7f00e0027c30 con 0x558fd7f341c0 2024-04-26T21:52:07.753 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.753+0000 7f00f1b64740 10 client.27127 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:52:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.753+0000 7f00f1b64740 10 client.27127 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:52:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.753+0000 7f00f1b64740 20 client.27127 populate_metadata read hostname 'smithi001' 2024-04-26T21:52:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.753+0000 7f00f1b64740 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558fd7f43d10 0x558fd7f461a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.753+0000 7f00f1b64740 4 client.27127 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-26T21:52:07.754 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.753+0000 7f00f1b64740 1 -- 172.21.15.1:0/4116519450 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fd7e9bf90 con 0x558fd7f43d10 2024-04-26T21:52:07.755 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.754+0000 7f00e7e00640 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558fd7f43d10 0x558fd7f461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.757 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.757+0000 7f00e7e00640 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558fd7f43d10 0x558fd7f461a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33af1c388c0fca25 server_cookie=435b10b511e0bf02 in_seq=0 out_seq=0 2024-04-26T21:52:07.758 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.757+0000 7f00d5600640 10 client.27127 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:52:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.759+0000 7f00d5600640 1 -- 172.21.15.1:0/4116519450 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558fd7e9bf90 con 0x558fd7f43d10 2024-04-26T21:52:07.759 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.759+0000 7f00d5600640 10 client.27127 handle_command_reply: tid=0 2024-04-26T21:52:07.761 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.761+0000 7f00f1b64740 1 client.27127 shutdown 2024-04-26T21:52:07.761 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.761+0000 7f00f1b64740 1 -- 172.21.15.1:0/4116519450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 msgr2=0x558fd7f31b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:07.761 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.761+0000 7f00f1b64740 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f31b40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f00e0010bf0 tx=0x7f00e0004a40 comp rx=0 tx=0).stop 2024-04-26T21:52:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.761+0000 7f00f1b64740 1 -- 172.21.15.1:0/4116519450 shutdown_connections 2024-04-26T21:52:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.761+0000 7f00f1b64740 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558fd7f43d10 0x558fd7f461a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.761+0000 7f00f1b64740 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f31b40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.761+0000 7f00f1b64740 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f31600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.761+0000 7f00f1b64740 1 --2- 172.21.15.1:0/4116519450 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f310c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.761+0000 7f00d5600640 0 client.27127 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:52:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.761+0000 7f00f1b64740 1 -- 172.21.15.1:0/4116519450 >> 172.21.15.1:0/4116519450 conn(0x558fd7f254f0 msgr2=0x558fd7f2b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:07.762 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.762+0000 7f00f1b64740 1 -- 172.21.15.1:0/4116519450 shutdown_connections 2024-04-26T21:52:07.763 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.763+0000 7f00f1b64740 1 -- 172.21.15.1:0/4116519450 wait complete. 2024-04-26T21:52:07.765 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.763+0000 7f00f1b64740 20 client.27127 trim_cache size 0 max 16384 2024-04-26T21:52:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.796+0000 7f00f1b64740 1 Processor -- start 2024-04-26T21:52:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.796+0000 7f00f1b64740 1 -- start start 2024-04-26T21:52:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f28fe0 0x558fd7f3f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f29900 0x558fd7f30260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f341c0 0x558fd7f2b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00f1b64740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b8e60 con 0x558fd7f341c0 2024-04-26T21:52:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00f1b64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b6730 con 0x558fd7f29900 2024-04-26T21:52:07.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00f1b64740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b45a0 con 0x558fd7f28fe0 2024-04-26T21:52:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00e7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f28fe0 0x558fd7f3f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00e7e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f28fe0 0x558fd7f3f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54480/0 (socket says 172.21.15.1:54480) 2024-04-26T21:52:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00e7e00640 1 -- 172.21.15.1:0/536196225 learned_addr learned my addr 172.21.15.1:0/536196225 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00dfe00640 1 --2- 172.21.15.1:0/536196225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f29900 0x558fd7f30260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.798 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.797+0000 7f00ecc00640 1 --2- 172.21.15.1:0/536196225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f341c0 0x558fd7f2b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.799 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.799+0000 7f00e7e00640 1 -- 172.21.15.1:0/536196225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f29900 msgr2=0x558fd7f30260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:52:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.799+0000 7f00e7e00640 1 --2- 172.21.15.1:0/536196225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f29900 0x558fd7f30260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.799+0000 7f00e7e00640 1 -- 172.21.15.1:0/536196225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f341c0 msgr2=0x558fd7f2b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:52:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.799+0000 7f00e7e00640 1 --2- 172.21.15.1:0/536196225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f341c0 0x558fd7f2b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.799+0000 7f00e7e00640 1 -- 172.21.15.1:0/536196225 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fd7c6c620 con 0x558fd7f28fe0 2024-04-26T21:52:07.800 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.800+0000 7f00e7e00640 1 --2- 172.21.15.1:0/536196225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f28fe0 0x558fd7f3f9e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x558fd7f46a20 tx=0x7f00d8005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a341a8d04a5ff0c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.802+0000 7f00d7e00640 1 -- 172.21.15.1:0/536196225 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8012900 con 0x558fd7f28fe0 2024-04-26T21:52:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.802+0000 7f00d7e00640 1 -- 172.21.15.1:0/536196225 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00d8018250 con 0x558fd7f28fe0 2024-04-26T21:52:07.802 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.802+0000 7f00d7e00640 1 -- 172.21.15.1:0/536196225 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d80106c0 con 0x558fd7f28fe0 2024-04-26T21:52:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.803+0000 7f00f1b64740 1 -- 172.21.15.1:0/536196225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f28fe0 msgr2=0x558fd7f3f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:07.803 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.803+0000 7f00f1b64740 1 --2- 172.21.15.1:0/536196225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f28fe0 0x558fd7f3f9e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x558fd7f46a20 tx=0x7f00d8005d50 comp rx=0 tx=0).stop 2024-04-26T21:52:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.805+0000 7f00f1b64740 1 -- 172.21.15.1:0/536196225 shutdown_connections 2024-04-26T21:52:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.805+0000 7f00f1b64740 1 --2- 172.21.15.1:0/536196225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f341c0 0x558fd7f2b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.805+0000 7f00f1b64740 1 --2- 172.21.15.1:0/536196225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f29900 0x558fd7f30260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.805+0000 7f00f1b64740 1 --2- 172.21.15.1:0/536196225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f28fe0 0x558fd7f3f9e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.805 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.805+0000 7f00f1b64740 1 -- 172.21.15.1:0/536196225 >> 172.21.15.1:0/536196225 conn(0x558fd7f254f0 msgr2=0x558fd7f23dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:07.806 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.805+0000 7f00f1b64740 1 -- 172.21.15.1:0/536196225 shutdown_connections 2024-04-26T21:52:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.805+0000 7f00f1b64740 1 -- 172.21.15.1:0/536196225 wait complete. 2024-04-26T21:52:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.806+0000 7f00f1b64740 1 -- start start 2024-04-26T21:52:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.806+0000 7f00f1b64740 1 Processor -- start 2024-04-26T21:52:07.809 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.809+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f3be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.809+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f3c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.810+0000 7f00f1b64740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f2d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.810 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.810+0000 7f00e7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f3be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.810+0000 7f00e7e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f3be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:57558/0 (socket says 172.21.15.1:57558) 2024-04-26T21:52:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.810+0000 7f00f1b64740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b4bf0 con 0x558fd7f29900 2024-04-26T21:52:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.810+0000 7f00f1b64740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b6060 con 0x558fd7f28fe0 2024-04-26T21:52:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.810+0000 7f00f1b64740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81b52c0 con 0x558fd7f341c0 2024-04-26T21:52:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.810+0000 7f00f1b64740 10 client.0 _ll_register_callbacks cb 0x7f00e0010870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-26T21:52:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.810+0000 7f00ecc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f2d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.810+0000 7f00ecc00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f2d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54496/0 (socket says 172.21.15.1:54496) 2024-04-26T21:52:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.810+0000 7f00ecc00640 1 -- 172.21.15.1:0/1658740048 learned_addr learned my addr 172.21.15.1:0/1658740048 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.811+0000 7f00dfe00640 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f3c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.811+0000 7f00e7e00640 1 -- 172.21.15.1:0/1658740048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 msgr2=0x558fd7f2d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:52:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.814+0000 7f00e7e00640 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f2d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.814+0000 7f00e7e00640 1 -- 172.21.15.1:0/1658740048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 msgr2=0x558fd7f3c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T21:52:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.814+0000 7f00e7e00640 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f3c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.814+0000 7f00e7e00640 1 -- 172.21.15.1:0/1658740048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fd7e48cb0 con 0x558fd7f28fe0 2024-04-26T21:52:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.814+0000 7f00ecc00640 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f2d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:52:07.815 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.814+0000 7f00dfe00640 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f3c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:52:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.815+0000 7f00e7e00640 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f3be90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f00d800ba40 tx=0x7f00d800bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc788a9f69e4867 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.815+0000 7f00d5600640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-04-26T21:52:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.815+0000 7f00d5600640 1 -- 172.21.15.1:0/1658740048 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8004240 con 0x558fd7f28fe0 2024-04-26T21:52:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.816+0000 7f00d5600640 1 -- 172.21.15.1:0/1658740048 <== mon.2 v2:172.21.15.38:3300/0 2 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7f00d80043e0 con 0x558fd7f28fe0 2024-04-26T21:52:07.816 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.816+0000 7f00f1b64740 1 -- 172.21.15.1:0/1658740048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00d8018250 con 0x558fd7f28fe0 2024-04-26T21:52:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.818+0000 7f00d5600640 1 -- 172.21.15.1:0/1658740048 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) ==== 24+0+0 (secure 0 0 0) 0x7f00d8030050 con 0x558fd7f28fe0 2024-04-26T21:52:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.818+0000 7f00f1b64740 10 client.27119 fetch_fsmap learned FSMap version 59 2024-04-26T21:52:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.818+0000 7f00f1b64740 1 -- 172.21.15.1:0/1658740048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) -- 0x558fd7fa9dc0 con 0x558fd7f28fe0 2024-04-26T21:52:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.819+0000 7f00d5600640 1 -- 172.21.15.1:0/1658740048 <== mon.2 v2:172.21.15.38:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f00d8018250 con 0x558fd7f28fe0 2024-04-26T21:52:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.819+0000 7f00d5600640 1 -- 172.21.15.1:0/1658740048 <== mon.2 v2:172.21.15.38:3300/0 5 ==== fsmap(e 59) ==== 8346+0+0 (secure 0 0 0) 0x7f00d80035f0 con 0x558fd7f28fe0 2024-04-26T21:52:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.819+0000 7f00f1b64740 10 client.27119 fetch_fsmap finished waiting for FSMap version 59 2024-04-26T21:52:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.819+0000 7f00f1b64740 10 client.27119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-26T21:52:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.819+0000 7f00f1b64740 20 client.27119 populate_metadata read hostname 'smithi001' 2024-04-26T21:52:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.819+0000 7f00f1b64740 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558fd7faa490 0x558fd7fac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.819+0000 7f00f1b64740 4 client.27119 mds_command: new command op to 24602 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-26T21:52:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.819+0000 7f00f1b64740 1 -- 172.21.15.1:0/1658740048 --> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558fd7e95e00 con 0x558fd7faa490 2024-04-26T21:52:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.823+0000 7f00dfe00640 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558fd7faa490 0x558fd7fac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.825+0000 7f00dfe00640 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558fd7faa490 0x558fd7fac8b0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c30b040c0444a806 server_cookie=b8d9cde463543c5a in_seq=0 out_seq=0 2024-04-26T21:52:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:07.825+0000 7f00d5600640 10 client.27119 ms_handle_connect on v2:172.21.15.38:6838/1675711617 2024-04-26T21:52:08.196 INFO:tasks.workunit.client.0.smithi001.stdout: 10 191246 20.13 MB/sec execute 365 sec latency 1099.262 ms 2024-04-26T21:52:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:08 smithi186 ceph-mon[32991]: pgmap v693: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.9 MiB/s wr, 494 op/s 2024-04-26T21:52:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:08 smithi186 ceph-mon[32991]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T21:52:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.789+0000 7f00d5600640 1 -- 172.21.15.1:0/1658740048 <== mds.0 v2:172.21.15.38:6838/1675711617 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558fd7e95e00 con 0x558fd7faa490 2024-04-26T21:52:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.789+0000 7f00d5600640 10 client.27119 handle_command_reply: tid=0 2024-04-26T21:52:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.789+0000 7f00f1b64740 1 client.27119 shutdown 2024-04-26T21:52:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.790+0000 7f00f1b64740 1 -- 172.21.15.1:0/1658740048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 msgr2=0x558fd7f3be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.790+0000 7f00f1b64740 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f3be90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f00d800ba40 tx=0x7f00d800bcf0 comp rx=0 tx=0).stop 2024-04-26T21:52:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.791+0000 7f00f1b64740 1 -- 172.21.15.1:0/1658740048 shutdown_connections 2024-04-26T21:52:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.791+0000 7f00f1b64740 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.38:6838/1675711617,v1:172.21.15.38:6839/1675711617] conn(0x558fd7faa490 0x558fd7fac8b0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.791+0000 7f00f1b64740 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558fd7f341c0 0x558fd7f2d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.791+0000 7f00f1b64740 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x558fd7f29900 0x558fd7f3c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.791+0000 7f00f1b64740 1 --2- 172.21.15.1:0/1658740048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558fd7f28fe0 0x558fd7f3be90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.791+0000 7f00f1b64740 1 -- 172.21.15.1:0/1658740048 >> 172.21.15.1:0/1658740048 conn(0x558fd7f254f0 msgr2=0x558fd7f24650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.791+0000 7f00d5600640 0 client.27119 ms_handle_reset on v2:172.21.15.38:6838/1675711617 2024-04-26T21:52:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.792+0000 7f00f1b64740 1 -- 172.21.15.1:0/1658740048 shutdown_connections 2024-04-26T21:52:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.792+0000 7f00f1b64740 1 -- 172.21.15.1:0/1658740048 wait complete. 2024-04-26T21:52:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.792+0000 7f00f1b64740 20 client.27119 trim_cache size 0 max 16384 2024-04-26T21:52:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:{ 2024-04-26T21:52:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "message": "", 2024-04-26T21:52:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout: "return_code": 0 2024-04-26T21:52:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stdout:} 2024-04-26T21:52:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.795+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f00cc0a3e60 msgr2=0x7f00cc0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.795+0000 7f00ed600640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f00cc0a3e60 0x7f00cc0a6320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00e0005fd0 tx=0x7f00e0009500 comp rx=0 tx=0).stop 2024-04-26T21:52:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.796+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8105320 msgr2=0x7f00e810e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.796+0000 7f00ed600640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8105320 0x7f00e810e8f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f00d002f550 tx=0x7f00d0003c00 comp rx=0 tx=0).stop 2024-04-26T21:52:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.796+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 shutdown_connections 2024-04-26T21:52:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.796+0000 7f00ed600640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f00cc0a3e60 0x7f00cc0a6320 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.796+0000 7f00ed600640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00e8105320 0x7f00e810e8f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.796+0000 7f00ed600640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00e810a970 0x7f00e810f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.796+0000 7f00ed600640 1 --2- 172.21.15.1:0/1693478940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00e8104960 0x7f00e81117b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.796+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 >> 172.21.15.1:0/1693478940 conn(0x7f00e8100430 msgr2=0x7f00e8100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.796+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 shutdown_connections 2024-04-26T21:52:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi001.front.sepia.ceph.com.smithi001.stderr:2024-04-26T21:52:08.797+0000 7f00ed600640 1 -- 172.21.15.1:0/1693478940 wait complete. 2024-04-26T21:52:08.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:08 smithi001 ceph-mon[28054]: pgmap v693: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.9 MiB/s wr, 494 op/s 2024-04-26T21:52:08.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:08 smithi001 ceph-mon[28054]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T21:52:08.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:08 smithi038 ceph-mon[32724]: pgmap v693: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.9 MiB/s wr, 494 op/s 2024-04-26T21:52:08.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:08 smithi038 ceph-mon[32724]: osdmap e133: 12 total, 12 up, 12 in 2024-04-26T21:52:09.196 INFO:tasks.workunit.client.0.smithi001.stdout: 10 191388 20.11 MB/sec execute 366 sec latency 1695.545 ms 2024-04-26T21:52:10.199 INFO:tasks.workunit.client.0.smithi001.stdout: 10 191508 20.07 MB/sec execute 367 sec latency 1338.145 ms 2024-04-26T21:52:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:10 smithi186 ceph-mon[32991]: pgmap v695: 129 pgs: 4 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 116 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.7 MiB/s wr, 483 op/s 2024-04-26T21:52:10.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:10 smithi001 ceph-mon[28054]: pgmap v695: 129 pgs: 4 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 116 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.7 MiB/s wr, 483 op/s 2024-04-26T21:52:10.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:10 smithi038 ceph-mon[32724]: pgmap v695: 129 pgs: 4 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 116 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.7 MiB/s wr, 483 op/s 2024-04-26T21:52:11.196 INFO:tasks.workunit.client.0.smithi001.stdout: 10 191587 20.03 MB/sec execute 368 sec latency 1492.666 ms 2024-04-26T21:52:12.196 INFO:tasks.workunit.client.0.smithi001.stdout: 10 191689 19.99 MB/sec execute 369 sec latency 1536.256 ms 2024-04-26T21:52:12.620 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:12 smithi001 ceph-mon[28054]: pgmap v696: 129 pgs: 17 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 75 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 478 op/s 2024-04-26T21:52:12.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:12 smithi186 ceph-mon[32991]: pgmap v696: 129 pgs: 17 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 75 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 478 op/s 2024-04-26T21:52:12.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:12 smithi038 ceph-mon[32724]: pgmap v696: 129 pgs: 17 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 75 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 478 op/s 2024-04-26T21:52:13.196 INFO:tasks.workunit.client.0.smithi001.stdout: 10 191840 19.96 MB/sec execute 370 sec latency 1203.418 ms 2024-04-26T21:52:14.196 INFO:tasks.workunit.client.0.smithi001.stdout: 10 192053 19.93 MB/sec execute 371 sec latency 174.987 ms 2024-04-26T21:52:14.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:14 smithi186 ceph-mon[32991]: pgmap v697: 129 pgs: 20 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.5 MiB/s wr, 459 op/s 2024-04-26T21:52:14.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:14 smithi001 ceph-mon[28054]: pgmap v697: 129 pgs: 20 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.5 MiB/s wr, 459 op/s 2024-04-26T21:52:14.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:14 smithi038 ceph-mon[32724]: pgmap v697: 129 pgs: 20 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 61 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.5 MiB/s wr, 459 op/s 2024-04-26T21:52:15.196 INFO:tasks.workunit.client.0.smithi001.stdout: 10 192329 19.90 MB/sec execute 372 sec latency 139.934 ms 2024-04-26T21:52:16.196 INFO:tasks.workunit.client.0.smithi001.stdout: 10 192711 19.90 MB/sec execute 373 sec latency 103.383 ms 2024-04-26T21:52:16.667 INFO:tasks.workunit.client.0.smithi001.stdout:[178890] write failed on handle 10755 (Resource temporarily unavailable) 2024-04-26T21:52:16.667 INFO:tasks.workunit.client.0.smithi001.stdout:Child failed with status 1 2024-04-26T21:52:16.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:16 smithi186 ceph-mon[32991]: pgmap v698: 129 pgs: 10 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 92 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 353 op/s 2024-04-26T21:52:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T21:52:16.804 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-04-26T21:52:16.804 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-26T21:52:16.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:16 smithi001 ceph-mon[28054]: pgmap v698: 129 pgs: 10 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 92 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 353 op/s 2024-04-26T21:52:16.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:16 smithi038 ceph-mon[32724]: pgmap v698: 129 pgs: 10 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 92 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 353 op/s 2024-04-26T21:52:18.638 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:18 smithi001 ceph-mon[28054]: pgmap v699: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 290 op/s 2024-04-26T21:52:18.644 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi001 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc612ffb9c1d5bc27536bc917a705262a36d4387 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-26T21:52:18.885 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=31de5266af954439839393a95344e27f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_bc612ffb9c1d5bc27536bc917a705262a36d4387/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi001 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc612ffb9c1d5bc27536bc917a705262a36d4387 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-26T21:52:18.888 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-26T21:52:18.900 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:52:18.905 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:18 smithi038 ceph-mon[32724]: pgmap v699: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 290 op/s 2024-04-26T21:52:18.905 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:18 smithi186 ceph-mon[32991]: pgmap v699: 129 pgs: 8 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 102 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 290 op/s 2024-04-26T21:52:18.927 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-26T21:52:18.937 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-26T21:52:18.937 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T21:52:18.937 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:52:18.938 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-26T21:52:19.088 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-26T21:52:19.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-26T21:52:19.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.421+0000 7effe2600640 1 -- 172.21.15.1:0/482854501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effdc102740 msgr2=0x7effdc102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.421+0000 7effe2600640 1 --2- 172.21.15.1:0/482854501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effdc102740 0x7effdc102bc0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7effc8009900 tx=0x7effc802a3f0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.422+0000 7effe2600640 1 -- 172.21.15.1:0/482854501 shutdown_connections 2024-04-26T21:52:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.422+0000 7effe2600640 1 --2- 172.21.15.1:0/482854501 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effdc103100 0x7effdc10f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.422+0000 7effe2600640 1 --2- 172.21.15.1:0/482854501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effdc102740 0x7effdc102bc0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.422+0000 7effe2600640 1 --2- 172.21.15.1:0/482854501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc108750 0x7effdc108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.422+0000 7effe2600640 1 -- 172.21.15.1:0/482854501 >> 172.21.15.1:0/482854501 conn(0x7effdc0fe270 msgr2=0x7effdc100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:19.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.422+0000 7effe2600640 1 -- 172.21.15.1:0/482854501 shutdown_connections 2024-04-26T21:52:19.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.422+0000 7effe2600640 1 -- 172.21.15.1:0/482854501 wait complete. 2024-04-26T21:52:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.423+0000 7effe2600640 1 Processor -- start 2024-04-26T21:52:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.423+0000 7effe2600640 1 -- start start 2024-04-26T21:52:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.423+0000 7effe2600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc102740 0x7effdc19b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.423+0000 7effe2600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effdc103100 0x7effdc19b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:19.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.423+0000 7effe2600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effdc108750 0x7effdc19fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.423+0000 7effe2600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7effdc112a10 con 0x7effdc102740 2024-04-26T21:52:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.423+0000 7effe2600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7effdc112890 con 0x7effdc103100 2024-04-26T21:52:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.423+0000 7effe2600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7effdc112b90 con 0x7effdc108750 2024-04-26T21:52:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effe1200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc102740 0x7effdc19b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effdbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effdc103100 0x7effdc19b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effe1200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc102740 0x7effdc19b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47482/0 (socket says 172.21.15.1:47482) 2024-04-26T21:52:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effe1200640 1 -- 172.21.15.1:0/98401291 learned_addr learned my addr 172.21.15.1:0/98401291 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effe1c00640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effdc108750 0x7effdc19fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:19.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effe1200640 1 -- 172.21.15.1:0/98401291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effdc108750 msgr2=0x7effdc19fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effe1200640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effdc108750 0x7effdc19fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effe1200640 1 -- 172.21.15.1:0/98401291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effdc103100 msgr2=0x7effdc19b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effe1200640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effdc103100 0x7effdc19b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effe1200640 1 -- 172.21.15.1:0/98401291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effdc1a0160 con 0x7effdc102740 2024-04-26T21:52:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.424+0000 7effdbe00640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effdc103100 0x7effdc19b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:52:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.425+0000 7effe1200640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc102740 0x7effdc19b170 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7effd000ea60 tx=0x7effd000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.425+0000 7effd9600640 1 -- 172.21.15.1:0/98401291 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd000cde0 con 0x7effdc102740 2024-04-26T21:52:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.425+0000 7effd9600640 1 -- 172.21.15.1:0/98401291 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effd00035f0 con 0x7effdc102740 2024-04-26T21:52:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.425+0000 7effd9600640 1 -- 172.21.15.1:0/98401291 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0010690 con 0x7effdc102740 2024-04-26T21:52:19.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.425+0000 7effe2600640 1 -- 172.21.15.1:0/98401291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effdc1a0450 con 0x7effdc102740 2024-04-26T21:52:19.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.425+0000 7effe2600640 1 -- 172.21.15.1:0/98401291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effdc1a7ce0 con 0x7effdc102740 2024-04-26T21:52:19.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.426+0000 7effe2600640 1 -- 172.21.15.1:0/98401291 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effdc19c4c0 con 0x7effdc102740 2024-04-26T21:52:19.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.429+0000 7effd9600640 1 -- 172.21.15.1:0/98401291 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7effd00037a0 con 0x7effdc102740 2024-04-26T21:52:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.431+0000 7effd9600640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7effc40a3d80 0x7effc40a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.431+0000 7effd9600640 1 -- 172.21.15.1:0/98401291 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7effd00d0eb0 con 0x7effdc102740 2024-04-26T21:52:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.431+0000 7effdbe00640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7effc40a3d80 0x7effc40a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.432+0000 7effdbe00640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7effc40a3d80 0x7effc40a6240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7effc8009230 tx=0x7effc80058d0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.437+0000 7effd9600640 1 -- 172.21.15.1:0/98401291 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7effd009d030 con 0x7effdc102740 2024-04-26T21:52:19.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.672+0000 7effe2600640 1 -- 172.21.15.1:0/98401291 --> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7effdc10e0e0 con 0x7effc40a3d80 2024-04-26T21:52:19.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.691+0000 7effd9600640 1 -- 172.21.15.1:0/98401291 <== mgr.15291 v2:172.21.15.1:6800/3236768718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+351 (secure 0 0 0) 0x7effdc10e0e0 con 0x7effc40a3d80 2024-04-26T21:52:19.692 INFO:teuthology.orchestra.run.smithi001.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-26T00:00:00", "created": "2024-04-26T21:30:19", "first": "2024-04-26T21:31:00", "last": "2024-04-26T21:52:00", "last_pruned": "2024-04-26T21:52:01", "created_count": 22, "pruned_count": 16, "active": true} 2024-04-26T21:52:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.694+0000 7effe2600640 1 -- 172.21.15.1:0/98401291 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7effc40a3d80 msgr2=0x7effc40a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.694+0000 7effe2600640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7effc40a3d80 0x7effc40a6240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7effc8009230 tx=0x7effc80058d0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.694+0000 7effe2600640 1 -- 172.21.15.1:0/98401291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc102740 msgr2=0x7effdc19b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.694+0000 7effe2600640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc102740 0x7effdc19b170 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7effd000ea60 tx=0x7effd000ef20 comp rx=0 tx=0).stop 2024-04-26T21:52:19.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.694+0000 7effe2600640 1 -- 172.21.15.1:0/98401291 shutdown_connections 2024-04-26T21:52:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.694+0000 7effe2600640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7effc40a3d80 0x7effc40a6240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.694+0000 7effe2600640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effdc108750 0x7effdc19fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.694+0000 7effe2600640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7effdc103100 0x7effdc19b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.694+0000 7effe2600640 1 --2- 172.21.15.1:0/98401291 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7effdc102740 0x7effdc19b170 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.694+0000 7effe2600640 1 -- 172.21.15.1:0/98401291 >> 172.21.15.1:0/98401291 conn(0x7effdc0fe270 msgr2=0x7effdc10d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.695+0000 7effe2600640 1 -- 172.21.15.1:0/98401291 shutdown_connections 2024-04-26T21:52:19.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.695+0000 7effe2600640 1 -- 172.21.15.1:0/98401291 wait complete. 2024-04-26T21:52:19.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-26T21:52:19.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.925+0000 7f3a36800640 1 -- 172.21.15.1:0/3547053831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a30108a00 msgr2=0x7f3a30108de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.925+0000 7f3a36800640 1 --2- 172.21.15.1:0/3547053831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a30108a00 0x7f3a30108de0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3a20009930 tx=0x7f3a2002a420 comp rx=0 tx=0).stop 2024-04-26T21:52:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.925+0000 7f3a36800640 1 -- 172.21.15.1:0/3547053831 shutdown_connections 2024-04-26T21:52:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.925+0000 7f3a36800640 1 --2- 172.21.15.1:0/3547053831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a3006b010 0x7f3a3006b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.925+0000 7f3a36800640 1 --2- 172.21.15.1:0/3547053831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a3006b9d0 0x7f3a3010f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.925+0000 7f3a36800640 1 --2- 172.21.15.1:0/3547053831 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a30108a00 0x7f3a30108de0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.925+0000 7f3a36800640 1 -- 172.21.15.1:0/3547053831 >> 172.21.15.1:0/3547053831 conn(0x7f3a30078930 msgr2=0x7f3a30078d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:19.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.926+0000 7f3a36800640 1 -- 172.21.15.1:0/3547053831 shutdown_connections 2024-04-26T21:52:19.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.926+0000 7f3a36800640 1 -- 172.21.15.1:0/3547053831 wait complete. 2024-04-26T21:52:19.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.927+0000 7f3a36800640 1 Processor -- start 2024-04-26T21:52:19.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.927+0000 7f3a36800640 1 -- start start 2024-04-26T21:52:19.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a36800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a3006b010 0x7f3a301992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:19.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a36800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a3006b9d0 0x7f3a30199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:19.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a36800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a3019db80 0x7f3a3019e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:19.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a36800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3a30112bd0 con 0x7f3a3006b010 2024-04-26T21:52:19.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a36800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3a30112a50 con 0x7f3a3006b9d0 2024-04-26T21:52:19.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a36800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3a30112d50 con 0x7f3a3019db80 2024-04-26T21:52:19.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a2f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a3006b010 0x7f3a301992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:19.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a2f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a3006b010 0x7f3a301992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47494/0 (socket says 172.21.15.1:47494) 2024-04-26T21:52:19.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a2f400640 1 -- 172.21.15.1:0/1703943771 learned_addr learned my addr 172.21.15.1:0/1703943771 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:19.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a2fe00640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a3019db80 0x7f3a3019e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:19.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a2ea00640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a3006b9d0 0x7f3a30199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:19.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a2f400640 1 -- 172.21.15.1:0/1703943771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a3019db80 msgr2=0x7f3a3019e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:19.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a2f400640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a3019db80 0x7f3a3019e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.928+0000 7f3a2f400640 1 -- 172.21.15.1:0/1703943771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a3006b9d0 msgr2=0x7f3a30199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:19.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.929+0000 7f3a2f400640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a3006b9d0 0x7f3a30199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:19.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.929+0000 7f3a2f400640 1 -- 172.21.15.1:0/1703943771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3019e670 con 0x7f3a3006b010 2024-04-26T21:52:19.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.929+0000 7f3a2fe00640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a3019db80 0x7f3a3019e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:52:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.929+0000 7f3a2f400640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a3006b010 0x7f3a301992c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3a20005940 tx=0x7f3a200042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.929+0000 7f3a27e00640 1 -- 172.21.15.1:0/1703943771 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2003f030 con 0x7f3a3006b010 2024-04-26T21:52:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.929+0000 7f3a36800640 1 -- 172.21.15.1:0/1703943771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3019e900 con 0x7f3a3006b010 2024-04-26T21:52:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.929+0000 7f3a27e00640 1 -- 172.21.15.1:0/1703943771 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a20003cb0 con 0x7f3a3006b010 2024-04-26T21:52:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.930+0000 7f3a27e00640 1 -- 172.21.15.1:0/1703943771 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20049060 con 0x7f3a3006b010 2024-04-26T21:52:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.930+0000 7f3a36800640 1 -- 172.21.15.1:0/1703943771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a301a60c0 con 0x7f3a3006b010 2024-04-26T21:52:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.931+0000 7f3a36800640 1 -- 172.21.15.1:0/1703943771 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a30050f10 con 0x7f3a3006b010 2024-04-26T21:52:19.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.933+0000 7f3a27e00640 1 -- 172.21.15.1:0/1703943771 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f3a20042030 con 0x7f3a3006b010 2024-04-26T21:52:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.934+0000 7f3a27e00640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3a140a3d10 0x7f3a140a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.934+0000 7f3a27e00640 1 -- 172.21.15.1:0/1703943771 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7f3a200f1340 con 0x7f3a3006b010 2024-04-26T21:52:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.935+0000 7f3a2ea00640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3a140a3d10 0x7f3a140a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.938+0000 7f3a2ea00640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3a140a3d10 0x7f3a140a61d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3a301013c0 tx=0x7f3a1c00b390 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:19.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:19.951+0000 7f3a27e00640 1 -- 172.21.15.1:0/1703943771 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3a200bce70 con 0x7f3a3006b010 2024-04-26T21:52:20.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.302+0000 7f3a36800640 1 -- 172.21.15.1:0/1703943771 --> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f3a30062d70 con 0x7f3a140a3d10 2024-04-26T21:52:20.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.315+0000 7f3a27e00640 1 -- 172.21.15.1:0/1703943771 <== mgr.15291 v2:172.21.15.1:6800/3236768718 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f3a30062d70 con 0x7f3a140a3d10 2024-04-26T21:52:20.315 INFO:teuthology.orchestra.run.smithi001.stdout:/ 1m 6m3h 2024-04-26T21:52:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.317+0000 7f3a36800640 1 -- 172.21.15.1:0/1703943771 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3a140a3d10 msgr2=0x7f3a140a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.317+0000 7f3a36800640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3a140a3d10 0x7f3a140a61d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3a301013c0 tx=0x7f3a1c00b390 comp rx=0 tx=0).stop 2024-04-26T21:52:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.317+0000 7f3a36800640 1 -- 172.21.15.1:0/1703943771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a3006b010 msgr2=0x7f3a301992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.317+0000 7f3a36800640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a3006b010 0x7f3a301992c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3a20005940 tx=0x7f3a200042f0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.317+0000 7f3a36800640 1 -- 172.21.15.1:0/1703943771 shutdown_connections 2024-04-26T21:52:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.317+0000 7f3a36800640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3a140a3d10 0x7f3a140a61d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.317+0000 7f3a36800640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a3019db80 0x7f3a3019e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.318+0000 7f3a36800640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3a3006b9d0 0x7f3a30199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.318+0000 7f3a36800640 1 --2- 172.21.15.1:0/1703943771 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a3006b010 0x7f3a301992c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.318+0000 7f3a36800640 1 -- 172.21.15.1:0/1703943771 >> 172.21.15.1:0/1703943771 conn(0x7f3a30078930 msgr2=0x7f3a300ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.318+0000 7f3a36800640 1 -- 172.21.15.1:0/1703943771 shutdown_connections 2024-04-26T21:52:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.318+0000 7f3a36800640 1 -- 172.21.15.1:0/1703943771 wait complete. 2024-04-26T21:52:20.336 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-26T21:52:20.576 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:52:20.662 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:20 smithi001 ceph-mon[28054]: pgmap v700: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 272 op/s 2024-04-26T21:52:20.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:20 smithi186 ceph-mon[32991]: pgmap v700: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 272 op/s 2024-04-26T21:52:20.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.815+0000 7fd48c600640 1 -- 172.21.15.1:0/322632170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd484109f00 msgr2=0x7fd48410a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:20.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.815+0000 7fd48c600640 1 --2- 172.21.15.1:0/322632170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd484109f00 0x7fd48410a360 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd470009ca0 tx=0x7fd47002a8d0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.815+0000 7fd48c600640 1 -- 172.21.15.1:0/322632170 shutdown_connections 2024-04-26T21:52:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.815+0000 7fd48c600640 1 --2- 172.21.15.1:0/322632170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd484109f00 0x7fd48410a360 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.815+0000 7fd48c600640 1 --2- 172.21.15.1:0/322632170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd48411c120 0x7fd48411e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.815+0000 7fd48c600640 1 --2- 172.21.15.1:0/322632170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd48410d070 0x7fd484109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.815+0000 7fd48c600640 1 -- 172.21.15.1:0/322632170 >> 172.21.15.1:0/322632170 conn(0x7fd48406de90 msgr2=0x7fd48406e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:20.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.816+0000 7fd48c600640 1 -- 172.21.15.1:0/322632170 shutdown_connections 2024-04-26T21:52:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.816+0000 7fd48c600640 1 -- 172.21.15.1:0/322632170 wait complete. 2024-04-26T21:52:20.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.818+0000 7fd48c600640 1 Processor -- start 2024-04-26T21:52:20.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.818+0000 7fd48c600640 1 -- start start 2024-04-26T21:52:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.821+0000 7fd48c600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd484109f00 0x7fd4841a8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.821+0000 7fd48c600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd48410d070 0x7fd4841a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:20.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.821+0000 7fd48c600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd48411c120 0x7fd4841ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.821+0000 7fd48c600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd484121b20 con 0x7fd48410d070 2024-04-26T21:52:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.821+0000 7fd48a600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd48411c120 0x7fd4841ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.822+0000 7fd48a600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd48411c120 0x7fd4841ac940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47530/0 (socket says 172.21.15.1:47530) 2024-04-26T21:52:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.822+0000 7fd48c600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd4841219a0 con 0x7fd48411c120 2024-04-26T21:52:20.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.822+0000 7fd48c600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd484121ca0 con 0x7fd484109f00 2024-04-26T21:52:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.822+0000 7fd48a600640 1 -- 172.21.15.1:0/493278726 learned_addr learned my addr 172.21.15.1:0/493278726 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:20.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.822+0000 7fd489200640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd48410d070 0x7fd4841a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:20.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:20 smithi038 ceph-mon[32724]: pgmap v700: 129 pgs: 3 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 119 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 272 op/s 2024-04-26T21:52:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.823+0000 7fd489c00640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd484109f00 0x7fd4841a8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.824+0000 7fd489200640 1 -- 172.21.15.1:0/493278726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd484109f00 msgr2=0x7fd4841a8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.824+0000 7fd489200640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd484109f00 0x7fd4841a8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.824+0000 7fd489200640 1 -- 172.21.15.1:0/493278726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd48411c120 msgr2=0x7fd4841ac940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.824+0000 7fd489200640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd48411c120 0x7fd4841ac940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.824+0000 7fd489200640 1 -- 172.21.15.1:0/493278726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4841ad070 con 0x7fd48410d070 2024-04-26T21:52:20.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.828+0000 7fd489200640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd48410d070 0x7fd4841a85c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd47002adf0 tx=0x7fd470003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.829+0000 7fd482000640 1 -- 172.21.15.1:0/493278726 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd470004040 con 0x7fd48410d070 2024-04-26T21:52:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.829+0000 7fd482000640 1 -- 172.21.15.1:0/493278726 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd470004690 con 0x7fd48410d070 2024-04-26T21:52:20.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.829+0000 7fd48c600640 1 -- 172.21.15.1:0/493278726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4841ad300 con 0x7fd48410d070 2024-04-26T21:52:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.829+0000 7fd482000640 1 -- 172.21.15.1:0/493278726 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd470030e80 con 0x7fd48410d070 2024-04-26T21:52:20.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.829+0000 7fd48c600640 1 -- 172.21.15.1:0/493278726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4841b4bf0 con 0x7fd48410d070 2024-04-26T21:52:20.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.830+0000 7fd48c600640 1 -- 172.21.15.1:0/493278726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4840697e0 con 0x7fd48410d070 2024-04-26T21:52:20.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.833+0000 7fd482000640 1 -- 172.21.15.1:0/493278726 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd4700305d0 con 0x7fd48410d070 2024-04-26T21:52:20.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.836+0000 7fd482000640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4680a3e70 0x7fd4680a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:20.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.836+0000 7fd482000640 1 -- 172.21.15.1:0/493278726 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7fd4700f1ff0 con 0x7fd48410d070 2024-04-26T21:52:20.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.837+0000 7fd489c00640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4680a3e70 0x7fd4680a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:20.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.839+0000 7fd489c00640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4680a3e70 0x7fd4680a6330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009990 tx=0x7fd47c009110 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:20.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:20.842+0000 7fd482000640 1 -- 172.21.15.1:0/493278726 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd4700be2e0 con 0x7fd48410d070 2024-04-26T21:52:21.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.059+0000 7fd48c600640 1 -- 172.21.15.1:0/493278726 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4841b4fb0 con 0x7fd48410d070 2024-04-26T21:52:21.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.067+0000 7fd482000640 1 -- 172.21.15.1:0/493278726 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v133) ==== 74+0+25524 (secure 0 0 0) 0x7fd4700bdbe0 con 0x7fd48410d070 2024-04-26T21:52:21.069 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:52:21.070 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":133,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:52:07.318369+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":133,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":133,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7},{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2275413710":"2024-04-27T21:30:18.553822+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2563183900":"2024-04-27T21:30:18.553822+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/4024107554":"2024-04-27T21:30:18.553822+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/2080672016":"2024-04-27T21:30:18.553822+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/2786715792":"2024-04-27T21:30:18.553822+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/2786715792":"2024-04-27T21:30:18.553822+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/590615206":"2024-04-27T21:30:18.553822+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]},{"pool":4,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]},{"pool":4,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:52:21.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.073+0000 7fd48c600640 1 -- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4680a3e70 msgr2=0x7fd4680a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.073+0000 7fd48c600640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4680a3e70 0x7fd4680a6330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009990 tx=0x7fd47c009110 comp rx=0 tx=0).stop 2024-04-26T21:52:21.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.073+0000 7fd48c600640 1 -- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd48410d070 msgr2=0x7fd4841a85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.073+0000 7fd48c600640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd48410d070 0x7fd4841a85c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd47002adf0 tx=0x7fd470003b80 comp rx=0 tx=0).stop 2024-04-26T21:52:21.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.074+0000 7fd48c600640 1 -- 172.21.15.1:0/493278726 shutdown_connections 2024-04-26T21:52:21.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.074+0000 7fd48c600640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd4680a3e70 0x7fd4680a6330 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.074+0000 7fd48c600640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd48411c120 0x7fd4841ac940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.074+0000 7fd48c600640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd48410d070 0x7fd4841a85c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.074+0000 7fd48c600640 1 --2- 172.21.15.1:0/493278726 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd484109f00 0x7fd4841a8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.074+0000 7fd48c600640 1 -- 172.21.15.1:0/493278726 >> 172.21.15.1:0/493278726 conn(0x7fd48406de90 msgr2=0x7fd48410aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:21.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.075+0000 7fd48c600640 1 -- 172.21.15.1:0/493278726 shutdown_connections 2024-04-26T21:52:21.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.075+0000 7fd48c600640 1 -- 172.21.15.1:0/493278726 wait complete. 2024-04-26T21:52:21.103 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:27:00.381106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:27:01.743898+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 39, 'snap_epoch': 133, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T21:27:05.341216+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 39, 'snap_epoch': 133, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T21:52:21.104 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:52:21.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.285+0000 7f7727e00640 1 -- 172.21.15.1:0/2169475597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7728102730 msgr2=0x7f7728102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.285+0000 7f7727e00640 1 --2- 172.21.15.1:0/2169475597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7728102730 0x7f7728102bb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7710009900 tx=0x7f771002a3f0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.285+0000 7f7727e00640 1 -- 172.21.15.1:0/2169475597 shutdown_connections 2024-04-26T21:52:21.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.285+0000 7f7727e00640 1 --2- 172.21.15.1:0/2169475597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77281030f0 0x7f772810f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.285+0000 7f7727e00640 1 --2- 172.21.15.1:0/2169475597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7728102730 0x7f7728102bb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.285+0000 7f7727e00640 1 --2- 172.21.15.1:0/2169475597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7728108740 0x7f7728108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.285+0000 7f7727e00640 1 -- 172.21.15.1:0/2169475597 >> 172.21.15.1:0/2169475597 conn(0x7f77280fe280 msgr2=0x7f77281006a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:21.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.285+0000 7f7727e00640 1 -- 172.21.15.1:0/2169475597 shutdown_connections 2024-04-26T21:52:21.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.285+0000 7f7727e00640 1 -- 172.21.15.1:0/2169475597 wait complete. 2024-04-26T21:52:21.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.286+0000 7f7727e00640 1 Processor -- start 2024-04-26T21:52:21.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7727e00640 1 -- start start 2024-04-26T21:52:21.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7727e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7728102730 0x7f772819b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:21.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7727e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77281030f0 0x7f772819b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:21.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7727e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7728108740 0x7f772819fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:21.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7727e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7728112a00 con 0x7f7728108740 2024-04-26T21:52:21.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7727e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7728112880 con 0x7f77281030f0 2024-04-26T21:52:21.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7727e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7728112b80 con 0x7f7728102730 2024-04-26T21:52:21.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7726a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7728102730 0x7f772819b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:21.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7727400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7728108740 0x7f772819fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7726a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7728102730 0x7f772819b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:56062/0 (socket says 172.21.15.1:56062) 2024-04-26T21:52:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7726a00640 1 -- 172.21.15.1:0/2429592627 learned_addr learned my addr 172.21.15.1:0/2429592627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.287+0000 7f7726000640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77281030f0 0x7f772819b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.288+0000 7f7726a00640 1 -- 172.21.15.1:0/2429592627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77281030f0 msgr2=0x7f772819b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.288+0000 7f7726a00640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77281030f0 0x7f772819b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.288+0000 7f7726a00640 1 -- 172.21.15.1:0/2429592627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7728108740 msgr2=0x7f772819fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.288+0000 7f7726a00640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7728108740 0x7f772819fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.288+0000 7f7726a00640 1 -- 172.21.15.1:0/2429592627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77281a0170 con 0x7f7728102730 2024-04-26T21:52:21.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.288+0000 7f7726a00640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7728102730 0x7f772819b180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f771800ea60 tx=0x7f771800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:21.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.288+0000 7f771f400640 1 -- 172.21.15.1:0/2429592627 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771800cde0 con 0x7f7728102730 2024-04-26T21:52:21.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.288+0000 7f7727e00640 1 -- 172.21.15.1:0/2429592627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77281a0460 con 0x7f7728102730 2024-04-26T21:52:21.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.288+0000 7f771f400640 1 -- 172.21.15.1:0/2429592627 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77180035f0 con 0x7f7728102730 2024-04-26T21:52:21.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.288+0000 7f771f400640 1 -- 172.21.15.1:0/2429592627 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7718010690 con 0x7f7728102730 2024-04-26T21:52:21.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.289+0000 7f7727e00640 1 -- 172.21.15.1:0/2429592627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77281a7cf0 con 0x7f7728102730 2024-04-26T21:52:21.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.289+0000 7f7727e00640 1 -- 172.21.15.1:0/2429592627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77280506c0 con 0x7f7728102730 2024-04-26T21:52:21.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.292+0000 7f771f400640 1 -- 172.21.15.1:0/2429592627 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f77180037a0 con 0x7f7728102730 2024-04-26T21:52:21.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.293+0000 7f771f400640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f770c0a3da0 0x7f770c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:21.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.293+0000 7f771f400640 1 -- 172.21.15.1:0/2429592627 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7f77180d0ed0 con 0x7f7728102730 2024-04-26T21:52:21.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.293+0000 7f7726000640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f770c0a3da0 0x7f770c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:21.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.294+0000 7f7726000640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f770c0a3da0 0x7f770c0a6260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7710009230 tx=0x7f77100058d0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:21.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.295+0000 7f771f400640 1 -- 172.21.15.1:0/2429592627 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f771809d050 con 0x7f7728102730 2024-04-26T21:52:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.516+0000 7f7727e00640 1 -- 172.21.15.1:0/2429592627 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f77281a8030 con 0x7f7728102730 2024-04-26T21:52:21.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.517+0000 7f771f400640 1 -- 172.21.15.1:0/2429592627 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v133) ==== 93+0+10 (secure 0 0 0) 0x7f771809c950 con 0x7f7728102730 2024-04-26T21:52:21.518 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:52:21.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.519+0000 7f7727e00640 1 -- 172.21.15.1:0/2429592627 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f770c0a3da0 msgr2=0x7f770c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.519+0000 7f7727e00640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f770c0a3da0 0x7f770c0a6260 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7710009230 tx=0x7f77100058d0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.519+0000 7f7727e00640 1 -- 172.21.15.1:0/2429592627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7728102730 msgr2=0x7f772819b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.519+0000 7f7727e00640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7728102730 0x7f772819b180 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f771800ea60 tx=0x7f771800ef20 comp rx=0 tx=0).stop 2024-04-26T21:52:21.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.520+0000 7f7727e00640 1 -- 172.21.15.1:0/2429592627 shutdown_connections 2024-04-26T21:52:21.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.520+0000 7f7727e00640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f770c0a3da0 0x7f770c0a6260 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.520+0000 7f7727e00640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7728108740 0x7f772819fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.520+0000 7f7727e00640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f77281030f0 0x7f772819b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.520+0000 7f7727e00640 1 --2- 172.21.15.1:0/2429592627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7728102730 0x7f772819b180 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.520+0000 7f7727e00640 1 -- 172.21.15.1:0/2429592627 >> 172.21.15.1:0/2429592627 conn(0x7f77280fe280 msgr2=0x7f772810df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:21.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.520+0000 7f7727e00640 1 -- 172.21.15.1:0/2429592627 shutdown_connections 2024-04-26T21:52:21.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.520+0000 7f7727e00640 1 -- 172.21.15.1:0/2429592627 wait complete. 2024-04-26T21:52:21.541 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:52:21.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:21 smithi186 ceph-mon[32991]: from='client.16776 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:52:21.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:21 smithi186 ceph-mon[32991]: from='client.16782 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:52:21.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:21 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/493278726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:52:21.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.749+0000 7fcbf5600640 1 -- 172.21.15.1:0/1887765019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbf01052d0 msgr2=0x7fcbf01117e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.749+0000 7fcbf5600640 1 --2- 172.21.15.1:0/1887765019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbf01052d0 0x7fcbf01117e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009cd0 tx=0x7fcbe802a8f0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.749+0000 7fcbf5600640 1 -- 172.21.15.1:0/1887765019 shutdown_connections 2024-04-26T21:52:21.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.749+0000 7fcbf5600640 1 --2- 172.21.15.1:0/1887765019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf0104910 0x7fcbf0104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.750+0000 7fcbf5600640 1 --2- 172.21.15.1:0/1887765019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbf01052d0 0x7fcbf01117e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.750+0000 7fcbf5600640 1 --2- 172.21.15.1:0/1887765019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf010a920 0x7fcbf010ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.750+0000 7fcbf5600640 1 -- 172.21.15.1:0/1887765019 >> 172.21.15.1:0/1887765019 conn(0x7fcbf0100400 msgr2=0x7fcbf0102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:21.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.750+0000 7fcbf5600640 1 -- 172.21.15.1:0/1887765019 shutdown_connections 2024-04-26T21:52:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.750+0000 7fcbf5600640 1 -- 172.21.15.1:0/1887765019 wait complete. 2024-04-26T21:52:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.751+0000 7fcbf5600640 1 Processor -- start 2024-04-26T21:52:21.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.751+0000 7fcbf5600640 1 -- start start 2024-04-26T21:52:21.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbf5600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf0104910 0x7fcbf019b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbf5600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf01052d0 0x7fcbf019b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbf5600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbf010a920 0x7fcbf019fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbf5600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf0114be0 con 0x7fcbf010a920 2024-04-26T21:52:21.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbf5600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf0114a60 con 0x7fcbf0104910 2024-04-26T21:52:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbf5600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf0114d60 con 0x7fcbf01052d0 2024-04-26T21:52:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbf4c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbf010a920 0x7fcbf019fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbefe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf0104910 0x7fcbf019b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:21.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbefe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf0104910 0x7fcbf019b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47578/0 (socket says 172.21.15.1:47578) 2024-04-26T21:52:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbefe00640 1 -- 172.21.15.1:0/3981337618 learned_addr learned my addr 172.21.15.1:0/3981337618 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbef400640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf01052d0 0x7fcbf019b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:21.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbefe00640 1 -- 172.21.15.1:0/3981337618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf01052d0 msgr2=0x7fcbf019b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbefe00640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf01052d0 0x7fcbf019b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbefe00640 1 -- 172.21.15.1:0/3981337618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbf010a920 msgr2=0x7fcbf019fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbefe00640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbf010a920 0x7fcbf019fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.752+0000 7fcbefe00640 1 -- 172.21.15.1:0/3981337618 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf01a0170 con 0x7fcbf0104910 2024-04-26T21:52:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.753+0000 7fcbefe00640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf0104910 0x7fcbf019b180 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000e9f0 tx=0x7fcbe000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.753+0000 7fcbecc00640 1 -- 172.21.15.1:0/3981337618 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe000cd80 con 0x7fcbf0104910 2024-04-26T21:52:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.753+0000 7fcbecc00640 1 -- 172.21.15.1:0/3981337618 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbe00108e0 con 0x7fcbf0104910 2024-04-26T21:52:21.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.753+0000 7fcbecc00640 1 -- 172.21.15.1:0/3981337618 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0018980 con 0x7fcbf0104910 2024-04-26T21:52:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.753+0000 7fcbf5600640 1 -- 172.21.15.1:0/3981337618 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf01a0460 con 0x7fcbf0104910 2024-04-26T21:52:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.754+0000 7fcbf5600640 1 -- 172.21.15.1:0/3981337618 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf01a7cf0 con 0x7fcbf0104910 2024-04-26T21:52:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.755+0000 7fcbf5600640 1 -- 172.21.15.1:0/3981337618 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbf00506c0 con 0x7fcbf0104910 2024-04-26T21:52:21.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.757+0000 7fcbecc00640 1 -- 172.21.15.1:0/3981337618 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fcbe0010430 con 0x7fcbf0104910 2024-04-26T21:52:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.758+0000 7fcbecc00640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcbd40a3da0 0x7fcbd40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:21.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.759+0000 7fcbecc00640 1 -- 172.21.15.1:0/3981337618 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7fcbe00d1250 con 0x7fcbf0104910 2024-04-26T21:52:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.759+0000 7fcbef400640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcbd40a3da0 0x7fcbd40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.760+0000 7fcbef400640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcbd40a3da0 0x7fcbd40a6260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0102b00 tx=0x7fcbdc005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:21.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.761+0000 7fcbecc00640 1 -- 172.21.15.1:0/3981337618 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcbe009d3d0 con 0x7fcbf0104910 2024-04-26T21:52:21.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:21 smithi001 ceph-mon[28054]: from='client.16776 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:52:21.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:21 smithi001 ceph-mon[28054]: from='client.16782 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:52:21.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:21 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/493278726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:52:21.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:21 smithi038 ceph-mon[32724]: from='client.16776 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:52:21.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:21 smithi038 ceph-mon[32724]: from='client.16782 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T21:52:21.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:21 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/493278726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:52:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.967+0000 7fcbf5600640 1 -- 172.21.15.1:0/3981337618 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcbf002dad0 con 0x7fcbf0104910 2024-04-26T21:52:21.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.968+0000 7fcbecc00640 1 -- 172.21.15.1:0/3981337618 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v133) ==== 104+0+11 (secure 0 0 0) 0x7fcbe009ccd0 con 0x7fcbf0104910 2024-04-26T21:52:21.968 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:52:21.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.970+0000 7fcbf5600640 1 -- 172.21.15.1:0/3981337618 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcbd40a3da0 msgr2=0x7fcbd40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.970+0000 7fcbf5600640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcbd40a3da0 0x7fcbd40a6260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0102b00 tx=0x7fcbdc005f90 comp rx=0 tx=0).stop 2024-04-26T21:52:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.970+0000 7fcbf5600640 1 -- 172.21.15.1:0/3981337618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf0104910 msgr2=0x7fcbf019b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.971+0000 7fcbf5600640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf0104910 0x7fcbf019b180 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000e9f0 tx=0x7fcbe000eeb0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.971+0000 7fcbf5600640 1 -- 172.21.15.1:0/3981337618 shutdown_connections 2024-04-26T21:52:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.971+0000 7fcbf5600640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcbd40a3da0 0x7fcbd40a6260 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.971+0000 7fcbf5600640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf01052d0 0x7fcbf019b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.971+0000 7fcbf5600640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbf010a920 0x7fcbf019fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.971+0000 7fcbf5600640 1 --2- 172.21.15.1:0/3981337618 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcbf0104910 0x7fcbf019b180 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:21.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.971+0000 7fcbf5600640 1 -- 172.21.15.1:0/3981337618 >> 172.21.15.1:0/3981337618 conn(0x7fcbf0100400 msgr2=0x7fcbf0100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:21.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.971+0000 7fcbf5600640 1 -- 172.21.15.1:0/3981337618 shutdown_connections 2024-04-26T21:52:21.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:21.971+0000 7fcbf5600640 1 -- 172.21.15.1:0/3981337618 wait complete. 2024-04-26T21:52:21.990 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:52:22.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.215+0000 7f625f200640 1 -- 172.21.15.1:0/1204651542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6260104920 msgr2=0x7f6260104da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:22.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.215+0000 7f625f200640 1 --2- 172.21.15.1:0/1204651542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6260104920 0x7f6260104da0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f624c009930 tx=0x7f624c02a420 comp rx=0 tx=0).stop 2024-04-26T21:52:22.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.215+0000 7f625f200640 1 -- 172.21.15.1:0/1204651542 shutdown_connections 2024-04-26T21:52:22.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.215+0000 7f625f200640 1 --2- 172.21.15.1:0/1204651542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62601052e0 0x7f62601117f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.215+0000 7f625f200640 1 --2- 172.21.15.1:0/1204651542 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6260104920 0x7f6260104da0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.215+0000 7f625f200640 1 --2- 172.21.15.1:0/1204651542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626010a930 0x7f626010ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.215+0000 7f625f200640 1 -- 172.21.15.1:0/1204651542 >> 172.21.15.1:0/1204651542 conn(0x7f62601003d0 msgr2=0x7f62601027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.216+0000 7f625f200640 1 -- 172.21.15.1:0/1204651542 shutdown_connections 2024-04-26T21:52:22.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.216+0000 7f625f200640 1 -- 172.21.15.1:0/1204651542 wait complete. 2024-04-26T21:52:22.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625f200640 1 Processor -- start 2024-04-26T21:52:22.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625f200640 1 -- start start 2024-04-26T21:52:22.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625f200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6260104920 0x7f626019b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:22.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625f200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62601052e0 0x7f626019b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:22.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625f200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626010a930 0x7f626019fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:22.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625f200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6260114c40 con 0x7f6260104920 2024-04-26T21:52:22.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625f200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f6260114ac0 con 0x7f62601052e0 2024-04-26T21:52:22.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625f200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6260114dc0 con 0x7f626010a930 2024-04-26T21:52:22.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6260104920 0x7f626019b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:22.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625de00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6260104920 0x7f626019b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47564/0 (socket says 172.21.15.1:47564) 2024-04-26T21:52:22.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625de00640 1 -- 172.21.15.1:0/2034051619 learned_addr learned my addr 172.21.15.1:0/2034051619 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:22.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.217+0000 7f625d400640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62601052e0 0x7f626019b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:22.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f625e800640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626010a930 0x7f626019fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:22.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f625de00640 1 -- 172.21.15.1:0/2034051619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626010a930 msgr2=0x7f626019fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:22.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f625de00640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626010a930 0x7f626019fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f625de00640 1 -- 172.21.15.1:0/2034051619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62601052e0 msgr2=0x7f626019b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:22.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f625de00640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62601052e0 0x7f626019b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f625de00640 1 -- 172.21.15.1:0/2034051619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62601a0130 con 0x7f6260104920 2024-04-26T21:52:22.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f625d400640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62601052e0 0x7f626019b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:52:22.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f625de00640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6260104920 0x7f626019b140 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f625000e9f0 tx=0x7f625000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:22.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f6256a00640 1 -- 172.21.15.1:0/2034051619 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625000ccc0 con 0x7f6260104920 2024-04-26T21:52:22.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f625f200640 1 -- 172.21.15.1:0/2034051619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62601a0420 con 0x7f6260104920 2024-04-26T21:52:22.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.218+0000 7f6256a00640 1 -- 172.21.15.1:0/2034051619 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62500108e0 con 0x7f6260104920 2024-04-26T21:52:22.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.219+0000 7f6256a00640 1 -- 172.21.15.1:0/2034051619 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6250018880 con 0x7f6260104920 2024-04-26T21:52:22.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.219+0000 7f625f200640 1 -- 172.21.15.1:0/2034051619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62601a7cb0 con 0x7f6260104920 2024-04-26T21:52:22.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.220+0000 7f625f200640 1 -- 172.21.15.1:0/2034051619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6260106060 con 0x7f6260104920 2024-04-26T21:52:22.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.222+0000 7f6256a00640 1 -- 172.21.15.1:0/2034051619 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f6250010430 con 0x7f6260104920 2024-04-26T21:52:22.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.223+0000 7f6256a00640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f62440a3d10 0x7f62440a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:22.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.223+0000 7f6256a00640 1 -- 172.21.15.1:0/2034051619 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7f62500d0ff0 con 0x7f6260104920 2024-04-26T21:52:22.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.223+0000 7f625d400640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f62440a3d10 0x7f62440a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:22.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.224+0000 7f625d400640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f62440a3d10 0x7f62440a61d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f624c031440 tx=0x7f624c005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:22.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.228+0000 7f6256a00640 1 -- 172.21.15.1:0/2034051619 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f625009d1b0 con 0x7f6260104920 2024-04-26T21:52:22.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.449+0000 7f625f200640 1 -- 172.21.15.1:0/2034051619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f626002dad0 con 0x7f6260104920 2024-04-26T21:52:22.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.450+0000 7f6256a00640 1 -- 172.21.15.1:0/2034051619 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v133) ==== 100+0+11 (secure 0 0 0) 0x7f625009cab0 con 0x7f6260104920 2024-04-26T21:52:22.450 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T21:52:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.452+0000 7f625f200640 1 -- 172.21.15.1:0/2034051619 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f62440a3d10 msgr2=0x7f62440a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.452+0000 7f625f200640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f62440a3d10 0x7f62440a61d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f624c031440 tx=0x7f624c005a20 comp rx=0 tx=0).stop 2024-04-26T21:52:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.452+0000 7f625f200640 1 -- 172.21.15.1:0/2034051619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6260104920 msgr2=0x7f626019b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.452+0000 7f625f200640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6260104920 0x7f626019b140 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f625000e9f0 tx=0x7f625000edf0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.453+0000 7f625f200640 1 -- 172.21.15.1:0/2034051619 shutdown_connections 2024-04-26T21:52:22.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.453+0000 7f625f200640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f62440a3d10 0x7f62440a61d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.453+0000 7f625f200640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f626010a930 0x7f626019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.453+0000 7f625f200640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62601052e0 0x7f626019b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.453+0000 7f625f200640 1 --2- 172.21.15.1:0/2034051619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6260104920 0x7f626019b140 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.453+0000 7f625f200640 1 -- 172.21.15.1:0/2034051619 >> 172.21.15.1:0/2034051619 conn(0x7f62601003d0 msgr2=0x7f6260100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:22.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.453+0000 7f625f200640 1 -- 172.21.15.1:0/2034051619 shutdown_connections 2024-04-26T21:52:22.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.453+0000 7f625f200640 1 -- 172.21.15.1:0/2034051619 wait complete. 2024-04-26T21:52:22.475 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T21:52:22.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.682+0000 7fd9faa00640 1 -- 172.21.15.1:0/3161095259 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f4104470 msgr2=0x7fd9f410c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:22.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.682+0000 7fd9faa00640 1 --2- 172.21.15.1:0/3161095259 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f4104470 0x7fd9f410c950 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009900 tx=0x7fd9e002a480 comp rx=0 tx=0).stop 2024-04-26T21:52:22.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.682+0000 7fd9faa00640 1 -- 172.21.15.1:0/3161095259 shutdown_connections 2024-04-26T21:52:22.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.682+0000 7fd9faa00640 1 --2- 172.21.15.1:0/3161095259 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f4104470 0x7fd9f410c950 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.682+0000 7fd9faa00640 1 --2- 172.21.15.1:0/3161095259 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f410cf20 0x7fd9f410f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.682+0000 7fd9faa00640 1 --2- 172.21.15.1:0/3161095259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9f4103b50 0x7fd9f4103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.682+0000 7fd9faa00640 1 -- 172.21.15.1:0/3161095259 >> 172.21.15.1:0/3161095259 conn(0x7fd9f4078a10 msgr2=0x7fd9f4078e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.682+0000 7fd9faa00640 1 -- 172.21.15.1:0/3161095259 shutdown_connections 2024-04-26T21:52:22.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.683+0000 7fd9faa00640 1 -- 172.21.15.1:0/3161095259 wait complete. 2024-04-26T21:52:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.683+0000 7fd9faa00640 1 Processor -- start 2024-04-26T21:52:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9faa00640 1 -- start start 2024-04-26T21:52:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9faa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f4103b50 0x7fd9f41a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9faa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4104470 0x7fd9f41a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:22.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9faa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9f410cf20 0x7fd9f41089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:22.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9faa00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f4112bf0 con 0x7fd9f4104470 2024-04-26T21:52:22.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9faa00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f4112a70 con 0x7fd9f410cf20 2024-04-26T21:52:22.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9faa00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f4112d70 con 0x7fd9f4103b50 2024-04-26T21:52:22.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9f3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f4103b50 0x7fd9f41a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:22.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9f8a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9f410cf20 0x7fd9f41089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:22.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9f3400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4104470 0x7fd9f41a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:22.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9f3e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f4103b50 0x7fd9f41a5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:56122/0 (socket says 172.21.15.1:56122) 2024-04-26T21:52:22.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.684+0000 7fd9f3e00640 1 -- 172.21.15.1:0/856998544 learned_addr learned my addr 172.21.15.1:0/856998544 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:22.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.685+0000 7fd9f8a00640 1 -- 172.21.15.1:0/856998544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f4103b50 msgr2=0x7fd9f41a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:22.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.685+0000 7fd9f8a00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f4103b50 0x7fd9f41a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.685+0000 7fd9f8a00640 1 -- 172.21.15.1:0/856998544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4104470 msgr2=0x7fd9f41a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:22.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.685+0000 7fd9f8a00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4104470 0x7fd9f41a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.685+0000 7fd9f8a00640 1 -- 172.21.15.1:0/856998544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f41092b0 con 0x7fd9f410cf20 2024-04-26T21:52:22.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.685+0000 7fd9f8a00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9f410cf20 0x7fd9f41089f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec005fd0 tx=0x7fd9ec00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:22.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.685+0000 7fd9f0c00640 1 -- 172.21.15.1:0/856998544 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec00cd80 con 0x7fd9f410cf20 2024-04-26T21:52:22.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.685+0000 7fd9f0c00640 1 -- 172.21.15.1:0/856998544 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9ec005020 con 0x7fd9f410cf20 2024-04-26T21:52:22.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.685+0000 7fd9faa00640 1 -- 172.21.15.1:0/856998544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f41095a0 con 0x7fd9f410cf20 2024-04-26T21:52:22.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.685+0000 7fd9f0c00640 1 -- 172.21.15.1:0/856998544 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec010690 con 0x7fd9f410cf20 2024-04-26T21:52:22.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.686+0000 7fd9faa00640 1 -- 172.21.15.1:0/856998544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f41ac3e0 con 0x7fd9f410cf20 2024-04-26T21:52:22.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.687+0000 7fd9faa00640 1 -- 172.21.15.1:0/856998544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f4069c30 con 0x7fd9f410cf20 2024-04-26T21:52:22.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.689+0000 7fd9f0c00640 1 -- 172.21.15.1:0/856998544 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fd9ec0054d0 con 0x7fd9f410cf20 2024-04-26T21:52:22.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.690+0000 7fd9f0c00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd9d80a3d20 0x7fd9d80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:22.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.690+0000 7fd9f0c00640 1 -- 172.21.15.1:0/856998544 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7fd9ec0d1310 con 0x7fd9f410cf20 2024-04-26T21:52:22.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.690+0000 7fd9f3e00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd9d80a3d20 0x7fd9d80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:22.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.691+0000 7fd9f3e00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd9d80a3d20 0x7fd9d80a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009c90 tx=0x7fd9e4005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:22.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.694+0000 7fd9f0c00640 1 -- 172.21.15.1:0/856998544 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fd9ec09d490 con 0x7fd9f410cf20 2024-04-26T21:52:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:22 smithi186 ceph-mon[32991]: pgmap v701: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 297 op/s 2024-04-26T21:52:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:22 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2429592627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:52:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:22 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3981337618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:52:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:22 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:52:22.727 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:22 smithi038 ceph-mon[32724]: pgmap v701: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 297 op/s 2024-04-26T21:52:22.727 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:22 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2429592627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:52:22.727 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:22 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3981337618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:52:22.727 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:22 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:52:22.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:22 smithi001 ceph-mon[28054]: pgmap v701: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 297 op/s 2024-04-26T21:52:22.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:22 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2429592627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:52:22.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:22 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3981337618' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:52:22.818 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:22 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T21:52:22.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.937+0000 7fd9faa00640 1 -- 172.21.15.1:0/856998544 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd9f41ac5b0 con 0x7fd9f410cf20 2024-04-26T21:52:22.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.937+0000 7fd9f0c00640 1 -- 172.21.15.1:0/856998544 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v133) ==== 103+0+11 (secure 0 0 0) 0x7fd9ec09cd90 con 0x7fd9f410cf20 2024-04-26T21:52:22.938 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:52:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.940+0000 7fd9faa00640 1 -- 172.21.15.1:0/856998544 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd9d80a3d20 msgr2=0x7fd9d80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.940+0000 7fd9faa00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd9d80a3d20 0x7fd9d80a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009c90 tx=0x7fd9e4005f90 comp rx=0 tx=0).stop 2024-04-26T21:52:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.940+0000 7fd9faa00640 1 -- 172.21.15.1:0/856998544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9f410cf20 msgr2=0x7fd9f41089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:22.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.940+0000 7fd9faa00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9f410cf20 0x7fd9f41089f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec005fd0 tx=0x7fd9ec00ee60 comp rx=0 tx=0).stop 2024-04-26T21:52:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.940+0000 7fd9faa00640 1 -- 172.21.15.1:0/856998544 shutdown_connections 2024-04-26T21:52:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.940+0000 7fd9faa00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fd9d80a3d20 0x7fd9d80a61e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.940+0000 7fd9faa00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9f410cf20 0x7fd9f41089f0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.940+0000 7fd9faa00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f4104470 0x7fd9f41a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.940+0000 7fd9faa00640 1 --2- 172.21.15.1:0/856998544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9f4103b50 0x7fd9f41a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.941+0000 7fd9faa00640 1 -- 172.21.15.1:0/856998544 >> 172.21.15.1:0/856998544 conn(0x7fd9f4078a10 msgr2=0x7fd9f410a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.941+0000 7fd9faa00640 1 -- 172.21.15.1:0/856998544 shutdown_connections 2024-04-26T21:52:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:22.941+0000 7fd9faa00640 1 -- 172.21.15.1:0/856998544 wait complete. 2024-04-26T21:52:22.961 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-26T21:52:23.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.172+0000 7f3b80a00640 1 -- 172.21.15.1:0/2919964890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b7c10e160 msgr2=0x7f3b7c115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:23.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.172+0000 7f3b80a00640 1 --2- 172.21.15.1:0/2919964890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b7c10e160 0x7f3b7c115d30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009ca0 tx=0x7f3b7402a8c0 comp rx=0 tx=0).stop 2024-04-26T21:52:23.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.172+0000 7f3b80a00640 1 -- 172.21.15.1:0/2919964890 shutdown_connections 2024-04-26T21:52:23.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.172+0000 7f3b80a00640 1 --2- 172.21.15.1:0/2919964890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b7c10e160 0x7f3b7c115d30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:23.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.172+0000 7f3b80a00640 1 --2- 172.21.15.1:0/2919964890 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b7c109a90 0x7f3b7c10da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:23.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.172+0000 7f3b80a00640 1 --2- 172.21.15.1:0/2919964890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b7c1090e0 0x7f3b7c1094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:23.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.172+0000 7f3b80a00640 1 -- 172.21.15.1:0/2919964890 >> 172.21.15.1:0/2919964890 conn(0x7f3b7c078700 msgr2=0x7f3b7c107750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:23.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.173+0000 7f3b80a00640 1 -- 172.21.15.1:0/2919964890 shutdown_connections 2024-04-26T21:52:23.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.173+0000 7f3b80a00640 1 -- 172.21.15.1:0/2919964890 wait complete. 2024-04-26T21:52:23.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.173+0000 7f3b80a00640 1 Processor -- start 2024-04-26T21:52:23.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b80a00640 1 -- start start 2024-04-26T21:52:23.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b80a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b7c1090e0 0x7f3b7c19f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:23.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b80a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b7c10e160 0x7f3b7c19fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b80a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b7c1a3e90 0x7f3b7c1a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b80a00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c119130 con 0x7f3b7c1a3e90 2024-04-26T21:52:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b80a00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c118fb0 con 0x7f3b7c1090e0 2024-04-26T21:52:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b80a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c1192b0 con 0x7f3b7c10e160 2024-04-26T21:52:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b7b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b7c1090e0 0x7f3b7c19f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b7b400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b7c1090e0 0x7f3b7c19f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:47620/0 (socket says 172.21.15.1:47620) 2024-04-26T21:52:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b7b400640 1 -- 172.21.15.1:0/2662490578 learned_addr learned my addr 172.21.15.1:0/2662490578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:52:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.174+0000 7f3b7be00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b7c1a3e90 0x7f3b7c1a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:23.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b73e00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b7c10e160 0x7f3b7c19fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:23.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b7b400640 1 -- 172.21.15.1:0/2662490578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b7c10e160 msgr2=0x7f3b7c19fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:23.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b7b400640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b7c10e160 0x7f3b7c19fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:23.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b7b400640 1 -- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b7c1a3e90 msgr2=0x7f3b7c1a4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:23.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b7b400640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b7c1a3e90 0x7f3b7c1a4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:23.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b7b400640 1 -- 172.21.15.1:0/2662490578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7c1a4980 con 0x7f3b7c1090e0 2024-04-26T21:52:23.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b7be00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b7c1a3e90 0x7f3b7c1a4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:52:23.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b7b400640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b7c1090e0 0x7f3b7c19f5d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00e9f0 tx=0x7f3b6c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:23.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b78c00640 1 -- 172.21.15.1:0/2662490578 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c00cd80 con 0x7f3b7c1090e0 2024-04-26T21:52:23.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b78c00640 1 -- 172.21.15.1:0/2662490578 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b6c0054d0 con 0x7f3b7c1090e0 2024-04-26T21:52:23.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b78c00640 1 -- 172.21.15.1:0/2662490578 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c010690 con 0x7f3b7c1090e0 2024-04-26T21:52:23.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b80a00640 1 -- 172.21.15.1:0/2662490578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b7c1a4c70 con 0x7f3b7c1090e0 2024-04-26T21:52:23.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.175+0000 7f3b80a00640 1 -- 172.21.15.1:0/2662490578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b7c1ac460 con 0x7f3b7c1090e0 2024-04-26T21:52:23.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.177+0000 7f3b80a00640 1 -- 172.21.15.1:0/2662490578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b7c050730 con 0x7f3b7c1090e0 2024-04-26T21:52:23.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.178+0000 7f3b78c00640 1 -- 172.21.15.1:0/2662490578 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f3b6c004e90 con 0x7f3b7c1090e0 2024-04-26T21:52:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.180+0000 7f3b78c00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b600a3d90 0x7f3b600a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:52:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.180+0000 7f3b78c00640 1 -- 172.21.15.1:0/2662490578 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(133..133 src has 1..133) ==== 9730+0+0 (secure 0 0 0) 0x7f3b6c0d11e0 con 0x7f3b7c1090e0 2024-04-26T21:52:23.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.180+0000 7f3b73e00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b600a3d90 0x7f3b600a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:52:23.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.181+0000 7f3b73e00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b600a3d90 0x7f3b600a6250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c10f3b0 tx=0x7f3b64005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:52:23.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.184+0000 7f3b78c00640 1 -- 172.21.15.1:0/2662490578 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f3b6c09d360 con 0x7f3b7c1090e0 2024-04-26T21:52:23.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.446+0000 7f3b80a00640 1 -- 172.21.15.1:0/2662490578 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f3b7c111380 con 0x7f3b7c1090e0 2024-04-26T21:52:23.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.458+0000 7f3b78c00640 1 -- 172.21.15.1:0/2662490578 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+746052 (secure 0 0 0) 0x7f3b6c0bbe20 con 0x7f3b7c1090e0 2024-04-26T21:52:23.470 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:52:23.470 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:52:23.470 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 26, 2024-04-26T21:52:23.470 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.471 INFO:teuthology.orchestra.run.smithi001.stdout: "active_gid": 15291, 2024-04-26T21:52:23.471 INFO:teuthology.orchestra.run.smithi001.stdout: "active_name": "x", 2024-04-26T21:52:23.471 INFO:teuthology.orchestra.run.smithi001.stdout: "active_addrs": { 2024-04-26T21:52:23.471 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-04-26T21:52:23.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.471 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-04-26T21:52:23.471 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6800", 2024-04-26T21:52:23.471 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3236768718 2024-04-26T21:52:23.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-04-26T21:52:23.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6801", 2024-04-26T21:52:23.472 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 3236768718 2024-04-26T21:52:23.472 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.472 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:23.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.472 INFO:teuthology.orchestra.run.smithi001.stdout: "active_addr": "172.21.15.1:6801/3236768718", 2024-04-26T21:52:23.473 INFO:teuthology.orchestra.run.smithi001.stdout: "active_change": "2024-04-26T21:30:18.554354+0000", 2024-04-26T21:52:23.473 INFO:teuthology.orchestra.run.smithi001.stdout: "active_mgr_features": 4540701547738038271, 2024-04-26T21:52:23.473 INFO:teuthology.orchestra.run.smithi001.stdout: "available": true, 2024-04-26T21:52:23.473 INFO:teuthology.orchestra.run.smithi001.stdout: "standbys": [ 2024-04-26T21:52:23.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.473 INFO:teuthology.orchestra.run.smithi001.stdout: "gid": 25411, 2024-04-26T21:52:23.473 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "z", 2024-04-26T21:52:23.473 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_features": 4540701547738038271, 2024-04-26T21:52:23.473 INFO:teuthology.orchestra.run.smithi001.stdout: "available_modules": [ 2024-04-26T21:52:23.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.474 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "alerts", 2024-04-26T21:52:23.474 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.474 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.474 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.474 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:23.474 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:23.474 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.474 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.475 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.475 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:23.475 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.475 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.475 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.475 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T21:52:23.475 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.475 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.475 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.476 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.476 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.476 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.476 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.476 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.476 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.476 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.476 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.476 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.477 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.477 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.477 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.477 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.477 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.477 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.477 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.477 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.477 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.478 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.478 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.478 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.478 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.478 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.478 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.478 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.479 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.479 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.479 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.479 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.479 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.479 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.479 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.479 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.480 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.480 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.480 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.480 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.480 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.480 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.480 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.480 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.480 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.480 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.481 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.481 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.481 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.481 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.481 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.481 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.481 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.481 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.481 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.482 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.482 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.482 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.482 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.482 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.482 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.482 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.483 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.483 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.483 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.483 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.483 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.483 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.483 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.483 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_destination": { 2024-04-26T21:52:23.484 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_destination", 2024-04-26T21:52:23.484 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.484 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.484 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.484 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.484 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.484 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.484 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.484 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Email address to send alerts to", 2024-04-26T21:52:23.485 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.485 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.485 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.485 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_from_name": { 2024-04-26T21:52:23.485 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_from_name", 2024-04-26T21:52:23.485 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.485 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.486 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.486 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "Ceph", 2024-04-26T21:52:23.486 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.486 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.486 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.486 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Email From: name", 2024-04-26T21:52:23.486 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.486 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.487 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.487 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_host": { 2024-04-26T21:52:23.487 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_host", 2024-04-26T21:52:23.487 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.487 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.487 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.487 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.488 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.488 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.488 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.488 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SMTP server", 2024-04-26T21:52:23.488 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.488 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.488 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.488 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_password": { 2024-04-26T21:52:23.489 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_password", 2024-04-26T21:52:23.489 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.489 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.489 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.489 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.489 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.489 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.489 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.489 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Password to authenticate with", 2024-04-26T21:52:23.490 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.490 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.490 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.490 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_port": { 2024-04-26T21:52:23.490 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_port", 2024-04-26T21:52:23.490 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.490 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.491 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.491 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "465", 2024-04-26T21:52:23.491 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.491 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.491 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.491 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SMTP port", 2024-04-26T21:52:23.491 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.491 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.491 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.492 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_sender": { 2024-04-26T21:52:23.492 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_sender", 2024-04-26T21:52:23.492 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.492 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.492 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.492 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.492 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.493 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.493 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.493 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SMTP envelope sender", 2024-04-26T21:52:23.493 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.493 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.493 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.493 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_ssl": { 2024-04-26T21:52:23.493 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_ssl", 2024-04-26T21:52:23.494 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.494 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.494 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.494 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.494 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.494 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.494 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.495 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T21:52:23.495 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.495 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.495 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.495 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_user": { 2024-04-26T21:52:23.495 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_user", 2024-04-26T21:52:23.495 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.496 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.496 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.496 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.496 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.496 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.496 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.496 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "User to authenticate as", 2024-04-26T21:52:23.496 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.496 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.497 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.497 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "balancer", 2024-04-26T21:52:23.497 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.497 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.497 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi001.stdout: "active": { 2024-04-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "active", 2024-04-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.499 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T21:52:23.499 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.499 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.499 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.499 INFO:teuthology.orchestra.run.smithi001.stdout: "begin_time": { 2024-04-26T21:52:23.499 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "begin_time", 2024-04-26T21:52:23.499 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.499 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.500 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.500 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0000", 2024-04-26T21:52:23.500 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.500 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.500 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.500 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T21:52:23.500 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T21:52:23.500 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.500 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.501 INFO:teuthology.orchestra.run.smithi001.stdout: "begin_weekday": { 2024-04-26T21:52:23.501 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "begin_weekday", 2024-04-26T21:52:23.501 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:23.501 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.501 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.501 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:23.501 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "0", 2024-04-26T21:52:23.502 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "6", 2024-04-26T21:52:23.502 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.502 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T21:52:23.502 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T21:52:23.502 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.502 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.502 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_compat_max_iterations": { 2024-04-26T21:52:23.502 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crush_compat_max_iterations", 2024-04-26T21:52:23.503 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:23.503 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.503 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.503 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "25", 2024-04-26T21:52:23.503 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:23.503 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "250", 2024-04-26T21:52:23.503 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.503 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T21:52:23.503 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.504 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.504 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.504 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_compat_metrics": { 2024-04-26T21:52:23.504 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crush_compat_metrics", 2024-04-26T21:52:23.504 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.504 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.504 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.504 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T21:52:23.505 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.505 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.505 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.505 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T21:52:23.505 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T21:52:23.505 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.505 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.505 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.505 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_compat_step": { 2024-04-26T21:52:23.506 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crush_compat_step", 2024-04-26T21:52:23.506 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:23.506 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.506 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.506 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0.5", 2024-04-26T21:52:23.506 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "0.001", 2024-04-26T21:52:23.506 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "0.999", 2024-04-26T21:52:23.507 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.507 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "aggressiveness of optimization", 2024-04-26T21:52:23.507 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T21:52:23.507 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.507 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.507 INFO:teuthology.orchestra.run.smithi001.stdout: "end_time": { 2024-04-26T21:52:23.507 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "end_time", 2024-04-26T21:52:23.507 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.508 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.508 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.508 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "2359", 2024-04-26T21:52:23.508 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.508 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.508 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.508 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T21:52:23.508 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T21:52:23.508 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.509 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.509 INFO:teuthology.orchestra.run.smithi001.stdout: "end_weekday": { 2024-04-26T21:52:23.509 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "end_weekday", 2024-04-26T21:52:23.509 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:23.509 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.509 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.509 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:23.509 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "0", 2024-04-26T21:52:23.510 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "6", 2024-04-26T21:52:23.510 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.510 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T21:52:23.510 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T21:52:23.510 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.510 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.510 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.510 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.511 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.512 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.513 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.513 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.513 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.513 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.513 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.513 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.513 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.513 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.514 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.515 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.515 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.515 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.515 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.515 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.515 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.515 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.515 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.515 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.515 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.516 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.516 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.516 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.516 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.516 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.516 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.516 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.516 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.516 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.516 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.517 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.517 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.517 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.517 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.517 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.517 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.517 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.517 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.517 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.518 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.518 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.518 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.518 INFO:teuthology.orchestra.run.smithi001.stdout: "min_score": { 2024-04-26T21:52:23.518 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "min_score", 2024-04-26T21:52:23.518 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:23.518 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.519 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.519 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:23.519 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.519 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.519 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.519 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T21:52:23.519 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.519 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.520 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.520 INFO:teuthology.orchestra.run.smithi001.stdout: "mode": { 2024-04-26T21:52:23.520 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mode", 2024-04-26T21:52:23.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.520 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.520 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.520 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "upmap", 2024-04-26T21:52:23.520 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.520 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.521 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.521 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-compat", 2024-04-26T21:52:23.521 INFO:teuthology.orchestra.run.smithi001.stdout: "none", 2024-04-26T21:52:23.521 INFO:teuthology.orchestra.run.smithi001.stdout: "read", 2024-04-26T21:52:23.521 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap", 2024-04-26T21:52:23.521 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap-read" 2024-04-26T21:52:23.521 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.521 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Balancer mode", 2024-04-26T21:52:23.521 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.522 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.522 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.522 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_ids": { 2024-04-26T21:52:23.522 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pool_ids", 2024-04-26T21:52:23.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.522 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.522 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.523 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.523 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.523 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.523 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.523 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T21:52:23.523 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.523 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.523 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.524 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:23.524 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:23.524 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.524 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.524 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.524 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:23.524 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.524 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.525 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.525 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T21:52:23.525 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.525 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.525 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.525 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap_max_deviation": { 2024-04-26T21:52:23.525 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "upmap_max_deviation", 2024-04-26T21:52:23.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.526 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.526 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.526 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:23.526 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:23.526 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.526 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.526 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T21:52:23.526 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T21:52:23.527 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.527 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.527 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap_max_optimizations": { 2024-04-26T21:52:23.527 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "upmap_max_optimizations", 2024-04-26T21:52:23.527 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:23.527 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.527 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.527 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:23.527 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.528 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.528 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.528 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T21:52:23.528 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.528 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.528 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephadm", 2024-04-26T21:52:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: "agent_down_multiplier": { 2024-04-26T21:52:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "agent_down_multiplier", 2024-04-26T21:52:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:23.529 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.530 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.530 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "3.0", 2024-04-26T21:52:23.530 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.530 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.530 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.530 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T21:52:23.530 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.530 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.531 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.531 INFO:teuthology.orchestra.run.smithi001.stdout: "agent_refresh_rate": { 2024-04-26T21:52:23.531 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "agent_refresh_rate", 2024-04-26T21:52:23.531 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.531 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.531 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.531 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "20", 2024-04-26T21:52:23.531 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.531 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.532 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.532 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T21:52:23.532 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.532 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.532 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.532 INFO:teuthology.orchestra.run.smithi001.stdout: "agent_starting_port": { 2024-04-26T21:52:23.532 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "agent_starting_port", 2024-04-26T21:52:23.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.533 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.533 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.533 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "4721", 2024-04-26T21:52:23.533 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.533 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.533 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.533 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T21:52:23.534 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.534 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.534 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.534 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_ptrace": { 2024-04-26T21:52:23.534 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "allow_ptrace", 2024-04-26T21:52:23.534 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.534 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.535 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.535 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.535 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.535 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.535 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.535 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T21:52:23.535 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T21:52:23.535 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.535 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.536 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.536 INFO:teuthology.orchestra.run.smithi001.stdout: "autotune_interval": { 2024-04-26T21:52:23.536 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "autotune_interval", 2024-04-26T21:52:23.536 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.536 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.536 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.536 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:23.536 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.536 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.537 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.537 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T21:52:23.537 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.537 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.537 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.537 INFO:teuthology.orchestra.run.smithi001.stdout: "autotune_memory_target_ratio": { 2024-04-26T21:52:23.537 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T21:52:23.538 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:23.538 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.538 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.538 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0.7", 2024-04-26T21:52:23.538 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.538 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.538 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.538 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T21:52:23.538 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.539 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.539 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.539 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm_log_destination": { 2024-04-26T21:52:23.539 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephadm_log_destination", 2024-04-26T21:52:23.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.539 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.539 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.539 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.540 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.540 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.540 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.540 INFO:teuthology.orchestra.run.smithi001.stdout: "file", 2024-04-26T21:52:23.540 INFO:teuthology.orchestra.run.smithi001.stdout: "file,syslog", 2024-04-26T21:52:23.540 INFO:teuthology.orchestra.run.smithi001.stdout: "syslog" 2024-04-26T21:52:23.540 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.540 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T21:52:23.540 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.541 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.541 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.541 INFO:teuthology.orchestra.run.smithi001.stdout: "cgroups_split": { 2024-04-26T21:52:23.541 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cgroups_split", 2024-04-26T21:52:23.541 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.541 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.541 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.541 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.541 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.542 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.542 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.542 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T21:52:23.542 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.542 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.542 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.543 INFO:teuthology.orchestra.run.smithi001.stdout: "config_checks_enabled": { 2024-04-26T21:52:23.543 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "config_checks_enabled", 2024-04-26T21:52:23.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.543 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.543 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.543 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.543 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.543 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.543 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.544 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T21:52:23.544 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.544 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.544 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.544 INFO:teuthology.orchestra.run.smithi001.stdout: "config_dashboard": { 2024-04-26T21:52:23.544 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "config_dashboard", 2024-04-26T21:52:23.544 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.544 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.545 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.545 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.545 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.545 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.545 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.545 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T21:52:23.545 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.545 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.545 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.546 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_alertmanager": { 2024-04-26T21:52:23.546 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_alertmanager", 2024-04-26T21:52:23.546 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.546 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.546 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.546 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T21:52:23.547 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.547 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.547 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.547 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:23.547 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.547 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.547 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.547 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_base": { 2024-04-26T21:52:23.548 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_base", 2024-04-26T21:52:23.548 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.548 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.548 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.548 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T21:52:23.548 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.548 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.548 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.548 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Container image name, without the tag", 2024-04-26T21:52:23.549 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.549 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.549 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.549 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_elasticsearch": { 2024-04-26T21:52:23.549 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_elasticsearch", 2024-04-26T21:52:23.549 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.549 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.549 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.550 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T21:52:23.550 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.550 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.550 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.550 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "elasticsearch container image", 2024-04-26T21:52:23.550 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.550 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.550 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.551 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_grafana": { 2024-04-26T21:52:23.551 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_grafana", 2024-04-26T21:52:23.551 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.551 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.551 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.551 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T21:52:23.551 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.551 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.552 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.552 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:23.552 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.552 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.552 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.552 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_haproxy": { 2024-04-26T21:52:23.552 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_haproxy", 2024-04-26T21:52:23.552 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.553 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.553 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.553 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T21:52:23.553 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.553 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.553 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.553 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "HAproxy container image", 2024-04-26T21:52:23.553 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.553 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.554 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.554 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_jaeger_agent": { 2024-04-26T21:52:23.554 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_jaeger_agent", 2024-04-26T21:52:23.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.554 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.554 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.554 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T21:52:23.555 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.555 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.555 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.555 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Jaeger agent container image", 2024-04-26T21:52:23.555 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.555 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.555 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.555 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_jaeger_collector": { 2024-04-26T21:52:23.555 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_jaeger_collector", 2024-04-26T21:52:23.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.556 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.556 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.556 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T21:52:23.556 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.556 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.556 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.556 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Jaeger collector container image", 2024-04-26T21:52:23.556 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.557 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.557 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.557 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_jaeger_query": { 2024-04-26T21:52:23.557 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_jaeger_query", 2024-04-26T21:52:23.557 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.557 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.557 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.557 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T21:52:23.558 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.558 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.558 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.558 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Jaeger query container image", 2024-04-26T21:52:23.558 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.558 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.558 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.559 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_keepalived": { 2024-04-26T21:52:23.559 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_keepalived", 2024-04-26T21:52:23.559 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.559 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.559 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.559 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T21:52:23.559 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.559 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.560 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.560 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Keepalived container image", 2024-04-26T21:52:23.560 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.560 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.560 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.560 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_loki": { 2024-04-26T21:52:23.560 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_loki", 2024-04-26T21:52:23.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.560 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.561 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.561 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T21:52:23.561 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.561 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.561 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.561 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Loki container image", 2024-04-26T21:52:23.561 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.561 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.561 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.562 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_node_exporter": { 2024-04-26T21:52:23.562 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_node_exporter", 2024-04-26T21:52:23.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.562 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.562 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.562 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T21:52:23.562 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.562 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.563 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.563 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:23.563 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.563 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.563 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.563 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_nvmeof": { 2024-04-26T21:52:23.563 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_nvmeof", 2024-04-26T21:52:23.563 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.563 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.564 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.564 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T21:52:23.564 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.564 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.564 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.564 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Nvme-of container image", 2024-04-26T21:52:23.564 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.564 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.565 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.565 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_prometheus": { 2024-04-26T21:52:23.565 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_prometheus", 2024-04-26T21:52:23.565 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.565 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.565 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.565 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T21:52:23.565 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.565 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.566 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.566 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:23.566 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.566 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.566 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.566 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_promtail": { 2024-04-26T21:52:23.566 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_promtail", 2024-04-26T21:52:23.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.567 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.567 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.567 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T21:52:23.567 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.567 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.567 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.567 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Promtail container image", 2024-04-26T21:52:23.567 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.568 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.568 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.568 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_samba": { 2024-04-26T21:52:23.568 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_samba", 2024-04-26T21:52:23.568 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.568 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.568 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.568 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T21:52:23.569 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.569 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.569 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.569 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Samba/SMB container image", 2024-04-26T21:52:23.569 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.569 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.569 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.569 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_snmp_gateway": { 2024-04-26T21:52:23.569 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_snmp_gateway", 2024-04-26T21:52:23.570 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.570 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.570 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.570 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T21:52:23.570 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.570 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.570 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.570 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SNMP Gateway container image", 2024-04-26T21:52:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "container_init": { 2024-04-26T21:52:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_init", 2024-04-26T21:52:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.571 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T21:52:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.572 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "daemon_cache_timeout": { 2024-04-26T21:52:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "daemon_cache_timeout", 2024-04-26T21:52:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.573 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T21:52:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "default_cephadm_command_timeout": { 2024-04-26T21:52:23.574 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T21:52:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "900", 2024-04-26T21:52:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T21:52:23.575 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "default_registry": { 2024-04-26T21:52:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "default_registry", 2024-04-26T21:52:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.576 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io", 2024-04-26T21:52:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T21:52:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "device_cache_timeout": { 2024-04-26T21:52:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "device_cache_timeout", 2024-04-26T21:52:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "1800", 2024-04-26T21:52:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.578 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "seconds to cache device inventory", 2024-04-26T21:52:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "device_enhanced_scan": { 2024-04-26T21:52:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "device_enhanced_scan", 2024-04-26T21:52:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.579 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T21:52:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "facts_cache_timeout": { 2024-04-26T21:52:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "facts_cache_timeout", 2024-04-26T21:52:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.581 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "seconds to cache host facts data", 2024-04-26T21:52:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "grafana_dashboards_path": { 2024-04-26T21:52:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "grafana_dashboards_path", 2024-04-26T21:52:23.582 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T21:52:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T21:52:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.583 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "host_check_interval": { 2024-04-26T21:52:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "host_check_interval", 2024-04-26T21:52:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.584 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to perform a host check", 2024-04-26T21:52:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "hw_monitoring": { 2024-04-26T21:52:23.585 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "hw_monitoring", 2024-04-26T21:52:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.586 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T21:52:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "inventory_list_all": { 2024-04-26T21:52:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "inventory_list_all", 2024-04-26T21:52:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.587 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T21:52:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.588 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.589 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.590 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "log_refresh_metadata": { 2024-04-26T21:52:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_refresh_metadata", 2024-04-26T21:52:23.591 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T21:52:23.592 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.593 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T21:52:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.595 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.596 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.597 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T21:52:23.598 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T21:52:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T21:52:23.599 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T21:52:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T21:52:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.600 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "*", 2024-04-26T21:52:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T21:52:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "max_count_per_host": { 2024-04-26T21:52:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_count_per_host", 2024-04-26T21:52:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.602 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "max number of daemons per service per host", 2024-04-26T21:52:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd_draining_count": { 2024-04-26T21:52:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_osd_draining_count", 2024-04-26T21:52:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.603 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T21:52:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.604 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "migration_current": { 2024-04-26T21:52:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "migration_current", 2024-04-26T21:52:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.605 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "internal - do not modify", 2024-04-26T21:52:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "mode": { 2024-04-26T21:52:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mode", 2024-04-26T21:52:23.606 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "root", 2024-04-26T21:52:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm-package", 2024-04-26T21:52:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: "root" 2024-04-26T21:52:23.607 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T21:52:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "oob_default_addr": { 2024-04-26T21:52:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "oob_default_addr", 2024-04-26T21:52:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.608 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "169.254.1.1", 2024-04-26T21:52:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T21:52:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.609 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "prometheus_alerts_path": { 2024-04-26T21:52:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "prometheus_alerts_path", 2024-04-26T21:52:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T21:52:23.610 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T21:52:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.611 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_insecure": { 2024-04-26T21:52:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_insecure", 2024-04-26T21:52:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.612 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T21:52:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_password": { 2024-04-26T21:52:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_password", 2024-04-26T21:52:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.613 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T21:52:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.614 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_url": { 2024-04-26T21:52:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_url", 2024-04-26T21:52:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.615 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T21:52:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_username": { 2024-04-26T21:52:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_username", 2024-04-26T21:52:23.616 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T21:52:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.617 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "secure_monitoring_stack": { 2024-04-26T21:52:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "secure_monitoring_stack", 2024-04-26T21:52:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.618 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T21:52:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "service_discovery_port": { 2024-04-26T21:52:23.619 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "service_discovery_port", 2024-04-26T21:52:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8765", 2024-04-26T21:52:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.620 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "cephadm service discovery port", 2024-04-26T21:52:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "ssh_config_file": { 2024-04-26T21:52:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssh_config_file", 2024-04-26T21:52:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.621 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T21:52:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.622 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "use_agent": { 2024-04-26T21:52:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "use_agent", 2024-04-26T21:52:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.623 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T21:52:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "use_repo_digest": { 2024-04-26T21:52:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "use_repo_digest", 2024-04-26T21:52:23.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.625 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T21:52:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_on_failed_host_check": { 2024-04-26T21:52:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_on_failed_host_check", 2024-04-26T21:52:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.626 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T21:52:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.627 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_on_stray_daemons": { 2024-04-26T21:52:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_on_stray_daemons", 2024-04-26T21:52:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.628 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T21:52:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_on_stray_hosts": { 2024-04-26T21:52:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_on_stray_hosts", 2024-04-26T21:52:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.629 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T21:52:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crash", 2024-04-26T21:52:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.631 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.632 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.633 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.634 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.635 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.636 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.638 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "retain_interval": { 2024-04-26T21:52:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "retain_interval", 2024-04-26T21:52:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.639 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "31536000", 2024-04-26T21:52:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T21:52:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.640 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_recent_interval": { 2024-04-26T21:52:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_recent_interval", 2024-04-26T21:52:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "1209600", 2024-04-26T21:52:23.641 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T21:52:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "dashboard", 2024-04-26T21:52:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T21:52:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T21:52:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.643 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.644 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T21:52:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T21:52:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.645 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T21:52:23.646 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T21:52:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.647 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "AUDIT_API_ENABLED": { 2024-04-26T21:52:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T21:52:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.648 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.649 INFO:teuthology.orchestra.run.smithi001.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T21:52:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T21:52:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.650 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T21:52:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T21:52:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.651 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.652 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T21:52:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T21:52:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.653 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T21:52:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T21:52:23.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.655 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T21:52:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T21:52:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T21:52:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T21:52:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.658 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T21:52:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T21:52:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.659 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.660 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T21:52:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T21:52:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.661 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T21:52:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T21:52:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.662 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T21:52:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T21:52:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.664 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T21:52:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T21:52:23.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "admin", 2024-04-26T21:52:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.666 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T21:52:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T21:52:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.667 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.668 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_URL": { 2024-04-26T21:52:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_URL", 2024-04-26T21:52:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.669 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T21:52:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T21:52:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.670 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "admin", 2024-04-26T21:52:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.671 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T21:52:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T21:52:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.672 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.673 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T21:52:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T21:52:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.674 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T21:52:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T21:52:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.675 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.676 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T21:52:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T21:52:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.677 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T21:52:23.678 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T21:52:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "[]", 2024-04-26T21:52:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.679 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T21:52:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T21:52:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.680 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "{}", 2024-04-26T21:52:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T21:52:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T21:52:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.682 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T21:52:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T21:52:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.683 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.684 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T21:52:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T21:52:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.685 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T21:52:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T21:52:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.687 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.688 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.689 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T21:52:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T21:52:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.690 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T21:52:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T21:52:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.691 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.692 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T21:52:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T21:52:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.693 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T21:52:23.694 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T21:52:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.695 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T21:52:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T21:52:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.696 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T21:52:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T21:52:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.698 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T21:52:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T21:52:23.699 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T21:52:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.700 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T21:52:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T21:52:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:23.701 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.702 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T21:52:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T21:52:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8", 2024-04-26T21:52:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.703 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T21:52:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T21:52:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.704 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "45", 2024-04-26T21:52:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.705 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T21:52:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T21:52:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.706 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T21:52:23.707 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T21:52:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "admin", 2024-04-26T21:52:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.708 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T21:52:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T21:52:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.709 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.710 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T21:52:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T21:52:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.711 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T21:52:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T21:52:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.712 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.713 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T21:52:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T21:52:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:23.714 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T21:52:23.715 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T21:52:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.716 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T21:52:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T21:52:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.717 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.718 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "cross_origin_url": { 2024-04-26T21:52:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cross_origin_url", 2024-04-26T21:52:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.719 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "crt_file": { 2024-04-26T21:52:23.720 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crt_file", 2024-04-26T21:52:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.721 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "debug": { 2024-04-26T21:52:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "debug", 2024-04-26T21:52:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.722 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Enable/disable debug options", 2024-04-26T21:52:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.724 INFO:teuthology.orchestra.run.smithi001.stdout: "jwt_token_ttl": { 2024-04-26T21:52:23.724 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "jwt_token_ttl", 2024-04-26T21:52:23.724 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.724 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.724 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.724 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "28800", 2024-04-26T21:52:23.724 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.724 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.724 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.725 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.725 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.725 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.725 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.725 INFO:teuthology.orchestra.run.smithi001.stdout: "key_file": { 2024-04-26T21:52:23.725 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "key_file", 2024-04-26T21:52:23.725 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.726 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.726 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.726 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.726 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.726 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.726 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.726 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.726 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.727 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.727 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.727 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.727 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.727 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.727 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.727 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.727 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.727 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.728 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.728 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.728 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.728 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.728 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.728 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.728 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.728 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.728 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.728 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.729 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.729 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.729 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.729 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.729 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.729 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.729 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.729 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.730 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.730 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.730 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.730 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.730 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.730 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.730 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.731 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.731 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.731 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.731 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.731 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.731 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.731 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.731 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.731 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.732 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.732 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.732 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.732 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.732 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.732 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.732 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.732 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.732 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.733 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.733 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.733 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.733 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.733 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.733 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.733 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.734 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.734 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.734 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.734 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.734 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.734 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.734 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.734 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.735 INFO:teuthology.orchestra.run.smithi001.stdout: "motd": { 2024-04-26T21:52:23.735 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "motd", 2024-04-26T21:52:23.735 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.735 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.735 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.735 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.735 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.735 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.736 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.736 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "The message of the day", 2024-04-26T21:52:23.736 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.736 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.736 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.736 INFO:teuthology.orchestra.run.smithi001.stdout: "redirect_resolve_ip_addr": { 2024-04-26T21:52:23.736 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T21:52:23.736 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.737 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.737 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.737 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.737 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.737 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.737 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.737 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.737 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.737 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.737 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.738 INFO:teuthology.orchestra.run.smithi001.stdout: "server_addr": { 2024-04-26T21:52:23.738 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_addr", 2024-04-26T21:52:23.738 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.738 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.738 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.738 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "::", 2024-04-26T21:52:23.738 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.738 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.739 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.739 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.739 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.739 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.739 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.739 INFO:teuthology.orchestra.run.smithi001.stdout: "server_port": { 2024-04-26T21:52:23.739 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_port", 2024-04-26T21:52:23.739 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.740 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.740 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.740 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8080", 2024-04-26T21:52:23.740 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.740 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.740 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.740 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.740 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.740 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.741 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.741 INFO:teuthology.orchestra.run.smithi001.stdout: "ssl": { 2024-04-26T21:52:23.741 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssl", 2024-04-26T21:52:23.741 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.741 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.741 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.741 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.741 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.741 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.742 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.742 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.742 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.742 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.742 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.742 INFO:teuthology.orchestra.run.smithi001.stdout: "ssl_server_port": { 2024-04-26T21:52:23.742 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssl_server_port", 2024-04-26T21:52:23.742 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.743 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.743 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.743 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8443", 2024-04-26T21:52:23.743 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.743 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.743 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.743 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.743 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.744 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.744 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.744 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_behaviour": { 2024-04-26T21:52:23.744 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_behaviour", 2024-04-26T21:52:23.744 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.744 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.744 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.744 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "redirect", 2024-04-26T21:52:23.745 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.745 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.745 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.745 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.745 INFO:teuthology.orchestra.run.smithi001.stdout: "redirect" 2024-04-26T21:52:23.745 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.745 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.745 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.746 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.746 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.746 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_error_status_code": { 2024-04-26T21:52:23.746 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_error_status_code", 2024-04-26T21:52:23.746 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.746 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.746 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.746 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "500", 2024-04-26T21:52:23.747 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "400", 2024-04-26T21:52:23.747 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "599", 2024-04-26T21:52:23.747 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.747 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.747 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.747 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.747 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.748 INFO:teuthology.orchestra.run.smithi001.stdout: "url_prefix": { 2024-04-26T21:52:23.748 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "url_prefix", 2024-04-26T21:52:23.748 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.748 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.748 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.748 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.748 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.748 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.748 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.749 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.749 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.749 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.749 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.749 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.749 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.749 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "devicehealth", 2024-04-26T21:52:23.750 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.750 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.750 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.750 INFO:teuthology.orchestra.run.smithi001.stdout: "enable_monitoring": { 2024-04-26T21:52:23.750 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enable_monitoring", 2024-04-26T21:52:23.750 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.750 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.750 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.751 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.751 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.751 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.751 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.751 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "monitor device health metrics", 2024-04-26T21:52:23.751 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.751 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.751 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.751 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.752 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.752 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.752 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.752 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.752 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.752 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.752 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.752 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.752 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.753 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.753 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.753 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.753 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.753 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.753 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.753 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.753 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.753 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.754 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.754 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.754 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.754 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.754 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.754 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.754 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.754 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.754 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.755 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.755 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.755 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.755 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.755 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.755 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.755 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.755 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.756 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.758 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.758 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.758 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.758 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.758 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.758 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.758 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.758 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.759 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.759 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.759 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.759 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.759 INFO:teuthology.orchestra.run.smithi001.stdout: "mark_out_threshold": { 2024-04-26T21:52:23.759 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mark_out_threshold", 2024-04-26T21:52:23.759 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.759 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.760 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.760 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "2419200", 2024-04-26T21:52:23.760 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.760 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.760 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.760 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T21:52:23.760 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.760 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.760 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.761 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": { 2024-04-26T21:52:23.761 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pool_name", 2024-04-26T21:52:23.761 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.761 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.761 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.761 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "device_health_metrics", 2024-04-26T21:52:23.761 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.761 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.761 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi001.stdout: "retention_period": { 2024-04-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "retention_period", 2024-04-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.763 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.763 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "15552000", 2024-04-26T21:52:23.763 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.763 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.763 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.763 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how long to retain device health metrics", 2024-04-26T21:52:23.763 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.763 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.764 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.764 INFO:teuthology.orchestra.run.smithi001.stdout: "scrape_frequency": { 2024-04-26T21:52:23.764 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "scrape_frequency", 2024-04-26T21:52:23.764 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.764 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.764 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.764 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "86400", 2024-04-26T21:52:23.764 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.764 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.765 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.765 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T21:52:23.765 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.765 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.765 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.765 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.765 INFO:teuthology.orchestra.run.smithi001.stdout: "self_heal": { 2024-04-26T21:52:23.765 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "self_heal", 2024-04-26T21:52:23.765 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.765 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.766 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.766 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.766 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.766 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.766 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.766 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T21:52:23.766 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.766 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.767 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.767 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:23.767 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:23.767 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.767 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.767 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.767 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:23.767 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.768 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.768 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.768 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T21:52:23.768 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.768 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.768 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.768 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.768 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_threshold": { 2024-04-26T21:52:23.768 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_threshold", 2024-04-26T21:52:23.769 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.769 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.769 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.769 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "7257600", 2024-04-26T21:52:23.769 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.769 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.769 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.769 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T21:52:23.769 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.770 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.770 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.770 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.770 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.770 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.770 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "diskprediction_local", 2024-04-26T21:52:23.770 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.770 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.771 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.771 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.771 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.771 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.771 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.771 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.771 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.771 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.771 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.772 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.772 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.772 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.772 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.772 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.772 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.772 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.772 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.773 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.773 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.773 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.773 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.773 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.773 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.773 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.773 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.773 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.774 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.774 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.774 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.774 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.774 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.774 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.774 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.774 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.775 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.775 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.775 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.775 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.775 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.775 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.775 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.775 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.776 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.776 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.776 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.776 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.776 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.776 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.776 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.776 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.776 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.777 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.777 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.777 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.777 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.777 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.777 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.777 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.777 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.778 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.778 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.778 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.778 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.778 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.778 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.778 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.778 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.779 INFO:teuthology.orchestra.run.smithi001.stdout: "predict_interval": { 2024-04-26T21:52:23.779 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "predict_interval", 2024-04-26T21:52:23.779 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.779 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.779 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.779 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "86400", 2024-04-26T21:52:23.779 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.779 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.779 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.780 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.780 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.780 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.780 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.780 INFO:teuthology.orchestra.run.smithi001.stdout: "predictor_model": { 2024-04-26T21:52:23.780 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "predictor_model", 2024-04-26T21:52:23.780 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.780 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.781 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.781 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "prophetstor", 2024-04-26T21:52:23.781 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.781 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.781 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.781 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.781 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.781 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.782 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.782 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.782 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:23.782 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:23.782 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.782 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.782 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.782 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:23.783 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.783 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.783 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.783 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.783 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.783 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.783 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.783 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.783 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.784 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.784 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "influx", 2024-04-26T21:52:23.784 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": false, 2024-04-26T21:52:23.784 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "influxdb python module not found", 2024-04-26T21:52:23.784 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.784 INFO:teuthology.orchestra.run.smithi001.stdout: "batch_size": { 2024-04-26T21:52:23.784 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "batch_size", 2024-04-26T21:52:23.784 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.785 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.785 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.785 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5000", 2024-04-26T21:52:23.785 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.785 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.785 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.785 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T21:52:23.785 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.786 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.786 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.786 INFO:teuthology.orchestra.run.smithi001.stdout: "database": { 2024-04-26T21:52:23.786 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "database", 2024-04-26T21:52:23.786 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.786 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.787 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.787 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "ceph", 2024-04-26T21:52:23.787 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.787 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.787 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.787 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T21:52:23.787 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.787 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.787 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.788 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": { 2024-04-26T21:52:23.788 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "hostname", 2024-04-26T21:52:23.788 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.788 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.788 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.788 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.788 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.788 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.788 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.789 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "InfluxDB server hostname", 2024-04-26T21:52:23.789 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.789 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.789 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.789 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:23.789 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:23.789 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.789 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.789 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.790 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "30", 2024-04-26T21:52:23.790 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "5", 2024-04-26T21:52:23.790 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.790 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.790 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T21:52:23.790 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.790 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.790 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.791 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.791 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.791 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.791 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.791 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.791 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.791 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.791 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.791 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.792 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.793 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.793 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.793 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.793 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.793 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.793 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.793 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.793 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.793 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.794 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.794 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.794 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.794 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.794 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.794 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.794 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.795 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.795 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.795 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.795 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.795 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.795 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.795 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.795 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.796 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.796 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.796 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.796 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.796 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.796 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.796 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.796 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.796 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.797 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.797 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.797 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.797 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.797 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.797 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.797 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.797 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.797 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.797 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.798 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.798 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.798 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.798 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.798 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.798 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.798 INFO:teuthology.orchestra.run.smithi001.stdout: "password": { 2024-04-26T21:52:23.798 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "password", 2024-04-26T21:52:23.799 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.799 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.799 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.799 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.799 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.799 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.799 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.799 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "password of InfluxDB server user", 2024-04-26T21:52:23.799 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.800 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.800 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.800 INFO:teuthology.orchestra.run.smithi001.stdout: "port": { 2024-04-26T21:52:23.800 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "port", 2024-04-26T21:52:23.800 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.800 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.800 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.800 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8086", 2024-04-26T21:52:23.800 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.801 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.801 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.801 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "InfluxDB server port", 2024-04-26T21:52:23.801 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.801 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.801 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.801 INFO:teuthology.orchestra.run.smithi001.stdout: "ssl": { 2024-04-26T21:52:23.801 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssl", 2024-04-26T21:52:23.802 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.802 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.802 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.802 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "false", 2024-04-26T21:52:23.802 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.802 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.802 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.802 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T21:52:23.803 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.803 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.803 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.803 INFO:teuthology.orchestra.run.smithi001.stdout: "threads": { 2024-04-26T21:52:23.803 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "threads", 2024-04-26T21:52:23.803 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.803 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.803 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.804 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:23.804 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:23.804 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "32", 2024-04-26T21:52:23.804 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.804 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T21:52:23.804 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.804 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.804 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.805 INFO:teuthology.orchestra.run.smithi001.stdout: "username": { 2024-04-26T21:52:23.805 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "username", 2024-04-26T21:52:23.805 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.805 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.805 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.805 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.805 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.805 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.805 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.806 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "username of InfluxDB server user", 2024-04-26T21:52:23.806 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.806 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.806 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.806 INFO:teuthology.orchestra.run.smithi001.stdout: "verify_ssl": { 2024-04-26T21:52:23.806 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "verify_ssl", 2024-04-26T21:52:23.806 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.806 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.807 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.807 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "true", 2024-04-26T21:52:23.807 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.807 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.807 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.807 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T21:52:23.807 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.807 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.807 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.807 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.808 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.808 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.808 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "insights", 2024-04-26T21:52:23.808 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.808 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.808 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.808 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.808 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.809 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.809 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.809 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.809 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.809 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.809 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.809 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.809 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.809 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.810 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.810 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.810 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.810 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.810 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.810 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.810 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.810 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.811 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.811 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.811 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.811 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.811 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.811 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.811 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.811 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.812 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.812 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.812 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.812 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.812 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.812 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.812 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.812 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.812 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.813 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.813 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.813 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.813 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.813 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.813 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.813 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.813 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.813 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.814 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.814 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.814 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.814 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.814 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.814 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.814 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.814 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.815 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.815 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.815 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.815 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.815 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.815 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.815 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.815 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.815 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.816 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.816 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.816 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.816 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.817 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "iostat", 2024-04-26T21:52:23.817 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.817 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.817 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.817 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.817 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.817 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.817 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.817 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.818 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.818 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.818 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.818 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.818 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.818 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.818 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.818 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.819 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.819 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.819 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.819 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.819 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.819 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.819 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.819 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.820 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.820 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.820 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.820 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.820 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.820 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.820 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.820 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.821 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.822 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.822 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.822 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.822 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.822 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.822 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.822 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.822 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.823 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.823 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.823 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.823 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.823 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.823 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.823 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.823 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.824 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.824 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.824 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.824 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.824 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.824 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.824 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.824 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.825 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.825 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.825 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.825 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.825 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.825 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.825 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.825 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.826 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.826 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.826 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "k8sevents", 2024-04-26T21:52:23.826 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.826 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.826 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.826 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_event_retention_days": { 2024-04-26T21:52:23.826 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ceph_event_retention_days", 2024-04-26T21:52:23.827 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.827 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.827 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.827 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "7", 2024-04-26T21:52:23.827 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.827 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.827 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.827 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T21:52:23.827 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.828 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.828 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.828 INFO:teuthology.orchestra.run.smithi001.stdout: "config_check_secs": { 2024-04-26T21:52:23.828 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "config_check_secs", 2024-04-26T21:52:23.828 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.828 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.828 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.828 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:23.829 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "10", 2024-04-26T21:52:23.829 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.829 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.829 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T21:52:23.829 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.829 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.829 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.829 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.829 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.830 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.830 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.830 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.830 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.830 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.830 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.830 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.830 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.831 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.831 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.831 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.831 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.831 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.831 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.831 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.831 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.831 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.832 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.832 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.832 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.832 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.832 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.832 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.832 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.832 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.832 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.833 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.833 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.833 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.833 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.833 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.833 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.833 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.834 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.834 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.834 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.834 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.834 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.834 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.834 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.834 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.835 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.835 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.835 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.835 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.835 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.835 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.835 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.835 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.835 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.836 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.837 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.837 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.837 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.837 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.837 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.837 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.837 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.837 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.838 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.838 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "localpool", 2024-04-26T21:52:23.838 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.838 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.838 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.838 INFO:teuthology.orchestra.run.smithi001.stdout: "failure_domain": { 2024-04-26T21:52:23.839 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "failure_domain", 2024-04-26T21:52:23.839 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.839 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.839 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.839 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "host", 2024-04-26T21:52:23.839 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.839 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.839 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.839 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "failure domain for any created local pool", 2024-04-26T21:52:23.840 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T21:52:23.840 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.840 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.840 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.840 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.840 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.840 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.840 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.841 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.841 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.841 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.841 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.841 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.841 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.841 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.841 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.841 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.842 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.842 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.842 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.842 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.842 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.842 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.842 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.842 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.843 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.844 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.845 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.845 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.845 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.845 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.845 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.845 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.845 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.845 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.845 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.846 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.846 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.846 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.846 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.846 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.846 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.846 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.847 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.847 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.847 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.847 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.847 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.847 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.847 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.847 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.848 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.848 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.848 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.848 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.848 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.848 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": { 2024-04-26T21:52:23.848 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "min_size", 2024-04-26T21:52:23.848 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.849 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.849 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.849 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.849 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.849 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.849 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.849 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "default min_size for any created local pool", 2024-04-26T21:52:23.849 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T21:52:23.849 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.850 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.850 INFO:teuthology.orchestra.run.smithi001.stdout: "num_rep": { 2024-04-26T21:52:23.850 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "num_rep", 2024-04-26T21:52:23.850 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.850 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.850 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.850 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "3", 2024-04-26T21:52:23.851 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.851 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.851 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.851 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "default replica count for any created local pool", 2024-04-26T21:52:23.851 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.851 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.851 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.851 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": { 2024-04-26T21:52:23.852 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pg_num", 2024-04-26T21:52:23.852 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.852 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.852 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.852 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "128", 2024-04-26T21:52:23.852 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.852 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.852 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.853 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T21:52:23.853 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.853 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.853 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.853 INFO:teuthology.orchestra.run.smithi001.stdout: "prefix": { 2024-04-26T21:52:23.853 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "prefix", 2024-04-26T21:52:23.853 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.853 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.854 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.854 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.854 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.854 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.854 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.854 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "name prefix for any created local pool", 2024-04-26T21:52:23.854 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.854 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.854 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.855 INFO:teuthology.orchestra.run.smithi001.stdout: "subtree": { 2024-04-26T21:52:23.855 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "subtree", 2024-04-26T21:52:23.855 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.855 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.855 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.855 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "rack", 2024-04-26T21:52:23.855 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.856 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.856 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.856 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T21:52:23.856 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T21:52:23.856 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.856 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.856 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.856 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.857 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mds_autoscaler", 2024-04-26T21:52:23.857 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.857 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.857 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.857 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.857 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.857 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.857 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.857 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.857 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.858 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.858 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.858 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.858 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.858 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.858 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.858 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.858 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.859 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.860 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.860 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.860 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.860 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.860 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.860 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.860 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.860 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.860 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.861 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.861 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.861 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.861 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.861 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.861 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.861 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.861 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.863 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.863 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.863 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.863 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.863 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.863 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.863 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.863 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.864 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.865 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.865 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mirroring", 2024-04-26T21:52:23.865 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.865 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.865 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.865 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.866 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.866 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.866 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.866 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.866 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.866 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.866 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.866 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.867 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.867 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.867 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.867 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.867 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.867 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.867 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.867 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.867 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.868 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.868 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.868 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.868 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.868 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.868 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.868 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.869 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.869 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.869 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.869 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.869 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.869 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.869 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.869 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.870 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.870 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.870 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.870 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.870 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.870 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.870 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.870 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.871 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.871 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.871 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.871 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.871 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.871 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.871 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.871 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.871 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.872 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.872 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.872 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.872 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.872 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.872 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.872 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.872 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.873 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.873 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.873 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.873 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.873 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.873 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.873 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.873 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.873 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.874 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.874 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.874 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "nfs", 2024-04-26T21:52:23.874 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.875 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.875 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.875 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.875 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.876 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.876 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.876 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.876 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.876 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.876 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.877 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.877 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.877 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.877 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.877 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.877 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.877 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.877 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.878 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.879 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.880 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.880 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.880 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.880 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.880 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.881 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.881 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.881 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.881 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.882 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.882 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.882 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.883 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.883 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.883 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.883 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.884 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.884 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.884 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.884 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.885 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.885 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.885 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.886 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.886 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.886 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.886 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.887 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.887 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.887 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.887 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.888 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.888 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.888 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.889 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.889 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.890 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.890 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "orchestrator", 2024-04-26T21:52:23.890 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.890 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.891 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.891 INFO:teuthology.orchestra.run.smithi001.stdout: "fail_fs": { 2024-04-26T21:52:23.891 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "fail_fs", 2024-04-26T21:52:23.891 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.892 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.892 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.892 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.892 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.892 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.893 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.893 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T21:52:23.893 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.893 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.894 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.894 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.894 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.895 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.895 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.895 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.895 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.895 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.896 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.896 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.897 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.897 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.897 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.897 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.897 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.898 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.898 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.898 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.898 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.899 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.899 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.899 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.899 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.900 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.900 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.900 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.900 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.901 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.901 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.901 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.901 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.901 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.902 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.902 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.902 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.902 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.903 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.903 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.903 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.903 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.904 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.904 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.904 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.904 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.905 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.905 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.905 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.905 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.905 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.906 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.906 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.906 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.907 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.907 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.907 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.907 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.908 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.908 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.908 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.908 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.909 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.909 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.909 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.909 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.909 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.910 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.910 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.910 INFO:teuthology.orchestra.run.smithi001.stdout: "orchestrator": { 2024-04-26T21:52:23.910 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "orchestrator", 2024-04-26T21:52:23.910 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.911 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.911 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.911 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.911 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.911 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.912 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.912 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm", 2024-04-26T21:52:23.912 INFO:teuthology.orchestra.run.smithi001.stdout: "rook", 2024-04-26T21:52:23.912 INFO:teuthology.orchestra.run.smithi001.stdout: "test_orchestrator" 2024-04-26T21:52:23.912 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.913 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Orchestrator backend", 2024-04-26T21:52:23.913 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.913 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.913 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.915 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "osd_perf_query", 2024-04-26T21:52:23.915 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.915 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.915 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.915 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.916 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.916 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.916 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.916 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.916 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.916 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.916 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.916 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.916 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.917 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.917 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.917 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.917 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.917 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.917 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.917 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.917 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.917 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.918 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.918 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.918 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.918 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.918 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.918 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.918 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.919 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.919 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.919 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.919 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.919 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.919 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.919 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.919 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.920 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.920 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.920 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.920 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.920 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.920 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.920 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.920 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.920 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.921 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.921 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.921 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.921 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.921 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.921 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.921 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.921 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.922 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.922 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.922 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.922 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.922 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.922 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.922 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.922 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.923 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.923 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.923 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.923 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.923 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.923 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.923 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.923 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "osd_support", 2024-04-26T21:52:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.924 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.925 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.925 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.925 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.925 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.925 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.925 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.925 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.925 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.926 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.926 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.926 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.926 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.926 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.926 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.926 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.926 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.927 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.927 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.927 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.927 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.927 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.927 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.927 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.927 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.928 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.928 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.928 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.928 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.928 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.928 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.928 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.928 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.929 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.929 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.929 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.929 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.929 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.929 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.929 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.929 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.929 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.930 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.930 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.930 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.930 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.930 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.930 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.930 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.930 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.931 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.931 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.931 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.931 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.931 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.931 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.931 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.931 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.932 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.932 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.932 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.932 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.932 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.932 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.932 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.933 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pg_autoscaler", 2024-04-26T21:52:23.933 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.933 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.933 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.933 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.933 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.934 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.934 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.934 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.934 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.934 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.934 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.934 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.935 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.935 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.935 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.935 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.935 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.935 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.935 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.935 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.935 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.936 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.936 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.936 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.936 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.936 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.936 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.936 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.937 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.937 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.937 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.937 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.937 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.937 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.937 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.937 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.938 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.938 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.938 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.938 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.938 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.938 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.938 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.938 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.939 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.939 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.939 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.939 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.939 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.939 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.939 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.939 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.939 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.940 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.940 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.940 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.940 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.940 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.940 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.940 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.940 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.940 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.941 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.941 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.941 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.941 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.941 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.941 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.941 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.941 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.942 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:23.942 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:23.942 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.942 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.942 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.942 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:23.942 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.942 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.943 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.943 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.943 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.943 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.943 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.943 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.943 INFO:teuthology.orchestra.run.smithi001.stdout: "threshold": { 2024-04-26T21:52:23.943 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "threshold", 2024-04-26T21:52:23.944 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:23.944 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.944 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.944 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "3.0", 2024-04-26T21:52:23.944 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1.0", 2024-04-26T21:52:23.944 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.944 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.944 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "scaling threshold", 2024-04-26T21:52:23.945 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T21:52:23.945 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.945 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.945 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.945 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.945 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.945 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.945 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "progress", 2024-04-26T21:52:23.945 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.946 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.946 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.946 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_pg_recovery_event": { 2024-04-26T21:52:23.946 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "allow_pg_recovery_event", 2024-04-26T21:52:23.946 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.946 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.946 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.946 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.947 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.947 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.947 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.947 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T21:52:23.947 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.947 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.947 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.947 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.947 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": { 2024-04-26T21:52:23.948 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enabled", 2024-04-26T21:52:23.948 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.948 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.948 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.948 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.948 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.948 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.948 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.948 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.949 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.949 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.949 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.949 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.949 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.949 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.949 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.949 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.950 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.950 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.950 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.950 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.950 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.950 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.950 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.950 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.950 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.951 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.951 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.951 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.951 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.951 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.951 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.951 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.952 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.952 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.952 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.952 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.952 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.952 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.952 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.952 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.952 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.953 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.953 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.953 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.953 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.953 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.953 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.953 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.953 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.953 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.954 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.954 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.954 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.954 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.954 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.954 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.954 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.954 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.955 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.955 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.955 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.955 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.955 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.955 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.955 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.956 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.956 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.956 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.956 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.956 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.956 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.956 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.956 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.956 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.957 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.957 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.957 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.957 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.957 INFO:teuthology.orchestra.run.smithi001.stdout: "max_completed_events": { 2024-04-26T21:52:23.957 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_completed_events", 2024-04-26T21:52:23.957 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.957 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.958 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.958 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "50", 2024-04-26T21:52:23.958 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.958 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.958 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.958 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "number of past completed events to remember", 2024-04-26T21:52:23.958 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.958 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.958 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.959 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:23.959 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:23.959 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:23.959 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.959 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.959 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:23.959 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.959 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.960 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.960 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how long the module is going to sleep", 2024-04-26T21:52:23.960 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.960 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.960 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.961 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "prometheus", 2024-04-26T21:52:23.961 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.961 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.961 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.961 INFO:teuthology.orchestra.run.smithi001.stdout: "cache": { 2024-04-26T21:52:23.961 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cache", 2024-04-26T21:52:23.961 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.961 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.962 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.962 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.962 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.962 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.962 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.962 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.962 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.962 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.962 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.963 INFO:teuthology.orchestra.run.smithi001.stdout: "exclude_perf_counters": { 2024-04-26T21:52:23.963 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "exclude_perf_counters", 2024-04-26T21:52:23.963 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.963 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.963 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.963 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:23.963 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.963 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.964 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.964 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T21:52:23.964 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T21:52:23.964 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.964 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.964 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.964 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.964 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.965 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.965 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.965 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.965 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.965 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.965 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.965 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.965 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.966 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.967 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.967 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.967 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.967 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.967 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.967 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.967 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.967 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.968 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.968 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.968 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.968 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.968 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.968 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.968 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.968 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.969 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.971 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.971 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.971 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.971 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.971 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.971 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.971 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.971 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.971 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.972 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.972 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.972 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.972 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.972 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_stats_pools": { 2024-04-26T21:52:23.972 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rbd_stats_pools", 2024-04-26T21:52:23.972 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.972 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.973 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.973 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.973 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.973 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.973 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.973 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.973 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.973 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.973 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T21:52:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T21:52:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "300", 2024-04-26T21:52:23.974 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: "scrape_interval": { 2024-04-26T21:52:23.975 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "scrape_interval", 2024-04-26T21:52:23.976 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:23.976 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.976 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.976 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "15.0", 2024-04-26T21:52:23.976 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.976 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.976 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.976 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.977 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.977 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.977 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.977 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.977 INFO:teuthology.orchestra.run.smithi001.stdout: "server_addr": { 2024-04-26T21:52:23.977 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_addr", 2024-04-26T21:52:23.977 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.977 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.977 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.978 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "::", 2024-04-26T21:52:23.978 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.978 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.978 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.978 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T21:52:23.978 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.978 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.978 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.979 INFO:teuthology.orchestra.run.smithi001.stdout: "server_port": { 2024-04-26T21:52:23.979 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_port", 2024-04-26T21:52:23.979 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.979 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.979 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.979 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "9283", 2024-04-26T21:52:23.979 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.979 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.980 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.980 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T21:52:23.980 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.980 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.980 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.980 INFO:teuthology.orchestra.run.smithi001.stdout: "stale_cache_strategy": { 2024-04-26T21:52:23.980 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "stale_cache_strategy", 2024-04-26T21:52:23.980 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.981 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.981 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.981 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "log", 2024-04-26T21:52:23.981 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.981 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.981 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.981 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.981 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.981 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.982 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.982 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_behaviour": { 2024-04-26T21:52:23.982 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_behaviour", 2024-04-26T21:52:23.982 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.982 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.982 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.982 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "default", 2024-04-26T21:52:23.982 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.983 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.983 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.983 INFO:teuthology.orchestra.run.smithi001.stdout: "default", 2024-04-26T21:52:23.983 INFO:teuthology.orchestra.run.smithi001.stdout: "error" 2024-04-26T21:52:23.983 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.983 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.983 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.983 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.984 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.984 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.984 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_error_status_code": { 2024-04-26T21:52:23.984 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_error_status_code", 2024-04-26T21:52:23.984 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.984 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.984 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.984 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "500", 2024-04-26T21:52:23.984 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "400", 2024-04-26T21:52:23.984 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "599", 2024-04-26T21:52:23.985 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.985 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.985 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.985 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.985 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.985 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.986 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:23.986 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rbd_support", 2024-04-26T21:52:23.986 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:23.986 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:23.986 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:23.986 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:23.986 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:23.986 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.986 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.987 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.987 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.987 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.987 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.987 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.987 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.987 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.987 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.988 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.988 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.988 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.988 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.988 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.988 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.988 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.988 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.989 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.989 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:23.989 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:23.989 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.989 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.989 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.989 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.989 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.989 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.990 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.990 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.990 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.990 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.990 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.990 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:23.990 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:23.990 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.991 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.991 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.991 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:23.991 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.991 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.991 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:23.991 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:23.991 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:23.991 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:23.992 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:23.992 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:23.992 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:23.992 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:23.992 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.992 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.992 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.992 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.993 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:23.993 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:23.993 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:23.993 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.993 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:23.993 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:23.993 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.993 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.993 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.994 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.994 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.994 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.994 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.994 INFO:teuthology.orchestra.run.smithi001.stdout: "max_concurrent_snap_create": { 2024-04-26T21:52:23.994 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_concurrent_snap_create", 2024-04-26T21:52:23.994 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:23.994 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.995 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.995 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:23.995 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.995 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.995 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.995 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.995 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.995 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "mirror_snapshot_schedule": { 2024-04-26T21:52:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T21:52:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "trash_purge_schedule": { 2024-04-26T21:52:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "trash_purge_schedule", 2024-04-26T21:52:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "restful", 2024-04-26T21:52:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "enable_auth": { 2024-04-26T21:52:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enable_auth", 2024-04-26T21:52:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "key_file": { 2024-04-26T21:52:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "key_file", 2024-04-26T21:52:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.009 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.009 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.009 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.009 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.009 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.009 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.009 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.009 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.009 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.010 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.010 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.010 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.010 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.010 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.010 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.010 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.010 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.011 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.011 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.011 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.011 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.011 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.011 INFO:teuthology.orchestra.run.smithi001.stdout: "server_addr": { 2024-04-26T21:52:24.011 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_addr", 2024-04-26T21:52:24.011 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.012 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.012 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.012 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.012 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.012 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.012 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.012 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.012 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.012 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.013 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.013 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.013 INFO:teuthology.orchestra.run.smithi001.stdout: "server_port": { 2024-04-26T21:52:24.013 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_port", 2024-04-26T21:52:24.013 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.013 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.013 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.013 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.013 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.014 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.014 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.014 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.014 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.014 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.014 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.014 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.015 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rgw", 2024-04-26T21:52:24.015 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.015 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.015 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.015 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.015 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.015 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.016 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.016 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.016 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.016 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.016 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.016 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.016 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.016 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.016 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.017 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.017 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.017 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.017 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.017 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.017 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.017 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.017 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.018 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.018 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.018 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.018 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.018 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.018 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.018 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.018 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.018 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.019 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.019 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.019 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.019 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.019 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.019 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.019 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.020 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.020 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.020 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.020 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.020 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.020 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.020 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.020 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.020 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.021 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.021 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.021 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.021 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.021 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.021 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.021 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.022 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.023 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.023 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.023 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.023 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.023 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.024 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rook", 2024-04-26T21:52:24.024 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.024 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.024 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.024 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.024 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.024 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.025 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.025 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.025 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.025 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.025 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.025 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.025 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.025 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.025 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.026 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.026 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.026 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.026 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.026 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.026 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.026 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.027 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.027 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.027 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.027 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.027 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.027 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.027 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.028 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.028 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.028 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.028 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.028 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.028 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.028 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.028 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.029 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.029 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.029 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.029 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.029 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.029 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.029 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.029 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.030 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.030 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.030 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.030 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.030 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.030 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.030 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.030 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.030 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.031 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.031 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.031 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.031 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.031 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.031 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.031 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.031 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.032 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.032 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.032 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.032 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.032 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.032 INFO:teuthology.orchestra.run.smithi001.stdout: "storage_class": { 2024-04-26T21:52:24.032 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "storage_class", 2024-04-26T21:52:24.032 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.033 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.033 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.033 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "local", 2024-04-26T21:52:24.033 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.033 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.033 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.033 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T21:52:24.033 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.033 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.034 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.034 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "selftest", 2024-04-26T21:52:24.034 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.035 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.035 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.035 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.035 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.035 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.035 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.035 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.036 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.036 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.036 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.036 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.036 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.036 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.036 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.036 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.037 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.037 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.037 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.037 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.037 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.037 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.037 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.037 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.037 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.038 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.038 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.038 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.038 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.038 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.038 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.038 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.038 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.039 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.039 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.039 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.039 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.039 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.039 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.039 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.039 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.040 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.040 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.040 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.040 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.040 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.040 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.040 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.040 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.040 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.041 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.041 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.041 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.041 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.041 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.041 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.041 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.041 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.042 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.042 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.042 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.042 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.042 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.042 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.042 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.042 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.043 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.043 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.043 INFO:teuthology.orchestra.run.smithi001.stdout: "roption1": { 2024-04-26T21:52:24.043 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "roption1", 2024-04-26T21:52:24.043 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.043 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.043 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.044 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.044 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.044 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.044 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.044 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.044 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.044 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.044 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.045 INFO:teuthology.orchestra.run.smithi001.stdout: "roption2": { 2024-04-26T21:52:24.045 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "roption2", 2024-04-26T21:52:24.045 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.045 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.045 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.045 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "xyz", 2024-04-26T21:52:24.045 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.045 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.046 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.046 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.046 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.046 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.046 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.046 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption1": { 2024-04-26T21:52:24.046 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption1", 2024-04-26T21:52:24.046 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.047 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.047 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.047 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.047 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.047 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.047 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.047 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.047 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.048 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.048 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.048 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption2": { 2024-04-26T21:52:24.048 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption2", 2024-04-26T21:52:24.048 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.048 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.048 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.048 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.049 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.049 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.049 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.049 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.049 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.049 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.049 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.049 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption3": { 2024-04-26T21:52:24.050 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption3", 2024-04-26T21:52:24.050 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.050 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.050 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.050 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.050 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.050 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.050 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.051 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.051 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.051 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.051 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.051 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption4": { 2024-04-26T21:52:24.051 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption4", 2024-04-26T21:52:24.051 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.051 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.052 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.052 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.052 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.052 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.052 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.052 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.052 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.052 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.052 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.053 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.053 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption5": { 2024-04-26T21:52:24.053 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption5", 2024-04-26T21:52:24.053 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.053 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.053 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.053 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.053 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.053 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.054 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.054 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.054 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.054 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.054 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.054 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption6": { 2024-04-26T21:52:24.054 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption6", 2024-04-26T21:52:24.054 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.055 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.055 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.055 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.055 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.055 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.055 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.055 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.055 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.055 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.056 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.056 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.056 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption7": { 2024-04-26T21:52:24.056 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption7", 2024-04-26T21:52:24.056 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.056 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.056 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.056 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.056 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:24.057 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "42", 2024-04-26T21:52:24.057 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.057 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.057 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.057 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.057 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.057 INFO:teuthology.orchestra.run.smithi001.stdout: "testkey": { 2024-04-26T21:52:24.057 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "testkey", 2024-04-26T21:52:24.058 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.058 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.058 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.058 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.058 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.058 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.058 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.058 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.058 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.059 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.059 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.059 INFO:teuthology.orchestra.run.smithi001.stdout: "testlkey": { 2024-04-26T21:52:24.059 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "testlkey", 2024-04-26T21:52:24.059 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.059 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.059 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.059 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.060 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.060 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.060 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.060 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.060 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.060 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.060 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.060 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.060 INFO:teuthology.orchestra.run.smithi001.stdout: "testnewline": { 2024-04-26T21:52:24.061 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "testnewline", 2024-04-26T21:52:24.061 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.061 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.061 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.061 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.061 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.061 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.061 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.061 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.062 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.062 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.062 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.062 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.062 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.062 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.062 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snap_schedule", 2024-04-26T21:52:24.063 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.063 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.063 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.063 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_m_granularity": { 2024-04-26T21:52:24.063 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "allow_m_granularity", 2024-04-26T21:52:24.063 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.063 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.063 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.063 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.064 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.064 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.064 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.064 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T21:52:24.064 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.064 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.064 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.064 INFO:teuthology.orchestra.run.smithi001.stdout: "dump_on_update": { 2024-04-26T21:52:24.065 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "dump_on_update", 2024-04-26T21:52:24.065 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.065 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.065 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.065 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.065 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.065 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.065 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.065 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "dump database to debug log on update", 2024-04-26T21:52:24.066 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.066 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.066 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.066 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.066 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.066 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.066 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.066 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.067 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.067 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.067 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.067 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.067 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.067 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.067 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.067 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.067 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.068 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.068 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.068 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.068 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.068 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.068 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.068 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.068 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.069 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.069 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.069 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.069 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.069 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.069 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.069 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.069 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.069 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.070 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.070 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.070 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.070 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.070 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.070 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.070 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.070 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.071 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.071 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.071 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.071 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.071 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.071 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.071 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.071 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.071 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.071 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.072 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.072 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.072 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.072 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.072 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.072 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.072 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.072 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.073 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "stats", 2024-04-26T21:52:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.074 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.075 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.075 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.075 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.075 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.075 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.075 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.075 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.075 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.075 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.076 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.076 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.076 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.076 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.076 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.076 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.076 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.076 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.076 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.077 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.077 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.077 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.077 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.077 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.077 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.077 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.077 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.078 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.078 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.078 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.078 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.078 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.078 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.078 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.078 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.078 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.079 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.079 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.079 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.079 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.079 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.079 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.079 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.079 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.079 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.080 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.080 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.080 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.080 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.080 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.080 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.080 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.080 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.080 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.081 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.081 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.081 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.081 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.081 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.081 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.081 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.081 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.082 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.082 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.082 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.082 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.082 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.082 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.082 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.082 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.082 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.083 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.083 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.083 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "status", 2024-04-26T21:52:24.083 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.083 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.083 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.083 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.083 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.083 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.084 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.084 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.084 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.084 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.084 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.084 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.084 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.084 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.084 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.085 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.085 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.085 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.085 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.085 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.085 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.085 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.085 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.086 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.086 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.086 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.086 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.086 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.086 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.086 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.086 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.086 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.087 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.087 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.087 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.087 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.087 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.087 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.087 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.087 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.088 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.088 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.088 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.088 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.088 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.088 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.088 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.089 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.089 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.089 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.089 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.089 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.089 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.089 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.089 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.090 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.090 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.090 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.090 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.090 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.090 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.090 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.090 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.090 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.091 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.091 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.091 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.091 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.091 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.091 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.091 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.091 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.091 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.092 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.092 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "telegraf", 2024-04-26T21:52:24.092 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.092 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.092 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.092 INFO:teuthology.orchestra.run.smithi001.stdout: "address": { 2024-04-26T21:52:24.092 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "address", 2024-04-26T21:52:24.093 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.093 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.093 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.093 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T21:52:24.093 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.093 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.093 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.093 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.093 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.094 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.094 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.094 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:24.094 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:24.094 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.094 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.094 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.094 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "15", 2024-04-26T21:52:24.095 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.095 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.095 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.095 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.095 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.095 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.095 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.096 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.096 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.096 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.096 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.096 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.096 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.096 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.096 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.097 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.097 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.097 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.097 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.097 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.097 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.097 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.097 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.097 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.098 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.098 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.098 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.098 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.098 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.098 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.098 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.098 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.099 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.099 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.099 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.099 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.099 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.099 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.099 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.099 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.100 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.100 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.100 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.100 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.100 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.100 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.100 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.100 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.100 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.101 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.101 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.101 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.101 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.101 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.101 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.101 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.101 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.101 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.102 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.102 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.102 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.102 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.102 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.102 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.102 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.102 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.103 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.103 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.103 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.103 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.103 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.103 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.103 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.103 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.103 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.104 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.104 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "telemetry", 2024-04-26T21:52:24.104 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.104 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.104 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.104 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_basic": { 2024-04-26T21:52:24.104 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_basic", 2024-04-26T21:52:24.104 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.105 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.105 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.105 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.105 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.105 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.105 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.105 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T21:52:24.105 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.105 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.106 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.106 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.106 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_crash": { 2024-04-26T21:52:24.106 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_crash", 2024-04-26T21:52:24.106 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.106 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.106 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.106 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.107 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.107 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.107 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.107 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T21:52:24.107 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.107 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.107 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.107 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_device": { 2024-04-26T21:52:24.108 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_device", 2024-04-26T21:52:24.108 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.108 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.108 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.108 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.108 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.108 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.108 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.108 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T21:52:24.109 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.109 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.109 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.109 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_ident": { 2024-04-26T21:52:24.109 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_ident", 2024-04-26T21:52:24.109 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.109 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.109 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.110 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.110 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.110 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.110 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.110 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T21:52:24.110 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.110 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.110 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.111 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_perf": { 2024-04-26T21:52:24.111 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_perf", 2024-04-26T21:52:24.111 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.111 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.111 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.111 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.111 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.111 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.112 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.112 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T21:52:24.112 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.112 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.112 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.112 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.112 INFO:teuthology.orchestra.run.smithi001.stdout: "contact": { 2024-04-26T21:52:24.112 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "contact", 2024-04-26T21:52:24.112 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.113 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.113 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.113 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.113 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.113 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.113 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.113 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.113 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.113 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.114 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.114 INFO:teuthology.orchestra.run.smithi001.stdout: "description": { 2024-04-26T21:52:24.114 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "description", 2024-04-26T21:52:24.114 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.114 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.114 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.114 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.115 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.115 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.115 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.115 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.115 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.115 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.115 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.115 INFO:teuthology.orchestra.run.smithi001.stdout: "device_url": { 2024-04-26T21:52:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "device_url", 2024-04-26T21:52:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T21:52:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.116 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.117 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.117 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.117 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.117 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.117 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": { 2024-04-26T21:52:24.117 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enabled", 2024-04-26T21:52:24.117 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.117 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.117 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.118 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.118 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.118 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.118 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.118 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.118 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.118 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.118 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.118 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.119 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:24.119 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:24.119 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.119 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.119 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.119 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "24", 2024-04-26T21:52:24.119 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "8", 2024-04-26T21:52:24.119 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.119 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.120 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.120 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.120 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.120 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.120 INFO:teuthology.orchestra.run.smithi001.stdout: "last_opt_revision": { 2024-04-26T21:52:24.120 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "last_opt_revision", 2024-04-26T21:52:24.120 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.120 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.121 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.121 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "1", 2024-04-26T21:52:24.121 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.121 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.121 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.121 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.121 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.121 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.122 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.122 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.122 INFO:teuthology.orchestra.run.smithi001.stdout: "leaderboard": { 2024-04-26T21:52:24.122 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "leaderboard", 2024-04-26T21:52:24.122 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.122 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.123 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.123 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.123 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.123 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.123 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.123 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.124 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.124 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.124 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.124 INFO:teuthology.orchestra.run.smithi001.stdout: "leaderboard_description": { 2024-04-26T21:52:24.125 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "leaderboard_description", 2024-04-26T21:52:24.125 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.125 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.125 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.125 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.125 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.126 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.126 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.126 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.126 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.126 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.127 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.127 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.127 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.127 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.128 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.128 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.128 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.128 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.128 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.128 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.128 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.129 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.129 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.129 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.129 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.129 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.129 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.130 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.130 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.130 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.130 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.130 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.131 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.131 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.131 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.131 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.131 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.131 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.131 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.131 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.131 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.132 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.132 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.132 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.132 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.132 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.132 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.133 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.133 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.133 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.133 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.133 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.133 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.134 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.134 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.134 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.134 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.134 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.134 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.135 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.135 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.135 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.135 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.135 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.136 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.136 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.136 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.136 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.137 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.137 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.137 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.137 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.137 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.137 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.137 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.137 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.137 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.138 INFO:teuthology.orchestra.run.smithi001.stdout: "organization": { 2024-04-26T21:52:24.138 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "organization", 2024-04-26T21:52:24.138 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.138 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.138 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.138 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.138 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.138 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.139 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.139 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.139 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.139 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.139 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.139 INFO:teuthology.orchestra.run.smithi001.stdout: "proxy": { 2024-04-26T21:52:24.139 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "proxy", 2024-04-26T21:52:24.140 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.140 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.140 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.140 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.140 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.140 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.140 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.140 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.140 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.141 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.141 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.141 INFO:teuthology.orchestra.run.smithi001.stdout: "url": { 2024-04-26T21:52:24.141 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "url", 2024-04-26T21:52:24.141 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.141 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.141 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.141 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T21:52:24.141 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.142 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.142 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.142 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.142 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.142 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.142 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.143 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "test_orchestrator", 2024-04-26T21:52:24.143 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.143 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.143 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.143 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.143 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.143 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.144 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.144 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.144 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.144 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.144 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.144 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.144 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.145 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.145 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.145 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.145 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.145 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.145 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.145 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.145 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.145 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.145 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.146 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.146 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.146 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.146 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.146 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.146 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.146 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.147 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.147 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.147 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.147 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.147 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.147 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.147 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.147 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.148 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.148 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.148 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.148 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.148 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.148 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.148 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.148 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.148 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.150 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.150 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.150 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.150 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.150 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.150 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.150 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.150 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.152 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "volumes", 2024-04-26T21:52:24.152 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.152 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.152 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.152 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.152 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.152 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.152 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.152 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.153 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.153 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.153 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.153 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.153 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.153 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.153 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.153 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.154 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.154 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.154 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.154 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.154 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.154 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.154 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.154 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.155 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.155 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.155 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.155 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.155 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.155 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.155 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.155 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.156 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.156 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.156 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.156 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.156 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.156 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.156 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.156 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.156 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.156 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.157 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.157 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.157 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.157 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.157 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.157 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.157 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.157 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.157 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.157 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.158 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.158 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.158 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.158 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.158 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.158 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.158 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.159 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.159 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.159 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.159 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.159 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.159 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.159 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.159 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.159 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.160 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.160 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.160 INFO:teuthology.orchestra.run.smithi001.stdout: "max_concurrent_clones": { 2024-04-26T21:52:24.160 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_concurrent_clones", 2024-04-26T21:52:24.160 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.160 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.160 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.160 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "4", 2024-04-26T21:52:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T21:52:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "periodic_async_work": { 2024-04-26T21:52:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "periodic_async_work", 2024-04-26T21:52:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Periodically check for async work", 2024-04-26T21:52:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "snapshot_clone_delay": { 2024-04-26T21:52:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snapshot_clone_delay", 2024-04-26T21:52:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T21:52:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "snapshot_clone_no_wait": { 2024-04-26T21:52:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T21:52:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T21:52:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix", 2024-04-26T21:52:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "discovery_interval": { 2024-04-26T21:52:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "discovery_interval", 2024-04-26T21:52:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "100", 2024-04-26T21:52:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "identifier": { 2024-04-26T21:52:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "identifier", 2024-04-26T21:52:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.173 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.173 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.173 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.173 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.173 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.173 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.173 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.173 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.174 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.174 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.174 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.174 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.174 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.174 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.174 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.174 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.175 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.175 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.175 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.175 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.175 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.175 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.175 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.175 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.175 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.176 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.176 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.176 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.176 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.176 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.176 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.176 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.176 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.177 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.177 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.177 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.177 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.177 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.177 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.177 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.177 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.177 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.178 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.178 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.178 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.178 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.178 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.178 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.178 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.178 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.179 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.179 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.179 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.179 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.179 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.179 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.179 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.179 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.180 INFO:teuthology.orchestra.run.smithi001.stdout: "zabbix_host": { 2024-04-26T21:52:24.180 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix_host", 2024-04-26T21:52:24.180 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.180 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.180 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.180 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.180 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.180 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.181 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.181 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.181 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.181 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.181 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.181 INFO:teuthology.orchestra.run.smithi001.stdout: "zabbix_port": { 2024-04-26T21:52:24.181 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix_port", 2024-04-26T21:52:24.181 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.182 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.182 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.182 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10051", 2024-04-26T21:52:24.182 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.182 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.182 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.182 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.182 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.183 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.183 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.183 INFO:teuthology.orchestra.run.smithi001.stdout: "zabbix_sender": { 2024-04-26T21:52:24.183 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix_sender", 2024-04-26T21:52:24.183 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.183 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.183 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.184 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T21:52:24.184 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.184 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.184 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.184 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.184 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.184 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.184 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.184 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.185 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.185 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.185 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:24.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.185 INFO:teuthology.orchestra.run.smithi001.stdout: "gid": 25424, 2024-04-26T21:52:24.185 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "y", 2024-04-26T21:52:24.185 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr_features": 4540701547738038271, 2024-04-26T21:52:24.185 INFO:teuthology.orchestra.run.smithi001.stdout: "available_modules": [ 2024-04-26T21:52:24.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.186 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "alerts", 2024-04-26T21:52:24.186 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.186 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.186 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.186 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:24.186 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:24.186 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.186 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.187 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.187 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:24.187 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.187 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.187 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.187 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T21:52:24.187 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.187 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.188 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.188 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.188 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.188 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.188 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.188 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.189 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.189 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.189 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.189 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.189 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.189 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.189 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.190 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.190 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.190 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.190 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.190 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.190 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.190 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.191 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.191 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.191 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.191 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.191 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.191 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.191 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.191 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.191 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.192 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.192 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.192 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.192 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.192 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.192 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.192 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.192 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.193 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.193 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.193 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.193 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.193 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.193 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.193 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.193 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.193 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.194 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.194 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.194 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.194 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.194 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.194 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.194 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.194 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.195 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.195 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.195 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.195 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.195 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.195 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.195 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.195 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.195 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.196 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.196 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.196 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.196 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_destination": { 2024-04-26T21:52:24.196 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_destination", 2024-04-26T21:52:24.196 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.196 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.197 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.197 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.197 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.197 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.197 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.197 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Email address to send alerts to", 2024-04-26T21:52:24.197 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.197 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.197 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.198 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_from_name": { 2024-04-26T21:52:24.198 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_from_name", 2024-04-26T21:52:24.198 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.198 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.198 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.198 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "Ceph", 2024-04-26T21:52:24.198 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.199 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.199 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.199 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Email From: name", 2024-04-26T21:52:24.199 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.199 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.199 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.199 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_host": { 2024-04-26T21:52:24.199 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_host", 2024-04-26T21:52:24.200 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.200 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.200 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.200 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.200 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.200 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.200 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.200 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SMTP server", 2024-04-26T21:52:24.200 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.201 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.201 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.201 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_password": { 2024-04-26T21:52:24.201 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_password", 2024-04-26T21:52:24.201 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.201 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.201 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.201 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.202 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.202 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.202 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.202 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Password to authenticate with", 2024-04-26T21:52:24.202 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.202 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.202 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.203 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_port": { 2024-04-26T21:52:24.203 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_port", 2024-04-26T21:52:24.203 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.203 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.203 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.203 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "465", 2024-04-26T21:52:24.203 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.203 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.204 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.204 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SMTP port", 2024-04-26T21:52:24.204 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.204 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.204 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.204 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_sender": { 2024-04-26T21:52:24.204 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_sender", 2024-04-26T21:52:24.204 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SMTP envelope sender", 2024-04-26T21:52:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.205 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.206 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.206 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_ssl": { 2024-04-26T21:52:24.206 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_ssl", 2024-04-26T21:52:24.206 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.206 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.206 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.206 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.206 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.207 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.207 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.207 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T21:52:24.207 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.207 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.207 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.207 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_user": { 2024-04-26T21:52:24.207 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_user", 2024-04-26T21:52:24.208 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.208 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.208 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.208 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.208 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.208 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.208 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.208 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "User to authenticate as", 2024-04-26T21:52:24.209 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.209 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.209 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.209 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.209 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.209 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "balancer", 2024-04-26T21:52:24.209 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.210 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.210 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.210 INFO:teuthology.orchestra.run.smithi001.stdout: "active": { 2024-04-26T21:52:24.210 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "active", 2024-04-26T21:52:24.210 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.210 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.210 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.210 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.210 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.211 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.211 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.211 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T21:52:24.211 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.211 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.211 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.211 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.211 INFO:teuthology.orchestra.run.smithi001.stdout: "begin_time": { 2024-04-26T21:52:24.211 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "begin_time", 2024-04-26T21:52:24.212 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.212 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.212 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.212 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0000", 2024-04-26T21:52:24.212 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.212 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.212 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.212 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T21:52:24.212 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T21:52:24.213 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.213 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.213 INFO:teuthology.orchestra.run.smithi001.stdout: "begin_weekday": { 2024-04-26T21:52:24.213 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "begin_weekday", 2024-04-26T21:52:24.213 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:24.213 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.213 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.213 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:24.214 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "0", 2024-04-26T21:52:24.214 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "6", 2024-04-26T21:52:24.214 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.214 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T21:52:24.214 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T21:52:24.214 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.214 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.214 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_compat_max_iterations": { 2024-04-26T21:52:24.215 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crush_compat_max_iterations", 2024-04-26T21:52:24.215 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:24.215 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.215 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.215 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "25", 2024-04-26T21:52:24.215 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:24.215 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "250", 2024-04-26T21:52:24.215 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.215 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T21:52:24.216 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.216 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.216 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.216 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.216 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_compat_metrics": { 2024-04-26T21:52:24.216 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crush_compat_metrics", 2024-04-26T21:52:24.216 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.216 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.217 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.217 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T21:52:24.217 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.217 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.217 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.217 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T21:52:24.217 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T21:52:24.217 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.217 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.218 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.218 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_compat_step": { 2024-04-26T21:52:24.218 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crush_compat_step", 2024-04-26T21:52:24.218 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.218 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.218 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.218 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0.5", 2024-04-26T21:52:24.218 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "0.001", 2024-04-26T21:52:24.218 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "0.999", 2024-04-26T21:52:24.219 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.219 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "aggressiveness of optimization", 2024-04-26T21:52:24.219 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T21:52:24.219 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.219 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.219 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.219 INFO:teuthology.orchestra.run.smithi001.stdout: "end_time": { 2024-04-26T21:52:24.219 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "end_time", 2024-04-26T21:52:24.219 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.220 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.220 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.220 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "2359", 2024-04-26T21:52:24.220 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.220 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.220 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.220 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T21:52:24.220 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T21:52:24.220 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.221 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.221 INFO:teuthology.orchestra.run.smithi001.stdout: "end_weekday": { 2024-04-26T21:52:24.221 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "end_weekday", 2024-04-26T21:52:24.221 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:24.221 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.221 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.221 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:24.222 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "0", 2024-04-26T21:52:24.222 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "6", 2024-04-26T21:52:24.222 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.222 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T21:52:24.222 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T21:52:24.222 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.222 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.222 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.223 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.223 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.223 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.223 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.223 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.223 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.223 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.223 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.223 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.224 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.224 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.224 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.224 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.224 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.224 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.224 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.224 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.225 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.225 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.225 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.225 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.225 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.225 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.225 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.226 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.226 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.226 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.226 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.226 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.226 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.226 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.226 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.227 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.227 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.227 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.227 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.227 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.227 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.227 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.227 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.228 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.228 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.228 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.228 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.228 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.228 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.228 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.229 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.229 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.229 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.229 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.229 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.229 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.229 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.230 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.230 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.230 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.230 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.230 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.230 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.230 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.230 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.230 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.231 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.231 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.231 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.231 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.231 INFO:teuthology.orchestra.run.smithi001.stdout: "min_score": { 2024-04-26T21:52:24.231 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "min_score", 2024-04-26T21:52:24.231 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.231 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.231 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.232 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:24.232 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.232 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.232 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.232 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T21:52:24.232 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.232 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.232 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.232 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.233 INFO:teuthology.orchestra.run.smithi001.stdout: "mode": { 2024-04-26T21:52:24.233 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mode", 2024-04-26T21:52:24.233 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.233 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.233 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.233 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "upmap", 2024-04-26T21:52:24.233 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.233 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.233 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.234 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-compat", 2024-04-26T21:52:24.234 INFO:teuthology.orchestra.run.smithi001.stdout: "none", 2024-04-26T21:52:24.234 INFO:teuthology.orchestra.run.smithi001.stdout: "read", 2024-04-26T21:52:24.234 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap", 2024-04-26T21:52:24.234 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap-read" 2024-04-26T21:52:24.234 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.234 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Balancer mode", 2024-04-26T21:52:24.234 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.234 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.235 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.235 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_ids": { 2024-04-26T21:52:24.235 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pool_ids", 2024-04-26T21:52:24.235 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.235 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.235 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.235 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.237 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:24.237 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:24.237 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.237 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.237 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.237 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:24.237 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.237 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.238 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.238 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T21:52:24.238 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.238 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.238 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.238 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap_max_deviation": { 2024-04-26T21:52:24.238 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "upmap_max_deviation", 2024-04-26T21:52:24.238 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.239 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.239 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.239 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:24.239 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:24.239 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.239 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.239 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T21:52:24.239 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T21:52:24.239 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.240 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.240 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap_max_optimizations": { 2024-04-26T21:52:24.240 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "upmap_max_optimizations", 2024-04-26T21:52:24.240 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:24.240 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.240 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.240 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:24.241 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.241 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.241 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.241 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T21:52:24.241 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.241 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.241 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.241 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.241 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.242 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.242 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.242 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephadm", 2024-04-26T21:52:24.242 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.242 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.242 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.242 INFO:teuthology.orchestra.run.smithi001.stdout: "agent_down_multiplier": { 2024-04-26T21:52:24.242 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "agent_down_multiplier", 2024-04-26T21:52:24.243 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.243 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.243 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.243 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "3.0", 2024-04-26T21:52:24.243 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.243 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.243 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.243 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T21:52:24.243 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.244 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.244 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.244 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.244 INFO:teuthology.orchestra.run.smithi001.stdout: "agent_refresh_rate": { 2024-04-26T21:52:24.244 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "agent_refresh_rate", 2024-04-26T21:52:24.244 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.244 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.245 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.245 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "20", 2024-04-26T21:52:24.245 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.245 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.245 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.245 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T21:52:24.245 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.245 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.245 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.246 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.246 INFO:teuthology.orchestra.run.smithi001.stdout: "agent_starting_port": { 2024-04-26T21:52:24.246 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "agent_starting_port", 2024-04-26T21:52:24.246 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.246 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.246 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.246 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "4721", 2024-04-26T21:52:24.246 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.247 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.247 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.247 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T21:52:24.247 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.247 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.247 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.247 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.247 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_ptrace": { 2024-04-26T21:52:24.247 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "allow_ptrace", 2024-04-26T21:52:24.248 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.248 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.248 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.248 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.248 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.248 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.248 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.248 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T21:52:24.249 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T21:52:24.249 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.249 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.249 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.249 INFO:teuthology.orchestra.run.smithi001.stdout: "autotune_interval": { 2024-04-26T21:52:24.249 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "autotune_interval", 2024-04-26T21:52:24.249 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.249 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.249 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.250 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:24.250 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.250 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.250 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.250 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T21:52:24.250 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.250 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.250 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.251 INFO:teuthology.orchestra.run.smithi001.stdout: "autotune_memory_target_ratio": { 2024-04-26T21:52:24.251 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T21:52:24.251 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.251 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.251 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.251 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0.7", 2024-04-26T21:52:24.251 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.251 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.251 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.252 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T21:52:24.252 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.252 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.252 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.252 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.252 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm_log_destination": { 2024-04-26T21:52:24.252 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephadm_log_destination", 2024-04-26T21:52:24.252 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.253 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.253 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.253 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.253 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.253 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.253 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.253 INFO:teuthology.orchestra.run.smithi001.stdout: "file", 2024-04-26T21:52:24.253 INFO:teuthology.orchestra.run.smithi001.stdout: "file,syslog", 2024-04-26T21:52:24.253 INFO:teuthology.orchestra.run.smithi001.stdout: "syslog" 2024-04-26T21:52:24.253 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.254 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T21:52:24.254 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.254 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.254 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.254 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.254 INFO:teuthology.orchestra.run.smithi001.stdout: "cgroups_split": { 2024-04-26T21:52:24.254 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cgroups_split", 2024-04-26T21:52:24.254 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.255 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.255 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.255 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.255 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.255 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.255 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.255 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T21:52:24.255 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.256 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.256 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.256 INFO:teuthology.orchestra.run.smithi001.stdout: "config_checks_enabled": { 2024-04-26T21:52:24.256 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "config_checks_enabled", 2024-04-26T21:52:24.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.256 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.256 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.256 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.257 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.257 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.257 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.257 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T21:52:24.257 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.257 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.257 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.257 INFO:teuthology.orchestra.run.smithi001.stdout: "config_dashboard": { 2024-04-26T21:52:24.257 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "config_dashboard", 2024-04-26T21:52:24.258 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.258 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.258 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.258 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.258 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.258 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.258 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.258 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T21:52:24.259 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.259 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.259 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.259 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_alertmanager": { 2024-04-26T21:52:24.259 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_alertmanager", 2024-04-26T21:52:24.259 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.259 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.259 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.260 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T21:52:24.260 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.260 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.260 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.260 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:24.260 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.260 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.260 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.260 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.261 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_base": { 2024-04-26T21:52:24.261 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_base", 2024-04-26T21:52:24.261 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.261 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.261 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.261 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T21:52:24.261 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.261 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.261 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.262 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Container image name, without the tag", 2024-04-26T21:52:24.262 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.262 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.262 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.262 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.262 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_elasticsearch": { 2024-04-26T21:52:24.262 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_elasticsearch", 2024-04-26T21:52:24.262 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.263 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.263 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.263 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T21:52:24.263 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.263 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.263 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.263 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "elasticsearch container image", 2024-04-26T21:52:24.263 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.263 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.264 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.264 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.264 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_grafana": { 2024-04-26T21:52:24.264 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_grafana", 2024-04-26T21:52:24.264 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.264 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.264 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.264 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T21:52:24.265 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.265 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.265 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.265 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:24.265 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.265 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.265 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.265 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_haproxy": { 2024-04-26T21:52:24.266 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_haproxy", 2024-04-26T21:52:24.266 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.266 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.266 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.266 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T21:52:24.266 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.266 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.266 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.267 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "HAproxy container image", 2024-04-26T21:52:24.267 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.267 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.267 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.267 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.267 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_jaeger_agent": { 2024-04-26T21:52:24.267 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_jaeger_agent", 2024-04-26T21:52:24.267 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.267 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.268 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.268 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T21:52:24.268 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.268 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.268 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.268 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Jaeger agent container image", 2024-04-26T21:52:24.268 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.268 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.269 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.269 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_jaeger_collector": { 2024-04-26T21:52:24.269 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_jaeger_collector", 2024-04-26T21:52:24.269 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.269 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.269 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.269 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T21:52:24.269 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.270 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.270 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.270 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Jaeger collector container image", 2024-04-26T21:52:24.270 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.270 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.270 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.270 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_jaeger_query": { 2024-04-26T21:52:24.270 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_jaeger_query", 2024-04-26T21:52:24.271 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.271 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.271 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.271 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T21:52:24.271 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.271 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.271 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.271 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Jaeger query container image", 2024-04-26T21:52:24.271 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.272 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.272 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.272 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_keepalived": { 2024-04-26T21:52:24.272 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_keepalived", 2024-04-26T21:52:24.272 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.272 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.272 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.272 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T21:52:24.273 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.273 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.273 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.273 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Keepalived container image", 2024-04-26T21:52:24.273 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.273 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.273 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.273 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_loki": { 2024-04-26T21:52:24.274 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_loki", 2024-04-26T21:52:24.274 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.274 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.274 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.274 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T21:52:24.274 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.274 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.274 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.275 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Loki container image", 2024-04-26T21:52:24.275 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.275 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.275 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.275 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_node_exporter": { 2024-04-26T21:52:24.275 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_node_exporter", 2024-04-26T21:52:24.275 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.275 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.276 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.276 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T21:52:24.276 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.276 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.276 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.276 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:24.276 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.276 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.277 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.277 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_nvmeof": { 2024-04-26T21:52:24.277 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_nvmeof", 2024-04-26T21:52:24.277 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.277 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.277 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.277 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T21:52:24.277 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Nvme-of container image", 2024-04-26T21:52:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.278 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_prometheus": { 2024-04-26T21:52:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_prometheus", 2024-04-26T21:52:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T21:52:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.279 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_promtail": { 2024-04-26T21:52:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_promtail", 2024-04-26T21:52:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.280 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T21:52:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Promtail container image", 2024-04-26T21:52:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.282 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_samba": { 2024-04-26T21:52:24.282 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_samba", 2024-04-26T21:52:24.282 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.282 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.282 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.282 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T21:52:24.282 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.283 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.283 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.283 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Samba/SMB container image", 2024-04-26T21:52:24.283 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.283 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.283 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.283 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_snmp_gateway": { 2024-04-26T21:52:24.283 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_snmp_gateway", 2024-04-26T21:52:24.284 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.284 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.284 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.284 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T21:52:24.284 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.284 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.284 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.284 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SNMP Gateway container image", 2024-04-26T21:52:24.284 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.285 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.285 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.285 INFO:teuthology.orchestra.run.smithi001.stdout: "container_init": { 2024-04-26T21:52:24.285 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_init", 2024-04-26T21:52:24.285 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.285 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.285 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.285 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.286 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.286 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.286 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.286 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T21:52:24.286 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.286 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.286 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.287 INFO:teuthology.orchestra.run.smithi001.stdout: "daemon_cache_timeout": { 2024-04-26T21:52:24.287 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "daemon_cache_timeout", 2024-04-26T21:52:24.287 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.287 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.287 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.287 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:24.287 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.287 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.287 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.288 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T21:52:24.288 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.288 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.288 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.288 INFO:teuthology.orchestra.run.smithi001.stdout: "default_cephadm_command_timeout": { 2024-04-26T21:52:24.288 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T21:52:24.288 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.289 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.289 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.289 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "900", 2024-04-26T21:52:24.289 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.289 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.289 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.289 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T21:52:24.289 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.290 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.290 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.290 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.290 INFO:teuthology.orchestra.run.smithi001.stdout: "default_registry": { 2024-04-26T21:52:24.290 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "default_registry", 2024-04-26T21:52:24.290 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.290 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.291 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.291 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io", 2024-04-26T21:52:24.291 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.291 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.291 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.291 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T21:52:24.291 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.291 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.291 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.292 INFO:teuthology.orchestra.run.smithi001.stdout: "device_cache_timeout": { 2024-04-26T21:52:24.292 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "device_cache_timeout", 2024-04-26T21:52:24.292 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.292 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.292 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.292 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "1800", 2024-04-26T21:52:24.292 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.292 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.293 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.293 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "seconds to cache device inventory", 2024-04-26T21:52:24.293 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.293 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.293 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.293 INFO:teuthology.orchestra.run.smithi001.stdout: "device_enhanced_scan": { 2024-04-26T21:52:24.293 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "device_enhanced_scan", 2024-04-26T21:52:24.294 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.294 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.294 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.294 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.294 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.294 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.294 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.294 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T21:52:24.295 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.295 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.295 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.295 INFO:teuthology.orchestra.run.smithi001.stdout: "facts_cache_timeout": { 2024-04-26T21:52:24.295 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "facts_cache_timeout", 2024-04-26T21:52:24.295 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.295 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.296 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.296 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:24.296 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.296 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.296 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.296 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "seconds to cache host facts data", 2024-04-26T21:52:24.296 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.296 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.297 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.297 INFO:teuthology.orchestra.run.smithi001.stdout: "grafana_dashboards_path": { 2024-04-26T21:52:24.297 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "grafana_dashboards_path", 2024-04-26T21:52:24.297 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.297 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.297 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.297 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T21:52:24.297 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.298 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.298 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.298 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T21:52:24.298 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.298 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.298 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.298 INFO:teuthology.orchestra.run.smithi001.stdout: "host_check_interval": { 2024-04-26T21:52:24.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "host_check_interval", 2024-04-26T21:52:24.299 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.299 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.299 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.299 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:24.299 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.299 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.299 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.299 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to perform a host check", 2024-04-26T21:52:24.300 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.300 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.300 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.300 INFO:teuthology.orchestra.run.smithi001.stdout: "hw_monitoring": { 2024-04-26T21:52:24.300 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "hw_monitoring", 2024-04-26T21:52:24.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.300 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.300 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.301 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.301 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.301 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.301 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.301 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T21:52:24.301 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.301 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.301 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.302 INFO:teuthology.orchestra.run.smithi001.stdout: "inventory_list_all": { 2024-04-26T21:52:24.302 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "inventory_list_all", 2024-04-26T21:52:24.302 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.302 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.302 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.302 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.302 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.303 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.303 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.303 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T21:52:24.303 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.303 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.306 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.307 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.307 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.307 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.307 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.307 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.307 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.307 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.308 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.308 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.308 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.308 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.308 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.308 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.308 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.308 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.308 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.309 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.309 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.309 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.309 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.309 INFO:teuthology.orchestra.run.smithi001.stdout: "log_refresh_metadata": { 2024-04-26T21:52:24.309 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_refresh_metadata", 2024-04-26T21:52:24.309 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.310 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.310 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.310 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.310 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.310 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.310 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.310 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T21:52:24.310 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.311 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.311 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.311 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.311 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.311 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.311 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.311 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.312 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.312 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.312 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.312 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.312 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T21:52:24.312 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.312 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.312 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.313 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.313 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.313 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.313 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.313 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.313 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.313 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.313 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.313 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.314 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.314 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.314 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.314 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.314 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.314 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.314 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.314 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.314 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.315 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.315 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.315 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.315 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.315 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.315 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.315 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.316 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.316 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.316 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.316 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.316 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.316 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.316 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.316 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.316 INFO:teuthology.orchestra.run.smithi001.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T21:52:24.317 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T21:52:24.317 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.317 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.317 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.317 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.317 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.317 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.317 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.317 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T21:52:24.318 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.318 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.318 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.318 INFO:teuthology.orchestra.run.smithi001.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T21:52:24.318 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T21:52:24.318 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.318 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.319 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.319 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "*", 2024-04-26T21:52:24.319 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.319 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.319 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.319 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T21:52:24.319 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.319 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.319 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.320 INFO:teuthology.orchestra.run.smithi001.stdout: "max_count_per_host": { 2024-04-26T21:52:24.320 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_count_per_host", 2024-04-26T21:52:24.320 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.320 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.320 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.320 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:24.320 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.320 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.321 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.321 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "max number of daemons per service per host", 2024-04-26T21:52:24.321 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.321 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.321 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.321 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd_draining_count": { 2024-04-26T21:52:24.321 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_osd_draining_count", 2024-04-26T21:52:24.321 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.321 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.322 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.322 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:24.322 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.322 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.322 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.322 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T21:52:24.322 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.323 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.323 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.323 INFO:teuthology.orchestra.run.smithi001.stdout: "migration_current": { 2024-04-26T21:52:24.323 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "migration_current", 2024-04-26T21:52:24.323 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.323 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.323 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.323 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.324 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.324 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.324 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.324 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "internal - do not modify", 2024-04-26T21:52:24.324 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.324 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.324 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi001.stdout: "mode": { 2024-04-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mode", 2024-04-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "root", 2024-04-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.325 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm-package", 2024-04-26T21:52:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "root" 2024-04-26T21:52:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T21:52:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "oob_default_addr": { 2024-04-26T21:52:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "oob_default_addr", 2024-04-26T21:52:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "169.254.1.1", 2024-04-26T21:52:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T21:52:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "prometheus_alerts_path": { 2024-04-26T21:52:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "prometheus_alerts_path", 2024-04-26T21:52:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T21:52:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T21:52:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_insecure": { 2024-04-26T21:52:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_insecure", 2024-04-26T21:52:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T21:52:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_password": { 2024-04-26T21:52:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_password", 2024-04-26T21:52:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T21:52:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_url": { 2024-04-26T21:52:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_url", 2024-04-26T21:52:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T21:52:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_username": { 2024-04-26T21:52:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_username", 2024-04-26T21:52:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T21:52:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "secure_monitoring_stack": { 2024-04-26T21:52:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "secure_monitoring_stack", 2024-04-26T21:52:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T21:52:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "service_discovery_port": { 2024-04-26T21:52:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "service_discovery_port", 2024-04-26T21:52:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8765", 2024-04-26T21:52:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "cephadm service discovery port", 2024-04-26T21:52:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: "ssh_config_file": { 2024-04-26T21:52:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssh_config_file", 2024-04-26T21:52:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.340 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T21:52:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: "use_agent": { 2024-04-26T21:52:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "use_agent", 2024-04-26T21:52:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.342 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T21:52:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: "use_repo_digest": { 2024-04-26T21:52:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "use_repo_digest", 2024-04-26T21:52:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.343 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T21:52:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.344 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_on_failed_host_check": { 2024-04-26T21:52:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_on_failed_host_check", 2024-04-26T21:52:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.345 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T21:52:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_on_stray_daemons": { 2024-04-26T21:52:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_on_stray_daemons", 2024-04-26T21:52:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.347 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T21:52:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_on_stray_hosts": { 2024-04-26T21:52:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_on_stray_hosts", 2024-04-26T21:52:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.348 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T21:52:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.349 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crash", 2024-04-26T21:52:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.350 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.351 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.352 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.353 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.353 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.353 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.353 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.353 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.353 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.353 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.353 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.353 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.354 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.354 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.354 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.354 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.354 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.354 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.354 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.355 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.355 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.355 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.355 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.355 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.355 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.355 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.355 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.355 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.356 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.356 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.356 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.356 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.356 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.356 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.356 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.356 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.356 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.357 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.357 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.357 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.357 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.357 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.357 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.357 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.358 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.358 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.358 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.358 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.358 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.358 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.358 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.358 INFO:teuthology.orchestra.run.smithi001.stdout: "retain_interval": { 2024-04-26T21:52:24.359 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "retain_interval", 2024-04-26T21:52:24.359 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.359 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.359 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.359 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "31536000", 2024-04-26T21:52:24.359 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.359 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.359 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.359 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T21:52:24.360 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.360 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.360 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.360 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_recent_interval": { 2024-04-26T21:52:24.360 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_recent_interval", 2024-04-26T21:52:24.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.360 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.360 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.361 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "1209600", 2024-04-26T21:52:24.361 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.361 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.361 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.361 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T21:52:24.361 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.361 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.361 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.361 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.362 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "dashboard", 2024-04-26T21:52:24.362 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.362 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.362 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.362 INFO:teuthology.orchestra.run.smithi001.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T21:52:24.362 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T21:52:24.363 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.363 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.363 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.363 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:24.363 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.363 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.363 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.363 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.364 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.364 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.364 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.364 INFO:teuthology.orchestra.run.smithi001.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T21:52:24.364 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T21:52:24.364 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.364 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.364 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.365 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.365 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.365 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.365 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.365 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.365 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.365 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.365 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.366 INFO:teuthology.orchestra.run.smithi001.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T21:52:24.366 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T21:52:24.366 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.366 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.366 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.366 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.366 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.366 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.367 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.367 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.367 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.367 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.367 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.367 INFO:teuthology.orchestra.run.smithi001.stdout: "AUDIT_API_ENABLED": { 2024-04-26T21:52:24.367 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T21:52:24.367 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.367 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.368 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.368 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.368 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.368 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.368 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.368 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.368 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.368 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.369 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.369 INFO:teuthology.orchestra.run.smithi001.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T21:52:24.369 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T21:52:24.369 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.369 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.369 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.369 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.369 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.370 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.370 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.370 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.370 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.370 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.370 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.370 INFO:teuthology.orchestra.run.smithi001.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T21:52:24.370 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T21:52:24.371 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.371 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.371 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.371 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.371 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.371 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.371 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.371 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.371 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.372 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.372 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.372 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T21:52:24.372 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T21:52:24.372 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.372 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.372 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.373 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.373 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.373 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.373 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.373 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.373 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.373 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.373 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.374 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T21:52:24.374 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T21:52:24.374 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.374 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.374 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.374 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.374 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.374 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.375 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.375 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.375 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.375 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.375 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.375 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T21:52:24.375 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T21:52:24.375 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.376 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.376 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.376 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.376 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.376 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.376 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.376 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.376 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.376 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.377 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.377 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T21:52:24.377 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T21:52:24.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.377 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.377 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.377 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.377 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.378 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.378 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.378 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.378 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.378 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.378 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.378 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T21:52:24.379 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T21:52:24.379 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.379 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.379 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.379 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.379 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.379 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.379 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.379 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.380 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.380 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.380 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.380 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T21:52:24.380 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T21:52:24.380 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.380 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.380 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.381 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.381 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.381 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.381 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.381 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.381 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.381 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.381 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.382 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T21:52:24.382 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T21:52:24.382 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.382 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.382 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.382 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.382 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.382 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.383 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.383 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.383 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.383 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.383 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.383 INFO:teuthology.orchestra.run.smithi001.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T21:52:24.383 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T21:52:24.383 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.384 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.384 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.384 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.384 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.384 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.384 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.384 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.384 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.384 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T21:52:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T21:52:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "admin", 2024-04-26T21:52:24.385 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.386 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.386 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.386 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.386 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.386 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.386 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.386 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T21:52:24.386 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T21:52:24.387 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.387 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.387 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.387 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.387 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.387 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.387 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.387 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.388 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.388 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.388 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.388 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.388 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_URL": { 2024-04-26T21:52:24.388 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_URL", 2024-04-26T21:52:24.388 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.388 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.389 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.389 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.389 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.389 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.389 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.389 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.389 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.389 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.389 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.390 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.390 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T21:52:24.390 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T21:52:24.390 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.390 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.390 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.390 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "admin", 2024-04-26T21:52:24.391 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.391 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.391 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.391 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.391 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.391 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.391 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.392 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.392 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T21:52:24.392 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T21:52:24.392 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.392 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.392 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.392 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.392 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.392 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.393 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.393 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.393 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.393 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.393 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.393 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.393 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T21:52:24.393 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T21:52:24.393 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.394 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.395 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.395 INFO:teuthology.orchestra.run.smithi001.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T21:52:24.395 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T21:52:24.395 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.395 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.395 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.395 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.395 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.396 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.396 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.396 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.396 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.396 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.396 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.396 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.397 INFO:teuthology.orchestra.run.smithi001.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T21:52:24.397 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T21:52:24.397 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.397 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.397 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.397 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.397 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.397 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.397 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.398 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.398 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.398 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.398 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.398 INFO:teuthology.orchestra.run.smithi001.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T21:52:24.398 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T21:52:24.398 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.399 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.399 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.399 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "[]", 2024-04-26T21:52:24.399 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.399 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.399 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.399 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.399 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.399 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.400 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.400 INFO:teuthology.orchestra.run.smithi001.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T21:52:24.400 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T21:52:24.400 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.400 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.400 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.400 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "{}", 2024-04-26T21:52:24.401 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.401 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.401 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.401 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.401 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.401 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.401 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.401 INFO:teuthology.orchestra.run.smithi001.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T21:52:24.401 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T21:52:24.402 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.402 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.402 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.402 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.402 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.402 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.402 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.402 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.403 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.403 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.403 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.403 INFO:teuthology.orchestra.run.smithi001.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T21:52:24.403 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T21:52:24.403 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.403 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.403 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.404 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.404 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.404 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.404 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.404 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.404 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.404 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.404 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.404 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.405 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T21:52:24.405 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T21:52:24.405 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.405 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.405 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.405 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.405 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.405 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.405 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.406 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.406 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.406 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.406 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.406 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T21:52:24.406 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T21:52:24.406 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.407 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.407 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.407 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.407 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.407 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.407 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.407 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.407 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.407 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.408 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.408 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T21:52:24.408 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T21:52:24.408 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.408 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.408 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.408 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.408 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.409 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.409 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.409 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.409 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.409 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.409 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.409 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T21:52:24.409 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T21:52:24.410 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.410 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.410 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.410 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.410 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.410 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.410 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.410 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.411 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.411 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.411 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.411 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T21:52:24.411 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T21:52:24.411 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.411 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.412 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.412 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.412 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.412 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.412 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.412 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.412 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.412 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.412 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.413 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.413 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T21:52:24.413 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T21:52:24.413 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.413 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.413 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.413 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.413 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.414 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.414 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.414 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.414 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.414 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.414 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.414 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T21:52:24.415 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T21:52:24.415 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.415 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.415 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.415 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.415 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.415 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.415 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.416 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.416 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.416 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.416 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.416 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.416 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T21:52:24.416 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T21:52:24.416 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.416 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.417 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.417 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.417 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.417 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.417 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.417 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.417 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.417 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.417 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.418 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T21:52:24.418 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T21:52:24.418 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.418 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.418 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.418 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T21:52:24.418 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.418 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.419 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.419 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.419 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.419 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.419 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.419 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T21:52:24.419 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T21:52:24.419 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.420 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.420 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.420 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:24.420 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.420 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.420 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.420 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.420 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.420 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.420 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.421 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T21:52:24.421 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T21:52:24.421 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.421 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.421 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.421 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8", 2024-04-26T21:52:24.421 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.421 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.422 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.422 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.422 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.422 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.422 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.422 INFO:teuthology.orchestra.run.smithi001.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T21:52:24.422 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T21:52:24.423 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.423 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.423 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.423 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "45", 2024-04-26T21:52:24.423 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.423 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.423 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.423 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.423 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.424 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.424 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.424 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.424 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T21:52:24.424 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T21:52:24.424 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.424 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.424 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.425 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.425 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.425 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.425 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.425 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.425 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.425 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.426 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.426 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.426 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T21:52:24.426 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T21:52:24.426 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.426 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.426 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.427 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "admin", 2024-04-26T21:52:24.427 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.427 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.427 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.427 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.427 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.427 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.427 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.428 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T21:52:24.428 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T21:52:24.428 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.428 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.428 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.428 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.428 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.428 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.429 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.429 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.429 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.429 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.429 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.429 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T21:52:24.429 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T21:52:24.429 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.430 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.430 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.430 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.430 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.430 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.430 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.430 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.431 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.431 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.431 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.431 INFO:teuthology.orchestra.run.smithi001.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T21:52:24.431 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T21:52:24.431 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.431 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.431 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.432 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.432 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.432 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.432 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.432 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.432 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.432 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.432 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.432 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.433 INFO:teuthology.orchestra.run.smithi001.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T21:52:24.433 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T21:52:24.433 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.433 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.433 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.433 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:24.433 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.433 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.434 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.434 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.434 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.434 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.434 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.434 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.434 INFO:teuthology.orchestra.run.smithi001.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T21:52:24.434 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T21:52:24.434 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.435 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.435 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.435 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:24.435 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.435 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.435 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.435 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.435 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.436 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.436 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.436 INFO:teuthology.orchestra.run.smithi001.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T21:52:24.436 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T21:52:24.436 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.436 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.436 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.437 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:24.437 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.437 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.437 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.437 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.437 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.437 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.437 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.437 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.437 INFO:teuthology.orchestra.run.smithi001.stdout: "cross_origin_url": { 2024-04-26T21:52:24.438 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cross_origin_url", 2024-04-26T21:52:24.438 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.438 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.438 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.438 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.438 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.438 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.438 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.439 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.439 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.439 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.439 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.439 INFO:teuthology.orchestra.run.smithi001.stdout: "crt_file": { 2024-04-26T21:52:24.439 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crt_file", 2024-04-26T21:52:24.440 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.440 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.440 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.440 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.440 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.440 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.440 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.440 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.440 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.441 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.441 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.441 INFO:teuthology.orchestra.run.smithi001.stdout: "debug": { 2024-04-26T21:52:24.441 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "debug", 2024-04-26T21:52:24.441 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.441 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.442 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.442 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.442 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.442 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.442 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.442 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Enable/disable debug options", 2024-04-26T21:52:24.442 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.442 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.443 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.443 INFO:teuthology.orchestra.run.smithi001.stdout: "jwt_token_ttl": { 2024-04-26T21:52:24.443 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "jwt_token_ttl", 2024-04-26T21:52:24.443 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.443 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.443 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.443 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "28800", 2024-04-26T21:52:24.443 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.444 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.444 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.444 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.444 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.444 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.444 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.444 INFO:teuthology.orchestra.run.smithi001.stdout: "key_file": { 2024-04-26T21:52:24.444 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "key_file", 2024-04-26T21:52:24.445 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.445 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.445 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.445 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.445 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.445 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.445 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.446 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.446 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.446 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.446 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.446 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.446 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.446 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.447 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.447 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.447 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.447 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.447 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.447 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.447 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.447 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.447 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.447 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.448 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.448 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.448 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.448 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.448 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.448 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.448 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.448 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.449 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.449 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.449 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.449 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.449 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.449 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.449 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.449 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.449 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.450 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.450 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.450 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.450 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.450 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.450 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.450 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.451 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.451 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.451 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.451 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.451 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.451 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.451 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.451 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.452 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.452 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.452 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.452 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.452 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.452 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.452 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.452 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.452 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.453 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.453 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.453 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.453 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.453 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.453 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.453 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.453 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.453 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.454 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.454 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.454 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.454 INFO:teuthology.orchestra.run.smithi001.stdout: "motd": { 2024-04-26T21:52:24.454 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "motd", 2024-04-26T21:52:24.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.454 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.455 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.455 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.455 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.455 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.455 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.455 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "The message of the day", 2024-04-26T21:52:24.455 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.455 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.455 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.456 INFO:teuthology.orchestra.run.smithi001.stdout: "redirect_resolve_ip_addr": { 2024-04-26T21:52:24.456 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T21:52:24.456 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.456 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.456 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.456 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.456 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.457 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.457 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.457 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.457 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.457 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.457 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.457 INFO:teuthology.orchestra.run.smithi001.stdout: "server_addr": { 2024-04-26T21:52:24.458 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_addr", 2024-04-26T21:52:24.458 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.458 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.458 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.458 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "::", 2024-04-26T21:52:24.458 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.458 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.458 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.458 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.459 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.459 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.459 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.459 INFO:teuthology.orchestra.run.smithi001.stdout: "server_port": { 2024-04-26T21:52:24.459 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_port", 2024-04-26T21:52:24.459 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.459 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.459 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.460 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8080", 2024-04-26T21:52:24.460 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.460 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.460 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.460 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.460 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.460 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.460 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.461 INFO:teuthology.orchestra.run.smithi001.stdout: "ssl": { 2024-04-26T21:52:24.461 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssl", 2024-04-26T21:52:24.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.461 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.461 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.461 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.461 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.461 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.462 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.462 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.462 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.462 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.462 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.462 INFO:teuthology.orchestra.run.smithi001.stdout: "ssl_server_port": { 2024-04-26T21:52:24.462 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssl_server_port", 2024-04-26T21:52:24.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.463 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.463 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.463 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8443", 2024-04-26T21:52:24.463 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.463 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.463 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.463 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.463 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.463 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.464 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.464 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_behaviour": { 2024-04-26T21:52:24.464 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_behaviour", 2024-04-26T21:52:24.464 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.464 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.464 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.464 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "redirect", 2024-04-26T21:52:24.465 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.465 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.465 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.465 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.465 INFO:teuthology.orchestra.run.smithi001.stdout: "redirect" 2024-04-26T21:52:24.465 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.465 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.465 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.465 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.465 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.466 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_error_status_code": { 2024-04-26T21:52:24.466 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_error_status_code", 2024-04-26T21:52:24.466 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.466 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.466 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.466 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "500", 2024-04-26T21:52:24.466 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "400", 2024-04-26T21:52:24.467 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "599", 2024-04-26T21:52:24.467 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.467 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.467 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.467 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.467 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.467 INFO:teuthology.orchestra.run.smithi001.stdout: "url_prefix": { 2024-04-26T21:52:24.467 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "url_prefix", 2024-04-26T21:52:24.467 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.468 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.468 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.468 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.468 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.468 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.468 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.468 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.468 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.468 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.469 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.469 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.469 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.469 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "devicehealth", 2024-04-26T21:52:24.469 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.469 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.469 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.470 INFO:teuthology.orchestra.run.smithi001.stdout: "enable_monitoring": { 2024-04-26T21:52:24.470 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enable_monitoring", 2024-04-26T21:52:24.470 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.470 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.470 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.470 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.470 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.470 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.470 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.471 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "monitor device health metrics", 2024-04-26T21:52:24.471 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.471 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.471 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.471 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.471 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.471 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.471 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.471 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.472 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.472 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.472 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.472 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.472 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.472 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.472 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.472 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.472 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.473 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.473 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.473 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.473 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.473 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.473 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.473 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.473 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.473 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.474 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.474 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.474 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.474 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.474 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.474 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.474 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.474 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.475 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.475 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.475 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.475 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.475 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.475 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.475 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.475 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.476 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.476 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.476 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.476 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.476 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.476 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.476 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.476 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.476 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.477 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.477 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.477 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.477 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.477 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.477 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.477 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.477 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.478 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.478 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.478 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.478 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.478 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.478 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.478 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.478 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.479 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.479 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.479 INFO:teuthology.orchestra.run.smithi001.stdout: "mark_out_threshold": { 2024-04-26T21:52:24.479 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mark_out_threshold", 2024-04-26T21:52:24.479 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.479 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.479 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.479 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "2419200", 2024-04-26T21:52:24.479 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.480 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.480 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.480 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T21:52:24.480 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.480 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.480 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.480 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": { 2024-04-26T21:52:24.481 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pool_name", 2024-04-26T21:52:24.481 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.481 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.481 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.481 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "device_health_metrics", 2024-04-26T21:52:24.481 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.481 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.481 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.481 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T21:52:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: "retention_period": { 2024-04-26T21:52:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "retention_period", 2024-04-26T21:52:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.482 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.483 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "15552000", 2024-04-26T21:52:24.483 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.483 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.483 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.483 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how long to retain device health metrics", 2024-04-26T21:52:24.483 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.483 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.483 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.484 INFO:teuthology.orchestra.run.smithi001.stdout: "scrape_frequency": { 2024-04-26T21:52:24.484 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "scrape_frequency", 2024-04-26T21:52:24.484 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.484 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.484 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.484 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "86400", 2024-04-26T21:52:24.484 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.484 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.484 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.484 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T21:52:24.485 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.485 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.485 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.485 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.485 INFO:teuthology.orchestra.run.smithi001.stdout: "self_heal": { 2024-04-26T21:52:24.485 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "self_heal", 2024-04-26T21:52:24.485 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.485 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.485 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.486 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.486 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.486 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.486 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.486 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T21:52:24.486 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.486 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.486 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.487 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:24.487 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:24.487 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.487 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.487 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.487 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:24.487 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.487 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.488 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.488 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T21:52:24.488 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.488 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.488 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.488 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_threshold": { 2024-04-26T21:52:24.489 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_threshold", 2024-04-26T21:52:24.489 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.489 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.489 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.489 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "7257600", 2024-04-26T21:52:24.489 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.489 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.489 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.489 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T21:52:24.490 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.490 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.490 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.490 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "diskprediction_local", 2024-04-26T21:52:24.491 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.491 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.491 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.491 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.491 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.491 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.491 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.491 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.492 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.492 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.492 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.492 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.492 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.492 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.492 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.492 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.493 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.493 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.493 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.493 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.493 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.493 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.493 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.493 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.494 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.494 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.494 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.494 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.494 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.494 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.494 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.494 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.494 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.495 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.495 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.495 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.495 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.495 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.495 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.495 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.495 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.496 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.496 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.496 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.496 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.496 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.496 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.496 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.496 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.497 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.497 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.497 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.497 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.497 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.497 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.497 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.498 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.498 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.498 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.498 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.498 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.498 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.498 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: "predict_interval": { 2024-04-26T21:52:24.499 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "predict_interval", 2024-04-26T21:52:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "86400", 2024-04-26T21:52:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.500 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.501 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.501 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.501 INFO:teuthology.orchestra.run.smithi001.stdout: "predictor_model": { 2024-04-26T21:52:24.501 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "predictor_model", 2024-04-26T21:52:24.501 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.501 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.501 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.502 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "prophetstor", 2024-04-26T21:52:24.502 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.502 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.502 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.502 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.502 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.502 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.502 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.503 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:24.503 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:24.503 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.503 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.503 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.503 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:24.503 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.503 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.503 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.504 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.504 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.504 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.504 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.504 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.504 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.504 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.504 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "influx", 2024-04-26T21:52:24.505 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": false, 2024-04-26T21:52:24.505 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "influxdb python module not found", 2024-04-26T21:52:24.505 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.505 INFO:teuthology.orchestra.run.smithi001.stdout: "batch_size": { 2024-04-26T21:52:24.505 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "batch_size", 2024-04-26T21:52:24.505 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.505 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.505 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.505 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5000", 2024-04-26T21:52:24.506 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.506 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.506 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.506 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T21:52:24.506 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.506 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.506 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.507 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.507 INFO:teuthology.orchestra.run.smithi001.stdout: "database": { 2024-04-26T21:52:24.507 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "database", 2024-04-26T21:52:24.507 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.510 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.511 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.511 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "ceph", 2024-04-26T21:52:24.511 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.511 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.511 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.511 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T21:52:24.511 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.511 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.512 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.512 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.512 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": { 2024-04-26T21:52:24.512 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "hostname", 2024-04-26T21:52:24.512 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.512 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.512 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.512 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.512 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.513 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.513 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.513 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "InfluxDB server hostname", 2024-04-26T21:52:24.513 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.513 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.513 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.513 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:24.514 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:24.514 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.514 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.514 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.514 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "30", 2024-04-26T21:52:24.514 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "5", 2024-04-26T21:52:24.514 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.514 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.514 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T21:52:24.515 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.515 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.515 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.515 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.515 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.515 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.515 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.515 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.515 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.515 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.516 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.516 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.516 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.516 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.516 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.516 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.516 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.516 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.516 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.516 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.517 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.517 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.517 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.517 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.517 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.517 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.517 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.518 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.518 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.518 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.518 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.518 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.518 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.518 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.518 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.519 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.519 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.519 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.519 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.519 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.519 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.519 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.520 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.520 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.520 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.520 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.520 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.520 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.520 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.520 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.520 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.521 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.521 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.521 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.521 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.521 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.521 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.521 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.521 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.522 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.522 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.522 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.522 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.522 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.522 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.522 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.523 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.523 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.523 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.523 INFO:teuthology.orchestra.run.smithi001.stdout: "password": { 2024-04-26T21:52:24.523 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "password", 2024-04-26T21:52:24.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.523 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.524 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.524 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.524 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.524 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.524 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.524 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "password of InfluxDB server user", 2024-04-26T21:52:24.524 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.524 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.525 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.525 INFO:teuthology.orchestra.run.smithi001.stdout: "port": { 2024-04-26T21:52:24.525 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "port", 2024-04-26T21:52:24.525 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.525 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.525 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.525 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8086", 2024-04-26T21:52:24.526 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.526 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.526 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.526 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "InfluxDB server port", 2024-04-26T21:52:24.526 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.526 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.527 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.527 INFO:teuthology.orchestra.run.smithi001.stdout: "ssl": { 2024-04-26T21:52:24.527 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssl", 2024-04-26T21:52:24.527 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.527 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.527 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.528 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "false", 2024-04-26T21:52:24.528 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.528 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.528 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.528 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T21:52:24.528 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.528 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.528 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.529 INFO:teuthology.orchestra.run.smithi001.stdout: "threads": { 2024-04-26T21:52:24.529 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "threads", 2024-04-26T21:52:24.529 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.529 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.529 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.529 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:24.529 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:24.529 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "32", 2024-04-26T21:52:24.530 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.530 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T21:52:24.530 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.530 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.530 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.530 INFO:teuthology.orchestra.run.smithi001.stdout: "username": { 2024-04-26T21:52:24.530 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "username", 2024-04-26T21:52:24.530 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.531 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.531 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.531 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.531 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.531 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.531 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.531 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "username of InfluxDB server user", 2024-04-26T21:52:24.532 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.532 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.532 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.532 INFO:teuthology.orchestra.run.smithi001.stdout: "verify_ssl": { 2024-04-26T21:52:24.532 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "verify_ssl", 2024-04-26T21:52:24.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.532 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.532 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.533 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "true", 2024-04-26T21:52:24.533 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.533 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.533 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.533 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T21:52:24.533 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.533 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.533 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.534 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.534 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.534 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.534 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "insights", 2024-04-26T21:52:24.534 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.534 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.534 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.534 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.535 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.535 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.535 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.535 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.535 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.535 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.535 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.535 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.535 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.536 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.536 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.536 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.536 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.536 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.536 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.536 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.536 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.537 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.537 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.537 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.537 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.537 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.537 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.537 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.537 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.538 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.538 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.538 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.538 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.538 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.538 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.538 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.538 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.539 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.539 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.539 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.539 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.539 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.539 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.539 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.539 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.539 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.540 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.540 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.540 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.540 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.540 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.540 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.540 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.541 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.541 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.541 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.541 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.541 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.541 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.541 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.542 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.542 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.542 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.542 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.542 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.542 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.542 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.543 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.543 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.543 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.543 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.543 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "iostat", 2024-04-26T21:52:24.543 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.543 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.544 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.544 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.544 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.544 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.544 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.544 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.544 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.544 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.544 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.545 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.545 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.545 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.545 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.545 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.545 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.545 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.545 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.546 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.546 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.546 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.546 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.546 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.546 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.546 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.546 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.547 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.547 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.547 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.547 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.547 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.547 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.547 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.547 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.548 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.548 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.548 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.548 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.548 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.548 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.548 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.548 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.549 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.549 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.549 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.549 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.549 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.549 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.549 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.549 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.549 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.550 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.550 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.550 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.550 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.550 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.550 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.550 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.551 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.551 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.551 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.551 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.551 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.551 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.551 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.551 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.551 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.552 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.552 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "k8sevents", 2024-04-26T21:52:24.552 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.552 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.552 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.553 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_event_retention_days": { 2024-04-26T21:52:24.553 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ceph_event_retention_days", 2024-04-26T21:52:24.553 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.553 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.553 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.553 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "7", 2024-04-26T21:52:24.553 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.553 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.553 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.553 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T21:52:24.554 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.554 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.554 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.554 INFO:teuthology.orchestra.run.smithi001.stdout: "config_check_secs": { 2024-04-26T21:52:24.554 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "config_check_secs", 2024-04-26T21:52:24.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.554 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.555 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.555 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:24.555 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "10", 2024-04-26T21:52:24.555 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.555 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.555 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T21:52:24.555 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.555 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.555 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.556 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.556 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.556 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.556 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.556 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.556 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.556 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.556 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.557 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.557 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.557 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.557 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.557 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.557 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.557 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.557 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.558 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.558 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.558 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.558 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.558 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.558 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.558 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.559 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.559 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.559 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.559 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.559 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.559 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.559 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.559 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.560 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.560 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.560 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.560 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.560 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.560 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.560 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.561 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.561 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.561 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.561 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.561 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.561 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.561 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.561 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.561 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.561 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.562 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.562 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.562 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.562 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.562 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.562 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.563 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.563 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.563 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.563 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.563 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.563 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.563 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.563 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.563 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.564 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "localpool", 2024-04-26T21:52:24.564 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.564 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.564 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.564 INFO:teuthology.orchestra.run.smithi001.stdout: "failure_domain": { 2024-04-26T21:52:24.565 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "failure_domain", 2024-04-26T21:52:24.565 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.565 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.565 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.565 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "host", 2024-04-26T21:52:24.565 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.565 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.565 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.565 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "failure domain for any created local pool", 2024-04-26T21:52:24.566 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T21:52:24.566 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.566 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.566 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.566 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.566 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.566 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.566 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.567 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.567 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.567 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.567 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.567 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.567 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.567 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.567 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.568 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.568 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.568 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.568 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.568 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.568 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.568 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.569 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.569 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.569 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.569 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.569 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.569 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.569 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.569 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.569 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.570 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.570 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.570 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.570 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.570 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.570 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.570 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.571 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.571 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.571 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.571 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.571 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.571 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.571 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.571 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.572 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.572 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.572 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.572 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.572 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.572 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.572 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.572 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.572 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.573 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.573 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.573 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.573 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.573 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.573 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.573 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.574 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.574 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.574 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.574 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.574 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.574 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.574 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": { 2024-04-26T21:52:24.574 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "min_size", 2024-04-26T21:52:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "default min_size for any created local pool", 2024-04-26T21:52:24.575 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T21:52:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "num_rep": { 2024-04-26T21:52:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "num_rep", 2024-04-26T21:52:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.576 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "3", 2024-04-26T21:52:24.577 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.577 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.577 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.577 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "default replica count for any created local pool", 2024-04-26T21:52:24.577 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.577 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.577 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.578 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": { 2024-04-26T21:52:24.578 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pg_num", 2024-04-26T21:52:24.578 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.578 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.578 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.578 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "128", 2024-04-26T21:52:24.578 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.578 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.578 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.579 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T21:52:24.579 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.579 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.579 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.579 INFO:teuthology.orchestra.run.smithi001.stdout: "prefix": { 2024-04-26T21:52:24.579 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "prefix", 2024-04-26T21:52:24.579 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.579 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.580 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.580 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.580 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.580 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.580 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.580 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "name prefix for any created local pool", 2024-04-26T21:52:24.580 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.580 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.580 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.581 INFO:teuthology.orchestra.run.smithi001.stdout: "subtree": { 2024-04-26T21:52:24.581 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "subtree", 2024-04-26T21:52:24.581 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.581 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.581 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.581 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "rack", 2024-04-26T21:52:24.581 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.581 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.582 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.582 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T21:52:24.582 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T21:52:24.582 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.582 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.583 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mds_autoscaler", 2024-04-26T21:52:24.583 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.583 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.583 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.583 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.583 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.583 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.583 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.583 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.584 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.585 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.585 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.585 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.585 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.585 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.585 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.585 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.585 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.586 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.586 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.586 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.586 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.586 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.586 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.586 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.586 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.587 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.587 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.587 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.587 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.587 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.587 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.587 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.588 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.588 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.588 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.588 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.588 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.588 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.588 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.588 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.588 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.589 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.589 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.589 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.589 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.589 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.589 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.589 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.590 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.590 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.590 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.590 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.590 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.590 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.590 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.591 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.591 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.591 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.591 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.592 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mirroring", 2024-04-26T21:52:24.592 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.592 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.592 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.592 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.592 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.592 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.593 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.593 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.593 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.593 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.593 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.593 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.593 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.593 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.594 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.594 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.594 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.594 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.594 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.594 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.594 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.594 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.595 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.595 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.595 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.595 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.595 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.595 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.595 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.595 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.596 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.596 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.596 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.596 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.596 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.596 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.596 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.597 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.597 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.597 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.597 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.597 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.597 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.597 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.597 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.598 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.598 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.598 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.598 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.598 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.598 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.598 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.598 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.599 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.599 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.599 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.599 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.599 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.599 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.599 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.599 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.599 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.600 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.600 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.600 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.600 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.600 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.600 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.600 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.601 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.601 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.601 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.601 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "nfs", 2024-04-26T21:52:24.601 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.602 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.602 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.602 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.602 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.602 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.602 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.602 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.602 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.603 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.604 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.604 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.604 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.604 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.604 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.604 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.604 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.605 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.605 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.605 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.605 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.605 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.605 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.605 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.605 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.605 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.605 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.606 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.606 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.606 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.606 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.606 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.606 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.607 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.607 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.607 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.607 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.607 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.607 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.607 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.607 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.607 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.608 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.608 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.608 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.608 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.608 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.608 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.608 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.609 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.609 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.609 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.609 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.609 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.609 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.609 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.609 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.609 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.610 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.610 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "orchestrator", 2024-04-26T21:52:24.610 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.611 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.611 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.611 INFO:teuthology.orchestra.run.smithi001.stdout: "fail_fs": { 2024-04-26T21:52:24.611 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "fail_fs", 2024-04-26T21:52:24.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.611 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.612 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.612 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.612 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.612 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.612 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.612 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T21:52:24.612 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.612 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.613 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.613 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.613 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.613 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.613 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.613 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.613 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.614 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.614 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.614 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.614 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.614 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.614 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.614 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.614 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.615 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.615 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.615 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.615 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.615 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.615 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.615 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.616 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.616 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.616 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.616 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.616 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.616 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.616 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.616 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.616 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.617 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.618 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.618 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.618 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.618 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.618 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.618 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.618 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.618 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.619 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.619 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.619 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.619 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.619 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.619 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.619 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.620 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.620 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.620 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.620 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.620 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.620 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.620 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.620 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.621 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.621 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.621 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.621 INFO:teuthology.orchestra.run.smithi001.stdout: "orchestrator": { 2024-04-26T21:52:24.621 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "orchestrator", 2024-04-26T21:52:24.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.621 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.621 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.622 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.622 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.622 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.622 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.622 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm", 2024-04-26T21:52:24.622 INFO:teuthology.orchestra.run.smithi001.stdout: "rook", 2024-04-26T21:52:24.622 INFO:teuthology.orchestra.run.smithi001.stdout: "test_orchestrator" 2024-04-26T21:52:24.622 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.623 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Orchestrator backend", 2024-04-26T21:52:24.623 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.623 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.623 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.623 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.623 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.624 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "osd_perf_query", 2024-04-26T21:52:24.624 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.624 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.624 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.624 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.624 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.624 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.624 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.625 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.625 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.625 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.625 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.625 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.625 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.625 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.625 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.626 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.626 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.626 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.626 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.626 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.626 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.626 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.626 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.627 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.627 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.627 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.627 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.627 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.627 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.627 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.628 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.628 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.628 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.628 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.628 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.628 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.628 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.628 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.629 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.629 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.629 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.629 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.629 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.629 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.629 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.629 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.629 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.630 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.630 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.630 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.630 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.630 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.630 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.630 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.630 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.631 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.631 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.631 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.631 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.631 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.631 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.631 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.631 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.631 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.632 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.632 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.632 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.633 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "osd_support", 2024-04-26T21:52:24.633 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.633 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.633 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.633 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.633 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.633 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.633 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.633 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.634 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.634 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.634 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.634 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.634 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.634 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.634 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.634 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.634 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.635 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.635 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.635 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.635 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.635 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.635 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.635 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.635 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.636 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.636 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.636 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.636 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.636 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.636 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.636 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.637 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.637 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.637 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.637 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.637 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.637 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.637 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.638 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.638 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.638 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.638 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.638 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.638 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.638 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.638 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.639 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.639 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.639 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.639 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.639 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.639 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.639 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.639 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.640 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.640 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.640 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.640 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.640 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.640 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.640 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.640 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.641 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.641 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.641 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.641 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.642 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pg_autoscaler", 2024-04-26T21:52:24.642 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.642 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.642 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.642 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.642 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.642 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.643 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.643 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.643 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.643 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.643 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.643 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.643 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.644 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.644 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.644 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.644 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.644 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.644 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.644 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.644 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.644 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.645 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.645 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.645 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.645 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.645 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.645 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.646 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.646 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.646 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.646 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.646 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.646 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.646 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.647 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.647 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.647 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.647 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.647 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.647 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.647 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.647 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.648 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.648 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.648 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.648 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.648 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.648 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.648 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.648 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.648 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.649 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.649 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.649 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.649 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.649 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.649 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.649 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.649 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.650 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.650 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.650 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.650 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.650 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.650 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.650 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.650 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:24.651 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:24.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.651 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.651 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.651 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:24.651 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.651 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.651 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.652 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.652 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.652 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.652 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.652 INFO:teuthology.orchestra.run.smithi001.stdout: "threshold": { 2024-04-26T21:52:24.652 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "threshold", 2024-04-26T21:52:24.652 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.652 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.652 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.653 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "3.0", 2024-04-26T21:52:24.653 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1.0", 2024-04-26T21:52:24.653 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.653 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.653 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "scaling threshold", 2024-04-26T21:52:24.653 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T21:52:24.653 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.653 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.654 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "progress", 2024-04-26T21:52:24.654 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.654 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.654 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.654 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_pg_recovery_event": { 2024-04-26T21:52:24.655 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "allow_pg_recovery_event", 2024-04-26T21:52:24.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.655 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.655 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.655 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.655 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.655 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.655 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.656 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T21:52:24.656 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.656 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.656 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.656 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": { 2024-04-26T21:52:24.656 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enabled", 2024-04-26T21:52:24.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.656 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.657 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.657 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.657 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.657 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.657 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.657 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.657 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.657 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.657 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.658 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.658 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.658 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.658 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.658 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.658 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.658 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.658 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.658 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.659 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.659 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.659 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.659 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.659 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.659 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.659 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.659 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.659 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.660 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.660 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.660 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.660 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.660 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.660 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.661 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.661 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.661 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.661 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.661 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.661 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.661 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.661 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.661 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.662 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.662 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.662 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.662 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.662 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.662 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.662 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.662 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.663 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.663 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.663 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.663 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.663 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.663 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.663 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.663 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.663 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.664 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.664 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.664 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.664 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.664 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.664 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.664 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.664 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.665 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.665 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.665 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.665 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.665 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.665 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.665 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.665 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.666 INFO:teuthology.orchestra.run.smithi001.stdout: "max_completed_events": { 2024-04-26T21:52:24.666 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_completed_events", 2024-04-26T21:52:24.666 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.666 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.666 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.666 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "50", 2024-04-26T21:52:24.666 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.666 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.666 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.667 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "number of past completed events to remember", 2024-04-26T21:52:24.667 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.667 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.667 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.667 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:24.667 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:24.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.667 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.668 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.668 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:24.668 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.668 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.668 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.668 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how long the module is going to sleep", 2024-04-26T21:52:24.668 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.668 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.669 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.669 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.669 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "prometheus", 2024-04-26T21:52:24.669 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.669 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.669 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.670 INFO:teuthology.orchestra.run.smithi001.stdout: "cache": { 2024-04-26T21:52:24.670 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cache", 2024-04-26T21:52:24.670 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.670 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.670 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.670 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.670 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.670 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.671 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.671 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.671 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.671 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.671 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.671 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.671 INFO:teuthology.orchestra.run.smithi001.stdout: "exclude_perf_counters": { 2024-04-26T21:52:24.671 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "exclude_perf_counters", 2024-04-26T21:52:24.671 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.671 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.672 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.672 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.672 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.672 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.672 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.672 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T21:52:24.672 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T21:52:24.672 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.672 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.673 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.673 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.673 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.673 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.673 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.673 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.673 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.673 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.673 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.674 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.674 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.674 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.674 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.674 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.674 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.674 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.674 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.675 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.675 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.675 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.675 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.675 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.675 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.675 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.675 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.676 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.676 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.676 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.676 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.676 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.676 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.676 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.676 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.676 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.677 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.677 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.677 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.677 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.677 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.677 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.677 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.678 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.678 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.678 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.678 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.678 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.678 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.678 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.678 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.679 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.679 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.679 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.679 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.679 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.679 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.679 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.680 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.680 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.680 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.680 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.680 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.680 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.680 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.680 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.680 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.681 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_stats_pools": { 2024-04-26T21:52:24.681 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rbd_stats_pools", 2024-04-26T21:52:24.681 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.681 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.681 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.681 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.681 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.682 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.682 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.682 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.682 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.682 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.682 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.682 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.683 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T21:52:24.683 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T21:52:24.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.683 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.683 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.683 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "300", 2024-04-26T21:52:24.683 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.683 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.683 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.684 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.684 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.684 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.684 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.684 INFO:teuthology.orchestra.run.smithi001.stdout: "scrape_interval": { 2024-04-26T21:52:24.684 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "scrape_interval", 2024-04-26T21:52:24.684 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.685 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.685 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.685 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "15.0", 2024-04-26T21:52:24.685 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.685 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.685 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.685 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.685 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.685 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.686 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.686 INFO:teuthology.orchestra.run.smithi001.stdout: "server_addr": { 2024-04-26T21:52:24.686 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_addr", 2024-04-26T21:52:24.686 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.686 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.686 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.686 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "::", 2024-04-26T21:52:24.687 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.687 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.687 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.687 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T21:52:24.687 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.687 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.687 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.687 INFO:teuthology.orchestra.run.smithi001.stdout: "server_port": { 2024-04-26T21:52:24.688 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_port", 2024-04-26T21:52:24.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.688 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.688 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.688 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "9283", 2024-04-26T21:52:24.688 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.688 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.688 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.688 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T21:52:24.689 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.689 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.689 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.689 INFO:teuthology.orchestra.run.smithi001.stdout: "stale_cache_strategy": { 2024-04-26T21:52:24.689 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "stale_cache_strategy", 2024-04-26T21:52:24.689 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.689 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.690 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.690 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "log", 2024-04-26T21:52:24.690 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.690 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.690 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.690 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.690 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.690 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.690 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.691 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_behaviour": { 2024-04-26T21:52:24.691 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_behaviour", 2024-04-26T21:52:24.691 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.691 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.691 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.691 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "default", 2024-04-26T21:52:24.691 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.691 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.692 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.692 INFO:teuthology.orchestra.run.smithi001.stdout: "default", 2024-04-26T21:52:24.692 INFO:teuthology.orchestra.run.smithi001.stdout: "error" 2024-04-26T21:52:24.692 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.692 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.692 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.692 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.692 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.693 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_error_status_code": { 2024-04-26T21:52:24.693 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_error_status_code", 2024-04-26T21:52:24.693 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.693 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.693 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.693 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "500", 2024-04-26T21:52:24.693 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "400", 2024-04-26T21:52:24.693 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "599", 2024-04-26T21:52:24.694 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.694 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.694 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.694 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.694 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.695 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rbd_support", 2024-04-26T21:52:24.695 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.695 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.695 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.695 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.695 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.695 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.695 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.696 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.696 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.696 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.696 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.696 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.696 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.696 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.696 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.696 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.697 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.697 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.697 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.697 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.697 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.697 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.697 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.698 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.698 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.698 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.698 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.698 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.698 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.698 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.698 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.699 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.699 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.699 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.699 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.699 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.699 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.699 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.699 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.700 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.700 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.700 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.700 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.700 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.700 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.700 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.700 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.700 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.701 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.701 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.701 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.701 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.701 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.701 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.701 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.701 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.702 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.702 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.702 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.702 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.702 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.702 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.702 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.702 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.703 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.703 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.703 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.703 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.703 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.703 INFO:teuthology.orchestra.run.smithi001.stdout: "max_concurrent_snap_create": { 2024-04-26T21:52:24.703 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_concurrent_snap_create", 2024-04-26T21:52:24.703 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.704 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.704 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.704 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:24.704 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.704 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.704 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.704 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.704 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.705 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.705 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.705 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.705 INFO:teuthology.orchestra.run.smithi001.stdout: "mirror_snapshot_schedule": { 2024-04-26T21:52:24.705 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T21:52:24.705 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.705 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.705 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.705 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.705 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.706 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.706 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.706 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.706 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.706 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.706 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.706 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.706 INFO:teuthology.orchestra.run.smithi001.stdout: "trash_purge_schedule": { 2024-04-26T21:52:24.707 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "trash_purge_schedule", 2024-04-26T21:52:24.707 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.707 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.707 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.707 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.707 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.707 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.707 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.707 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.708 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.708 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.708 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.708 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.708 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.708 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "restful", 2024-04-26T21:52:24.708 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.709 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.709 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.709 INFO:teuthology.orchestra.run.smithi001.stdout: "enable_auth": { 2024-04-26T21:52:24.709 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enable_auth", 2024-04-26T21:52:24.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.709 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.709 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.709 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.709 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.710 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.710 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.710 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.710 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.710 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.710 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.711 INFO:teuthology.orchestra.run.smithi001.stdout: "key_file": { 2024-04-26T21:52:24.711 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "key_file", 2024-04-26T21:52:24.711 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.715 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.715 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.715 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.715 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.716 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.716 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.716 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.716 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.716 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.716 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.717 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.717 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.717 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.717 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.717 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.717 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.717 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.717 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.718 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.718 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.718 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.718 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.718 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.718 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.718 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.718 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.719 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.719 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.719 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.719 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.719 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.719 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.719 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.719 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.720 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.720 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.720 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.720 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.720 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.720 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.720 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.720 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.720 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.721 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.721 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.721 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.721 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.721 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.721 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.721 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.722 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.722 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.722 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.722 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.722 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.722 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.722 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.722 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.722 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.723 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.723 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.723 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.723 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.723 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.723 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.723 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.724 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.724 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.724 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.724 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.724 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.724 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.724 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.724 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.724 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.725 INFO:teuthology.orchestra.run.smithi001.stdout: "server_addr": { 2024-04-26T21:52:24.725 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_addr", 2024-04-26T21:52:24.725 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.725 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.725 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.725 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.725 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.726 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.726 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.726 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.726 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.726 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.726 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.726 INFO:teuthology.orchestra.run.smithi001.stdout: "server_port": { 2024-04-26T21:52:24.727 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_port", 2024-04-26T21:52:24.727 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.727 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.727 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.727 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.727 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.727 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.727 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.727 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.728 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.728 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.728 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.728 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.728 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.729 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rgw", 2024-04-26T21:52:24.729 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.729 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.729 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.729 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.729 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.729 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.729 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.730 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.730 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.730 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.730 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.730 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.730 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.730 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.730 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.730 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.731 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.731 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.731 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.731 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.731 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.731 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.731 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.731 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.732 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.732 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.732 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.732 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.732 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.732 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.732 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.732 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.733 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.733 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.733 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.733 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.733 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.733 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.733 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.734 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.734 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.734 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.734 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.734 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.734 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.734 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.734 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.735 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.735 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.735 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.735 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.735 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.735 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.735 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.735 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.736 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.736 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.736 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.736 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.736 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.736 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.736 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.736 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.737 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.737 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.737 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.737 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.737 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.738 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rook", 2024-04-26T21:52:24.738 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.738 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.738 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.738 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.738 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.738 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.738 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.738 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.739 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.739 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.739 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.739 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.739 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.739 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.739 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.739 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.739 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.740 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.740 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.740 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.740 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.740 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.740 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.740 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.741 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.741 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.741 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.741 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.741 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.741 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.741 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.741 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.741 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.742 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.742 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.742 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.742 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.742 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.742 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.743 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.743 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.743 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.743 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.743 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.743 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.743 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.743 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.743 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.744 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.744 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.744 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.744 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.744 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.744 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.744 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.744 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.745 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.745 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.745 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.745 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.745 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.745 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.745 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.745 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.746 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.746 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.746 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.746 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.746 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.746 INFO:teuthology.orchestra.run.smithi001.stdout: "storage_class": { 2024-04-26T21:52:24.746 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "storage_class", 2024-04-26T21:52:24.747 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.747 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.747 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.747 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "local", 2024-04-26T21:52:24.747 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.747 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.747 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.747 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T21:52:24.747 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.748 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.748 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.748 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.748 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.748 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "selftest", 2024-04-26T21:52:24.748 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.748 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.749 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.749 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.749 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.749 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.749 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.749 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.749 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.749 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.749 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.750 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.750 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.750 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.750 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.750 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.750 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.750 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.750 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.750 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.751 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.751 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.751 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.751 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.751 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.751 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.751 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.751 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.752 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.752 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.752 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.752 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.752 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.752 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.752 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.752 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.752 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.753 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.753 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.753 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.753 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.753 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.753 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.753 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.753 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.753 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.754 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.754 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.754 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.754 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.754 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.754 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.754 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.754 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.754 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.755 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.755 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.755 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.755 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.755 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.755 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.755 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.755 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.756 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.756 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.756 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.756 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.756 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.756 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.756 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.756 INFO:teuthology.orchestra.run.smithi001.stdout: "roption1": { 2024-04-26T21:52:24.756 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "roption1", 2024-04-26T21:52:24.757 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.757 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.757 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.757 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.757 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.757 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.757 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.757 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.757 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.758 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.758 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.758 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.758 INFO:teuthology.orchestra.run.smithi001.stdout: "roption2": { 2024-04-26T21:52:24.758 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "roption2", 2024-04-26T21:52:24.758 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.758 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.758 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.758 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "xyz", 2024-04-26T21:52:24.759 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.759 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.759 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.759 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.759 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.759 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.759 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.759 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.759 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption1": { 2024-04-26T21:52:24.760 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption1", 2024-04-26T21:52:24.760 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.760 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.760 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.760 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.760 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.760 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.760 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.760 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.761 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.761 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.761 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.761 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption2": { 2024-04-26T21:52:24.761 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption2", 2024-04-26T21:52:24.761 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.761 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.761 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.761 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.762 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.762 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.762 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.762 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.762 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.762 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.762 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.762 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.763 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption3": { 2024-04-26T21:52:24.763 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption3", 2024-04-26T21:52:24.763 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.763 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.763 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.763 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.763 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.763 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.763 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.764 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.764 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.764 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.764 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.764 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption4": { 2024-04-26T21:52:24.764 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption4", 2024-04-26T21:52:24.764 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.764 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.765 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.765 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.765 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.765 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.765 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.765 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.765 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.765 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.766 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.766 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.766 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption5": { 2024-04-26T21:52:24.766 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption5", 2024-04-26T21:52:24.766 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.766 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.766 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.766 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.767 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.767 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.767 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.767 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.767 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.767 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.767 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.767 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.767 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption6": { 2024-04-26T21:52:24.768 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption6", 2024-04-26T21:52:24.768 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.768 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.768 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.768 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.768 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.768 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.768 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.768 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.769 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.769 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.769 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.769 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.769 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption7": { 2024-04-26T21:52:24.769 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption7", 2024-04-26T21:52:24.769 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.769 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.769 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.770 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.770 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:24.770 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "42", 2024-04-26T21:52:24.770 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.770 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.770 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.770 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.770 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.771 INFO:teuthology.orchestra.run.smithi001.stdout: "testkey": { 2024-04-26T21:52:24.771 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "testkey", 2024-04-26T21:52:24.771 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.771 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.771 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.771 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.771 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.771 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.771 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.771 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.772 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.772 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.772 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.772 INFO:teuthology.orchestra.run.smithi001.stdout: "testlkey": { 2024-04-26T21:52:24.772 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "testlkey", 2024-04-26T21:52:24.772 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.772 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.772 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.773 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.773 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.773 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.773 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.773 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.773 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.773 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.773 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.773 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.773 INFO:teuthology.orchestra.run.smithi001.stdout: "testnewline": { 2024-04-26T21:52:24.774 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "testnewline", 2024-04-26T21:52:24.774 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.774 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.774 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.774 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.774 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.774 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.774 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.774 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.775 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.775 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.775 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.775 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.775 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.775 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.775 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.775 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snap_schedule", 2024-04-26T21:52:24.775 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.775 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.776 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.776 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_m_granularity": { 2024-04-26T21:52:24.776 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "allow_m_granularity", 2024-04-26T21:52:24.776 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.776 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.776 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.776 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.776 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.777 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.777 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.777 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T21:52:24.777 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.777 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.777 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.777 INFO:teuthology.orchestra.run.smithi001.stdout: "dump_on_update": { 2024-04-26T21:52:24.777 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "dump_on_update", 2024-04-26T21:52:24.777 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.778 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.778 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.778 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.778 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.778 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.778 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.778 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "dump database to debug log on update", 2024-04-26T21:52:24.778 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.779 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.779 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.779 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.779 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.779 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.779 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.779 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.779 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.779 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.780 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.780 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.780 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.780 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.780 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.780 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.780 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.780 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.780 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.781 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.781 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.781 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.781 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.781 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.781 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.781 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.781 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.782 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.782 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.782 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.782 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.782 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.782 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.782 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.782 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.782 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.783 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.783 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.783 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.783 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.783 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.783 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.783 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.783 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.784 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.784 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.784 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.784 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.784 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.784 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.784 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.784 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.784 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.785 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.785 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.785 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.785 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.785 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.785 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.785 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.785 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.785 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.786 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.787 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.787 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.787 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "stats", 2024-04-26T21:52:24.787 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.787 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.787 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.787 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.788 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.788 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.788 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.788 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.788 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.788 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.788 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.788 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.788 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.789 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.789 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.789 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.789 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.789 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.789 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.789 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.789 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.789 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.790 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.790 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.790 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.790 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.790 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.790 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.791 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.791 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.791 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.791 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.791 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.791 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.791 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.792 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.792 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.792 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.792 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.792 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.792 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.792 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.792 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.792 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.793 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.793 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.793 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.793 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.793 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.793 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.793 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.793 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.793 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.794 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.794 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.794 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.794 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.794 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.794 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.795 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.795 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.795 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.795 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.795 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.795 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.795 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.795 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.795 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.796 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.796 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.796 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.796 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "status", 2024-04-26T21:52:24.796 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.796 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.796 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.797 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.797 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.797 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.797 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.797 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.797 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.797 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.797 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.798 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.798 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.798 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.798 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.798 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.798 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.798 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.798 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.799 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.799 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.799 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.799 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.799 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.799 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.799 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.800 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.800 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.800 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.800 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.800 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.800 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.800 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.800 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.800 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.800 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.801 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.801 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.801 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.801 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.801 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.801 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.801 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.801 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.802 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.802 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.802 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.802 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.802 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.802 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.802 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.802 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.802 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.803 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.803 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.803 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.803 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.803 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.803 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.803 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.803 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.804 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.804 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.804 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.804 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.804 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.804 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.804 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.804 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.804 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.804 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.805 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "telegraf", 2024-04-26T21:52:24.805 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.805 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.805 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.805 INFO:teuthology.orchestra.run.smithi001.stdout: "address": { 2024-04-26T21:52:24.805 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "address", 2024-04-26T21:52:24.805 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.806 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.806 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.806 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T21:52:24.806 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.806 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.806 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.806 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.806 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.806 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.807 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.807 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:24.807 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:24.807 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.807 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.807 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.807 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "15", 2024-04-26T21:52:24.807 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.807 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.808 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.808 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.808 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.808 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.808 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.808 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.808 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.808 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.808 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.809 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.809 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.809 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.809 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.809 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.809 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.809 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.809 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.809 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.810 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.810 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.810 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.810 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.810 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.810 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.810 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.810 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.811 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.811 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.811 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.811 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.811 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.811 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.811 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.811 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.811 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.812 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.812 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.812 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.812 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.812 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.812 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.812 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.812 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.813 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.813 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.813 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.813 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.813 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.813 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.813 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.813 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.813 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.813 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.814 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.814 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.814 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.814 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.814 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.814 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.814 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.814 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.815 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.815 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.815 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.815 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.815 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.815 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.815 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.815 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.815 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.816 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.816 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.816 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "telemetry", 2024-04-26T21:52:24.817 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.817 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.817 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.817 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_basic": { 2024-04-26T21:52:24.817 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_basic", 2024-04-26T21:52:24.817 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.817 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.817 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.817 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.818 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.818 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.818 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.818 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T21:52:24.818 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.818 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.818 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.818 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.818 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_crash": { 2024-04-26T21:52:24.819 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_crash", 2024-04-26T21:52:24.819 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.819 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.819 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.819 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.819 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.819 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.819 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.819 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T21:52:24.820 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.820 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.820 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.820 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.820 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_device": { 2024-04-26T21:52:24.820 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_device", 2024-04-26T21:52:24.820 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.820 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.821 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.821 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.821 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.821 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.821 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.821 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T21:52:24.821 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.821 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.821 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.822 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.822 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_ident": { 2024-04-26T21:52:24.822 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_ident", 2024-04-26T21:52:24.822 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.822 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.822 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.822 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.822 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.822 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.823 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.823 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T21:52:24.823 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.823 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.823 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.823 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.823 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_perf": { 2024-04-26T21:52:24.824 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_perf", 2024-04-26T21:52:24.824 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.824 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.824 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.824 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.824 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.824 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.824 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.824 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T21:52:24.824 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.825 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.825 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.825 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.825 INFO:teuthology.orchestra.run.smithi001.stdout: "contact": { 2024-04-26T21:52:24.825 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "contact", 2024-04-26T21:52:24.825 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.825 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.825 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.825 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.826 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.826 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.826 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.826 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.826 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.826 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.826 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.827 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.827 INFO:teuthology.orchestra.run.smithi001.stdout: "description": { 2024-04-26T21:52:24.827 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "description", 2024-04-26T21:52:24.827 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.827 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.827 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.827 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.827 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.827 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.828 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.828 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.828 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.828 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.828 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.828 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.828 INFO:teuthology.orchestra.run.smithi001.stdout: "device_url": { 2024-04-26T21:52:24.828 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "device_url", 2024-04-26T21:52:24.829 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.829 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.829 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.829 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T21:52:24.829 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.829 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.829 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.829 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.829 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.830 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.830 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.830 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": { 2024-04-26T21:52:24.830 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enabled", 2024-04-26T21:52:24.830 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.830 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.830 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.830 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:24.831 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:24.832 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.832 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.832 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.832 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "24", 2024-04-26T21:52:24.832 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "8", 2024-04-26T21:52:24.832 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.832 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.832 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.833 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.833 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.833 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.833 INFO:teuthology.orchestra.run.smithi001.stdout: "last_opt_revision": { 2024-04-26T21:52:24.833 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "last_opt_revision", 2024-04-26T21:52:24.833 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.833 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.833 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.834 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "1", 2024-04-26T21:52:24.834 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.834 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.834 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.834 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.834 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.834 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.834 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.835 INFO:teuthology.orchestra.run.smithi001.stdout: "leaderboard": { 2024-04-26T21:52:24.835 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "leaderboard", 2024-04-26T21:52:24.835 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.835 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.835 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.835 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.835 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.835 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.835 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.836 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.836 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.836 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.836 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.836 INFO:teuthology.orchestra.run.smithi001.stdout: "leaderboard_description": { 2024-04-26T21:52:24.836 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "leaderboard_description", 2024-04-26T21:52:24.836 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.837 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.837 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.837 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.837 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.837 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.837 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.837 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.837 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.837 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.838 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.838 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.838 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.838 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.838 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.838 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.838 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.839 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.839 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.839 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.839 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.839 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.839 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.839 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.839 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.839 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.840 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.840 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.840 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.840 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.840 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.840 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.840 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.841 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.841 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.841 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.841 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.841 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.841 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.841 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.841 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.841 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.842 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.842 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.842 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.842 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.842 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.842 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.842 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.842 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.843 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.844 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.845 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: "organization": { 2024-04-26T21:52:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "organization", 2024-04-26T21:52:24.846 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.847 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: "proxy": { 2024-04-26T21:52:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "proxy", 2024-04-26T21:52:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.848 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.849 INFO:teuthology.orchestra.run.smithi001.stdout: "url": { 2024-04-26T21:52:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "url", 2024-04-26T21:52:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T21:52:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.850 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.851 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "test_orchestrator", 2024-04-26T21:52:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.852 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.853 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.854 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.855 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.856 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.856 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.856 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.856 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.856 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.856 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.856 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.856 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.857 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.857 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.857 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.857 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.857 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.857 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.857 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.857 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.857 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.858 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.858 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.858 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.858 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.858 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.858 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.858 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.859 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.859 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.859 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.859 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.859 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.859 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.859 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.859 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.859 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.860 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.860 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.860 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.860 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.860 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "volumes", 2024-04-26T21:52:24.860 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.860 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.860 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.861 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.861 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.861 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.861 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.861 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.861 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.861 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.861 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.861 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.862 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.862 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.862 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.862 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.862 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.862 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.862 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.862 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.863 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.863 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.863 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.863 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.863 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.863 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.863 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.863 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.864 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.864 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.864 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.864 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.864 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.864 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.864 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.864 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.865 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.865 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.865 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.865 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.865 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.865 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.865 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.865 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.866 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.866 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.866 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.866 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.866 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.866 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.866 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.866 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.867 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.867 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.867 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.867 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.867 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.867 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.867 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.868 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.868 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.868 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.868 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.868 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.868 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.868 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.868 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.869 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.869 INFO:teuthology.orchestra.run.smithi001.stdout: "max_concurrent_clones": { 2024-04-26T21:52:24.869 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_concurrent_clones", 2024-04-26T21:52:24.869 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.869 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.869 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.869 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "4", 2024-04-26T21:52:24.869 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.870 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.870 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.870 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T21:52:24.870 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.870 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.870 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.870 INFO:teuthology.orchestra.run.smithi001.stdout: "periodic_async_work": { 2024-04-26T21:52:24.870 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "periodic_async_work", 2024-04-26T21:52:24.871 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.871 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.871 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.871 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.871 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.871 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.871 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.871 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Periodically check for async work", 2024-04-26T21:52:24.871 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.872 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.872 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.872 INFO:teuthology.orchestra.run.smithi001.stdout: "snapshot_clone_delay": { 2024-04-26T21:52:24.872 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snapshot_clone_delay", 2024-04-26T21:52:24.872 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.872 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.872 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.872 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:24.873 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.873 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.873 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.873 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T21:52:24.873 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.873 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.873 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.873 INFO:teuthology.orchestra.run.smithi001.stdout: "snapshot_clone_no_wait": { 2024-04-26T21:52:24.874 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T21:52:24.874 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.874 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.874 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.874 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.874 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.874 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.874 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.874 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T21:52:24.875 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.875 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.875 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.875 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.875 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix", 2024-04-26T21:52:24.875 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.876 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.876 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.876 INFO:teuthology.orchestra.run.smithi001.stdout: "discovery_interval": { 2024-04-26T21:52:24.876 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "discovery_interval", 2024-04-26T21:52:24.876 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:24.876 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.876 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.876 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "100", 2024-04-26T21:52:24.877 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.877 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.877 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.877 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.877 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.877 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.877 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.877 INFO:teuthology.orchestra.run.smithi001.stdout: "identifier": { 2024-04-26T21:52:24.878 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "identifier", 2024-04-26T21:52:24.878 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.878 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.878 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.878 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.878 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.878 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.878 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.878 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.879 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.879 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.879 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.879 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:24.879 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:24.879 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.879 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.880 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.880 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:24.880 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.880 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.880 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.880 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.880 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.880 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.880 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.881 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.881 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.881 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.881 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.882 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.882 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.882 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.882 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.883 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.883 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.883 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.883 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.883 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.883 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.884 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.884 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.884 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.884 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.884 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.884 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.885 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.885 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.885 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.885 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.885 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.885 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.885 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.885 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.886 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.886 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.886 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.886 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.886 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.886 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.886 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.886 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.887 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.887 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.887 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.887 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.887 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.887 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.887 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.887 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.888 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.888 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.888 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.888 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.888 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.888 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.888 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.888 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.888 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.889 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.889 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.889 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.889 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.889 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.889 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.889 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.889 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.890 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.890 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.890 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.890 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.890 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.890 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.890 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.891 INFO:teuthology.orchestra.run.smithi001.stdout: "zabbix_host": { 2024-04-26T21:52:24.891 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix_host", 2024-04-26T21:52:24.891 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.891 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.891 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.891 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.891 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.891 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.892 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.892 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.892 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.892 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.892 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.892 INFO:teuthology.orchestra.run.smithi001.stdout: "zabbix_port": { 2024-04-26T21:52:24.893 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix_port", 2024-04-26T21:52:24.893 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.893 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.893 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.893 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10051", 2024-04-26T21:52:24.893 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.893 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.893 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.893 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.894 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.894 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.894 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.894 INFO:teuthology.orchestra.run.smithi001.stdout: "zabbix_sender": { 2024-04-26T21:52:24.894 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix_sender", 2024-04-26T21:52:24.894 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.894 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.895 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.895 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T21:52:24.895 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.895 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.895 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.895 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.896 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.896 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.896 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.896 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:24.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.896 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.897 INFO:teuthology.orchestra.run.smithi001.stdout: "modules": [ 2024-04-26T21:52:24.897 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm", 2024-04-26T21:52:24.897 INFO:teuthology.orchestra.run.smithi001.stdout: "dashboard", 2024-04-26T21:52:24.897 INFO:teuthology.orchestra.run.smithi001.stdout: "iostat", 2024-04-26T21:52:24.897 INFO:teuthology.orchestra.run.smithi001.stdout: "nfs", 2024-04-26T21:52:24.897 INFO:teuthology.orchestra.run.smithi001.stdout: "restful", 2024-04-26T21:52:24.897 INFO:teuthology.orchestra.run.smithi001.stdout: "snap_schedule" 2024-04-26T21:52:24.897 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.898 INFO:teuthology.orchestra.run.smithi001.stdout: "available_modules": [ 2024-04-26T21:52:24.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.898 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "alerts", 2024-04-26T21:52:24.898 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.898 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.898 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.898 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:24.898 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:24.899 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.899 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.899 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.899 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:24.899 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.899 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.899 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.899 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How frequently to reexamine health status", 2024-04-26T21:52:24.899 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.900 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.900 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.900 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.900 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.900 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.900 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.900 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.901 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.901 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.901 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.901 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.901 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.901 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.901 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.901 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.901 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.902 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.902 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.902 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.902 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.902 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.902 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.902 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.903 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.903 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.903 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.903 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.903 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.904 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.904 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.904 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.904 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.904 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.905 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.905 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.905 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.906 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.906 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.906 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.906 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.907 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.907 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.907 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.907 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.908 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.908 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.908 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.908 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.909 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.909 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.909 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.909 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.909 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.910 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.910 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.910 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.911 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.911 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.911 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.911 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.911 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.912 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.912 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.912 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.912 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.913 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.913 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.913 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.914 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_destination": { 2024-04-26T21:52:24.914 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_destination", 2024-04-26T21:52:24.914 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.917 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.918 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.918 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.918 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.918 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.919 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.919 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Email address to send alerts to", 2024-04-26T21:52:24.919 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.919 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.919 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.919 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_from_name": { 2024-04-26T21:52:24.919 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_from_name", 2024-04-26T21:52:24.919 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.920 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.920 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.920 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "Ceph", 2024-04-26T21:52:24.920 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.920 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.920 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.920 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Email From: name", 2024-04-26T21:52:24.920 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.920 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.921 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.921 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.921 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_host": { 2024-04-26T21:52:24.921 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_host", 2024-04-26T21:52:24.921 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.921 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.921 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.921 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.922 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.922 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.922 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.922 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SMTP server", 2024-04-26T21:52:24.922 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.923 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.923 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.923 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.923 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_password": { 2024-04-26T21:52:24.923 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_password", 2024-04-26T21:52:24.924 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.924 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.924 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.924 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.924 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.924 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.924 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.924 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Password to authenticate with", 2024-04-26T21:52:24.924 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.925 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.925 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.925 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.925 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_port": { 2024-04-26T21:52:24.925 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_port", 2024-04-26T21:52:24.925 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.925 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.925 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.926 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "465", 2024-04-26T21:52:24.926 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.926 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.926 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.926 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SMTP port", 2024-04-26T21:52:24.926 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.926 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.926 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.927 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.927 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_sender": { 2024-04-26T21:52:24.927 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_sender", 2024-04-26T21:52:24.927 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.927 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.927 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.927 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.927 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SMTP envelope sender", 2024-04-26T21:52:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.928 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_ssl": { 2024-04-26T21:52:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_ssl", 2024-04-26T21:52:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.929 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-26T21:52:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "smtp_user": { 2024-04-26T21:52:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "smtp_user", 2024-04-26T21:52:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.930 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.931 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.931 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.931 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.931 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.931 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.931 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "User to authenticate as", 2024-04-26T21:52:24.931 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.931 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.932 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.932 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "balancer", 2024-04-26T21:52:24.932 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.932 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.932 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.933 INFO:teuthology.orchestra.run.smithi001.stdout: "active": { 2024-04-26T21:52:24.933 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "active", 2024-04-26T21:52:24.933 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.933 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.933 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.933 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.934 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.934 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.934 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.934 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "automatically balance PGs across cluster", 2024-04-26T21:52:24.934 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.934 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.935 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.935 INFO:teuthology.orchestra.run.smithi001.stdout: "begin_time": { 2024-04-26T21:52:24.935 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "begin_time", 2024-04-26T21:52:24.935 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.936 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.936 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.936 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0000", 2024-04-26T21:52:24.936 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.936 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.936 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.937 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "beginning time of day to automatically balance", 2024-04-26T21:52:24.937 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T21:52:24.937 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.937 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.938 INFO:teuthology.orchestra.run.smithi001.stdout: "begin_weekday": { 2024-04-26T21:52:24.938 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "begin_weekday", 2024-04-26T21:52:24.938 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:24.938 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.938 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.938 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:24.938 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "0", 2024-04-26T21:52:24.939 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "6", 2024-04-26T21:52:24.939 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.939 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-26T21:52:24.939 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T21:52:24.939 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.939 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.939 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_compat_max_iterations": { 2024-04-26T21:52:24.939 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crush_compat_max_iterations", 2024-04-26T21:52:24.940 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:24.940 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.940 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.940 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "25", 2024-04-26T21:52:24.940 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:24.940 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "250", 2024-04-26T21:52:24.940 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.940 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-26T21:52:24.940 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.941 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.941 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.941 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_compat_metrics": { 2024-04-26T21:52:24.941 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crush_compat_metrics", 2024-04-26T21:52:24.941 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.941 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.941 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.941 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "pgs,objects,bytes", 2024-04-26T21:52:24.942 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.942 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.942 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.942 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-26T21:52:24.942 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-26T21:52:24.942 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.942 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.942 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.943 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_compat_step": { 2024-04-26T21:52:24.943 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crush_compat_step", 2024-04-26T21:52:24.943 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.943 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.943 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.943 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0.5", 2024-04-26T21:52:24.943 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "0.001", 2024-04-26T21:52:24.943 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "0.999", 2024-04-26T21:52:24.943 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.944 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "aggressiveness of optimization", 2024-04-26T21:52:24.944 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-26T21:52:24.944 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.944 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.944 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.944 INFO:teuthology.orchestra.run.smithi001.stdout: "end_time": { 2024-04-26T21:52:24.944 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "end_time", 2024-04-26T21:52:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "2359", 2024-04-26T21:52:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "ending time of day to automatically balance", 2024-04-26T21:52:24.945 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-26T21:52:24.946 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.946 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.946 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.946 INFO:teuthology.orchestra.run.smithi001.stdout: "end_weekday": { 2024-04-26T21:52:24.946 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "end_weekday", 2024-04-26T21:52:24.946 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:24.946 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.947 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.947 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:24.947 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "0", 2024-04-26T21:52:24.947 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "6", 2024-04-26T21:52:24.947 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.947 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-26T21:52:24.947 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-26T21:52:24.947 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.947 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.948 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:24.948 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:24.948 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.948 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.948 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.948 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.948 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.948 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.949 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.949 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.949 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.949 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.949 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.949 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.949 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.949 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.950 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.950 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.950 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.950 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.950 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:24.950 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:24.950 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.951 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.951 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.951 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.951 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.951 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.951 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.951 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.951 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.951 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.952 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.952 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:24.952 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:24.952 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.952 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.952 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.952 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:24.952 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.952 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.953 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.953 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:24.953 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:24.953 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:24.953 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:24.953 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:24.953 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:24.953 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.954 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.954 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.954 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.954 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.954 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:24.954 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:24.954 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.955 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.955 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.955 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.955 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.955 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.955 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.955 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:24.955 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.955 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.956 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.956 INFO:teuthology.orchestra.run.smithi001.stdout: "min_score": { 2024-04-26T21:52:24.956 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "min_score", 2024-04-26T21:52:24.956 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.956 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.956 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.956 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:24.956 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.957 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.957 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.957 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-26T21:52:24.957 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.957 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.957 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mode": { 2024-04-26T21:52:24.957 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mode", 2024-04-26T21:52:24.958 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.958 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.958 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.958 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "upmap", 2024-04-26T21:52:24.958 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.958 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.958 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.958 INFO:teuthology.orchestra.run.smithi001.stdout: "crush-compat", 2024-04-26T21:52:24.959 INFO:teuthology.orchestra.run.smithi001.stdout: "none", 2024-04-26T21:52:24.959 INFO:teuthology.orchestra.run.smithi001.stdout: "read", 2024-04-26T21:52:24.959 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap", 2024-04-26T21:52:24.959 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap-read" 2024-04-26T21:52:24.959 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.959 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Balancer mode", 2024-04-26T21:52:24.959 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.959 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.959 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.960 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_ids": { 2024-04-26T21:52:24.960 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pool_ids", 2024-04-26T21:52:24.960 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.960 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.960 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.960 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.960 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.960 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.961 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.961 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-26T21:52:24.961 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.961 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.961 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.962 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.962 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:24.962 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:24.962 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.962 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.963 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.963 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:24.963 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.963 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.963 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.964 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-26T21:52:24.964 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.964 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.964 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.964 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap_max_deviation": { 2024-04-26T21:52:24.964 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "upmap_max_deviation", 2024-04-26T21:52:24.964 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.965 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.965 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.965 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:24.965 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:24.965 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.965 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.965 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-26T21:52:24.965 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-26T21:52:24.966 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.966 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.966 INFO:teuthology.orchestra.run.smithi001.stdout: "upmap_max_optimizations": { 2024-04-26T21:52:24.966 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "upmap_max_optimizations", 2024-04-26T21:52:24.966 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:24.966 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.967 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.967 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:24.967 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.967 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.967 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.967 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-26T21:52:24.967 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.967 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.967 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:24.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:24.968 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephadm", 2024-04-26T21:52:24.968 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:24.968 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:24.968 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:24.968 INFO:teuthology.orchestra.run.smithi001.stdout: "agent_down_multiplier": { 2024-04-26T21:52:24.969 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "agent_down_multiplier", 2024-04-26T21:52:24.969 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.969 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.969 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.969 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "3.0", 2024-04-26T21:52:24.969 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.969 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.969 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.970 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-26T21:52:24.970 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.970 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.970 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.970 INFO:teuthology.orchestra.run.smithi001.stdout: "agent_refresh_rate": { 2024-04-26T21:52:24.970 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "agent_refresh_rate", 2024-04-26T21:52:24.970 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.970 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.971 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.971 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "20", 2024-04-26T21:52:24.971 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.971 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.971 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.971 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-26T21:52:24.971 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.971 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.971 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.972 INFO:teuthology.orchestra.run.smithi001.stdout: "agent_starting_port": { 2024-04-26T21:52:24.972 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "agent_starting_port", 2024-04-26T21:52:24.972 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:24.972 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.972 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.972 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "4721", 2024-04-26T21:52:24.972 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.973 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.973 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.973 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-26T21:52:24.973 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.973 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.973 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.973 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_ptrace": { 2024-04-26T21:52:24.973 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "allow_ptrace", 2024-04-26T21:52:24.974 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.974 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.974 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.974 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.974 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.974 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.974 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.974 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-26T21:52:24.975 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-26T21:52:24.975 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.975 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.975 INFO:teuthology.orchestra.run.smithi001.stdout: "autotune_interval": { 2024-04-26T21:52:24.975 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "autotune_interval", 2024-04-26T21:52:24.975 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:24.975 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.975 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.976 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:24.976 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.976 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.976 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.976 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-26T21:52:24.976 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.976 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.976 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: "autotune_memory_target_ratio": { 2024-04-26T21:52:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "autotune_memory_target_ratio", 2024-04-26T21:52:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0.7", 2024-04-26T21:52:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.977 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-26T21:52:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm_log_destination": { 2024-04-26T21:52:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cephadm_log_destination", 2024-04-26T21:52:24.978 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: "file", 2024-04-26T21:52:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: "file,syslog", 2024-04-26T21:52:24.979 INFO:teuthology.orchestra.run.smithi001.stdout: "syslog" 2024-04-26T21:52:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-26T21:52:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: "cgroups_split": { 2024-04-26T21:52:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cgroups_split", 2024-04-26T21:52:24.980 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.981 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.981 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.981 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.981 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.981 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.981 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.981 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-26T21:52:24.981 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.982 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.982 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.982 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.982 INFO:teuthology.orchestra.run.smithi001.stdout: "config_checks_enabled": { 2024-04-26T21:52:24.982 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "config_checks_enabled", 2024-04-26T21:52:24.982 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.982 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.983 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.983 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:24.983 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.983 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.983 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.984 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-26T21:52:24.984 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.984 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.984 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.985 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.985 INFO:teuthology.orchestra.run.smithi001.stdout: "config_dashboard": { 2024-04-26T21:52:24.985 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "config_dashboard", 2024-04-26T21:52:24.985 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:24.985 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.986 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.986 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:24.986 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.986 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.986 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.987 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-26T21:52:24.987 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.987 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.987 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.987 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.988 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_alertmanager": { 2024-04-26T21:52:24.988 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_alertmanager", 2024-04-26T21:52:24.988 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.988 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.988 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.989 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-26T21:52:24.989 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.989 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.989 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.989 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:24.989 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.989 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.990 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.990 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.990 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_base": { 2024-04-26T21:52:24.990 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_base", 2024-04-26T21:52:24.990 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.991 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.991 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:24.991 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-26T21:52:24.991 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.991 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.991 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.991 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Container image name, without the tag", 2024-04-26T21:52:24.991 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.991 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.992 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.992 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.992 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_elasticsearch": { 2024-04-26T21:52:24.992 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_elasticsearch", 2024-04-26T21:52:24.992 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.992 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.993 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.993 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-26T21:52:24.993 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.993 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.993 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.993 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "elasticsearch container image", 2024-04-26T21:52:24.994 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.994 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.994 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.994 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.994 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_grafana": { 2024-04-26T21:52:24.994 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_grafana", 2024-04-26T21:52:24.994 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.995 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.995 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.995 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-26T21:52:24.995 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.995 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.996 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.996 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:24.996 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.996 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.996 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.996 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.997 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_haproxy": { 2024-04-26T21:52:24.997 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_haproxy", 2024-04-26T21:52:24.997 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.997 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.997 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.997 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-26T21:52:24.997 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.997 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.998 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.998 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "HAproxy container image", 2024-04-26T21:52:24.998 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:24.998 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:24.998 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:24.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:24.998 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_jaeger_agent": { 2024-04-26T21:52:24.998 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_jaeger_agent", 2024-04-26T21:52:24.999 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:24.999 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:24.999 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:24.999 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-26T21:52:24.999 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:24.999 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:24.999 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:24.999 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Jaeger agent container image", 2024-04-26T21:52:25.000 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.000 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.000 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.000 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_jaeger_collector": { 2024-04-26T21:52:25.000 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_jaeger_collector", 2024-04-26T21:52:25.001 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.001 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.001 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.001 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-26T21:52:25.002 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.002 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.002 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.002 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Jaeger collector container image", 2024-04-26T21:52:25.002 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.003 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.003 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.003 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_jaeger_query": { 2024-04-26T21:52:25.003 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_jaeger_query", 2024-04-26T21:52:25.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.004 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.004 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.004 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-26T21:52:25.004 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.004 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.004 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.005 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Jaeger query container image", 2024-04-26T21:52:25.005 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.005 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.005 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.005 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_keepalived": { 2024-04-26T21:52:25.005 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_keepalived", 2024-04-26T21:52:25.005 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.005 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.005 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.006 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-26T21:52:25.006 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.006 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.006 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.006 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Keepalived container image", 2024-04-26T21:52:25.006 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.006 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.006 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.007 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_loki": { 2024-04-26T21:52:25.007 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_loki", 2024-04-26T21:52:25.007 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.007 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.007 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.007 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-26T21:52:25.007 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.008 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.008 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.008 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Loki container image", 2024-04-26T21:52:25.008 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.008 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.008 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.008 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_node_exporter": { 2024-04-26T21:52:25.008 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_node_exporter", 2024-04-26T21:52:25.009 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.009 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.009 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.009 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-26T21:52:25.009 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.009 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.009 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.009 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:25.009 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.010 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.010 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.010 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.010 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_nvmeof": { 2024-04-26T21:52:25.010 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_nvmeof", 2024-04-26T21:52:25.010 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.010 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.010 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.011 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-26T21:52:25.011 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.011 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.011 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.011 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Nvme-of container image", 2024-04-26T21:52:25.011 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.011 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.011 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.011 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.012 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_prometheus": { 2024-04-26T21:52:25.012 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_prometheus", 2024-04-26T21:52:25.012 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.012 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.013 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.013 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-26T21:52:25.013 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.013 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.014 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.014 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Prometheus container image", 2024-04-26T21:52:25.014 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.014 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.015 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.015 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_promtail": { 2024-04-26T21:52:25.016 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_promtail", 2024-04-26T21:52:25.016 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.016 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.016 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.017 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-26T21:52:25.017 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.017 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.017 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.018 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Promtail container image", 2024-04-26T21:52:25.018 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.018 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.018 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.019 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_samba": { 2024-04-26T21:52:25.019 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_samba", 2024-04-26T21:52:25.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.020 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.020 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.020 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-26T21:52:25.020 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.021 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.021 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.021 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Samba/SMB container image", 2024-04-26T21:52:25.021 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.021 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.022 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.022 INFO:teuthology.orchestra.run.smithi001.stdout: "container_image_snmp_gateway": { 2024-04-26T21:52:25.022 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_image_snmp_gateway", 2024-04-26T21:52:25.023 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.023 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.023 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.023 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-26T21:52:25.023 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.024 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.024 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.024 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "SNMP Gateway container image", 2024-04-26T21:52:25.024 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.025 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.025 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.025 INFO:teuthology.orchestra.run.smithi001.stdout: "container_init": { 2024-04-26T21:52:25.025 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "container_init", 2024-04-26T21:52:25.026 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.026 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.026 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.026 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.027 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.027 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.027 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.027 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Run podman/docker with `--init`", 2024-04-26T21:52:25.027 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.028 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.028 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.028 INFO:teuthology.orchestra.run.smithi001.stdout: "daemon_cache_timeout": { 2024-04-26T21:52:25.028 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "daemon_cache_timeout", 2024-04-26T21:52:25.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.029 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.029 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.029 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:25.029 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.030 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.030 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.030 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-26T21:52:25.030 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.031 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.031 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.031 INFO:teuthology.orchestra.run.smithi001.stdout: "default_cephadm_command_timeout": { 2024-04-26T21:52:25.031 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "default_cephadm_command_timeout", 2024-04-26T21:52:25.032 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.032 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.032 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.032 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "900", 2024-04-26T21:52:25.033 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.033 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.033 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.033 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-26T21:52:25.033 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.034 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.034 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.035 INFO:teuthology.orchestra.run.smithi001.stdout: "default_registry": { 2024-04-26T21:52:25.035 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "default_registry", 2024-04-26T21:52:25.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.035 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.036 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.036 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "docker.io", 2024-04-26T21:52:25.036 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.036 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.036 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.037 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-26T21:52:25.037 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.037 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.037 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.037 INFO:teuthology.orchestra.run.smithi001.stdout: "device_cache_timeout": { 2024-04-26T21:52:25.038 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "device_cache_timeout", 2024-04-26T21:52:25.038 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.038 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.038 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.038 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "1800", 2024-04-26T21:52:25.038 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.039 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.039 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.039 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "seconds to cache device inventory", 2024-04-26T21:52:25.039 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.039 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.040 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.040 INFO:teuthology.orchestra.run.smithi001.stdout: "device_enhanced_scan": { 2024-04-26T21:52:25.040 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "device_enhanced_scan", 2024-04-26T21:52:25.041 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.041 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.041 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.041 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.042 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.042 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.042 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.042 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-26T21:52:25.043 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.043 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.043 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.043 INFO:teuthology.orchestra.run.smithi001.stdout: "facts_cache_timeout": { 2024-04-26T21:52:25.043 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "facts_cache_timeout", 2024-04-26T21:52:25.043 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.043 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.043 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.043 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:25.044 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.044 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.044 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.044 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "seconds to cache host facts data", 2024-04-26T21:52:25.044 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.044 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.044 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.044 INFO:teuthology.orchestra.run.smithi001.stdout: "grafana_dashboards_path": { 2024-04-26T21:52:25.045 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "grafana_dashboards_path", 2024-04-26T21:52:25.045 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.045 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.045 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.045 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-26T21:52:25.045 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.045 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.045 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.045 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-26T21:52:25.046 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.046 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.046 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.046 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.046 INFO:teuthology.orchestra.run.smithi001.stdout: "host_check_interval": { 2024-04-26T21:52:25.046 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "host_check_interval", 2024-04-26T21:52:25.046 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.047 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.047 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.047 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:25.047 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.047 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.047 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.047 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to perform a host check", 2024-04-26T21:52:25.047 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.047 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.048 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.048 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.048 INFO:teuthology.orchestra.run.smithi001.stdout: "hw_monitoring": { 2024-04-26T21:52:25.048 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "hw_monitoring", 2024-04-26T21:52:25.048 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.048 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.048 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.048 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.048 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.049 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.049 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.049 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-26T21:52:25.049 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.049 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.049 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.049 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.049 INFO:teuthology.orchestra.run.smithi001.stdout: "inventory_list_all": { 2024-04-26T21:52:25.049 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "inventory_list_all", 2024-04-26T21:52:25.050 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.050 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.050 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.050 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.050 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.050 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.050 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.050 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-26T21:52:25.051 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.051 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.051 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.051 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.051 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.051 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.051 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.051 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.051 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.051 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.052 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.052 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.052 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.052 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.052 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.052 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.052 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.052 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.053 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.053 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.053 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.053 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.054 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.054 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.054 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.054 INFO:teuthology.orchestra.run.smithi001.stdout: "log_refresh_metadata": { 2024-04-26T21:52:25.054 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_refresh_metadata", 2024-04-26T21:52:25.054 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.054 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.054 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.055 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.055 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.055 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.055 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.055 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-26T21:52:25.055 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.055 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.055 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.055 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.055 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.056 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.056 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.056 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.056 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.056 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.056 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.056 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.056 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.056 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-26T21:52:25.057 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.057 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.057 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.057 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.057 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.057 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.057 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.057 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.057 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.058 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.058 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.058 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.058 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.058 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.058 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.058 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.058 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.059 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.059 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.059 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.059 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.059 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.059 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.059 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.059 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.059 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.059 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.060 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.060 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.060 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.060 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.060 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.060 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.060 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.060 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.060 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.061 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.061 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.061 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.061 INFO:teuthology.orchestra.run.smithi001.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-26T21:52:25.061 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-26T21:52:25.061 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.061 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.061 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.061 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.062 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.062 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.062 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.062 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-26T21:52:25.062 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.062 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.062 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.062 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.063 INFO:teuthology.orchestra.run.smithi001.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-26T21:52:25.063 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-26T21:52:25.063 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.063 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.063 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.063 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "*", 2024-04-26T21:52:25.063 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.063 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.063 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.064 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-26T21:52:25.064 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.064 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.064 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.064 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.065 INFO:teuthology.orchestra.run.smithi001.stdout: "max_count_per_host": { 2024-04-26T21:52:25.065 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_count_per_host", 2024-04-26T21:52:25.065 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.065 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.065 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.066 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:25.066 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.066 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.066 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.066 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "max number of daemons per service per host", 2024-04-26T21:52:25.066 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.066 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.066 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.067 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.067 INFO:teuthology.orchestra.run.smithi001.stdout: "max_osd_draining_count": { 2024-04-26T21:52:25.067 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_osd_draining_count", 2024-04-26T21:52:25.067 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.067 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.067 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.067 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:25.067 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.067 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.067 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-26T21:52:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: "migration_current": { 2024-04-26T21:52:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "migration_current", 2024-04-26T21:52:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.068 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.069 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.069 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.069 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.069 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.069 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.069 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "internal - do not modify", 2024-04-26T21:52:25.069 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.069 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.069 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.070 INFO:teuthology.orchestra.run.smithi001.stdout: "mode": { 2024-04-26T21:52:25.070 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mode", 2024-04-26T21:52:25.070 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.070 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.070 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.070 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "root", 2024-04-26T21:52:25.070 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.070 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.071 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.071 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm-package", 2024-04-26T21:52:25.071 INFO:teuthology.orchestra.run.smithi001.stdout: "root" 2024-04-26T21:52:25.071 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.071 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "mode for remote execution of cephadm", 2024-04-26T21:52:25.071 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.071 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.071 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.072 INFO:teuthology.orchestra.run.smithi001.stdout: "oob_default_addr": { 2024-04-26T21:52:25.072 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "oob_default_addr", 2024-04-26T21:52:25.072 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.072 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.072 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.072 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "169.254.1.1", 2024-04-26T21:52:25.072 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.072 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.072 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.072 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-26T21:52:25.073 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.073 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.073 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.073 INFO:teuthology.orchestra.run.smithi001.stdout: "prometheus_alerts_path": { 2024-04-26T21:52:25.073 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "prometheus_alerts_path", 2024-04-26T21:52:25.073 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.073 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.073 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.074 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-26T21:52:25.074 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.074 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.074 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.074 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-26T21:52:25.074 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.074 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.074 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.075 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_insecure": { 2024-04-26T21:52:25.075 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_insecure", 2024-04-26T21:52:25.075 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.075 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.075 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.075 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.076 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.076 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.076 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.076 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-26T21:52:25.077 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.077 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.077 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.078 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_password": { 2024-04-26T21:52:25.078 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_password", 2024-04-26T21:52:25.078 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.078 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.079 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.079 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.079 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.079 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.080 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.080 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-26T21:52:25.080 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.080 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.081 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.081 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.081 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_url": { 2024-04-26T21:52:25.082 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_url", 2024-04-26T21:52:25.082 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.082 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.082 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.083 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.083 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.083 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.083 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.084 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-26T21:52:25.084 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.084 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.084 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.085 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.085 INFO:teuthology.orchestra.run.smithi001.stdout: "registry_username": { 2024-04-26T21:52:25.085 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "registry_username", 2024-04-26T21:52:25.085 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.085 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.086 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.086 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.086 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.086 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.087 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.087 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-26T21:52:25.087 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.087 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.088 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.088 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.088 INFO:teuthology.orchestra.run.smithi001.stdout: "secure_monitoring_stack": { 2024-04-26T21:52:25.088 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "secure_monitoring_stack", 2024-04-26T21:52:25.088 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.089 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.089 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.089 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.089 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.089 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.090 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.090 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-26T21:52:25.090 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.090 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.091 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.091 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.091 INFO:teuthology.orchestra.run.smithi001.stdout: "service_discovery_port": { 2024-04-26T21:52:25.091 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "service_discovery_port", 2024-04-26T21:52:25.091 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.092 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.092 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.092 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8765", 2024-04-26T21:52:25.092 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.092 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.093 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.093 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "cephadm service discovery port", 2024-04-26T21:52:25.093 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.093 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.093 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.094 INFO:teuthology.orchestra.run.smithi001.stdout: "ssh_config_file": { 2024-04-26T21:52:25.094 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssh_config_file", 2024-04-26T21:52:25.094 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.095 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.095 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.095 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.095 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.095 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.096 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.096 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-26T21:52:25.096 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.096 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.097 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.097 INFO:teuthology.orchestra.run.smithi001.stdout: "use_agent": { 2024-04-26T21:52:25.097 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "use_agent", 2024-04-26T21:52:25.098 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.098 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.098 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.099 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.099 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.099 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.099 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.099 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-26T21:52:25.099 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.100 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.100 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.100 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.100 INFO:teuthology.orchestra.run.smithi001.stdout: "use_repo_digest": { 2024-04-26T21:52:25.100 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "use_repo_digest", 2024-04-26T21:52:25.101 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.101 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.101 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.101 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.101 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.102 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.102 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.102 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-26T21:52:25.103 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.103 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.104 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.104 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.104 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_on_failed_host_check": { 2024-04-26T21:52:25.104 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_on_failed_host_check", 2024-04-26T21:52:25.105 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.105 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.105 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.105 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.106 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.106 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.106 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.106 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise a health warning if the host check fails", 2024-04-26T21:52:25.106 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.106 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.107 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.107 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.107 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_on_stray_daemons": { 2024-04-26T21:52:25.107 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_on_stray_daemons", 2024-04-26T21:52:25.107 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.107 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.107 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.107 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.108 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.108 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.108 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.108 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-26T21:52:25.108 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.108 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.108 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.109 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.109 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_on_stray_hosts": { 2024-04-26T21:52:25.109 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_on_stray_hosts", 2024-04-26T21:52:25.109 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.109 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.109 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.109 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.110 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.110 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.110 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.110 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-26T21:52:25.110 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.110 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.110 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.110 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.111 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.111 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.111 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.111 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crash", 2024-04-26T21:52:25.111 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.111 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.111 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.111 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.112 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.112 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.112 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.112 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.112 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.112 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.112 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.112 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.113 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.113 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.113 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.113 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.113 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.113 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.113 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.113 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.113 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.114 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.114 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.114 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.114 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.114 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.114 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.114 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.114 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.115 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.115 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.115 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.115 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.115 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.115 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.115 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.115 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.115 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.115 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.116 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.116 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.116 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.116 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.117 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.117 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.117 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.117 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.117 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.117 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.117 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.117 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.118 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.118 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.118 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.118 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.118 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.118 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.119 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.119 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.119 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.119 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.119 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.119 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.119 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.120 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.120 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.122 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.122 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.122 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.122 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.122 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: "retain_interval": { 2024-04-26T21:52:25.123 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "retain_interval", 2024-04-26T21:52:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "31536000", 2024-04-26T21:52:25.124 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-26T21:52:25.125 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_recent_interval": { 2024-04-26T21:52:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_recent_interval", 2024-04-26T21:52:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.126 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "1209600", 2024-04-26T21:52:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.127 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-26T21:52:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "dashboard", 2024-04-26T21:52:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-26T21:52:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-26T21:52:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.129 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.130 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "ALERTMANAGER_API_HOST": { 2024-04-26T21:52:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-26T21:52:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.131 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-26T21:52:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-26T21:52:25.132 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.133 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "AUDIT_API_ENABLED": { 2024-04-26T21:52:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "AUDIT_API_ENABLED", 2024-04-26T21:52:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.134 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-26T21:52:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-26T21:52:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.136 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "ENABLE_BROWSABLE_API": { 2024-04-26T21:52:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-26T21:52:25.137 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.138 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-26T21:52:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-26T21:52:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.139 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.140 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-26T21:52:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-26T21:52:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.141 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-26T21:52:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-26T21:52:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.142 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-26T21:52:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-26T21:52:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.144 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.145 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.145 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.145 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.145 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.145 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.145 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-26T21:52:25.145 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-26T21:52:25.145 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.145 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.146 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.146 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.146 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.146 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.146 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.146 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.146 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.146 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.147 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.147 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.147 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-26T21:52:25.147 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-26T21:52:25.147 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.147 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.147 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.148 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.148 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.148 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.148 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.148 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.148 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.149 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.149 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.149 INFO:teuthology.orchestra.run.smithi001.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-26T21:52:25.149 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-26T21:52:25.149 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.149 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.149 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.149 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.150 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.150 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.150 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.150 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.150 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.150 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.150 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.151 INFO:teuthology.orchestra.run.smithi001.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-26T21:52:25.151 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-26T21:52:25.151 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.151 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.151 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.151 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.151 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.151 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.151 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.152 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.152 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.152 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.152 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.152 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_PASSWORD": { 2024-04-26T21:52:25.152 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-26T21:52:25.152 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.152 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.152 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.153 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "admin", 2024-04-26T21:52:25.153 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.153 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.153 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.153 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.153 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.153 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.153 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.154 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-26T21:52:25.154 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-26T21:52:25.154 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.154 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.154 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.154 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.154 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.154 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.154 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.155 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.155 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.155 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.155 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.155 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_URL": { 2024-04-26T21:52:25.155 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_URL", 2024-04-26T21:52:25.155 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.155 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.156 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.156 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.156 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.156 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.156 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.156 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.156 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.156 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.157 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.157 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_API_USERNAME": { 2024-04-26T21:52:25.157 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-26T21:52:25.157 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.157 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.157 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.157 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "admin", 2024-04-26T21:52:25.157 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.158 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.158 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.158 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.158 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.158 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.158 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.158 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.159 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-26T21:52:25.159 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-26T21:52:25.159 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.159 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.159 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.159 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.159 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.159 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.160 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.160 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.160 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.160 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.160 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.160 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.160 INFO:teuthology.orchestra.run.smithi001.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-26T21:52:25.160 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-26T21:52:25.160 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.161 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.161 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.161 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.161 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.161 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.161 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.161 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.161 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.162 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.162 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.162 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.162 INFO:teuthology.orchestra.run.smithi001.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-26T21:52:25.162 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-26T21:52:25.162 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.162 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.162 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.162 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.163 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.163 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.163 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.163 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.163 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.163 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.163 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.163 INFO:teuthology.orchestra.run.smithi001.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-26T21:52:25.164 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-26T21:52:25.164 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.164 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.164 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.164 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.164 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.164 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.164 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.164 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.165 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.165 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.165 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.165 INFO:teuthology.orchestra.run.smithi001.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-26T21:52:25.165 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-26T21:52:25.165 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.165 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.166 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.166 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "[]", 2024-04-26T21:52:25.166 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.166 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.166 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.166 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.166 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.166 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.167 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.167 INFO:teuthology.orchestra.run.smithi001.stdout: "MULTICLUSTER_CONFIG": { 2024-04-26T21:52:25.167 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-26T21:52:25.167 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.167 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.167 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.167 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "{}", 2024-04-26T21:52:25.167 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.167 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.168 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.168 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.168 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.168 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.168 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.168 INFO:teuthology.orchestra.run.smithi001.stdout: "PROMETHEUS_API_HOST": { 2024-04-26T21:52:25.168 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-26T21:52:25.169 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.169 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.169 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.169 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.169 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.169 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.169 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-26T21:52:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-26T21:52:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.170 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.171 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-26T21:52:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-26T21:52:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.172 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-26T21:52:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-26T21:52:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.173 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.174 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.175 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.175 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.175 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-26T21:52:25.175 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-26T21:52:25.175 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.175 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.175 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.175 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.175 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.176 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.176 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.176 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.176 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.176 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.176 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.176 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.176 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-26T21:52:25.177 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-26T21:52:25.177 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.177 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.177 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.177 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.177 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.177 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.177 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.177 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.177 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.178 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.178 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.178 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.178 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-26T21:52:25.178 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-26T21:52:25.178 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.178 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.178 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.179 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.179 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.179 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.179 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.179 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.179 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.179 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.179 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.180 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.180 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-26T21:52:25.180 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-26T21:52:25.180 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.180 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.180 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.180 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.180 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.181 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.181 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.181 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.181 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.181 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.181 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.181 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.181 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-26T21:52:25.181 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-26T21:52:25.181 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.182 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.182 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.182 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.182 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.182 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.182 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.182 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.182 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.183 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.183 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.183 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_ENABLED": { 2024-04-26T21:52:25.183 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-26T21:52:25.183 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.183 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.183 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.183 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.184 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.184 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.184 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.184 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.184 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.184 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.184 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.185 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-26T21:52:25.185 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-26T21:52:25.185 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.185 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.185 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.185 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-26T21:52:25.185 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.185 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.186 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.186 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.186 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.186 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.186 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.186 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-26T21:52:25.186 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-26T21:52:25.187 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.187 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.187 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.187 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:25.187 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.187 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.187 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.187 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.188 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.188 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.188 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.188 INFO:teuthology.orchestra.run.smithi001.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-26T21:52:25.188 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-26T21:52:25.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.188 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.188 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.189 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8", 2024-04-26T21:52:25.189 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.189 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.189 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.189 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.189 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.189 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.189 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-26T21:52:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-26T21:52:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "45", 2024-04-26T21:52:25.190 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.191 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.191 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.191 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.191 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.191 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.191 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.191 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_ACCESS_KEY": { 2024-04-26T21:52:25.192 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-26T21:52:25.192 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.192 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.192 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.192 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.192 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.192 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.192 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.192 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.193 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.193 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.193 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.193 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-26T21:52:25.193 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-26T21:52:25.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.193 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.193 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.194 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "admin", 2024-04-26T21:52:25.194 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.194 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.194 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.194 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.194 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.194 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.194 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.195 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_SECRET_KEY": { 2024-04-26T21:52:25.195 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-26T21:52:25.195 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.195 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.195 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.195 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.195 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.195 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.195 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: "RGW_API_SSL_VERIFY": { 2024-04-26T21:52:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-26T21:52:25.196 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.197 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.197 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.197 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.197 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.197 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.197 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.197 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.197 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.197 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.198 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.198 INFO:teuthology.orchestra.run.smithi001.stdout: "UNSAFE_TLS_v1_2": { 2024-04-26T21:52:25.198 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-26T21:52:25.198 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.198 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.198 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.198 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.199 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.199 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.199 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.199 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.199 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.199 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.199 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.199 INFO:teuthology.orchestra.run.smithi001.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-26T21:52:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-26T21:52:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-26T21:52:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-26T21:52:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-26T21:52:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-26T21:52:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "cross_origin_url": { 2024-04-26T21:52:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cross_origin_url", 2024-04-26T21:52:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "crt_file": { 2024-04-26T21:52:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "crt_file", 2024-04-26T21:52:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.214 INFO:teuthology.orchestra.run.smithi001.stdout: "debug": { 2024-04-26T21:52:25.214 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "debug", 2024-04-26T21:52:25.214 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.214 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.215 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.215 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.215 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.215 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.216 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.216 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Enable/disable debug options", 2024-04-26T21:52:25.216 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.216 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.217 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.217 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.217 INFO:teuthology.orchestra.run.smithi001.stdout: "jwt_token_ttl": { 2024-04-26T21:52:25.217 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "jwt_token_ttl", 2024-04-26T21:52:25.218 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.218 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.218 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.218 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "28800", 2024-04-26T21:52:25.219 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.219 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.219 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.219 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.220 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.220 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.220 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.220 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.221 INFO:teuthology.orchestra.run.smithi001.stdout: "key_file": { 2024-04-26T21:52:25.221 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "key_file", 2024-04-26T21:52:25.221 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.221 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.221 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.222 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.222 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.222 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.223 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.223 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.223 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.223 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.224 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.224 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.224 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.224 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.224 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.225 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.225 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.225 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.225 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.225 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.225 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.226 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.226 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.226 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.226 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.226 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.227 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.227 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.227 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.227 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.227 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.228 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.228 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.228 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.228 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.228 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.228 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.229 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.229 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.229 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.229 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.229 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.229 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.229 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.229 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.229 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.230 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.230 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.230 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.230 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.230 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.230 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.230 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.230 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.231 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.231 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.231 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.231 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.231 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.231 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.231 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.231 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.231 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.232 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.232 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.232 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.232 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.232 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.232 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.232 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.232 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.232 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.233 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.233 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.233 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.233 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.233 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.233 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.233 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.233 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.233 INFO:teuthology.orchestra.run.smithi001.stdout: "motd": { 2024-04-26T21:52:25.234 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "motd", 2024-04-26T21:52:25.234 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.234 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.234 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.234 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.234 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.234 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.234 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.235 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "The message of the day", 2024-04-26T21:52:25.235 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.235 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.235 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.235 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.235 INFO:teuthology.orchestra.run.smithi001.stdout: "redirect_resolve_ip_addr": { 2024-04-26T21:52:25.235 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "redirect_resolve_ip_addr", 2024-04-26T21:52:25.235 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.235 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.235 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.236 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.236 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.236 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.236 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.236 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.236 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.236 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.236 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.236 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.237 INFO:teuthology.orchestra.run.smithi001.stdout: "server_addr": { 2024-04-26T21:52:25.237 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_addr", 2024-04-26T21:52:25.237 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.237 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.237 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.237 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "::", 2024-04-26T21:52:25.237 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.237 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.237 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.238 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.238 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.238 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.238 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.238 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.238 INFO:teuthology.orchestra.run.smithi001.stdout: "server_port": { 2024-04-26T21:52:25.238 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_port", 2024-04-26T21:52:25.238 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.239 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.239 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.239 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8080", 2024-04-26T21:52:25.239 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.239 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.239 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.239 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.239 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.239 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.240 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.240 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.240 INFO:teuthology.orchestra.run.smithi001.stdout: "ssl": { 2024-04-26T21:52:25.240 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssl", 2024-04-26T21:52:25.240 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.240 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.240 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.240 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.240 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.241 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.241 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.241 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.241 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.241 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.241 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.241 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.241 INFO:teuthology.orchestra.run.smithi001.stdout: "ssl_server_port": { 2024-04-26T21:52:25.241 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssl_server_port", 2024-04-26T21:52:25.241 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.242 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.242 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.242 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8443", 2024-04-26T21:52:25.242 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.242 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.242 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.242 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.242 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.243 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.243 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.243 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.243 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_behaviour": { 2024-04-26T21:52:25.243 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_behaviour", 2024-04-26T21:52:25.243 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.243 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.243 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.243 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "redirect", 2024-04-26T21:52:25.244 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.244 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.244 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.244 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.244 INFO:teuthology.orchestra.run.smithi001.stdout: "redirect" 2024-04-26T21:52:25.244 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.244 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.245 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.245 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.245 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.245 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.245 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_error_status_code": { 2024-04-26T21:52:25.246 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_error_status_code", 2024-04-26T21:52:25.246 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.246 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.246 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.246 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "500", 2024-04-26T21:52:25.247 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "400", 2024-04-26T21:52:25.247 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "599", 2024-04-26T21:52:25.247 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.247 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.249 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.249 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.250 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.250 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.250 INFO:teuthology.orchestra.run.smithi001.stdout: "url_prefix": { 2024-04-26T21:52:25.250 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "url_prefix", 2024-04-26T21:52:25.251 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.251 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.251 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.251 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.251 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.252 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.252 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.252 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.252 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.252 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.253 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.253 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.253 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.253 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.254 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.254 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "devicehealth", 2024-04-26T21:52:25.254 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.254 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.255 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.255 INFO:teuthology.orchestra.run.smithi001.stdout: "enable_monitoring": { 2024-04-26T21:52:25.255 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enable_monitoring", 2024-04-26T21:52:25.255 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.256 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.256 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.256 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.256 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.256 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.256 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.256 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "monitor device health metrics", 2024-04-26T21:52:25.257 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.257 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.257 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.257 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.257 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.257 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.257 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.257 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.257 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.258 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.258 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.258 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.258 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.258 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.258 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.258 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.258 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.259 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.259 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.259 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.259 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.259 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.259 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.259 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.259 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.259 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.260 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.260 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.260 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.260 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.260 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.260 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.260 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.260 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.260 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.261 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.261 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.261 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.261 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.261 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.261 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.261 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.261 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.261 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.261 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.262 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.262 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.262 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.262 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.262 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.262 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.262 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.262 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.263 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.263 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.263 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.263 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.263 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.263 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.263 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.263 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.263 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.264 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.264 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.264 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.264 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.264 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.264 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.264 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.264 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.264 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.265 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.265 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.265 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.265 INFO:teuthology.orchestra.run.smithi001.stdout: "mark_out_threshold": { 2024-04-26T21:52:25.265 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mark_out_threshold", 2024-04-26T21:52:25.265 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.265 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.265 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.265 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "2419200", 2024-04-26T21:52:25.266 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.266 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.266 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.266 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-26T21:52:25.266 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.266 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.266 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.266 INFO:teuthology.orchestra.run.smithi001.stdout: "pool_name": { 2024-04-26T21:52:25.267 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pool_name", 2024-04-26T21:52:25.267 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.267 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.267 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.267 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "device_health_metrics", 2024-04-26T21:52:25.267 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.267 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.267 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.267 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-26T21:52:25.268 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.268 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.268 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.268 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.268 INFO:teuthology.orchestra.run.smithi001.stdout: "retention_period": { 2024-04-26T21:52:25.268 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "retention_period", 2024-04-26T21:52:25.268 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.268 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.268 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.269 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "15552000", 2024-04-26T21:52:25.269 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.269 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.269 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.269 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how long to retain device health metrics", 2024-04-26T21:52:25.269 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.270 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.270 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.270 INFO:teuthology.orchestra.run.smithi001.stdout: "scrape_frequency": { 2024-04-26T21:52:25.270 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "scrape_frequency", 2024-04-26T21:52:25.270 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.270 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.270 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.271 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "86400", 2024-04-26T21:52:25.271 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.271 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.271 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.271 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-26T21:52:25.271 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.271 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.271 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.272 INFO:teuthology.orchestra.run.smithi001.stdout: "self_heal": { 2024-04-26T21:52:25.272 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "self_heal", 2024-04-26T21:52:25.272 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.272 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.272 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.272 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.272 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.272 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.272 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.273 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-26T21:52:25.273 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.273 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.273 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.273 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:25.273 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:25.273 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.273 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.274 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.274 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:25.274 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.274 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.274 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.274 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how frequently to wake up and check device health", 2024-04-26T21:52:25.274 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.274 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.275 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.275 INFO:teuthology.orchestra.run.smithi001.stdout: "warn_threshold": { 2024-04-26T21:52:25.275 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "warn_threshold", 2024-04-26T21:52:25.275 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.275 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.275 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.275 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "7257600", 2024-04-26T21:52:25.275 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.276 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.276 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.276 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-26T21:52:25.276 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.276 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.276 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.276 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.276 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.276 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.277 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "diskprediction_local", 2024-04-26T21:52:25.277 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.277 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.277 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.277 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.277 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.277 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.277 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.277 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.277 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.278 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.278 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.278 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.278 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.278 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.278 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.278 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.278 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.279 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.280 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.280 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.280 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.280 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.280 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.280 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.280 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.280 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.280 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.280 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.281 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.281 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.281 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.281 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.281 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.281 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.281 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.281 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.282 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.282 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.282 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.282 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.282 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.282 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.282 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.282 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.283 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.283 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.283 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.283 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.283 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.283 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.283 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.283 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "predict_interval": { 2024-04-26T21:52:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "predict_interval", 2024-04-26T21:52:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "86400", 2024-04-26T21:52:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.285 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "predictor_model": { 2024-04-26T21:52:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "predictor_model", 2024-04-26T21:52:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "prophetstor", 2024-04-26T21:52:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.287 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "600", 2024-04-26T21:52:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.288 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.289 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "influx", 2024-04-26T21:52:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": false, 2024-04-26T21:52:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "influxdb python module not found", 2024-04-26T21:52:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "batch_size": { 2024-04-26T21:52:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "batch_size", 2024-04-26T21:52:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.290 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5000", 2024-04-26T21:52:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-26T21:52:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.291 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "database": { 2024-04-26T21:52:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "database", 2024-04-26T21:52:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "ceph", 2024-04-26T21:52:25.292 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-26T21:52:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": { 2024-04-26T21:52:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "hostname", 2024-04-26T21:52:25.293 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "InfluxDB server hostname", 2024-04-26T21:52:25.294 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.295 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "30", 2024-04-26T21:52:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "5", 2024-04-26T21:52:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-26T21:52:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.296 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.297 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.298 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.299 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.301 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.302 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.303 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.305 INFO:teuthology.orchestra.run.smithi001.stdout: "password": { 2024-04-26T21:52:25.305 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "password", 2024-04-26T21:52:25.305 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.305 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.305 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.305 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.305 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.305 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.305 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.306 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "password of InfluxDB server user", 2024-04-26T21:52:25.306 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.306 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.306 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.306 INFO:teuthology.orchestra.run.smithi001.stdout: "port": { 2024-04-26T21:52:25.306 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "port", 2024-04-26T21:52:25.306 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.306 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.307 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.307 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "8086", 2024-04-26T21:52:25.307 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.307 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.307 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.307 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "InfluxDB server port", 2024-04-26T21:52:25.307 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.307 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.307 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.308 INFO:teuthology.orchestra.run.smithi001.stdout: "ssl": { 2024-04-26T21:52:25.308 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ssl", 2024-04-26T21:52:25.308 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.308 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.308 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.308 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "false", 2024-04-26T21:52:25.308 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.308 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T21:52:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: "threads": { 2024-04-26T21:52:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "threads", 2024-04-26T21:52:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.309 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.310 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.310 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:25.310 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:25.310 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "32", 2024-04-26T21:52:25.310 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.310 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-26T21:52:25.310 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.310 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.311 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.311 INFO:teuthology.orchestra.run.smithi001.stdout: "username": { 2024-04-26T21:52:25.311 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "username", 2024-04-26T21:52:25.311 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.311 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.311 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.311 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.311 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.312 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.312 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.312 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "username of InfluxDB server user", 2024-04-26T21:52:25.312 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.312 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.312 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.312 INFO:teuthology.orchestra.run.smithi001.stdout: "verify_ssl": { 2024-04-26T21:52:25.312 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "verify_ssl", 2024-04-26T21:52:25.313 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.313 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.313 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.313 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "true", 2024-04-26T21:52:25.313 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.313 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.313 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.313 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-26T21:52:25.313 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.313 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.314 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.314 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.314 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.314 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "insights", 2024-04-26T21:52:25.314 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.314 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.314 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.314 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.315 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.315 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.315 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.315 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.315 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.315 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.315 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.315 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.315 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.316 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.316 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.316 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.316 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.316 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.316 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.316 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.316 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.316 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.316 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.317 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.317 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.317 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.317 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.317 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.317 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.317 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.317 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.318 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.318 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.318 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.318 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.318 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.318 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.318 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.318 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.319 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.319 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.319 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.319 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.319 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.319 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.319 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.319 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.319 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.320 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.320 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.320 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.320 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.320 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.320 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.320 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.320 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.320 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.321 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.321 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.321 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.321 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.321 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.321 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.321 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.321 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.321 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.321 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.322 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.322 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.322 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "iostat", 2024-04-26T21:52:25.322 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.323 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.323 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.323 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.323 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.323 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.323 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.323 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.324 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.324 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.324 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.324 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.324 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.324 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.324 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.324 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.324 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.325 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.325 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.325 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.325 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.325 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.325 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.325 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.325 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.326 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.326 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.326 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.326 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.326 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.326 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.326 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.326 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.327 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.327 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.327 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.327 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.327 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.327 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.327 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.327 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.327 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.328 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.328 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.328 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.328 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.328 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.328 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.328 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.328 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.328 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.329 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.329 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.329 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.329 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.329 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.329 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.329 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.329 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.330 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.330 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.330 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.330 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.330 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.330 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.330 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.330 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.331 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.331 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.331 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.331 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.331 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "k8sevents", 2024-04-26T21:52:25.331 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.332 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.332 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.332 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_event_retention_days": { 2024-04-26T21:52:25.332 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "ceph_event_retention_days", 2024-04-26T21:52:25.332 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.332 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.332 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.332 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "7", 2024-04-26T21:52:25.333 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.333 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.333 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.333 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-26T21:52:25.333 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.333 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.333 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.334 INFO:teuthology.orchestra.run.smithi001.stdout: "config_check_secs": { 2024-04-26T21:52:25.334 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "config_check_secs", 2024-04-26T21:52:25.334 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.334 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.334 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.334 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:25.334 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "10", 2024-04-26T21:52:25.334 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.335 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.335 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-26T21:52:25.335 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.335 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.335 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.335 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.335 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.335 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.336 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.336 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.336 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.336 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.336 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.336 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.336 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.336 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.336 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.336 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.337 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.337 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.337 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.337 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.337 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.337 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.337 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.337 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.338 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.338 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.338 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.338 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.338 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.338 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.338 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.339 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.339 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.339 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.339 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.339 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.339 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.339 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.339 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.339 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.340 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.340 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.340 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.340 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.340 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.340 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.340 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.340 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.341 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.342 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.342 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.342 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.342 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.342 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.342 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.342 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.342 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.343 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.343 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.343 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.343 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.343 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.343 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.343 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "localpool", 2024-04-26T21:52:25.344 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.344 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.344 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.344 INFO:teuthology.orchestra.run.smithi001.stdout: "failure_domain": { 2024-04-26T21:52:25.344 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "failure_domain", 2024-04-26T21:52:25.344 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.344 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.344 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.344 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "host", 2024-04-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "failure domain for any created local pool", 2024-04-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.346 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.346 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.346 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.346 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.346 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.346 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.346 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.347 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.347 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.347 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.347 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.347 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.347 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.347 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.347 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.348 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.348 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.348 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.348 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.348 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.348 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.348 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.348 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.348 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.349 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.349 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.349 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.349 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.349 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.349 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.349 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.349 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.349 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.350 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.350 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.350 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.350 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.350 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.350 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.350 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.350 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.351 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.351 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.351 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.351 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.351 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.351 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.351 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.351 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.351 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.352 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.352 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.352 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.352 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.352 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.353 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.353 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.353 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.353 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.353 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.353 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.353 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.353 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.353 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.354 INFO:teuthology.orchestra.run.smithi001.stdout: "min_size": { 2024-04-26T21:52:25.354 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "min_size", 2024-04-26T21:52:25.354 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.354 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.354 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.354 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.355 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.355 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.355 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.355 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "default min_size for any created local pool", 2024-04-26T21:52:25.355 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-26T21:52:25.355 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.355 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.355 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.355 INFO:teuthology.orchestra.run.smithi001.stdout: "num_rep": { 2024-04-26T21:52:25.356 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "num_rep", 2024-04-26T21:52:25.356 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.356 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.356 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.356 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "3", 2024-04-26T21:52:25.356 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.356 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.356 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.356 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "default replica count for any created local pool", 2024-04-26T21:52:25.357 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.357 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.357 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.357 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_num": { 2024-04-26T21:52:25.357 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pg_num", 2024-04-26T21:52:25.357 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.358 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.358 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.358 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "128", 2024-04-26T21:52:25.358 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.358 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.358 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.359 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "default pg_num for any created local pool", 2024-04-26T21:52:25.359 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.359 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.359 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.359 INFO:teuthology.orchestra.run.smithi001.stdout: "prefix": { 2024-04-26T21:52:25.359 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "prefix", 2024-04-26T21:52:25.359 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.359 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.360 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.360 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.360 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.360 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.360 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.360 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "name prefix for any created local pool", 2024-04-26T21:52:25.360 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.360 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.360 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.361 INFO:teuthology.orchestra.run.smithi001.stdout: "subtree": { 2024-04-26T21:52:25.361 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "subtree", 2024-04-26T21:52:25.361 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.361 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.361 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.361 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "rack", 2024-04-26T21:52:25.361 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.361 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.361 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.361 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-26T21:52:25.362 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-26T21:52:25.362 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.362 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.362 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mds_autoscaler", 2024-04-26T21:52:25.363 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.363 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.363 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.363 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.363 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.363 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.363 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.363 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.363 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.363 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.364 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.365 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.365 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.365 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.365 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.365 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.365 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.365 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.365 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.365 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.366 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.366 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.366 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.366 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.366 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.366 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.366 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.366 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mirroring", 2024-04-26T21:52:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.380 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.380 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.380 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.380 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.380 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.380 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.380 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.381 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "nfs", 2024-04-26T21:52:25.381 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.381 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.381 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.381 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.381 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.381 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.381 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.382 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.382 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.382 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.382 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.382 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.382 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.382 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.382 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.382 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.383 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.383 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.383 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.383 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.383 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.383 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.383 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.383 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.384 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.384 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.384 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.384 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.384 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.384 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.384 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.384 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.384 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.385 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.385 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.385 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.385 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.385 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.385 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.385 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.385 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.386 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.386 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.386 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.386 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.386 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.386 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.386 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.386 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.386 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.387 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.387 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.387 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.387 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.387 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.387 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.387 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.387 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.387 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.388 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.388 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.388 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.388 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.388 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.388 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.388 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.388 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.388 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.389 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.389 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.389 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.389 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "orchestrator", 2024-04-26T21:52:25.390 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.390 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.390 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.390 INFO:teuthology.orchestra.run.smithi001.stdout: "fail_fs": { 2024-04-26T21:52:25.390 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "fail_fs", 2024-04-26T21:52:25.390 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.390 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.390 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.390 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.391 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.391 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.391 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.391 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-26T21:52:25.391 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.391 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.391 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.391 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.392 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.392 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.392 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.392 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.392 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.392 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.392 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.392 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.393 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.393 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.393 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.393 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.393 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.393 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.393 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.393 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.393 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.393 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.394 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.394 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.394 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.394 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.394 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.394 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.394 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.394 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.394 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.395 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.395 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.395 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.395 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.395 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.395 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.395 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.395 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.395 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.396 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.396 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.396 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.396 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.396 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.396 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.396 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.396 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.396 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.397 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.397 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.397 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.397 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.397 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.397 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.397 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.397 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.397 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.398 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.398 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.398 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.398 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.398 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.398 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.398 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.398 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.399 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.399 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.399 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.399 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.399 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.399 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.400 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.400 INFO:teuthology.orchestra.run.smithi001.stdout: "orchestrator": { 2024-04-26T21:52:25.400 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "orchestrator", 2024-04-26T21:52:25.400 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.400 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.400 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.400 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.400 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.400 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.401 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.401 INFO:teuthology.orchestra.run.smithi001.stdout: "cephadm", 2024-04-26T21:52:25.401 INFO:teuthology.orchestra.run.smithi001.stdout: "rook", 2024-04-26T21:52:25.401 INFO:teuthology.orchestra.run.smithi001.stdout: "test_orchestrator" 2024-04-26T21:52:25.401 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.401 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Orchestrator backend", 2024-04-26T21:52:25.401 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.401 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.401 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.401 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.402 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.402 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "osd_perf_query", 2024-04-26T21:52:25.402 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.402 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.402 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.402 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.403 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.403 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.403 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.403 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.403 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.403 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.403 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.403 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.403 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.404 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.404 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.404 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.404 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.404 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.404 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.404 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.404 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.404 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.405 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.405 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.405 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.405 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.405 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.405 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.405 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.405 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.406 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.406 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.406 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.406 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.406 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.406 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.406 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.406 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.407 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.407 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.407 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.407 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.407 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.407 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.407 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.407 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.408 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.408 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.408 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.408 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.408 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.408 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.408 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.408 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.409 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.410 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.410 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.410 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.410 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.410 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.410 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.411 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.411 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.411 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.411 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "osd_support", 2024-04-26T21:52:25.411 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.411 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.411 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.411 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.412 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.412 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.412 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.412 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.412 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.412 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.412 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.412 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.412 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.413 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.413 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.413 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.413 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.413 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.413 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.413 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.413 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.413 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.413 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.414 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.414 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.414 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.414 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.414 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.414 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.414 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.414 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.415 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.415 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.415 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.415 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.415 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.415 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.415 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.415 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.416 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.416 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.416 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.416 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.416 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.416 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.416 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.416 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.416 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.417 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.417 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.417 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.417 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.417 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.417 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.417 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.417 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.417 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.418 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.418 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.418 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.418 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.418 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.418 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.419 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "pg_autoscaler", 2024-04-26T21:52:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.421 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.422 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.423 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.424 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.425 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.426 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.427 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.428 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.429 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.430 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "threshold": { 2024-04-26T21:52:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "threshold", 2024-04-26T21:52:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.431 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "3.0", 2024-04-26T21:52:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1.0", 2024-04-26T21:52:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "scaling threshold", 2024-04-26T21:52:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-26T21:52:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.432 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "progress", 2024-04-26T21:52:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_pg_recovery_event": { 2024-04-26T21:52:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "allow_pg_recovery_event", 2024-04-26T21:52:25.433 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.434 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-26T21:52:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": { 2024-04-26T21:52:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enabled", 2024-04-26T21:52:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.435 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.436 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.437 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.438 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.439 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.439 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.439 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.439 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.439 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.439 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.439 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.440 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.440 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.440 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.440 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.440 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.440 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.440 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.440 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.440 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.440 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.441 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.441 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.441 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.441 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.441 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.441 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.442 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.442 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.442 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.442 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.442 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.442 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.442 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.443 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.443 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.443 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.443 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.443 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.443 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.443 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.443 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.444 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.444 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.444 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.444 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.444 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.444 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.444 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.444 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.445 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.445 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.445 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.445 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.445 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.445 INFO:teuthology.orchestra.run.smithi001.stdout: "max_completed_events": { 2024-04-26T21:52:25.445 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_completed_events", 2024-04-26T21:52:25.446 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.446 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.446 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.446 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "50", 2024-04-26T21:52:25.447 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.447 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.447 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.447 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "number of past completed events to remember", 2024-04-26T21:52:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "sleep_interval": { 2024-04-26T21:52:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "sleep_interval", 2024-04-26T21:52:25.448 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "5", 2024-04-26T21:52:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "how long the module is going to sleep", 2024-04-26T21:52:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.449 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "prometheus", 2024-04-26T21:52:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.450 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "cache": { 2024-04-26T21:52:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "cache", 2024-04-26T21:52:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.451 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "exclude_perf_counters": { 2024-04-26T21:52:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "exclude_perf_counters", 2024-04-26T21:52:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.453 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-26T21:52:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-26T21:52:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.455 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.457 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.458 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.459 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.461 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.462 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_stats_pools": { 2024-04-26T21:52:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rbd_stats_pools", 2024-04-26T21:52:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.463 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-26T21:52:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-26T21:52:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.464 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "300", 2024-04-26T21:52:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.465 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "scrape_interval": { 2024-04-26T21:52:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "scrape_interval", 2024-04-26T21:52:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "15.0", 2024-04-26T21:52:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.466 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "server_addr": { 2024-04-26T21:52:25.467 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_addr", 2024-04-26T21:52:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "::", 2024-04-26T21:52:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-26T21:52:25.468 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.469 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.469 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.469 INFO:teuthology.orchestra.run.smithi001.stdout: "server_port": { 2024-04-26T21:52:25.469 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_port", 2024-04-26T21:52:25.469 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.469 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.469 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.469 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "9283", 2024-04-26T21:52:25.470 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.470 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.470 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.470 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-26T21:52:25.470 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.470 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.470 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: "stale_cache_strategy": { 2024-04-26T21:52:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "stale_cache_strategy", 2024-04-26T21:52:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "log", 2024-04-26T21:52:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.472 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.472 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.472 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.472 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.472 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_behaviour": { 2024-04-26T21:52:25.472 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_behaviour", 2024-04-26T21:52:25.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.472 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.473 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.473 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "default", 2024-04-26T21:52:25.473 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.473 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.473 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.473 INFO:teuthology.orchestra.run.smithi001.stdout: "default", 2024-04-26T21:52:25.473 INFO:teuthology.orchestra.run.smithi001.stdout: "error" 2024-04-26T21:52:25.474 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.474 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.474 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.474 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.474 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.474 INFO:teuthology.orchestra.run.smithi001.stdout: "standby_error_status_code": { 2024-04-26T21:52:25.474 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "standby_error_status_code", 2024-04-26T21:52:25.474 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.475 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.475 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.475 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "500", 2024-04-26T21:52:25.475 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "400", 2024-04-26T21:52:25.475 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "599", 2024-04-26T21:52:25.475 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.475 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.475 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.475 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.475 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.476 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rbd_support", 2024-04-26T21:52:25.476 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.476 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.476 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.476 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.477 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.477 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.477 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.477 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.477 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.477 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.477 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.477 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.477 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.478 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.478 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.478 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.478 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.478 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.478 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.478 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.478 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.478 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.479 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.479 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.479 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.479 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.479 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.479 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.479 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.479 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.480 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.480 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.480 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.480 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.480 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.480 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.480 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.480 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.480 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.481 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.481 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.481 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.481 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.481 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.481 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.481 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.481 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.481 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.482 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.482 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.482 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.482 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.482 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.482 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.482 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.482 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.483 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.483 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.483 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.483 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.483 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.483 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.484 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.484 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.484 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.484 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.484 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.484 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.484 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.484 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.485 INFO:teuthology.orchestra.run.smithi001.stdout: "max_concurrent_snap_create": { 2024-04-26T21:52:25.485 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_concurrent_snap_create", 2024-04-26T21:52:25.485 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.485 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.485 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.485 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10", 2024-04-26T21:52:25.485 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.485 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.485 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.486 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.486 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.486 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.486 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.486 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.486 INFO:teuthology.orchestra.run.smithi001.stdout: "mirror_snapshot_schedule": { 2024-04-26T21:52:25.486 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "mirror_snapshot_schedule", 2024-04-26T21:52:25.486 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.487 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.487 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.487 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.487 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.487 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.487 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.487 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.487 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.488 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.488 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.488 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.488 INFO:teuthology.orchestra.run.smithi001.stdout: "trash_purge_schedule": { 2024-04-26T21:52:25.488 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "trash_purge_schedule", 2024-04-26T21:52:25.488 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.488 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.488 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.488 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.489 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.489 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.489 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.489 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.489 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.489 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.489 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.489 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.490 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "restful", 2024-04-26T21:52:25.490 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.490 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.490 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.490 INFO:teuthology.orchestra.run.smithi001.stdout: "enable_auth": { 2024-04-26T21:52:25.491 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enable_auth", 2024-04-26T21:52:25.491 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.491 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.491 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.491 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.491 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.491 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.491 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.492 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.492 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.492 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.492 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.492 INFO:teuthology.orchestra.run.smithi001.stdout: "key_file": { 2024-04-26T21:52:25.492 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "key_file", 2024-04-26T21:52:25.492 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.492 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.493 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.493 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.493 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.493 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.493 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.493 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.493 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.494 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.494 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.494 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.494 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.494 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.494 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.495 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.495 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.495 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.495 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.495 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.495 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.495 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.495 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.496 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.496 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.496 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.496 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.496 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.496 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.496 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.496 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.497 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.497 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.497 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.497 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.497 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.497 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.497 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.497 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.497 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.498 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.498 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.498 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.498 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.498 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.498 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.498 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.498 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.499 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.499 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.499 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.499 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.499 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.499 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.499 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.499 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.500 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.500 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.500 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.500 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.500 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.500 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.500 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.501 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.501 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.501 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.501 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.501 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.501 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.501 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.501 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.501 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.501 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.502 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.502 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.502 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.502 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.502 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.502 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.502 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.502 INFO:teuthology.orchestra.run.smithi001.stdout: "server_addr": { 2024-04-26T21:52:25.502 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_addr", 2024-04-26T21:52:25.503 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.503 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.503 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.503 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.503 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.503 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.503 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.503 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.504 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.504 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.504 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.504 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.504 INFO:teuthology.orchestra.run.smithi001.stdout: "server_port": { 2024-04-26T21:52:25.504 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "server_port", 2024-04-26T21:52:25.504 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.505 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.505 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.505 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.505 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.505 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.505 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.505 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.505 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.505 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.505 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.506 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.506 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.506 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.506 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.506 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rgw", 2024-04-26T21:52:25.506 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.506 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.506 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.506 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.507 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.508 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.508 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.508 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.508 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.508 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.508 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.508 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.509 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.509 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.509 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.509 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.509 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.509 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.509 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.509 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.509 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.510 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.510 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.510 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.510 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.510 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.510 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.510 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.510 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.510 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.511 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.511 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.511 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.511 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.511 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.511 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.511 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.511 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.511 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.511 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.512 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.513 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.514 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.514 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.514 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.514 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.514 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.514 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.514 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.515 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.515 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rook", 2024-04-26T21:52:25.515 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.515 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.515 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.515 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.515 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.516 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.516 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.516 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.516 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.516 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.516 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.516 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.516 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.516 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.517 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.517 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.517 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.517 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.517 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.517 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.517 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.517 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.517 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.518 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.518 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.518 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.518 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.518 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.518 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.518 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.519 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.519 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.519 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.519 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.519 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.519 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.519 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.519 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.520 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.520 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.520 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.520 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.520 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.520 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.520 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.520 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.521 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.521 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.521 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.521 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.521 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.521 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.521 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.521 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.521 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.522 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.522 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.522 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.522 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.522 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.522 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.522 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.523 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.523 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.523 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.523 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.523 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.523 INFO:teuthology.orchestra.run.smithi001.stdout: "storage_class": { 2024-04-26T21:52:25.523 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "storage_class", 2024-04-26T21:52:25.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.523 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.524 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.524 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "local", 2024-04-26T21:52:25.524 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.524 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.524 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.524 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-26T21:52:25.524 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.524 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.525 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.526 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "selftest", 2024-04-26T21:52:25.526 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.526 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.526 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.526 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.526 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.526 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.526 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.527 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.527 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.527 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.527 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.527 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.527 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.527 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.527 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.527 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.528 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.528 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.528 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.528 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.528 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.528 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.528 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.528 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.528 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.529 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.529 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.529 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.529 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.529 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.529 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.529 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.529 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.529 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.530 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.530 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.530 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.530 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.530 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.530 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.530 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.530 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.531 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.531 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.531 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.531 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.531 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.531 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.531 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.531 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.531 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.532 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.532 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.532 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.532 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.532 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.532 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.532 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.532 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.533 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.533 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.533 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.533 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.533 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.533 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.533 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.533 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.533 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.534 INFO:teuthology.orchestra.run.smithi001.stdout: "roption1": { 2024-04-26T21:52:25.534 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "roption1", 2024-04-26T21:52:25.534 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.534 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.534 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.534 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.534 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.534 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.534 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.535 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.535 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.538 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.538 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.539 INFO:teuthology.orchestra.run.smithi001.stdout: "roption2": { 2024-04-26T21:52:25.539 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "roption2", 2024-04-26T21:52:25.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.539 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.539 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.539 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "xyz", 2024-04-26T21:52:25.539 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption1": { 2024-04-26T21:52:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption1", 2024-04-26T21:52:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption2": { 2024-04-26T21:52:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption2", 2024-04-26T21:52:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption3": { 2024-04-26T21:52:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption3", 2024-04-26T21:52:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "float", 2024-04-26T21:52:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption4": { 2024-04-26T21:52:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption4", 2024-04-26T21:52:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption5": { 2024-04-26T21:52:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption5", 2024-04-26T21:52:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption6": { 2024-04-26T21:52:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption6", 2024-04-26T21:52:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "rwoption7": { 2024-04-26T21:52:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rwoption7", 2024-04-26T21:52:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "1", 2024-04-26T21:52:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "42", 2024-04-26T21:52:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "testkey": { 2024-04-26T21:52:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "testkey", 2024-04-26T21:52:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.554 INFO:teuthology.orchestra.run.smithi001.stdout: "testlkey": { 2024-04-26T21:52:25.554 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "testlkey", 2024-04-26T21:52:25.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.554 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.554 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.554 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.554 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.555 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.555 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.555 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.555 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.555 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.555 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.555 INFO:teuthology.orchestra.run.smithi001.stdout: "testnewline": { 2024-04-26T21:52:25.555 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "testnewline", 2024-04-26T21:52:25.556 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.556 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.556 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.556 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.556 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.556 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.556 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.556 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.556 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.557 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.557 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.557 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snap_schedule", 2024-04-26T21:52:25.557 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.557 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.558 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.558 INFO:teuthology.orchestra.run.smithi001.stdout: "allow_m_granularity": { 2024-04-26T21:52:25.558 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "allow_m_granularity", 2024-04-26T21:52:25.558 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.558 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.558 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.558 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.558 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.558 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.559 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.559 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "allow minute scheduled snapshots", 2024-04-26T21:52:25.559 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.559 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.559 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.559 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.560 INFO:teuthology.orchestra.run.smithi001.stdout: "dump_on_update": { 2024-04-26T21:52:25.560 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "dump_on_update", 2024-04-26T21:52:25.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.560 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.560 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.560 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.560 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.561 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.561 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.561 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "dump database to debug log on update", 2024-04-26T21:52:25.561 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.561 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.561 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.561 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.561 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.562 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.562 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.562 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.562 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.562 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.562 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.562 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.562 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.563 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.563 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.563 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.563 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.563 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.563 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.563 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.563 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.564 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.564 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.564 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.564 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.564 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.564 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.565 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.565 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.565 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.565 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.565 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.565 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.565 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.565 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.566 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.566 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.566 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.566 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.566 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.566 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.566 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.566 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.567 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.567 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.567 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.567 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.567 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.567 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.567 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.568 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.568 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.568 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.568 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.568 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.568 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.568 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.568 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.569 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.569 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.569 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.569 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.569 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.569 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.569 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.569 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.569 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.570 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "stats", 2024-04-26T21:52:25.570 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.570 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.571 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.572 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.572 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.572 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.572 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.572 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.572 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.573 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.573 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.573 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.573 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.573 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.573 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.573 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.574 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.575 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.576 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.576 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.576 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.576 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.576 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.576 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.576 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.576 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.576 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.577 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.577 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.577 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.577 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.577 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.577 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.577 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.578 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.578 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.578 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.578 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.578 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.578 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.578 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.578 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.579 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.579 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.579 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.579 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.579 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.579 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "status", 2024-04-26T21:52:25.580 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.580 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.580 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.580 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.580 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.580 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.581 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.582 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.583 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.584 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.585 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.586 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.587 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.588 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "telegraf", 2024-04-26T21:52:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.589 INFO:teuthology.orchestra.run.smithi001.stdout: "address": { 2024-04-26T21:52:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "address", 2024-04-26T21:52:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-26T21:52:25.590 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.591 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "15", 2024-04-26T21:52:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.592 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.594 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.595 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.596 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.597 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.598 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.599 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.600 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.601 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.601 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.601 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.601 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.601 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.601 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.602 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.602 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.602 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.602 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.602 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.602 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "telemetry", 2024-04-26T21:52:25.602 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.603 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.603 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.603 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_basic": { 2024-04-26T21:52:25.603 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_basic", 2024-04-26T21:52:25.603 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.603 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.603 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.603 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.603 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.604 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.604 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.604 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-26T21:52:25.604 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.604 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.604 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.604 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.604 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_crash": { 2024-04-26T21:52:25.604 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_crash", 2024-04-26T21:52:25.605 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.605 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.605 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.605 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.605 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.605 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.605 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.605 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-26T21:52:25.605 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.606 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.606 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.606 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_device": { 2024-04-26T21:52:25.606 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_device", 2024-04-26T21:52:25.606 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.606 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.606 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.607 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.607 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.607 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.607 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.607 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-26T21:52:25.607 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.607 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.607 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.608 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_ident": { 2024-04-26T21:52:25.608 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_ident", 2024-04-26T21:52:25.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.608 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.608 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.608 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.608 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.608 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.608 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.609 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-26T21:52:25.609 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.609 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.609 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.609 INFO:teuthology.orchestra.run.smithi001.stdout: "channel_perf": { 2024-04-26T21:52:25.609 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "channel_perf", 2024-04-26T21:52:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.610 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-26T21:52:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: "contact": { 2024-04-26T21:52:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "contact", 2024-04-26T21:52:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.611 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.612 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.612 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.612 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.612 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.612 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.612 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "description": { 2024-04-26T21:52:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "description", 2024-04-26T21:52:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.613 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.614 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "device_url": { 2024-04-26T21:52:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "device_url", 2024-04-26T21:52:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-26T21:52:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.615 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "enabled": { 2024-04-26T21:52:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "enabled", 2024-04-26T21:52:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.616 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.617 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "24", 2024-04-26T21:52:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "8", 2024-04-26T21:52:25.618 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "last_opt_revision": { 2024-04-26T21:52:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "last_opt_revision", 2024-04-26T21:52:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.619 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "1", 2024-04-26T21:52:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.620 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "leaderboard": { 2024-04-26T21:52:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "leaderboard", 2024-04-26T21:52:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.621 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "leaderboard_description": { 2024-04-26T21:52:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "leaderboard_description", 2024-04-26T21:52:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.623 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.624 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.625 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.626 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.627 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.628 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.629 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.631 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "organization": { 2024-04-26T21:52:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "organization", 2024-04-26T21:52:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.632 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.633 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.633 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.633 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.633 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.633 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.633 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.633 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.634 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.634 INFO:teuthology.orchestra.run.smithi001.stdout: "proxy": { 2024-04-26T21:52:25.634 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "proxy", 2024-04-26T21:52:25.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.634 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.634 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.634 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.635 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.635 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.635 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.635 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.635 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.635 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.635 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.635 INFO:teuthology.orchestra.run.smithi001.stdout: "url": { 2024-04-26T21:52:25.635 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "url", 2024-04-26T21:52:25.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.636 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.636 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.636 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-26T21:52:25.636 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.636 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.636 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.636 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "test_orchestrator", 2024-04-26T21:52:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.637 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.638 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.638 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.638 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.638 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.638 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.638 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.638 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.639 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.639 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.639 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.639 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.639 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.639 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.639 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.639 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.640 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.640 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.640 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.640 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.640 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.640 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.640 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.641 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.641 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.641 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.641 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.641 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.641 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.641 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.641 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.641 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.641 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.642 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.642 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.642 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.642 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.642 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.642 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.643 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.643 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.643 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.643 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.643 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.643 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.644 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.644 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.644 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.644 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.644 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.644 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.644 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.644 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.645 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.645 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.645 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.645 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.645 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.645 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.645 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.645 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.645 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.646 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.646 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.646 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.646 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "volumes", 2024-04-26T21:52:25.647 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.647 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.647 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.647 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.647 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.647 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.648 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.648 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.648 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.648 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.648 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.648 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.648 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.648 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.648 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.649 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.649 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.649 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.649 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.649 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.649 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.649 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.649 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.650 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.650 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.650 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.650 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.650 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.650 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.650 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.650 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.650 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.651 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.651 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.651 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.651 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.651 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.651 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.651 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.652 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.652 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.652 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.652 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.652 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.652 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.652 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.652 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.653 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.653 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.653 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.653 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.653 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.653 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.653 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.653 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.654 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.654 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.654 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.654 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.654 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.655 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.655 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.655 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.655 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.655 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.655 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.655 INFO:teuthology.orchestra.run.smithi001.stdout: "max_concurrent_clones": { 2024-04-26T21:52:25.656 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "max_concurrent_clones", 2024-04-26T21:52:25.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.656 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.656 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.656 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "4", 2024-04-26T21:52:25.656 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.656 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.656 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.656 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-26T21:52:25.657 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.657 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.657 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.657 INFO:teuthology.orchestra.run.smithi001.stdout: "periodic_async_work": { 2024-04-26T21:52:25.657 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "periodic_async_work", 2024-04-26T21:52:25.657 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.657 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.657 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.658 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.658 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.658 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.658 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.658 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Periodically check for async work", 2024-04-26T21:52:25.658 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.658 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.658 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.659 INFO:teuthology.orchestra.run.smithi001.stdout: "snapshot_clone_delay": { 2024-04-26T21:52:25.659 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snapshot_clone_delay", 2024-04-26T21:52:25.659 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.659 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.659 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.659 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "0", 2024-04-26T21:52:25.659 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.659 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.660 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.660 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-26T21:52:25.660 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.660 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.660 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.660 INFO:teuthology.orchestra.run.smithi001.stdout: "snapshot_clone_no_wait": { 2024-04-26T21:52:25.660 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snapshot_clone_no_wait", 2024-04-26T21:52:25.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.661 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.661 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.661 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "True", 2024-04-26T21:52:25.661 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.661 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.661 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.661 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-26T21:52:25.661 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.661 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.662 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.662 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix", 2024-04-26T21:52:25.662 INFO:teuthology.orchestra.run.smithi001.stdout: "can_run": true, 2024-04-26T21:52:25.662 INFO:teuthology.orchestra.run.smithi001.stdout: "error_string": "", 2024-04-26T21:52:25.662 INFO:teuthology.orchestra.run.smithi001.stdout: "module_options": { 2024-04-26T21:52:25.663 INFO:teuthology.orchestra.run.smithi001.stdout: "discovery_interval": { 2024-04-26T21:52:25.663 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "discovery_interval", 2024-04-26T21:52:25.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "uint", 2024-04-26T21:52:25.663 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.663 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.663 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "100", 2024-04-26T21:52:25.663 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.664 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.664 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.664 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.664 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.664 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.664 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.665 INFO:teuthology.orchestra.run.smithi001.stdout: "identifier": { 2024-04-26T21:52:25.665 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "identifier", 2024-04-26T21:52:25.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.665 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.666 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.666 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.666 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.666 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.666 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.667 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.667 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.667 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.667 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.667 INFO:teuthology.orchestra.run.smithi001.stdout: "interval": { 2024-04-26T21:52:25.668 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "interval", 2024-04-26T21:52:25.668 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "secs", 2024-04-26T21:52:25.668 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.668 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.668 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "60", 2024-04-26T21:52:25.669 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.669 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.669 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.669 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.669 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.670 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.670 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.670 INFO:teuthology.orchestra.run.smithi001.stdout: "log_level": { 2024-04-26T21:52:25.670 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_level", 2024-04-26T21:52:25.670 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.671 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.671 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.671 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.671 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.671 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.672 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.672 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.672 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.672 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.672 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.672 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.673 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.673 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.673 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.673 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.673 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.674 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.674 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster": { 2024-04-26T21:52:25.674 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster", 2024-04-26T21:52:25.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.675 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.675 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.675 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.675 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.675 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.676 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.676 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.676 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.676 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.676 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.677 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_cluster_level": { 2024-04-26T21:52:25.677 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_cluster_level", 2024-04-26T21:52:25.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.677 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.677 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.677 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "info", 2024-04-26T21:52:25.678 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.678 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.678 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [ 2024-04-26T21:52:25.678 INFO:teuthology.orchestra.run.smithi001.stdout: "", 2024-04-26T21:52:25.678 INFO:teuthology.orchestra.run.smithi001.stdout: "critical", 2024-04-26T21:52:25.678 INFO:teuthology.orchestra.run.smithi001.stdout: "debug", 2024-04-26T21:52:25.679 INFO:teuthology.orchestra.run.smithi001.stdout: "error", 2024-04-26T21:52:25.679 INFO:teuthology.orchestra.run.smithi001.stdout: "info", 2024-04-26T21:52:25.679 INFO:teuthology.orchestra.run.smithi001.stdout: "warning" 2024-04-26T21:52:25.679 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.679 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.679 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.679 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.679 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.680 INFO:teuthology.orchestra.run.smithi001.stdout: "log_to_file": { 2024-04-26T21:52:25.680 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "log_to_file", 2024-04-26T21:52:25.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "bool", 2024-04-26T21:52:25.680 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.680 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 1, 2024-04-26T21:52:25.680 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "False", 2024-04-26T21:52:25.680 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.681 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.681 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.681 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.681 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.681 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.681 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.681 INFO:teuthology.orchestra.run.smithi001.stdout: "zabbix_host": { 2024-04-26T21:52:25.681 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix_host", 2024-04-26T21:52:25.682 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.682 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.682 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.682 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "", 2024-04-26T21:52:25.682 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.682 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.682 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.682 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.682 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.683 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.683 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.683 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.683 INFO:teuthology.orchestra.run.smithi001.stdout: "zabbix_port": { 2024-04-26T21:52:25.683 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix_port", 2024-04-26T21:52:25.683 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "int", 2024-04-26T21:52:25.683 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.683 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.684 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "10051", 2024-04-26T21:52:25.684 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.684 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.684 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.684 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.684 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.684 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.684 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.685 INFO:teuthology.orchestra.run.smithi001.stdout: "zabbix_sender": { 2024-04-26T21:52:25.685 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "zabbix_sender", 2024-04-26T21:52:25.685 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "str", 2024-04-26T21:52:25.685 INFO:teuthology.orchestra.run.smithi001.stdout: "level": "advanced", 2024-04-26T21:52:25.685 INFO:teuthology.orchestra.run.smithi001.stdout: "flags": 0, 2024-04-26T21:52:25.685 INFO:teuthology.orchestra.run.smithi001.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-26T21:52:25.686 INFO:teuthology.orchestra.run.smithi001.stdout: "min": "", 2024-04-26T21:52:25.686 INFO:teuthology.orchestra.run.smithi001.stdout: "max": "", 2024-04-26T21:52:25.686 INFO:teuthology.orchestra.run.smithi001.stdout: "enum_allowed": [], 2024-04-26T21:52:25.686 INFO:teuthology.orchestra.run.smithi001.stdout: "desc": "", 2024-04-26T21:52:25.686 INFO:teuthology.orchestra.run.smithi001.stdout: "long_desc": "", 2024-04-26T21:52:25.687 INFO:teuthology.orchestra.run.smithi001.stdout: "tags": [], 2024-04-26T21:52:25.687 INFO:teuthology.orchestra.run.smithi001.stdout: "see_also": [] 2024-04-26T21:52:25.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.687 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.687 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.687 INFO:teuthology.orchestra.run.smithi001.stdout: "services": { 2024-04-26T21:52:25.687 INFO:teuthology.orchestra.run.smithi001.stdout: "dashboard": "https://172.21.15.1:8443/" 2024-04-26T21:52:25.687 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.688 INFO:teuthology.orchestra.run.smithi001.stdout: "always_on_modules": { 2024-04-26T21:52:25.688 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus": [ 2024-04-26T21:52:25.688 INFO:teuthology.orchestra.run.smithi001.stdout: "balancer", 2024-04-26T21:52:25.688 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-04-26T21:52:25.688 INFO:teuthology.orchestra.run.smithi001.stdout: "devicehealth", 2024-04-26T21:52:25.688 INFO:teuthology.orchestra.run.smithi001.stdout: "orchestrator", 2024-04-26T21:52:25.688 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscaler", 2024-04-26T21:52:25.688 INFO:teuthology.orchestra.run.smithi001.stdout: "progress", 2024-04-26T21:52:25.689 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_support", 2024-04-26T21:52:25.689 INFO:teuthology.orchestra.run.smithi001.stdout: "status", 2024-04-26T21:52:25.689 INFO:teuthology.orchestra.run.smithi001.stdout: "telemetry", 2024-04-26T21:52:25.689 INFO:teuthology.orchestra.run.smithi001.stdout: "volumes" 2024-04-26T21:52:25.689 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.689 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific": [ 2024-04-26T21:52:25.689 INFO:teuthology.orchestra.run.smithi001.stdout: "balancer", 2024-04-26T21:52:25.689 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-04-26T21:52:25.689 INFO:teuthology.orchestra.run.smithi001.stdout: "devicehealth", 2024-04-26T21:52:25.690 INFO:teuthology.orchestra.run.smithi001.stdout: "orchestrator", 2024-04-26T21:52:25.690 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscaler", 2024-04-26T21:52:25.690 INFO:teuthology.orchestra.run.smithi001.stdout: "progress", 2024-04-26T21:52:25.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_support", 2024-04-26T21:52:25.690 INFO:teuthology.orchestra.run.smithi001.stdout: "status", 2024-04-26T21:52:25.690 INFO:teuthology.orchestra.run.smithi001.stdout: "telemetry", 2024-04-26T21:52:25.690 INFO:teuthology.orchestra.run.smithi001.stdout: "volumes" 2024-04-26T21:52:25.690 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.690 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy": [ 2024-04-26T21:52:25.691 INFO:teuthology.orchestra.run.smithi001.stdout: "balancer", 2024-04-26T21:52:25.691 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-04-26T21:52:25.691 INFO:teuthology.orchestra.run.smithi001.stdout: "devicehealth", 2024-04-26T21:52:25.691 INFO:teuthology.orchestra.run.smithi001.stdout: "orchestrator", 2024-04-26T21:52:25.691 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscaler", 2024-04-26T21:52:25.691 INFO:teuthology.orchestra.run.smithi001.stdout: "progress", 2024-04-26T21:52:25.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_support", 2024-04-26T21:52:25.691 INFO:teuthology.orchestra.run.smithi001.stdout: "status", 2024-04-26T21:52:25.692 INFO:teuthology.orchestra.run.smithi001.stdout: "telemetry", 2024-04-26T21:52:25.692 INFO:teuthology.orchestra.run.smithi001.stdout: "volumes" 2024-04-26T21:52:25.692 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.692 INFO:teuthology.orchestra.run.smithi001.stdout: "reef": [ 2024-04-26T21:52:25.692 INFO:teuthology.orchestra.run.smithi001.stdout: "balancer", 2024-04-26T21:52:25.693 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-04-26T21:52:25.693 INFO:teuthology.orchestra.run.smithi001.stdout: "devicehealth", 2024-04-26T21:52:25.693 INFO:teuthology.orchestra.run.smithi001.stdout: "orchestrator", 2024-04-26T21:52:25.693 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscaler", 2024-04-26T21:52:25.693 INFO:teuthology.orchestra.run.smithi001.stdout: "progress", 2024-04-26T21:52:25.693 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_support", 2024-04-26T21:52:25.693 INFO:teuthology.orchestra.run.smithi001.stdout: "status", 2024-04-26T21:52:25.694 INFO:teuthology.orchestra.run.smithi001.stdout: "telemetry", 2024-04-26T21:52:25.694 INFO:teuthology.orchestra.run.smithi001.stdout: "volumes" 2024-04-26T21:52:25.694 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-04-26T21:52:25.694 INFO:teuthology.orchestra.run.smithi001.stdout: "squid": [ 2024-04-26T21:52:25.694 INFO:teuthology.orchestra.run.smithi001.stdout: "balancer", 2024-04-26T21:52:25.694 INFO:teuthology.orchestra.run.smithi001.stdout: "crash", 2024-04-26T21:52:25.694 INFO:teuthology.orchestra.run.smithi001.stdout: "devicehealth", 2024-04-26T21:52:25.694 INFO:teuthology.orchestra.run.smithi001.stdout: "orchestrator", 2024-04-26T21:52:25.695 INFO:teuthology.orchestra.run.smithi001.stdout: "pg_autoscaler", 2024-04-26T21:52:25.695 INFO:teuthology.orchestra.run.smithi001.stdout: "progress", 2024-04-26T21:52:25.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rbd_support", 2024-04-26T21:52:25.695 INFO:teuthology.orchestra.run.smithi001.stdout: "status", 2024-04-26T21:52:25.695 INFO:teuthology.orchestra.run.smithi001.stdout: "telemetry", 2024-04-26T21:52:25.695 INFO:teuthology.orchestra.run.smithi001.stdout: "volumes" 2024-04-26T21:52:25.695 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:25.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.695 INFO:teuthology.orchestra.run.smithi001.stdout: "last_failure_osd_epoch": 96, 2024-04-26T21:52:25.696 INFO:teuthology.orchestra.run.smithi001.stdout: "active_clients": [ 2024-04-26T21:52:25.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.696 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "devicehealth", 2024-04-26T21:52:25.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-04-26T21:52:25.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-04-26T21:52:25.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:0", 2024-04-26T21:52:25.697 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2705143372 2024-04-26T21:52:25.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:25.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.697 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "libcephsqlite", 2024-04-26T21:52:25.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-04-26T21:52:25.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.698 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-04-26T21:52:25.698 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:0", 2024-04-26T21:52:25.698 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2979800538 2024-04-26T21:52:25.698 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.698 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:25.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.698 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "rbd_support", 2024-04-26T21:52:25.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-04-26T21:52:25.699 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.699 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-04-26T21:52:25.699 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:0", 2024-04-26T21:52:25.699 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 2944028457 2024-04-26T21:52:25.699 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.699 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:25.699 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.700 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snap_schedule", 2024-04-26T21:52:25.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-04-26T21:52:25.700 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.700 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-04-26T21:52:25.700 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:0", 2024-04-26T21:52:25.700 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 4226497665 2024-04-26T21:52:25.700 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.700 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:25.701 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.701 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "snap_schedule", 2024-04-26T21:52:25.701 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-04-26T21:52:25.701 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.701 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-04-26T21:52:25.702 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:0", 2024-04-26T21:52:25.702 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 866777756 2024-04-26T21:52:25.702 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.702 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "volumes", 2024-04-26T21:52:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-04-26T21:52:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-04-26T21:52:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-04-26T21:52:25.703 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:0", 2024-04-26T21:52:25.704 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 1413563965 2024-04-26T21:52:25.704 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.704 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:25.704 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:25.704 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-04-26T21:52:25.704 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:52:25.706 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:23 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2034051619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:52:25.706 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:23 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/856998544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:52:25.706 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:24 smithi038 ceph-mon[32724]: pgmap v702: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 178 op/s 2024-04-26T21:52:25.706 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:24 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2662490578' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T21:52:25.706 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:52:25.706 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:52:25.707 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:24 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:52:25.707 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:24 smithi038 ceph-mon[32724]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T21:52:25.707 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:23 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2034051619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:52:25.707 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:23 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/856998544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:52:25.707 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:24 smithi186 ceph-mon[32991]: pgmap v702: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 178 op/s 2024-04-26T21:52:25.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:24 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2662490578' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T21:52:25.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:52:25.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:52:25.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:24 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:52:25.708 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:24 smithi186 ceph-mon[32991]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T21:52:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.468+0000 7f3b80a00640 1 -- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b600a3d90 msgr2=0x7f3b600a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.468+0000 7f3b80a00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b600a3d90 0x7f3b600a6250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c10f3b0 tx=0x7f3b64005f90 comp rx=0 tx=0).stop 2024-04-26T21:52:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.468+0000 7f3b80a00640 1 -- 172.21.15.1:0/2662490578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b7c1090e0 msgr2=0x7f3b7c19f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:52:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.468+0000 7f3b80a00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b7c1090e0 0x7f3b7c19f5d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00e9f0 tx=0x7f3b6c00edf0 comp rx=0 tx=0).stop 2024-04-26T21:52:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.469+0000 7f3b80a00640 1 -- 172.21.15.1:0/2662490578 shutdown_connections 2024-04-26T21:52:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.469+0000 7f3b80a00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f3b600a3d90 0x7f3b600a6250 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:25.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.469+0000 7f3b80a00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b7c10e160 0x7f3b7c19fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.469+0000 7f3b80a00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3b7c1a3e90 0x7f3b7c1a4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.469+0000 7f3b80a00640 1 --2- 172.21.15.1:0/2662490578 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b7c1090e0 0x7f3b7c19f5d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:52:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.469+0000 7f3b80a00640 1 -- 172.21.15.1:0/2662490578 >> 172.21.15.1:0/2662490578 conn(0x7f3b7c078700 msgr2=0x7f3b7c07a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:52:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.469+0000 7f3b80a00640 1 -- 172.21.15.1:0/2662490578 shutdown_connections 2024-04-26T21:52:25.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:52:23.469+0000 7f3b80a00640 1 -- 172.21.15.1:0/2662490578 wait complete. 2024-04-26T21:52:25.717 DEBUG:tasks.check_counter:Getting stats from a 2024-04-26T21:52:25.718 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-26T21:52:25.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:23 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2034051619' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:52:25.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:23 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/856998544' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:52:25.718 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:24 smithi001 ceph-mon[28054]: pgmap v702: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 178 op/s 2024-04-26T21:52:25.719 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:24 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2662490578' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-26T21:52:25.719 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T21:52:25.719 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T21:52:25.719 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:24 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' 2024-04-26T21:52:25.719 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:24 smithi001 ceph-mon[28054]: osdmap e134: 12 total, 12 up, 12 in 2024-04-26T21:52:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:52:26.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:26 smithi186 ceph-mon[32991]: pgmap v704: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 135 op/s 2024-04-26T21:52:26.820 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:26 smithi001 ceph-mon[28054]: pgmap v704: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 135 op/s 2024-04-26T21:52:26.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:26 smithi038 ceph-mon[32724]: pgmap v704: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.3 MiB/s wr, 135 op/s 2024-04-26T21:52:28.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:28 smithi186 ceph-mon[32991]: pgmap v705: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 134 op/s 2024-04-26T21:52:28.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:28 smithi001 ceph-mon[28054]: pgmap v705: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 134 op/s 2024-04-26T21:52:28.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:28 smithi038 ceph-mon[32724]: pgmap v705: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.8 MiB/s wr, 134 op/s 2024-04-26T21:52:28.847 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:52:28.847 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:52:28.847 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 0, 2024-04-26T21:52:28.847 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 2, 2024-04-26T21:52:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 0, 2024-04-26T21:52:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 0, 2024-04-26T21:52:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 4, 2024-04-26T21:52:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 0, 2024-04-26T21:52:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 0.006487872, 2024-04-26T21:52:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T21:52:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 0.000784071, 2024-04-26T21:52:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:52:28.848 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:28.849 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:28.849 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:28.849 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:28.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.849 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:28.849 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:28.849 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:28.849 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:28.849 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.849 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T21:52:28.850 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T21:52:28.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.850 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:52:28.850 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 418, 2024-04-26T21:52:28.850 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 407, 2024-04-26T21:52:28.850 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 1646048, 2024-04-26T21:52:28.850 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 251936, 2024-04-26T21:52:28.850 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 2, 2024-04-26T21:52:28.850 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 1, 2024-04-26T21:52:28.851 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 0.159747251, 2024-04-26T21:52:28.851 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 0.061129197, 2024-04-26T21:52:28.851 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 0.096014837, 2024-04-26T21:52:28.851 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 0.018699130, 2024-04-26T21:52:28.851 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:28.851 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 406, 2024-04-26T21:52:28.851 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.056641921, 2024-04-26T21:52:28.851 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000139512 2024-04-26T21:52:28.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 1646048, 2024-04-26T21:52:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 251936 2024-04-26T21:52:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:52:28.852 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 7, 2024-04-26T21:52:28.853 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 314, 2024-04-26T21:52:28.853 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 3472, 2024-04-26T21:52:28.853 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 291280, 2024-04-26T21:52:28.853 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 6, 2024-04-26T21:52:28.853 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 3, 2024-04-26T21:52:28.853 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 0.062278042, 2024-04-26T21:52:28.853 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 0.049195434, 2024-04-26T21:52:28.853 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 0.005274956, 2024-04-26T21:52:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:52:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 312, 2024-04-26T21:52:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.065579369, 2024-04-26T21:52:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000210190 2024-04-26T21:52:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:28.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.855 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-26T21:52:28.855 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 291280 2024-04-26T21:52:28.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.855 INFO:teuthology.orchestra.run.smithi001.stdout: "cct": { 2024-04-26T21:52:28.855 INFO:teuthology.orchestra.run.smithi001.stdout: "total_workers": 0, 2024-04-26T21:52:28.855 INFO:teuthology.orchestra.run.smithi001.stdout: "unhealthy_workers": 0 2024-04-26T21:52:28.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.855 INFO:teuthology.orchestra.run.smithi001.stdout: "mempool": { 2024-04-26T21:52:28.855 INFO:teuthology.orchestra.run.smithi001.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:52:28.856 INFO:teuthology.orchestra.run.smithi001.stdout: "bloom_filter_items": 0, 2024-04-26T21:52:28.856 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:52:28.856 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:52:28.856 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:52:28.856 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:52:28.856 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:52:28.856 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:52:28.856 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:52:28.856 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:52:28.856 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:52:28.857 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:52:28.857 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:52:28.857 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:52:28.857 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:52:28.857 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_items": 0, 2024-04-26T21:52:28.857 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:52:28.857 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_blob_items": 0, 2024-04-26T21:52:28.857 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:52:28.857 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:52:28.857 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:52:28.858 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:52:28.858 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:52:28.858 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:52:28.858 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:52:28.858 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_txc_items": 0, 2024-04-26T21:52:28.858 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:52:28.858 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:52:28.858 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:52:28.858 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_items": 0, 2024-04-26T21:52:28.858 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_bytes": 0, 2024-04-26T21:52:28.859 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_items": 0, 2024-04-26T21:52:28.859 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:52:28.859 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:52:28.859 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:52:28.859 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:52:28.859 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_anon_bytes": 148731, 2024-04-26T21:52:28.859 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_anon_items": 31, 2024-04-26T21:52:28.859 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_meta_bytes": 0, 2024-04-26T21:52:28.859 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_meta_items": 0, 2024-04-26T21:52:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bytes": 0, 2024-04-26T21:52:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_items": 0, 2024-04-26T21:52:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:52:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mapbl_items": 0, 2024-04-26T21:52:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:52:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pglog_items": 0, 2024-04-26T21:52:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_bytes": 0, 2024-04-26T21:52:28.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_items": 0, 2024-04-26T21:52:28.861 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:52:28.861 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:52:28.861 INFO:teuthology.orchestra.run.smithi001.stdout: "pgmap_bytes": 0, 2024-04-26T21:52:28.861 INFO:teuthology.orchestra.run.smithi001.stdout: "pgmap_items": 0, 2024-04-26T21:52:28.861 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_co_bytes": 928, 2024-04-26T21:52:28.861 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_co_items": 2, 2024-04-26T21:52:28.861 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_1_bytes": 0, 2024-04-26T21:52:28.861 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_1_items": 0, 2024-04-26T21:52:28.861 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_2_bytes": 0, 2024-04-26T21:52:28.861 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_2_items": 0 2024-04-26T21:52:28.862 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:28.862 INFO:teuthology.orchestra.run.smithi001.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:52:28.862 INFO:teuthology.orchestra.run.smithi001.stdout: "val": 0, 2024-04-26T21:52:28.862 INFO:teuthology.orchestra.run.smithi001.stdout: "max": 104857600, 2024-04-26T21:52:28.862 INFO:teuthology.orchestra.run.smithi001.stdout: "get_started": 0, 2024-04-26T21:52:28.862 INFO:teuthology.orchestra.run.smithi001.stdout: "get": 421, 2024-04-26T21:52:28.862 INFO:teuthology.orchestra.run.smithi001.stdout: "get_sum": 1587017, 2024-04-26T21:52:28.863 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:28.863 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_success": 421, 2024-04-26T21:52:28.863 INFO:teuthology.orchestra.run.smithi001.stdout: "take": 0, 2024-04-26T21:52:28.863 INFO:teuthology.orchestra.run.smithi001.stdout: "take_sum": 0, 2024-04-26T21:52:28.863 INFO:teuthology.orchestra.run.smithi001.stdout: "put": 421, 2024-04-26T21:52:28.863 INFO:teuthology.orchestra.run.smithi001.stdout: "put_sum": 1587017, 2024-04-26T21:52:28.863 INFO:teuthology.orchestra.run.smithi001.stdout: "wait": { 2024-04-26T21:52:28.863 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:28.864 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:28.864 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:28.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:28.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:28.864 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:52:29.668 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-26T21:52:29.669 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-26T21:52:29.669 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-04-26T21:52:29.669 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-04-26T21:52:29.669 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-26T21:52:29.669 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-26T21:52:29.669 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-26T21:52:29.669 DEBUG:tasks.check_counter:Getting stats from d 2024-04-26T21:52:29.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-26T21:52:30.199 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:52:30.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:30 smithi186 ceph-mon[32991]: pgmap v706: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 110 op/s 2024-04-26T21:52:30.765 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:30 smithi001 ceph-mon[28054]: pgmap v706: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 110 op/s 2024-04-26T21:52:30.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:30 smithi038 ceph-mon[32724]: pgmap v706: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.6 MiB/s wr, 110 op/s 2024-04-26T21:52:32.630 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:32 smithi001 ceph-mon[28054]: pgmap v707: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 3.0 MiB/s wr, 87 op/s 2024-04-26T21:52:32.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:32 smithi186 ceph-mon[32991]: pgmap v707: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 3.0 MiB/s wr, 87 op/s 2024-04-26T21:52:32.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:32 smithi038 ceph-mon[32724]: pgmap v707: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 3.0 MiB/s wr, 87 op/s 2024-04-26T21:52:32.951 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:52:32.951 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:52:32.951 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 425, 2024-04-26T21:52:32.951 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 414, 2024-04-26T21:52:32.952 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 1647952, 2024-04-26T21:52:32.952 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 256416, 2024-04-26T21:52:32.952 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 4, 2024-04-26T21:52:32.952 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 0, 2024-04-26T21:52:32.952 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 0.166268460, 2024-04-26T21:52:32.952 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 0.041890200, 2024-04-26T21:52:32.952 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 0.141963754, 2024-04-26T21:52:32.952 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 0.019758260, 2024-04-26T21:52:32.952 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:32.953 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 413, 2024-04-26T21:52:32.953 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.056346431, 2024-04-26T21:52:32.953 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000136432 2024-04-26T21:52:32.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.953 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:32.953 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:32.953 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:32.953 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:32.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.954 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 1647952, 2024-04-26T21:52:32.954 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 256416 2024-04-26T21:52:32.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.954 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:52:32.954 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 3, 2024-04-26T21:52:32.954 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 317, 2024-04-26T21:52:32.954 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 480, 2024-04-26T21:52:32.955 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 291712, 2024-04-26T21:52:32.955 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 6, 2024-04-26T21:52:32.955 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 3, 2024-04-26T21:52:32.955 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 0.059523802, 2024-04-26T21:52:32.955 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 0.045279874, 2024-04-26T21:52:32.955 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 0.006518073, 2024-04-26T21:52:32.955 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:52:32.955 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:32.956 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 314, 2024-04-26T21:52:32.956 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.062114767, 2024-04-26T21:52:32.956 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000197817 2024-04-26T21:52:32.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.956 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:32.956 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:32.956 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:32.956 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:32.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.957 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T21:52:32.957 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 291712 2024-04-26T21:52:32.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.957 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:52:32.957 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 4, 2024-04-26T21:52:32.957 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 4, 2024-04-26T21:52:32.957 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 2992, 2024-04-26T21:52:32.957 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 448, 2024-04-26T21:52:32.958 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 2, 2024-04-26T21:52:32.958 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 1, 2024-04-26T21:52:32.958 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 0.001371419, 2024-04-26T21:52:32.958 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 0.000066285, 2024-04-26T21:52:32.958 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 0.001209133, 2024-04-26T21:52:32.958 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:52:32.958 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:32.958 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 3, 2024-04-26T21:52:32.959 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.001351918, 2024-04-26T21:52:32.959 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000450639 2024-04-26T21:52:32.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.959 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:32.959 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:32.959 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:32.959 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:32.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.959 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T21:52:32.959 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T21:52:32.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.960 INFO:teuthology.orchestra.run.smithi001.stdout: "cct": { 2024-04-26T21:52:32.960 INFO:teuthology.orchestra.run.smithi001.stdout: "total_workers": 0, 2024-04-26T21:52:32.960 INFO:teuthology.orchestra.run.smithi001.stdout: "unhealthy_workers": 0 2024-04-26T21:52:32.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mempool": { 2024-04-26T21:52:32.960 INFO:teuthology.orchestra.run.smithi001.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:52:32.960 INFO:teuthology.orchestra.run.smithi001.stdout: "bloom_filter_items": 0, 2024-04-26T21:52:32.960 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:52:32.961 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:52:32.961 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:52:32.961 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:52:32.961 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:52:32.961 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:52:32.961 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:52:32.961 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:52:32.961 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:52:32.962 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:52:32.962 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:52:32.962 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:52:32.962 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:52:32.962 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_items": 0, 2024-04-26T21:52:32.962 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:52:32.962 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_blob_items": 0, 2024-04-26T21:52:32.962 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:52:32.963 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:52:32.963 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:52:32.963 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:52:32.963 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:52:32.963 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:52:32.963 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:52:32.963 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_txc_items": 0, 2024-04-26T21:52:32.963 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:52:32.963 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:52:32.964 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:52:32.964 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_items": 0, 2024-04-26T21:52:32.964 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_bytes": 0, 2024-04-26T21:52:32.964 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_items": 0, 2024-04-26T21:52:32.964 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:52:32.964 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:52:32.964 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:52:32.964 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:52:32.964 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_anon_bytes": 149745, 2024-04-26T21:52:32.964 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_anon_items": 31, 2024-04-26T21:52:32.965 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_meta_bytes": 0, 2024-04-26T21:52:32.965 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_meta_items": 0, 2024-04-26T21:52:32.965 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bytes": 0, 2024-04-26T21:52:32.965 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_items": 0, 2024-04-26T21:52:32.965 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:52:32.965 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mapbl_items": 0, 2024-04-26T21:52:32.965 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:52:32.965 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pglog_items": 0, 2024-04-26T21:52:32.966 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_bytes": 0, 2024-04-26T21:52:32.966 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_items": 0, 2024-04-26T21:52:32.966 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:52:32.966 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:52:32.966 INFO:teuthology.orchestra.run.smithi001.stdout: "pgmap_bytes": 0, 2024-04-26T21:52:32.966 INFO:teuthology.orchestra.run.smithi001.stdout: "pgmap_items": 0, 2024-04-26T21:52:32.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_co_bytes": 928, 2024-04-26T21:52:32.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_co_items": 2, 2024-04-26T21:52:32.966 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_1_bytes": 0, 2024-04-26T21:52:32.967 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_1_items": 0, 2024-04-26T21:52:32.967 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_2_bytes": 0, 2024-04-26T21:52:32.967 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_2_items": 0 2024-04-26T21:52:32.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:32.967 INFO:teuthology.orchestra.run.smithi001.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:52:32.967 INFO:teuthology.orchestra.run.smithi001.stdout: "val": 0, 2024-04-26T21:52:32.967 INFO:teuthology.orchestra.run.smithi001.stdout: "max": 104857600, 2024-04-26T21:52:32.967 INFO:teuthology.orchestra.run.smithi001.stdout: "get_started": 0, 2024-04-26T21:52:32.967 INFO:teuthology.orchestra.run.smithi001.stdout: "get": 428, 2024-04-26T21:52:32.967 INFO:teuthology.orchestra.run.smithi001.stdout: "get_sum": 1587927, 2024-04-26T21:52:32.968 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:32.968 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_success": 428, 2024-04-26T21:52:32.968 INFO:teuthology.orchestra.run.smithi001.stdout: "take": 0, 2024-04-26T21:52:32.968 INFO:teuthology.orchestra.run.smithi001.stdout: "take_sum": 0, 2024-04-26T21:52:32.968 INFO:teuthology.orchestra.run.smithi001.stdout: "put": 428, 2024-04-26T21:52:32.968 INFO:teuthology.orchestra.run.smithi001.stdout: "put_sum": 1587927, 2024-04-26T21:52:32.968 INFO:teuthology.orchestra.run.smithi001.stdout: "wait": { 2024-04-26T21:52:32.968 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:32.968 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:32.968 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:32.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:32.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:32.969 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:52:33.919 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-26T21:52:33.920 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-26T21:52:33.920 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-04-26T21:52:33.920 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-04-26T21:52:33.920 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-26T21:52:33.920 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-26T21:52:33.920 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-26T21:52:33.921 DEBUG:tasks.check_counter:Getting stats from g 2024-04-26T21:52:33.921 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-26T21:52:34.386 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:52:34.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:34 smithi186 ceph-mon[32991]: pgmap v708: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 3.7 MiB/s wr, 109 op/s 2024-04-26T21:52:34.716 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:34 smithi001 ceph-mon[28054]: pgmap v708: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 3.7 MiB/s wr, 109 op/s 2024-04-26T21:52:34.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:34 smithi038 ceph-mon[32724]: pgmap v708: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 3.7 MiB/s wr, 109 op/s 2024-04-26T21:52:36.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:36 smithi186 ceph-mon[32991]: pgmap v709: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 4.7 MiB/s wr, 154 op/s 2024-04-26T21:52:36.706 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:36 smithi001 ceph-mon[28054]: pgmap v709: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 4.7 MiB/s wr, 154 op/s 2024-04-26T21:52:36.711 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:52:36.711 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:52:36.711 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 446861, 2024-04-26T21:52:36.711 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 601172, 2024-04-26T21:52:36.711 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 151056691, 2024-04-26T21:52:36.711 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 691532563, 2024-04-26T21:52:36.712 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 10, 2024-04-26T21:52:36.712 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 5, 2024-04-26T21:52:36.712 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 131.369777334, 2024-04-26T21:52:36.712 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 88.018701300, 2024-04-26T21:52:36.712 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 49.987601768, 2024-04-26T21:52:36.712 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 4.085995282, 2024-04-26T21:52:36.712 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:36.712 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 601170, 2024-04-26T21:52:36.712 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 116.439645537, 2024-04-26T21:52:36.712 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000193688 2024-04-26T21:52:36.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.713 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:36.713 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 875706, 2024-04-26T21:52:36.713 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.415281275, 2024-04-26T21:52:36.713 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000474 2024-04-26T21:52:36.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.713 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 116441984, 2024-04-26T21:52:36.713 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 360862096 2024-04-26T21:52:36.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.714 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:52:36.714 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 29250, 2024-04-26T21:52:36.714 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 29245, 2024-04-26T21:52:36.714 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 9093190, 2024-04-26T21:52:36.714 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 187333299, 2024-04-26T21:52:36.714 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 13, 2024-04-26T21:52:36.714 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 8, 2024-04-26T21:52:36.714 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 6.190633173, 2024-04-26T21:52:36.714 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 1.636415268, 2024-04-26T21:52:36.715 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 1.151328989, 2024-04-26T21:52:36.715 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 2.947342318, 2024-04-26T21:52:36.715 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:36.715 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 29243, 2024-04-26T21:52:36.715 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 8.075457668, 2024-04-26T21:52:36.715 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000276150 2024-04-26T21:52:36.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.715 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:36.715 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 72, 2024-04-26T21:52:36.715 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000028596, 2024-04-26T21:52:36.716 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000397 2024-04-26T21:52:36.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.716 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 1871680, 2024-04-26T21:52:36.716 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 281712 2024-04-26T21:52:36.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.716 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:52:36.716 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 322086, 2024-04-26T21:52:36.716 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 285021, 2024-04-26T21:52:36.716 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 100400280, 2024-04-26T21:52:36.716 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 301144700, 2024-04-26T21:52:36.717 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 10, 2024-04-26T21:52:36.717 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 7, 2024-04-26T21:52:36.717 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 51.807081139, 2024-04-26T21:52:36.717 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 22.180905398, 2024-04-26T21:52:36.717 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 31.294012785, 2024-04-26T21:52:36.717 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 4.337260845, 2024-04-26T21:52:36.717 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:36.717 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 285020, 2024-04-26T21:52:36.717 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 48.371081344, 2024-04-26T21:52:36.717 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000169711 2024-04-26T21:52:36.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.718 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:36.718 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 513504, 2024-04-26T21:52:36.718 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.214680709, 2024-04-26T21:52:36.718 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000418 2024-04-26T21:52:36.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.718 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T21:52:36.718 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 715552 2024-04-26T21:52:36.718 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.719 INFO:teuthology.orchestra.run.smithi001.stdout: "cct": { 2024-04-26T21:52:36.719 INFO:teuthology.orchestra.run.smithi001.stdout: "total_workers": 1, 2024-04-26T21:52:36.719 INFO:teuthology.orchestra.run.smithi001.stdout: "unhealthy_workers": 0 2024-04-26T21:52:36.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.719 INFO:teuthology.orchestra.run.smithi001.stdout: "finisher-MDSRank": { 2024-04-26T21:52:36.719 INFO:teuthology.orchestra.run.smithi001.stdout: "queue_len": 0, 2024-04-26T21:52:36.719 INFO:teuthology.orchestra.run.smithi001.stdout: "complete_latency": { 2024-04-26T21:52:36.719 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 69216, 2024-04-26T21:52:36.719 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 180.762996414, 2024-04-26T21:52:36.719 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.002611578 2024-04-26T21:52:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: "finisher-PurgeQueue": { 2024-04-26T21:52:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: "queue_len": 0, 2024-04-26T21:52:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: "complete_latency": { 2024-04-26T21:52:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 50040, 2024-04-26T21:52:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 17.566087460, 2024-04-26T21:52:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000351040 2024-04-26T21:52:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:36.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.721 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": { 2024-04-26T21:52:36.721 INFO:teuthology.orchestra.run.smithi001.stdout: "request": 223110, 2024-04-26T21:52:36.721 INFO:teuthology.orchestra.run.smithi001.stdout: "reply": 192225, 2024-04-26T21:52:36.721 INFO:teuthology.orchestra.run.smithi001.stdout: "reply_latency": { 2024-04-26T21:52:36.721 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 192225, 2024-04-26T21:52:36.721 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 1157.186911365, 2024-04-26T21:52:36.721 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.006019960 2024-04-26T21:52:36.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.721 INFO:teuthology.orchestra.run.smithi001.stdout: "slow_reply": 0, 2024-04-26T21:52:36.722 INFO:teuthology.orchestra.run.smithi001.stdout: "forward": 30884, 2024-04-26T21:52:36.722 INFO:teuthology.orchestra.run.smithi001.stdout: "dir_fetch_complete": 0, 2024-04-26T21:52:36.722 INFO:teuthology.orchestra.run.smithi001.stdout: "dir_fetch_keys": 0, 2024-04-26T21:52:36.722 INFO:teuthology.orchestra.run.smithi001.stdout: "dir_commit": 85, 2024-04-26T21:52:36.722 INFO:teuthology.orchestra.run.smithi001.stdout: "dir_split": 5, 2024-04-26T21:52:36.722 INFO:teuthology.orchestra.run.smithi001.stdout: "dir_merge": 1, 2024-04-26T21:52:36.722 INFO:teuthology.orchestra.run.smithi001.stdout: "inodes": 2432, 2024-04-26T21:52:36.722 INFO:teuthology.orchestra.run.smithi001.stdout: "inodes_top": 850, 2024-04-26T21:52:36.722 INFO:teuthology.orchestra.run.smithi001.stdout: "inodes_bottom": 1560, 2024-04-26T21:52:36.723 INFO:teuthology.orchestra.run.smithi001.stdout: "inodes_pin_tail": 22, 2024-04-26T21:52:36.723 INFO:teuthology.orchestra.run.smithi001.stdout: "inodes_pinned": 1218, 2024-04-26T21:52:36.723 INFO:teuthology.orchestra.run.smithi001.stdout: "inodes_expired": 533, 2024-04-26T21:52:36.723 INFO:teuthology.orchestra.run.smithi001.stdout: "inodes_with_caps": 216, 2024-04-26T21:52:36.723 INFO:teuthology.orchestra.run.smithi001.stdout: "caps": 216, 2024-04-26T21:52:36.723 INFO:teuthology.orchestra.run.smithi001.stdout: "subtrees": 12, 2024-04-26T21:52:36.723 INFO:teuthology.orchestra.run.smithi001.stdout: "traverse": 378686, 2024-04-26T21:52:36.723 INFO:teuthology.orchestra.run.smithi001.stdout: "traverse_hit": 183347, 2024-04-26T21:52:36.723 INFO:teuthology.orchestra.run.smithi001.stdout: "traverse_forward": 30882, 2024-04-26T21:52:36.723 INFO:teuthology.orchestra.run.smithi001.stdout: "traverse_discover": 252, 2024-04-26T21:52:36.724 INFO:teuthology.orchestra.run.smithi001.stdout: "traverse_dir_fetch": 0, 2024-04-26T21:52:36.724 INFO:teuthology.orchestra.run.smithi001.stdout: "traverse_remote_ino": 0, 2024-04-26T21:52:36.724 INFO:teuthology.orchestra.run.smithi001.stdout: "traverse_lock": 3321, 2024-04-26T21:52:36.724 INFO:teuthology.orchestra.run.smithi001.stdout: "load_cent": 112716, 2024-04-26T21:52:36.724 INFO:teuthology.orchestra.run.smithi001.stdout: "q": 0, 2024-04-26T21:52:36.724 INFO:teuthology.orchestra.run.smithi001.stdout: "exported": 7, 2024-04-26T21:52:36.724 INFO:teuthology.orchestra.run.smithi001.stdout: "exported_inodes": 350, 2024-04-26T21:52:36.724 INFO:teuthology.orchestra.run.smithi001.stdout: "imported": 12, 2024-04-26T21:52:36.724 INFO:teuthology.orchestra.run.smithi001.stdout: "imported_inodes": 773, 2024-04-26T21:52:36.725 INFO:teuthology.orchestra.run.smithi001.stdout: "openino_dir_fetch": 0, 2024-04-26T21:52:36.725 INFO:teuthology.orchestra.run.smithi001.stdout: "openino_backtrace_fetch": 0, 2024-04-26T21:52:36.725 INFO:teuthology.orchestra.run.smithi001.stdout: "openino_peer_discover": 0, 2024-04-26T21:52:36.725 INFO:teuthology.orchestra.run.smithi001.stdout: "root_rfiles": 4, 2024-04-26T21:52:36.725 INFO:teuthology.orchestra.run.smithi001.stdout: "root_rbytes": 234, 2024-04-26T21:52:36.725 INFO:teuthology.orchestra.run.smithi001.stdout: "root_rsnaps": 0, 2024-04-26T21:52:36.725 INFO:teuthology.orchestra.run.smithi001.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T21:52:36.725 INFO:teuthology.orchestra.run.smithi001.stdout: "scrub_set_tag": 0, 2024-04-26T21:52:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T21:52:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "scrub_inotable_repaired": 0, 2024-04-26T21:52:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "scrub_dir_inodes": 0, 2024-04-26T21:52:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T21:52:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T21:52:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "scrub_file_inodes": 0, 2024-04-26T21:52:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "handle_inode_file_caps": 8, 2024-04-26T21:52:36.726 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_cap_op_revoke": 22397, 2024-04-26T21:52:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_cap_op_grant": 116810, 2024-04-26T21:52:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T21:52:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T21:52:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T21:52:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "handle_client_caps": 62440, 2024-04-26T21:52:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "handle_client_caps_dirty": 21065, 2024-04-26T21:52:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "handle_client_cap_release": 9312, 2024-04-26T21:52:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: "process_request_cap_release": 148354 2024-04-26T21:52:36.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.728 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_cache": { 2024-04-26T21:52:36.728 INFO:teuthology.orchestra.run.smithi001.stdout: "dir_update": 459, 2024-04-26T21:52:36.728 INFO:teuthology.orchestra.run.smithi001.stdout: "dir_update_receipt": 1312, 2024-04-26T21:52:36.728 INFO:teuthology.orchestra.run.smithi001.stdout: "dir_try_discover": 736, 2024-04-26T21:52:36.728 INFO:teuthology.orchestra.run.smithi001.stdout: "dir_send_discover": 1283, 2024-04-26T21:52:36.728 INFO:teuthology.orchestra.run.smithi001.stdout: "dir_handle_discover": 516, 2024-04-26T21:52:36.728 INFO:teuthology.orchestra.run.smithi001.stdout: "num_strays": 0, 2024-04-26T21:52:36.728 INFO:teuthology.orchestra.run.smithi001.stdout: "num_strays_delayed": 0, 2024-04-26T21:52:36.728 INFO:teuthology.orchestra.run.smithi001.stdout: "num_strays_enqueuing": 0, 2024-04-26T21:52:36.728 INFO:teuthology.orchestra.run.smithi001.stdout: "strays_created": 29723, 2024-04-26T21:52:36.729 INFO:teuthology.orchestra.run.smithi001.stdout: "strays_enqueued": 29723, 2024-04-26T21:52:36.729 INFO:teuthology.orchestra.run.smithi001.stdout: "strays_reintegrated": 0, 2024-04-26T21:52:36.729 INFO:teuthology.orchestra.run.smithi001.stdout: "strays_migrated": 0, 2024-04-26T21:52:36.729 INFO:teuthology.orchestra.run.smithi001.stdout: "num_recovering_processing": 0, 2024-04-26T21:52:36.729 INFO:teuthology.orchestra.run.smithi001.stdout: "num_recovering_enqueued": 0, 2024-04-26T21:52:36.729 INFO:teuthology.orchestra.run.smithi001.stdout: "num_recovering_prioritized": 0, 2024-04-26T21:52:36.729 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_started": 0, 2024-04-26T21:52:36.729 INFO:teuthology.orchestra.run.smithi001.stdout: "recovery_completed": 0, 2024-04-26T21:52:36.729 INFO:teuthology.orchestra.run.smithi001.stdout: "ireq_quiesce_path": 0, 2024-04-26T21:52:36.730 INFO:teuthology.orchestra.run.smithi001.stdout: "ireq_quiesce_inode": 0, 2024-04-26T21:52:36.730 INFO:teuthology.orchestra.run.smithi001.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T21:52:36.730 INFO:teuthology.orchestra.run.smithi001.stdout: "ireq_exportdir": 12, 2024-04-26T21:52:36.730 INFO:teuthology.orchestra.run.smithi001.stdout: "ireq_flush": 0, 2024-04-26T21:52:36.730 INFO:teuthology.orchestra.run.smithi001.stdout: "ireq_fragmentdir": 6, 2024-04-26T21:52:36.730 INFO:teuthology.orchestra.run.smithi001.stdout: "ireq_fragstats": 0, 2024-04-26T21:52:36.730 INFO:teuthology.orchestra.run.smithi001.stdout: "ireq_inodestats": 0 2024-04-26T21:52:36.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.730 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_log": { 2024-04-26T21:52:36.730 INFO:teuthology.orchestra.run.smithi001.stdout: "evlrg": 0, 2024-04-26T21:52:36.731 INFO:teuthology.orchestra.run.smithi001.stdout: "evadd": 149343, 2024-04-26T21:52:36.731 INFO:teuthology.orchestra.run.smithi001.stdout: "evex": 70456, 2024-04-26T21:52:36.731 INFO:teuthology.orchestra.run.smithi001.stdout: "evtrm": 61451, 2024-04-26T21:52:36.731 INFO:teuthology.orchestra.run.smithi001.stdout: "ev": 87892, 2024-04-26T21:52:36.731 INFO:teuthology.orchestra.run.smithi001.stdout: "evexg": 0, 2024-04-26T21:52:36.731 INFO:teuthology.orchestra.run.smithi001.stdout: "evexd": 9005, 2024-04-26T21:52:36.731 INFO:teuthology.orchestra.run.smithi001.stdout: "segadd": 216, 2024-04-26T21:52:36.731 INFO:teuthology.orchestra.run.smithi001.stdout: "segex": 88, 2024-04-26T21:52:36.731 INFO:teuthology.orchestra.run.smithi001.stdout: "segtrm": 76, 2024-04-26T21:52:36.732 INFO:teuthology.orchestra.run.smithi001.stdout: "seg": 140, 2024-04-26T21:52:36.732 INFO:teuthology.orchestra.run.smithi001.stdout: "segmjr": 8, 2024-04-26T21:52:36.732 INFO:teuthology.orchestra.run.smithi001.stdout: "segexg": 0, 2024-04-26T21:52:36.732 INFO:teuthology.orchestra.run.smithi001.stdout: "segexd": 12, 2024-04-26T21:52:36.732 INFO:teuthology.orchestra.run.smithi001.stdout: "expos": 152553741, 2024-04-26T21:52:36.732 INFO:teuthology.orchestra.run.smithi001.stdout: "wrpos": 694585667, 2024-04-26T21:52:36.732 INFO:teuthology.orchestra.run.smithi001.stdout: "rdpos": 0, 2024-04-26T21:52:36.732 INFO:teuthology.orchestra.run.smithi001.stdout: "jlat": { 2024-04-26T21:52:36.732 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 37498, 2024-04-26T21:52:36.732 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 357.276233179, 2024-04-26T21:52:36.733 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.009527874 2024-04-26T21:52:36.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.733 INFO:teuthology.orchestra.run.smithi001.stdout: "replayed": 0 2024-04-26T21:52:36.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.733 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_mem": { 2024-04-26T21:52:36.733 INFO:teuthology.orchestra.run.smithi001.stdout: "ino": 1966, 2024-04-26T21:52:36.733 INFO:teuthology.orchestra.run.smithi001.stdout: "ino+": 31958, 2024-04-26T21:52:36.733 INFO:teuthology.orchestra.run.smithi001.stdout: "ino-": 29992, 2024-04-26T21:52:36.733 INFO:teuthology.orchestra.run.smithi001.stdout: "dir": 224, 2024-04-26T21:52:36.734 INFO:teuthology.orchestra.run.smithi001.stdout: "dir+": 838, 2024-04-26T21:52:36.734 INFO:teuthology.orchestra.run.smithi001.stdout: "dir-": 614, 2024-04-26T21:52:36.734 INFO:teuthology.orchestra.run.smithi001.stdout: "dn": 2420, 2024-04-26T21:52:36.734 INFO:teuthology.orchestra.run.smithi001.stdout: "dn+": 57690, 2024-04-26T21:52:36.734 INFO:teuthology.orchestra.run.smithi001.stdout: "dn-": 55270, 2024-04-26T21:52:36.734 INFO:teuthology.orchestra.run.smithi001.stdout: "cap": 216, 2024-04-26T21:52:36.734 INFO:teuthology.orchestra.run.smithi001.stdout: "cap+": 33158, 2024-04-26T21:52:36.734 INFO:teuthology.orchestra.run.smithi001.stdout: "cap-": 32942, 2024-04-26T21:52:36.734 INFO:teuthology.orchestra.run.smithi001.stdout: "rss": 85496, 2024-04-26T21:52:36.734 INFO:teuthology.orchestra.run.smithi001.stdout: "heap": 223608 2024-04-26T21:52:36.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.735 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_server": { 2024-04-26T21:52:36.735 INFO:teuthology.orchestra.run.smithi001.stdout: "dispatch_client_request": 332111, 2024-04-26T21:52:36.735 INFO:teuthology.orchestra.run.smithi001.stdout: "dispatch_server_request": 48, 2024-04-26T21:52:36.735 INFO:teuthology.orchestra.run.smithi001.stdout: "handle_client_request": 223110, 2024-04-26T21:52:36.735 INFO:teuthology.orchestra.run.smithi001.stdout: "handle_client_session": 3209, 2024-04-26T21:52:36.735 INFO:teuthology.orchestra.run.smithi001.stdout: "handle_peer_request": 22092, 2024-04-26T21:52:36.735 INFO:teuthology.orchestra.run.smithi001.stdout: "req_create_latency": { 2024-04-26T21:52:36.735 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 29641, 2024-04-26T21:52:36.736 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 244.355702039, 2024-04-26T21:52:36.736 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.008243841 2024-04-26T21:52:36.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.736 INFO:teuthology.orchestra.run.smithi001.stdout: "req_getattr_latency": { 2024-04-26T21:52:36.736 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 2996, 2024-04-26T21:52:36.736 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 56.329115666, 2024-04-26T21:52:36.736 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.018801440 2024-04-26T21:52:36.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.736 INFO:teuthology.orchestra.run.smithi001.stdout: "req_getfilelock_latency": { 2024-04-26T21:52:36.736 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.737 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.737 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.737 INFO:teuthology.orchestra.run.smithi001.stdout: "req_link_latency": { 2024-04-26T21:52:36.737 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.737 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.737 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.737 INFO:teuthology.orchestra.run.smithi001.stdout: "req_lookup_latency": { 2024-04-26T21:52:36.737 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 59414, 2024-04-26T21:52:36.738 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 182.517344817, 2024-04-26T21:52:36.738 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.003071958 2024-04-26T21:52:36.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.738 INFO:teuthology.orchestra.run.smithi001.stdout: "req_lookuphash_latency": { 2024-04-26T21:52:36.738 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.738 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.738 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.738 INFO:teuthology.orchestra.run.smithi001.stdout: "req_lookupino_latency": { 2024-04-26T21:52:36.739 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.739 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.739 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.739 INFO:teuthology.orchestra.run.smithi001.stdout: "req_lookupname_latency": { 2024-04-26T21:52:36.739 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.739 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.739 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.740 INFO:teuthology.orchestra.run.smithi001.stdout: "req_lookupparent_latency": { 2024-04-26T21:52:36.740 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.740 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.740 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.740 INFO:teuthology.orchestra.run.smithi001.stdout: "req_lookupsnap_latency": { 2024-04-26T21:52:36.741 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.741 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.741 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.741 INFO:teuthology.orchestra.run.smithi001.stdout: "req_lssnap_latency": { 2024-04-26T21:52:36.741 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.742 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.742 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.742 INFO:teuthology.orchestra.run.smithi001.stdout: "req_mkdir_latency": { 2024-04-26T21:52:36.742 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.742 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.742 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.743 INFO:teuthology.orchestra.run.smithi001.stdout: "req_mknod_latency": { 2024-04-26T21:52:36.743 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.743 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.743 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.743 INFO:teuthology.orchestra.run.smithi001.stdout: "req_mksnap_latency": { 2024-04-26T21:52:36.743 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.743 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.743 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.744 INFO:teuthology.orchestra.run.smithi001.stdout: "req_open_latency": { 2024-04-26T21:52:36.744 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.744 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.744 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.744 INFO:teuthology.orchestra.run.smithi001.stdout: "req_readdir_latency": { 2024-04-26T21:52:36.744 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 63785, 2024-04-26T21:52:36.744 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 443.147973948, 2024-04-26T21:52:36.745 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.006947526 2024-04-26T21:52:36.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.745 INFO:teuthology.orchestra.run.smithi001.stdout: "req_rename_latency": { 2024-04-26T21:52:36.745 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 5933, 2024-04-26T21:52:36.745 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 47.191284492, 2024-04-26T21:52:36.745 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.007954034 2024-04-26T21:52:36.745 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.745 INFO:teuthology.orchestra.run.smithi001.stdout: "req_renamesnap_latency": { 2024-04-26T21:52:36.745 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.746 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.746 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.746 INFO:teuthology.orchestra.run.smithi001.stdout: "req_snapdiff_latency": { 2024-04-26T21:52:36.746 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.746 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.746 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.746 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.747 INFO:teuthology.orchestra.run.smithi001.stdout: "req_rmdir_latency": { 2024-04-26T21:52:36.747 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.747 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.747 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.747 INFO:teuthology.orchestra.run.smithi001.stdout: "req_rmsnap_latency": { 2024-04-26T21:52:36.747 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.747 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.748 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.748 INFO:teuthology.orchestra.run.smithi001.stdout: "req_rmxattr_latency": { 2024-04-26T21:52:36.748 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.748 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.748 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.748 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.748 INFO:teuthology.orchestra.run.smithi001.stdout: "req_setattr_latency": { 2024-04-26T21:52:36.748 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 105, 2024-04-26T21:52:36.749 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 2.885297632, 2024-04-26T21:52:36.749 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.027479025 2024-04-26T21:52:36.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.749 INFO:teuthology.orchestra.run.smithi001.stdout: "req_setdirlayout_latency": { 2024-04-26T21:52:36.749 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.749 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.749 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.749 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: "req_setfilelock_latency": { 2024-04-26T21:52:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 628, 2024-04-26T21:52:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 2.103090412, 2024-04-26T21:52:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.003348870 2024-04-26T21:52:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: "req_setlayout_latency": { 2024-04-26T21:52:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.750 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: "req_setxattr_latency": { 2024-04-26T21:52:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.751 INFO:teuthology.orchestra.run.smithi001.stdout: "req_symlink_latency": { 2024-04-26T21:52:36.752 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.752 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.752 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.752 INFO:teuthology.orchestra.run.smithi001.stdout: "req_unlink_latency": { 2024-04-26T21:52:36.752 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 29723, 2024-04-26T21:52:36.752 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 178.657102359, 2024-04-26T21:52:36.752 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.006010735 2024-04-26T21:52:36.752 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.753 INFO:teuthology.orchestra.run.smithi001.stdout: "cap_revoke_eviction": 0, 2024-04-26T21:52:36.753 INFO:teuthology.orchestra.run.smithi001.stdout: "cap_acquisition_throttle": 0, 2024-04-26T21:52:36.753 INFO:teuthology.orchestra.run.smithi001.stdout: "req_getvxattr_latency": { 2024-04-26T21:52:36.753 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.753 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.753 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.753 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:36.753 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.754 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_sessions": { 2024-04-26T21:52:36.754 INFO:teuthology.orchestra.run.smithi001.stdout: "session_count": 2, 2024-04-26T21:52:36.754 INFO:teuthology.orchestra.run.smithi001.stdout: "session_add": 2, 2024-04-26T21:52:36.754 INFO:teuthology.orchestra.run.smithi001.stdout: "session_remove": 0, 2024-04-26T21:52:36.754 INFO:teuthology.orchestra.run.smithi001.stdout: "sessions_open": 2, 2024-04-26T21:52:36.754 INFO:teuthology.orchestra.run.smithi001.stdout: "sessions_stale": 0, 2024-04-26T21:52:36.754 INFO:teuthology.orchestra.run.smithi001.stdout: "total_load": 23658, 2024-04-26T21:52:36.754 INFO:teuthology.orchestra.run.smithi001.stdout: "average_load": 11829, 2024-04-26T21:52:36.755 INFO:teuthology.orchestra.run.smithi001.stdout: "avg_session_uptime": 1263, 2024-04-26T21:52:36.755 INFO:teuthology.orchestra.run.smithi001.stdout: "mdthresh_evicted": 0 2024-04-26T21:52:36.755 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.755 INFO:teuthology.orchestra.run.smithi001.stdout: "mempool": { 2024-04-26T21:52:36.755 INFO:teuthology.orchestra.run.smithi001.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:52:36.755 INFO:teuthology.orchestra.run.smithi001.stdout: "bloom_filter_items": 0, 2024-04-26T21:52:36.755 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:52:36.755 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:52:36.755 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:52:36.755 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:52:36.756 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:52:36.756 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:52:36.756 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:52:36.756 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:52:36.756 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:52:36.756 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:52:36.756 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:52:36.756 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:52:36.756 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:52:36.757 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_items": 0, 2024-04-26T21:52:36.757 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:52:36.757 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_blob_items": 0, 2024-04-26T21:52:36.757 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:52:36.757 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:52:36.757 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:52:36.757 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:52:36.757 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:52:36.757 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:52:36.758 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:52:36.758 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_txc_items": 0, 2024-04-26T21:52:36.758 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:52:36.758 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:52:36.758 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:52:36.758 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_items": 0, 2024-04-26T21:52:36.758 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_bytes": 0, 2024-04-26T21:52:36.758 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_items": 0, 2024-04-26T21:52:36.759 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:52:36.759 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:52:36.759 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:52:36.759 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:52:36.759 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_anon_bytes": 4637265, 2024-04-26T21:52:36.759 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_anon_items": 2044, 2024-04-26T21:52:36.759 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_meta_bytes": 792, 2024-04-26T21:52:36.759 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_meta_items": 9, 2024-04-26T21:52:36.759 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bytes": 0, 2024-04-26T21:52:36.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_items": 0, 2024-04-26T21:52:36.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:52:36.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mapbl_items": 0, 2024-04-26T21:52:36.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:52:36.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pglog_items": 0, 2024-04-26T21:52:36.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_bytes": 10040, 2024-04-26T21:52:36.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_items": 211, 2024-04-26T21:52:36.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:52:36.760 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:52:36.761 INFO:teuthology.orchestra.run.smithi001.stdout: "pgmap_bytes": 0, 2024-04-26T21:52:36.761 INFO:teuthology.orchestra.run.smithi001.stdout: "pgmap_items": 0, 2024-04-26T21:52:36.761 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_co_bytes": 7666767, 2024-04-26T21:52:36.761 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_co_items": 92201, 2024-04-26T21:52:36.761 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_1_bytes": 0, 2024-04-26T21:52:36.761 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_1_items": 0, 2024-04-26T21:52:36.761 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_2_bytes": 0, 2024-04-26T21:52:36.761 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_2_items": 0 2024-04-26T21:52:36.761 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.762 INFO:teuthology.orchestra.run.smithi001.stdout: "objecter": { 2024-04-26T21:52:36.762 INFO:teuthology.orchestra.run.smithi001.stdout: "op_active": 0, 2024-04-26T21:52:36.762 INFO:teuthology.orchestra.run.smithi001.stdout: "op_laggy": 0, 2024-04-26T21:52:36.762 INFO:teuthology.orchestra.run.smithi001.stdout: "op_send": 101920, 2024-04-26T21:52:36.762 INFO:teuthology.orchestra.run.smithi001.stdout: "op_send_bytes": 698106473, 2024-04-26T21:52:36.762 INFO:teuthology.orchestra.run.smithi001.stdout: "op_resend": 28, 2024-04-26T21:52:36.762 INFO:teuthology.orchestra.run.smithi001.stdout: "op_reply": 101920, 2024-04-26T21:52:36.762 INFO:teuthology.orchestra.run.smithi001.stdout: "op_latency": { 2024-04-26T21:52:36.763 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 101920, 2024-04-26T21:52:36.763 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 1720.554665214, 2024-04-26T21:52:36.763 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.016881423 2024-04-26T21:52:36.763 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.763 INFO:teuthology.orchestra.run.smithi001.stdout: "op_inflight": 0, 2024-04-26T21:52:36.763 INFO:teuthology.orchestra.run.smithi001.stdout: "oplen_avg": { 2024-04-26T21:52:36.763 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 101920, 2024-04-26T21:52:36.763 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 104437 2024-04-26T21:52:36.764 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.764 INFO:teuthology.orchestra.run.smithi001.stdout: "op": 101920, 2024-04-26T21:52:36.764 INFO:teuthology.orchestra.run.smithi001.stdout: "op_r": 1264, 2024-04-26T21:52:36.764 INFO:teuthology.orchestra.run.smithi001.stdout: "op_w": 100656, 2024-04-26T21:52:36.764 INFO:teuthology.orchestra.run.smithi001.stdout: "op_rmw": 0, 2024-04-26T21:52:36.764 INFO:teuthology.orchestra.run.smithi001.stdout: "op_pg": 0, 2024-04-26T21:52:36.764 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_stat": 0, 2024-04-26T21:52:36.764 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_create": 387, 2024-04-26T21:52:36.764 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_read": 1264, 2024-04-26T21:52:36.764 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_write": 38762, 2024-04-26T21:52:36.765 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_writefull": 635, 2024-04-26T21:52:36.765 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_writesame": 0, 2024-04-26T21:52:36.765 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_append": 0, 2024-04-26T21:52:36.765 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_zero": 0, 2024-04-26T21:52:36.765 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_truncate": 5, 2024-04-26T21:52:36.765 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_delete": 59665, 2024-04-26T21:52:36.765 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_mapext": 0, 2024-04-26T21:52:36.765 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_sparse_read": 0, 2024-04-26T21:52:36.765 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_clonerange": 0, 2024-04-26T21:52:36.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_getxattr": 0, 2024-04-26T21:52:36.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_setxattr": 606, 2024-04-26T21:52:36.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_cmpxattr": 0, 2024-04-26T21:52:36.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_rmxattr": 0, 2024-04-26T21:52:36.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_resetxattrs": 0, 2024-04-26T21:52:36.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_call": 0, 2024-04-26T21:52:36.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_watch": 0, 2024-04-26T21:52:36.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_notify": 0, 2024-04-26T21:52:36.766 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T21:52:36.767 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_pgls": 0, 2024-04-26T21:52:36.767 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_pgls_filter": 0, 2024-04-26T21:52:36.767 INFO:teuthology.orchestra.run.smithi001.stdout: "osdop_other": 0, 2024-04-26T21:52:36.767 INFO:teuthology.orchestra.run.smithi001.stdout: "linger_active": 0, 2024-04-26T21:52:36.767 INFO:teuthology.orchestra.run.smithi001.stdout: "linger_send": 0, 2024-04-26T21:52:36.767 INFO:teuthology.orchestra.run.smithi001.stdout: "linger_resend": 0, 2024-04-26T21:52:36.767 INFO:teuthology.orchestra.run.smithi001.stdout: "linger_ping": 0, 2024-04-26T21:52:36.767 INFO:teuthology.orchestra.run.smithi001.stdout: "poolop_active": 0, 2024-04-26T21:52:36.767 INFO:teuthology.orchestra.run.smithi001.stdout: "poolop_send": 0, 2024-04-26T21:52:36.768 INFO:teuthology.orchestra.run.smithi001.stdout: "poolop_resend": 0, 2024-04-26T21:52:36.768 INFO:teuthology.orchestra.run.smithi001.stdout: "poolstat_active": 0, 2024-04-26T21:52:36.768 INFO:teuthology.orchestra.run.smithi001.stdout: "poolstat_send": 0, 2024-04-26T21:52:36.768 INFO:teuthology.orchestra.run.smithi001.stdout: "poolstat_resend": 0, 2024-04-26T21:52:36.768 INFO:teuthology.orchestra.run.smithi001.stdout: "statfs_active": 0, 2024-04-26T21:52:36.768 INFO:teuthology.orchestra.run.smithi001.stdout: "statfs_send": 0, 2024-04-26T21:52:36.768 INFO:teuthology.orchestra.run.smithi001.stdout: "statfs_resend": 0, 2024-04-26T21:52:36.768 INFO:teuthology.orchestra.run.smithi001.stdout: "command_active": 0, 2024-04-26T21:52:36.768 INFO:teuthology.orchestra.run.smithi001.stdout: "command_send": 0, 2024-04-26T21:52:36.769 INFO:teuthology.orchestra.run.smithi001.stdout: "command_resend": 0, 2024-04-26T21:52:36.769 INFO:teuthology.orchestra.run.smithi001.stdout: "map_epoch": 134, 2024-04-26T21:52:36.769 INFO:teuthology.orchestra.run.smithi001.stdout: "map_full": 0, 2024-04-26T21:52:36.769 INFO:teuthology.orchestra.run.smithi001.stdout: "map_inc": 43, 2024-04-26T21:52:36.769 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_sessions": 12, 2024-04-26T21:52:36.769 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_session_open": 12, 2024-04-26T21:52:36.769 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_session_close": 0, 2024-04-26T21:52:36.769 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_laggy": 0, 2024-04-26T21:52:36.769 INFO:teuthology.orchestra.run.smithi001.stdout: "omap_wr": 2364, 2024-04-26T21:52:36.769 INFO:teuthology.orchestra.run.smithi001.stdout: "omap_rd": 0, 2024-04-26T21:52:36.770 INFO:teuthology.orchestra.run.smithi001.stdout: "omap_del": 749 2024-04-26T21:52:36.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.770 INFO:teuthology.orchestra.run.smithi001.stdout: "oft": { 2024-04-26T21:52:36.770 INFO:teuthology.orchestra.run.smithi001.stdout: "omap_total_objs": 1, 2024-04-26T21:52:36.770 INFO:teuthology.orchestra.run.smithi001.stdout: "omap_total_kv_pairs": 0, 2024-04-26T21:52:36.770 INFO:teuthology.orchestra.run.smithi001.stdout: "omap_total_updates": 0, 2024-04-26T21:52:36.770 INFO:teuthology.orchestra.run.smithi001.stdout: "omap_total_removes": 0 2024-04-26T21:52:36.770 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.770 INFO:teuthology.orchestra.run.smithi001.stdout: "osd-slow-ops": { 2024-04-26T21:52:36.771 INFO:teuthology.orchestra.run.smithi001.stdout: "slow_ops_count": 3 2024-04-26T21:52:36.771 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.771 INFO:teuthology.orchestra.run.smithi001.stdout: "purge_queue": { 2024-04-26T21:52:36.771 INFO:teuthology.orchestra.run.smithi001.stdout: "pq_executing_ops": 0, 2024-04-26T21:52:36.771 INFO:teuthology.orchestra.run.smithi001.stdout: "pq_executing_ops_high_water": 16, 2024-04-26T21:52:36.771 INFO:teuthology.orchestra.run.smithi001.stdout: "pq_executing": 0, 2024-04-26T21:52:36.771 INFO:teuthology.orchestra.run.smithi001.stdout: "pq_executing_high_water": 8, 2024-04-26T21:52:36.771 INFO:teuthology.orchestra.run.smithi001.stdout: "pq_executed_ops": 59446, 2024-04-26T21:52:36.771 INFO:teuthology.orchestra.run.smithi001.stdout: "pq_executed": 29723, 2024-04-26T21:52:36.772 INFO:teuthology.orchestra.run.smithi001.stdout: "pq_item_in_journal": 0 2024-04-26T21:52:36.772 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.772 INFO:teuthology.orchestra.run.smithi001.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:52:36.772 INFO:teuthology.orchestra.run.smithi001.stdout: "val": 0, 2024-04-26T21:52:36.772 INFO:teuthology.orchestra.run.smithi001.stdout: "max": 104857600, 2024-04-26T21:52:36.772 INFO:teuthology.orchestra.run.smithi001.stdout: "get_started": 0, 2024-04-26T21:52:36.772 INFO:teuthology.orchestra.run.smithi001.stdout: "get": 798193, 2024-04-26T21:52:36.772 INFO:teuthology.orchestra.run.smithi001.stdout: "get_sum": 174313215, 2024-04-26T21:52:36.773 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:36.773 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_success": 798193, 2024-04-26T21:52:36.773 INFO:teuthology.orchestra.run.smithi001.stdout: "take": 0, 2024-04-26T21:52:36.773 INFO:teuthology.orchestra.run.smithi001.stdout: "take_sum": 0, 2024-04-26T21:52:36.773 INFO:teuthology.orchestra.run.smithi001.stdout: "put": 798193, 2024-04-26T21:52:36.773 INFO:teuthology.orchestra.run.smithi001.stdout: "put_sum": 174313215, 2024-04-26T21:52:36.773 INFO:teuthology.orchestra.run.smithi001.stdout: "wait": { 2024-04-26T21:52:36.773 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.773 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.774 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.774 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:36.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.774 INFO:teuthology.orchestra.run.smithi001.stdout: "throttle-objecter_bytes": { 2024-04-26T21:52:36.774 INFO:teuthology.orchestra.run.smithi001.stdout: "val": 0, 2024-04-26T21:52:36.774 INFO:teuthology.orchestra.run.smithi001.stdout: "max": 104857600, 2024-04-26T21:52:36.774 INFO:teuthology.orchestra.run.smithi001.stdout: "get_started": 0, 2024-04-26T21:52:36.774 INFO:teuthology.orchestra.run.smithi001.stdout: "get": 0, 2024-04-26T21:52:36.774 INFO:teuthology.orchestra.run.smithi001.stdout: "get_sum": 0, 2024-04-26T21:52:36.775 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:36.775 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:36.775 INFO:teuthology.orchestra.run.smithi001.stdout: "take": 101920, 2024-04-26T21:52:36.775 INFO:teuthology.orchestra.run.smithi001.stdout: "take_sum": 701350552, 2024-04-26T21:52:36.775 INFO:teuthology.orchestra.run.smithi001.stdout: "put": 101920, 2024-04-26T21:52:36.775 INFO:teuthology.orchestra.run.smithi001.stdout: "put_sum": 701350552, 2024-04-26T21:52:36.775 INFO:teuthology.orchestra.run.smithi001.stdout: "wait": { 2024-04-26T21:52:36.775 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.775 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.776 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.776 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:36.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.776 INFO:teuthology.orchestra.run.smithi001.stdout: "throttle-objecter_ops": { 2024-04-26T21:52:36.776 INFO:teuthology.orchestra.run.smithi001.stdout: "val": 0, 2024-04-26T21:52:36.776 INFO:teuthology.orchestra.run.smithi001.stdout: "max": 1024, 2024-04-26T21:52:36.776 INFO:teuthology.orchestra.run.smithi001.stdout: "get_started": 0, 2024-04-26T21:52:36.776 INFO:teuthology.orchestra.run.smithi001.stdout: "get": 0, 2024-04-26T21:52:36.776 INFO:teuthology.orchestra.run.smithi001.stdout: "get_sum": 0, 2024-04-26T21:52:36.776 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:36.777 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:36.777 INFO:teuthology.orchestra.run.smithi001.stdout: "take": 101920, 2024-04-26T21:52:36.777 INFO:teuthology.orchestra.run.smithi001.stdout: "take_sum": 101920, 2024-04-26T21:52:36.777 INFO:teuthology.orchestra.run.smithi001.stdout: "put": 101920, 2024-04-26T21:52:36.777 INFO:teuthology.orchestra.run.smithi001.stdout: "put_sum": 101920, 2024-04-26T21:52:36.777 INFO:teuthology.orchestra.run.smithi001.stdout: "wait": { 2024-04-26T21:52:36.777 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.777 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.777 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.778 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:36.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.778 INFO:teuthology.orchestra.run.smithi001.stdout: "throttle-write_buf_throttle": { 2024-04-26T21:52:36.778 INFO:teuthology.orchestra.run.smithi001.stdout: "val": 0, 2024-04-26T21:52:36.778 INFO:teuthology.orchestra.run.smithi001.stdout: "max": 3758096384, 2024-04-26T21:52:36.778 INFO:teuthology.orchestra.run.smithi001.stdout: "get_started": 0, 2024-04-26T21:52:36.778 INFO:teuthology.orchestra.run.smithi001.stdout: "get": 29723, 2024-04-26T21:52:36.778 INFO:teuthology.orchestra.run.smithi001.stdout: "get_sum": 3244079, 2024-04-26T21:52:36.778 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:36.778 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_success": 29723, 2024-04-26T21:52:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: "take": 0, 2024-04-26T21:52:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: "take_sum": 0, 2024-04-26T21:52:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: "put": 1264, 2024-04-26T21:52:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: "put_sum": 3244079, 2024-04-26T21:52:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: "wait": { 2024-04-26T21:52:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:36.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:36.780 INFO:teuthology.orchestra.run.smithi001.stdout: "throttle-write_buf_throttle-0x55d2a11c43c0": { 2024-04-26T21:52:36.780 INFO:teuthology.orchestra.run.smithi001.stdout: "val": 83835, 2024-04-26T21:52:36.780 INFO:teuthology.orchestra.run.smithi001.stdout: "max": 3758096384, 2024-04-26T21:52:36.780 INFO:teuthology.orchestra.run.smithi001.stdout: "get_started": 0, 2024-04-26T21:52:36.780 INFO:teuthology.orchestra.run.smithi001.stdout: "get": 149343, 2024-04-26T21:52:36.780 INFO:teuthology.orchestra.run.smithi001.stdout: "get_sum": 690391363, 2024-04-26T21:52:36.780 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:36.780 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_success": 149343, 2024-04-26T21:52:36.780 INFO:teuthology.orchestra.run.smithi001.stdout: "take": 0, 2024-04-26T21:52:36.780 INFO:teuthology.orchestra.run.smithi001.stdout: "take_sum": 0, 2024-04-26T21:52:36.781 INFO:teuthology.orchestra.run.smithi001.stdout: "put": 37498, 2024-04-26T21:52:36.781 INFO:teuthology.orchestra.run.smithi001.stdout: "put_sum": 690307528, 2024-04-26T21:52:36.781 INFO:teuthology.orchestra.run.smithi001.stdout: "wait": { 2024-04-26T21:52:36.781 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:36.781 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:36.781 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:36.781 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:36.781 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:36.781 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:52:36.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:36 smithi038 ceph-mon[32724]: pgmap v709: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 4.7 MiB/s wr, 154 op/s 2024-04-26T21:52:37.636 INFO:tasks.check_counter:Daemon mds.g mds.exported=7 2024-04-26T21:52:37.637 INFO:tasks.check_counter:Daemon mds.g mds.imported=12 2024-04-26T21:52:37.637 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=459 2024-04-26T21:52:37.637 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=1312 2024-04-26T21:52:37.637 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-04-26T21:52:37.637 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-04-26T21:52:37.637 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T21:52:37.637 DEBUG:tasks.check_counter:Getting stats from j 2024-04-26T21:52:37.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-26T21:52:38.098 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T21:52:38.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:38 smithi001 ceph-mon[28054]: pgmap v710: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 3.7 MiB/s wr, 142 op/s 2024-04-26T21:52:38.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:38 smithi186 ceph-mon[32991]: pgmap v710: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 3.7 MiB/s wr, 142 op/s 2024-04-26T21:52:38.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:38 smithi038 ceph-mon[32724]: pgmap v710: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 3.7 MiB/s wr, 142 op/s 2024-04-26T21:52:40.440 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-04-26T21:52:40.440 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:52:40.440 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 4, 2024-04-26T21:52:40.440 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 4, 2024-04-26T21:52:40.440 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 2992, 2024-04-26T21:52:40.440 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 448, 2024-04-26T21:52:40.441 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 4, 2024-04-26T21:52:40.441 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 0, 2024-04-26T21:52:40.441 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 0.004654232, 2024-04-26T21:52:40.441 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 0.000089018, 2024-04-26T21:52:40.441 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 0.001564450, 2024-04-26T21:52:40.441 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:52:40.441 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:40.441 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 3, 2024-04-26T21:52:40.442 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.001660842, 2024-04-26T21:52:40.442 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000553614 2024-04-26T21:52:40.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.442 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:40.442 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:40.442 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:40.442 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:40.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.443 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T21:52:40.443 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T21:52:40.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.443 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:52:40.443 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 2, 2024-04-26T21:52:40.443 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 319, 2024-04-26T21:52:40.443 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 320, 2024-04-26T21:52:40.443 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 213248, 2024-04-26T21:52:40.443 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 5, 2024-04-26T21:52:40.444 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 3, 2024-04-26T21:52:40.444 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 0.062241183, 2024-04-26T21:52:40.444 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 0.048667728, 2024-04-26T21:52:40.444 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 0.005852846, 2024-04-26T21:52:40.444 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:52:40.444 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:40.444 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 316, 2024-04-26T21:52:40.444 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.057858815, 2024-04-26T21:52:40.444 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000183097 2024-04-26T21:52:40.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.445 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:40.445 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:40.445 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:40.445 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:40.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.445 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T21:52:40.445 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 213248 2024-04-26T21:52:40.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.446 INFO:teuthology.orchestra.run.smithi001.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:52:40.446 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_messages": 428, 2024-04-26T21:52:40.446 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages": 417, 2024-04-26T21:52:40.446 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_bytes": 1648768, 2024-04-26T21:52:40.446 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_bytes": 258336, 2024-04-26T21:52:40.446 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_created_connections": 2, 2024-04-26T21:52:40.446 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_active_connections": 1, 2024-04-26T21:52:40.446 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_total_time": 0.154977121, 2024-04-26T21:52:40.447 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_send_time": 0.063053195, 2024-04-26T21:52:40.447 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_recv_time": 0.086429337, 2024-04-26T21:52:40.447 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_running_fast_dispatch_time": 0.020109081, 2024-04-26T21:52:40.447 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:40.447 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 416, 2024-04-26T21:52:40.447 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.047883264, 2024-04-26T21:52:40.447 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000115104 2024-04-26T21:52:40.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.447 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:40.448 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:40.448 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:40.448 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:40.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.448 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_recv_encrypted_bytes": 1648768, 2024-04-26T21:52:40.448 INFO:teuthology.orchestra.run.smithi001.stdout: "msgr_send_encrypted_bytes": 258336 2024-04-26T21:52:40.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.448 INFO:teuthology.orchestra.run.smithi001.stdout: "cct": { 2024-04-26T21:52:40.448 INFO:teuthology.orchestra.run.smithi001.stdout: "total_workers": 0, 2024-04-26T21:52:40.449 INFO:teuthology.orchestra.run.smithi001.stdout: "unhealthy_workers": 0 2024-04-26T21:52:40.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.449 INFO:teuthology.orchestra.run.smithi001.stdout: "mempool": { 2024-04-26T21:52:40.449 INFO:teuthology.orchestra.run.smithi001.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:52:40.449 INFO:teuthology.orchestra.run.smithi001.stdout: "bloom_filter_items": 0, 2024-04-26T21:52:40.449 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:52:40.449 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:52:40.449 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:52:40.449 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:52:40.450 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:52:40.450 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:52:40.450 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:52:40.450 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:52:40.450 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:52:40.450 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:52:40.450 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:52:40.450 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:52:40.451 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:52:40.451 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_extent_items": 0, 2024-04-26T21:52:40.451 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:52:40.451 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_blob_items": 0, 2024-04-26T21:52:40.451 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:52:40.451 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:52:40.451 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:52:40.451 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:52:40.452 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:52:40.452 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:52:40.452 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:52:40.452 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_txc_items": 0, 2024-04-26T21:52:40.452 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:52:40.452 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:52:40.452 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:52:40.452 INFO:teuthology.orchestra.run.smithi001.stdout: "bluestore_writing_items": 0, 2024-04-26T21:52:40.452 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_bytes": 0, 2024-04-26T21:52:40.453 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_items": 0, 2024-04-26T21:52:40.453 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:52:40.453 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:52:40.453 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:52:40.453 INFO:teuthology.orchestra.run.smithi001.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:52:40.453 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_anon_bytes": 149745, 2024-04-26T21:52:40.453 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_anon_items": 31, 2024-04-26T21:52:40.453 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_meta_bytes": 0, 2024-04-26T21:52:40.453 INFO:teuthology.orchestra.run.smithi001.stdout: "buffer_meta_items": 0, 2024-04-26T21:52:40.454 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_bytes": 0, 2024-04-26T21:52:40.454 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_items": 0, 2024-04-26T21:52:40.454 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:52:40.454 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_mapbl_items": 0, 2024-04-26T21:52:40.454 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:52:40.454 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_pglog_items": 0, 2024-04-26T21:52:40.454 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_bytes": 0, 2024-04-26T21:52:40.454 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_items": 0, 2024-04-26T21:52:40.455 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:52:40.455 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:52:40.455 INFO:teuthology.orchestra.run.smithi001.stdout: "pgmap_bytes": 0, 2024-04-26T21:52:40.455 INFO:teuthology.orchestra.run.smithi001.stdout: "pgmap_items": 0, 2024-04-26T21:52:40.455 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_co_bytes": 928, 2024-04-26T21:52:40.455 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_co_items": 2, 2024-04-26T21:52:40.455 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_1_bytes": 0, 2024-04-26T21:52:40.455 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_1_items": 0, 2024-04-26T21:52:40.455 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_2_bytes": 0, 2024-04-26T21:52:40.456 INFO:teuthology.orchestra.run.smithi001.stdout: "unittest_2_items": 0 2024-04-26T21:52:40.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-04-26T21:52:40.456 INFO:teuthology.orchestra.run.smithi001.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:52:40.456 INFO:teuthology.orchestra.run.smithi001.stdout: "val": 0, 2024-04-26T21:52:40.456 INFO:teuthology.orchestra.run.smithi001.stdout: "max": 104857600, 2024-04-26T21:52:40.456 INFO:teuthology.orchestra.run.smithi001.stdout: "get_started": 0, 2024-04-26T21:52:40.456 INFO:teuthology.orchestra.run.smithi001.stdout: "get": 430, 2024-04-26T21:52:40.456 INFO:teuthology.orchestra.run.smithi001.stdout: "get_sum": 1588290, 2024-04-26T21:52:40.456 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:40.457 INFO:teuthology.orchestra.run.smithi001.stdout: "get_or_fail_success": 430, 2024-04-26T21:52:40.457 INFO:teuthology.orchestra.run.smithi001.stdout: "take": 0, 2024-04-26T21:52:40.457 INFO:teuthology.orchestra.run.smithi001.stdout: "take_sum": 0, 2024-04-26T21:52:40.457 INFO:teuthology.orchestra.run.smithi001.stdout: "put": 430, 2024-04-26T21:52:40.457 INFO:teuthology.orchestra.run.smithi001.stdout: "put_sum": 1588290, 2024-04-26T21:52:40.457 INFO:teuthology.orchestra.run.smithi001.stdout: "wait": { 2024-04-26T21:52:40.457 INFO:teuthology.orchestra.run.smithi001.stdout: "avgcount": 0, 2024-04-26T21:52:40.457 INFO:teuthology.orchestra.run.smithi001.stdout: "sum": 0.000000000, 2024-04-26T21:52:40.458 INFO:teuthology.orchestra.run.smithi001.stdout: "avgtime": 0.000000000 2024-04-26T21:52:40.458 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:40.458 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-04-26T21:52:40.458 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-04-26T21:52:40.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:40 smithi001 ceph-mon[28054]: pgmap v711: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 3.1 MiB/s wr, 140 op/s 2024-04-26T21:52:40.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:40 smithi186 ceph-mon[32991]: pgmap v711: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 3.1 MiB/s wr, 140 op/s 2024-04-26T21:52:40.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:40 smithi038 ceph-mon[32724]: pgmap v711: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 3.1 MiB/s wr, 140 op/s 2024-04-26T21:52:41.268 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-26T21:52:41.269 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-26T21:52:41.269 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-04-26T21:52:41.269 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-04-26T21:52:41.269 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-26T21:52:41.269 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-26T21:52:41.269 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-26T21:52:41.269 DEBUG:tasks.check_counter:Getting stats from b 2024-04-26T21:52:41.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-26T21:52:41.596 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:52:42.689 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:42 smithi001 ceph-mon[28054]: pgmap v712: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.9 MiB/s wr, 143 op/s 2024-04-26T21:52:42.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:42 smithi186 ceph-mon[32991]: pgmap v712: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.9 MiB/s wr, 143 op/s 2024-04-26T21:52:42.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:42 smithi038 ceph-mon[32724]: pgmap v712: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.9 MiB/s wr, 143 op/s 2024-04-26T21:52:43.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:43 smithi038 ceph-mon[32724]: pgmap v713: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 2.2 MiB/s wr, 112 op/s 2024-04-26T21:52:43.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:43 smithi186 ceph-mon[32991]: pgmap v713: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 2.2 MiB/s wr, 112 op/s 2024-04-26T21:52:43.777 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-26T21:52:43.777 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:52:43.777 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 153780, 2024-04-26T21:52:43.777 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 150562, 2024-04-26T21:52:43.777 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 45210896, 2024-04-26T21:52:43.777 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 206125930, 2024-04-26T21:52:43.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 17, 2024-04-26T21:52:43.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 4, 2024-04-26T21:52:43.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 10.886797517, 2024-04-26T21:52:43.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 3.789844014, 2024-04-26T21:52:43.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 5.457495339, 2024-04-26T21:52:43.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 2.210012537, 2024-04-26T21:52:43.778 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:43.778 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 150565, 2024-04-26T21:52:43.779 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 33.505405584, 2024-04-26T21:52:43.779 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000222531 2024-04-26T21:52:43.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.779 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:43.779 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 204438, 2024-04-26T21:52:43.779 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.044543844, 2024-04-26T21:52:43.779 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000217 2024-04-26T21:52:43.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.779 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1884368, 2024-04-26T21:52:43.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 277248 2024-04-26T21:52:43.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.780 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:52:43.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 316659, 2024-04-26T21:52:43.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 269795, 2024-04-26T21:52:43.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 98472631, 2024-04-26T21:52:43.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 266198444, 2024-04-26T21:52:43.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 190, 2024-04-26T21:52:43.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 5, 2024-04-26T21:52:43.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 22.094759166, 2024-04-26T21:52:43.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 9.550129297, 2024-04-26T21:52:43.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 12.389935016, 2024-04-26T21:52:43.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 2.320400154, 2024-04-26T21:52:43.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:43.781 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 269794, 2024-04-26T21:52:43.781 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 28.899649498, 2024-04-26T21:52:43.781 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000107117 2024-04-26T21:52:43.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:43.782 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 484012, 2024-04-26T21:52:43.782 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.087734597, 2024-04-26T21:52:43.782 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000181 2024-04-26T21:52:43.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 42704, 2024-04-26T21:52:43.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 810144 2024-04-26T21:52:43.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.783 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:52:43.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 334583, 2024-04-26T21:52:43.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 460162, 2024-04-26T21:52:43.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 122674988, 2024-04-26T21:52:43.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 453521658, 2024-04-26T21:52:43.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 23, 2024-04-26T21:52:43.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 8, 2024-04-26T21:52:43.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 27.542592165, 2024-04-26T21:52:43.784 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 11.902301510, 2024-04-26T21:52:43.784 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 16.784451025, 2024-04-26T21:52:43.784 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 1.970688562, 2024-04-26T21:52:43.784 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:43.784 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 460154, 2024-04-26T21:52:43.784 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 56.318251804, 2024-04-26T21:52:43.784 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000122390 2024-04-26T21:52:43.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.785 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:43.785 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 603844, 2024-04-26T21:52:43.785 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.140527729, 2024-04-26T21:52:43.785 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000232 2024-04-26T21:52:43.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.785 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 112525456, 2024-04-26T21:52:43.785 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 347596704 2024-04-26T21:52:43.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.786 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-26T21:52:43.786 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-26T21:52:43.786 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-26T21:52:43.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.786 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-26T21:52:43.786 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-26T21:52:43.786 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-26T21:52:43.786 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 78757, 2024-04-26T21:52:43.787 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 127.533524268, 2024-04-26T21:52:43.787 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001619329 2024-04-26T21:52:43.787 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:43.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.787 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-26T21:52:43.787 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-26T21:52:43.787 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-26T21:52:43.788 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 52088, 2024-04-26T21:52:43.788 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 8.693570350, 2024-04-26T21:52:43.788 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000166901 2024-04-26T21:52:43.788 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:43.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.788 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-26T21:52:43.788 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 213005, 2024-04-26T21:52:43.788 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 181807, 2024-04-26T21:52:43.789 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-26T21:52:43.789 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 181807, 2024-04-26T21:52:43.789 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 967.391981397, 2024-04-26T21:52:43.789 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.005320983 2024-04-26T21:52:43.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.789 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-26T21:52:43.789 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 31196, 2024-04-26T21:52:43.789 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 6, 2024-04-26T21:52:43.790 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-26T21:52:43.790 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 2496, 2024-04-26T21:52:43.790 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 6, 2024-04-26T21:52:43.790 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 2, 2024-04-26T21:52:43.790 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 2245, 2024-04-26T21:52:43.790 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 1083, 2024-04-26T21:52:43.790 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 1147, 2024-04-26T21:52:43.790 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 15, 2024-04-26T21:52:43.791 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 699, 2024-04-26T21:52:43.791 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 336, 2024-04-26T21:52:43.791 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 195, 2024-04-26T21:52:43.791 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 196, 2024-04-26T21:52:43.791 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 11, 2024-04-26T21:52:43.791 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 357856, 2024-04-26T21:52:43.791 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 185080, 2024-04-26T21:52:43.791 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 31194, 2024-04-26T21:52:43.792 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 148, 2024-04-26T21:52:43.792 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 0, 2024-04-26T21:52:43.792 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-26T21:52:43.792 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 3975, 2024-04-26T21:52:43.792 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 17308, 2024-04-26T21:52:43.792 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-26T21:52:43.792 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 8, 2024-04-26T21:52:43.792 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 258, 2024-04-26T21:52:43.792 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 9, 2024-04-26T21:52:43.793 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 607, 2024-04-26T21:52:43.793 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-26T21:52:43.793 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 0, 2024-04-26T21:52:43.793 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-26T21:52:43.793 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 645, 2024-04-26T21:52:43.793 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 212344417, 2024-04-26T21:52:43.793 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 6, 2024-04-26T21:52:43.793 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T21:52:43.793 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-26T21:52:43.794 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T21:52:43.794 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-26T21:52:43.794 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 0, 2024-04-26T21:52:43.794 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T21:52:43.794 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T21:52:43.794 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 0, 2024-04-26T21:52:43.794 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 16, 2024-04-26T21:52:43.795 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 12197, 2024-04-26T21:52:43.795 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 100534, 2024-04-26T21:52:43.795 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T21:52:43.795 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T21:52:43.795 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T21:52:43.795 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 59048, 2024-04-26T21:52:43.795 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 25893, 2024-04-26T21:52:43.795 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 9603, 2024-04-26T21:52:43.796 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 141734 2024-04-26T21:52:43.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.796 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-26T21:52:43.796 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 443, 2024-04-26T21:52:43.796 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 1229, 2024-04-26T21:52:43.796 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 671, 2024-04-26T21:52:43.796 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 1170, 2024-04-26T21:52:43.797 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 578, 2024-04-26T21:52:43.797 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 0, 2024-04-26T21:52:43.797 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-26T21:52:43.797 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-26T21:52:43.797 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 27886, 2024-04-26T21:52:43.797 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 27886, 2024-04-26T21:52:43.797 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-26T21:52:43.797 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-26T21:52:43.797 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-26T21:52:43.798 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-26T21:52:43.798 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-26T21:52:43.798 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 2, 2024-04-26T21:52:43.798 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 2, 2024-04-26T21:52:43.798 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-26T21:52:43.798 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-26T21:52:43.798 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T21:52:43.798 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 11, 2024-04-26T21:52:43.799 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-26T21:52:43.799 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 8, 2024-04-26T21:52:43.799 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-26T21:52:43.799 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-26T21:52:43.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.799 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-26T21:52:43.799 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-26T21:52:43.799 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 147792, 2024-04-26T21:52:43.799 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 146211, 2024-04-26T21:52:43.800 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 146211, 2024-04-26T21:52:43.800 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 1581, 2024-04-26T21:52:43.800 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-26T21:52:43.800 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 1919, 2024-04-26T21:52:43.800 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 247, 2024-04-26T21:52:43.800 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 245, 2024-04-26T21:52:43.800 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 245, 2024-04-26T21:52:43.800 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 2, 2024-04-26T21:52:43.800 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 1, 2024-04-26T21:52:43.801 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-26T21:52:43.801 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 4, 2024-04-26T21:52:43.801 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 424421466, 2024-04-26T21:52:43.801 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 430721313, 2024-04-26T21:52:43.801 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 0, 2024-04-26T21:52:43.801 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-26T21:52:43.801 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 40883, 2024-04-26T21:52:43.801 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 365.608592222, 2024-04-26T21:52:43.801 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.008942802 2024-04-26T21:52:43.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.802 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 0 2024-04-26T21:52:43.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.802 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-26T21:52:43.802 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 1743, 2024-04-26T21:52:43.802 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 29913, 2024-04-26T21:52:43.802 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 28170, 2024-04-26T21:52:43.802 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 224, 2024-04-26T21:52:43.802 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 767, 2024-04-26T21:52:43.803 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 543, 2024-04-26T21:52:43.803 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 2257, 2024-04-26T21:52:43.803 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 54841, 2024-04-26T21:52:43.803 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 52584, 2024-04-26T21:52:43.803 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 181, 2024-04-26T21:52:43.803 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 31707, 2024-04-26T21:52:43.803 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 31526, 2024-04-26T21:52:43.803 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 86744, 2024-04-26T21:52:43.803 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 231800 2024-04-26T21:52:43.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.804 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-26T21:52:43.804 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 308290, 2024-04-26T21:52:43.804 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 15, 2024-04-26T21:52:43.804 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 213005, 2024-04-26T21:52:43.804 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 4164, 2024-04-26T21:52:43.804 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 14918, 2024-04-26T21:52:43.804 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-26T21:52:43.805 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 27790, 2024-04-26T21:52:43.805 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 207.375571219, 2024-04-26T21:52:43.805 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.007462237 2024-04-26T21:52:43.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.805 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-26T21:52:43.805 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 7783, 2024-04-26T21:52:43.805 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 122.238533872, 2024-04-26T21:52:43.805 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.015705837 2024-04-26T21:52:43.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.806 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-26T21:52:43.806 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.806 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.806 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.806 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-26T21:52:43.806 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.806 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.806 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.807 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-26T21:52:43.807 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 52410, 2024-04-26T21:52:43.807 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 138.392547828, 2024-04-26T21:52:43.807 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.002640575 2024-04-26T21:52:43.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.807 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-26T21:52:43.807 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.808 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.808 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.808 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-26T21:52:43.808 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2, 2024-04-26T21:52:43.808 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000281400, 2024-04-26T21:52:43.808 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000140700 2024-04-26T21:52:43.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.808 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-26T21:52:43.808 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.809 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.809 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.809 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-26T21:52:43.809 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.809 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.809 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.810 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-26T21:52:43.810 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.811 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.811 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.811 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-26T21:52:43.811 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 22, 2024-04-26T21:52:43.811 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.012234487, 2024-04-26T21:52:43.811 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000556113 2024-04-26T21:52:43.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.811 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-26T21:52:43.812 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 7, 2024-04-26T21:52:43.812 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.011272256, 2024-04-26T21:52:43.812 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001610322 2024-04-26T21:52:43.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.812 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-26T21:52:43.812 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.812 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.812 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.813 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-26T21:52:43.813 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 22, 2024-04-26T21:52:43.813 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 11.867184446, 2024-04-26T21:52:43.813 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.539417474 2024-04-26T21:52:43.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.813 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-26T21:52:43.813 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1, 2024-04-26T21:52:43.813 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000170358, 2024-04-26T21:52:43.814 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000170358 2024-04-26T21:52:43.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.814 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-26T21:52:43.814 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 58136, 2024-04-26T21:52:43.814 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 300.904206417, 2024-04-26T21:52:43.814 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.005175867 2024-04-26T21:52:43.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.814 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-26T21:52:43.815 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 6367, 2024-04-26T21:52:43.815 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 47.898910792, 2024-04-26T21:52:43.815 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.007522995 2024-04-26T21:52:43.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.815 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-26T21:52:43.815 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.815 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.816 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.816 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-26T21:52:43.816 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.816 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.816 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.816 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-26T21:52:43.817 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.817 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.817 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.817 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-26T21:52:43.817 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 16, 2024-04-26T21:52:43.817 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 6.374762893, 2024-04-26T21:52:43.817 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.398422680 2024-04-26T21:52:43.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.818 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-26T21:52:43.818 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.818 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.818 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.818 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-26T21:52:43.818 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 93, 2024-04-26T21:52:43.818 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 2.096060955, 2024-04-26T21:52:43.819 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.022538289 2024-04-26T21:52:43.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.819 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-26T21:52:43.819 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.819 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.819 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.819 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-26T21:52:43.819 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1264, 2024-04-26T21:52:43.820 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 2.006635061, 2024-04-26T21:52:43.820 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001587527 2024-04-26T21:52:43.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.820 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-26T21:52:43.820 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.820 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.820 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.821 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-26T21:52:43.821 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 8, 2024-04-26T21:52:43.821 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.022031632, 2024-04-26T21:52:43.821 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.002753954 2024-04-26T21:52:43.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.821 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-26T21:52:43.821 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.821 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.821 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.822 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-26T21:52:43.822 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 27886, 2024-04-26T21:52:43.822 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 128.191577781, 2024-04-26T21:52:43.822 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.004596986 2024-04-26T21:52:43.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.822 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-26T21:52:43.822 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-26T21:52:43.823 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-26T21:52:43.823 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.823 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.823 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:43.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.823 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-26T21:52:43.823 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 2, 2024-04-26T21:52:43.823 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 6, 2024-04-26T21:52:43.824 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 4, 2024-04-26T21:52:43.824 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 2, 2024-04-26T21:52:43.824 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-26T21:52:43.824 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 24512, 2024-04-26T21:52:43.824 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 12256, 2024-04-26T21:52:43.824 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 1323, 2024-04-26T21:52:43.824 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-26T21:52:43.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.824 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-26T21:52:43.825 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:52:43.825 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-26T21:52:43.825 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:52:43.825 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:52:43.825 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:52:43.825 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:52:43.825 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:52:43.825 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:52:43.826 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:52:43.826 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:52:43.826 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:52:43.826 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:52:43.826 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:52:43.826 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:52:43.826 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:52:43.826 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-26T21:52:43.827 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:52:43.827 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-26T21:52:43.827 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:52:43.827 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:52:43.827 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:52:43.827 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:52:43.827 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:52:43.827 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:52:43.827 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:52:43.828 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-26T21:52:43.828 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:52:43.828 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:52:43.828 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:52:43.828 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-26T21:52:43.828 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-26T21:52:43.828 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-26T21:52:43.828 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:52:43.828 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:52:43.829 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:52:43.829 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:52:43.829 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 322930, 2024-04-26T21:52:43.829 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 846, 2024-04-26T21:52:43.829 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 440, 2024-04-26T21:52:43.829 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 5, 2024-04-26T21:52:43.829 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-26T21:52:43.829 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-26T21:52:43.829 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:52:43.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-26T21:52:43.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:52:43.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-26T21:52:43.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 10040, 2024-04-26T21:52:43.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 211, 2024-04-26T21:52:43.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:52:43.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:52:43.830 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-26T21:52:43.831 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-26T21:52:43.831 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 6171137, 2024-04-26T21:52:43.831 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 48083, 2024-04-26T21:52:43.831 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-26T21:52:43.831 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-26T21:52:43.831 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-26T21:52:43.831 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-26T21:52:43.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.831 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-26T21:52:43.832 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-26T21:52:43.832 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-26T21:52:43.832 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 128913, 2024-04-26T21:52:43.832 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 454621950, 2024-04-26T21:52:43.832 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 32, 2024-04-26T21:52:43.832 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 128913, 2024-04-26T21:52:43.832 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-26T21:52:43.832 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 128913, 2024-04-26T21:52:43.833 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 5900.827951580, 2024-04-26T21:52:43.833 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.045773722 2024-04-26T21:52:43.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.833 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-26T21:52:43.833 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-26T21:52:43.833 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 128913, 2024-04-26T21:52:43.833 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 173895 2024-04-26T21:52:43.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.834 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 128913, 2024-04-26T21:52:43.834 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 1181, 2024-04-26T21:52:43.834 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 127732, 2024-04-26T21:52:43.834 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-26T21:52:43.834 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-26T21:52:43.834 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 2392, 2024-04-26T21:52:43.834 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 24357, 2024-04-26T21:52:43.834 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 1173, 2024-04-26T21:52:43.835 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 42056, 2024-04-26T21:52:43.835 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 1022, 2024-04-26T21:52:43.835 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-26T21:52:43.835 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-26T21:52:43.835 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-26T21:52:43.835 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 6, 2024-04-26T21:52:43.835 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 56501, 2024-04-26T21:52:43.835 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-26T21:52:43.835 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-26T21:52:43.836 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-26T21:52:43.836 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 5, 2024-04-26T21:52:43.836 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 36848, 2024-04-26T21:52:43.836 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-26T21:52:43.836 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-26T21:52:43.836 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-26T21:52:43.836 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-26T21:52:43.836 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-26T21:52:43.837 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-26T21:52:43.837 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T21:52:43.837 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-26T21:52:43.837 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-26T21:52:43.837 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-26T21:52:43.837 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-26T21:52:43.837 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-26T21:52:43.837 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-26T21:52:43.837 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-26T21:52:43.838 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-26T21:52:43.838 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-26T21:52:43.838 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-26T21:52:43.838 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-26T21:52:43.838 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-26T21:52:43.838 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-26T21:52:43.838 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-26T21:52:43.838 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-26T21:52:43.839 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-26T21:52:43.839 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-26T21:52:43.839 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-26T21:52:43.839 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-26T21:52:43.839 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 134, 2024-04-26T21:52:43.839 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-26T21:52:43.839 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 48, 2024-04-26T21:52:43.839 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 12, 2024-04-26T21:52:43.839 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 12, 2024-04-26T21:52:43.840 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-26T21:52:43.840 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-26T21:52:43.840 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 7077, 2024-04-26T21:52:43.840 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 12, 2024-04-26T21:52:43.840 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 2446 2024-04-26T21:52:43.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.840 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-26T21:52:43.840 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 1, 2024-04-26T21:52:43.840 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-26T21:52:43.841 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-26T21:52:43.841 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-26T21:52:43.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.841 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-26T21:52:43.841 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 2 2024-04-26T21:52:43.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.841 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-26T21:52:43.841 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-26T21:52:43.841 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 16, 2024-04-26T21:52:43.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-26T21:52:43.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 8, 2024-04-26T21:52:43.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 55772, 2024-04-26T21:52:43.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 27886, 2024-04-26T21:52:43.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-26T21:52:43.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.842 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:52:43.842 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:43.843 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-26T21:52:43.843 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:43.843 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 805018, 2024-04-26T21:52:43.843 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 179979533, 2024-04-26T21:52:43.843 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:43.843 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 805018, 2024-04-26T21:52:43.843 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-26T21:52:43.843 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-26T21:52:43.843 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 805018, 2024-04-26T21:52:43.844 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 179979533, 2024-04-26T21:52:43.844 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:43.844 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.844 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.844 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.844 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:43.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.844 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-26T21:52:43.844 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-26T21:52:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-26T21:52:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-26T21:52:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 128913, 2024-04-26T21:52:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 457666444, 2024-04-26T21:52:43.845 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 128913, 2024-04-26T21:52:43.846 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 457666444, 2024-04-26T21:52:43.846 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:43.846 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.846 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.846 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:43.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.846 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-26T21:52:43.847 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:43.847 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-26T21:52:43.847 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:43.847 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-26T21:52:43.847 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-26T21:52:43.847 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:43.847 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:43.847 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 128913, 2024-04-26T21:52:43.847 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 128913, 2024-04-26T21:52:43.848 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 128913, 2024-04-26T21:52:43.848 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 128913, 2024-04-26T21:52:43.848 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:43.848 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.848 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.848 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.848 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:43.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.848 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-26T21:52:43.849 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:43.849 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-26T21:52:43.849 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:43.849 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 27886, 2024-04-26T21:52:43.849 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 3044590, 2024-04-26T21:52:43.849 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:43.849 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 27886, 2024-04-26T21:52:43.849 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-26T21:52:43.849 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-26T21:52:43.850 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 1173, 2024-04-26T21:52:43.850 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 3044590, 2024-04-26T21:52:43.850 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:43.850 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.850 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.850 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.850 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:43.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:43.851 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x55a5dbad4460": { 2024-04-26T21:52:43.851 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 83697, 2024-04-26T21:52:43.851 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-26T21:52:43.851 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:43.851 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 147792, 2024-04-26T21:52:43.851 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 426527009, 2024-04-26T21:52:43.851 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:43.851 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 147792, 2024-04-26T21:52:43.851 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-26T21:52:43.852 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-26T21:52:43.852 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 40883, 2024-04-26T21:52:43.852 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 426443312, 2024-04-26T21:52:43.852 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:43.852 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:43.852 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:43.852 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:43.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:43.853 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:43.853 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-26T21:52:43.853 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:43 smithi001 ceph-mon[28054]: pgmap v713: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 2.2 MiB/s wr, 112 op/s 2024-04-26T21:52:44.717 INFO:tasks.check_counter:Daemon mds.b mds.exported=8 2024-04-26T21:52:44.718 INFO:tasks.check_counter:Daemon mds.b mds.imported=9 2024-04-26T21:52:44.718 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=443 2024-04-26T21:52:44.718 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=1229 2024-04-26T21:52:44.718 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-26T21:52:44.718 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=22 2024-04-26T21:52:44.718 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=16 2024-04-26T21:52:44.718 DEBUG:tasks.check_counter:Getting stats from e 2024-04-26T21:52:44.719 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-26T21:52:44.967 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:52:46.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:46 smithi001 ceph-mon[28054]: pgmap v714: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-04-26T21:52:46.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:46 smithi038 ceph-mon[32724]: pgmap v714: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-04-26T21:52:46.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:46 smithi186 ceph-mon[32991]: pgmap v714: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-04-26T21:52:47.433 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-26T21:52:47.433 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:52:47.433 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 2004, 2024-04-26T21:52:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 1954, 2024-04-26T21:52:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 356277160, 2024-04-26T21:52:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 740056, 2024-04-26T21:52:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 7, 2024-04-26T21:52:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-26T21:52:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.383849664, 2024-04-26T21:52:47.434 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.052123815, 2024-04-26T21:52:47.435 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.283454527, 2024-04-26T21:52:47.435 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.046407963, 2024-04-26T21:52:47.435 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:47.435 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1954, 2024-04-26T21:52:47.435 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.083114828, 2024-04-26T21:52:47.435 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000042535 2024-04-26T21:52:47.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.435 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:47.436 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.436 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.436 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.436 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1860032, 2024-04-26T21:52:47.436 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 267184 2024-04-26T21:52:47.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.437 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:52:47.437 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 2086, 2024-04-26T21:52:47.437 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2403, 2024-04-26T21:52:47.437 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 470316651, 2024-04-26T21:52:47.437 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 1461231, 2024-04-26T21:52:47.437 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 10, 2024-04-26T21:52:47.437 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 7, 2024-04-26T21:52:47.437 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.469699162, 2024-04-26T21:52:47.438 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.049227289, 2024-04-26T21:52:47.438 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.348819819, 2024-04-26T21:52:47.438 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.053336061, 2024-04-26T21:52:47.438 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:47.438 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 2400, 2024-04-26T21:52:47.438 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.117460782, 2024-04-26T21:52:47.438 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000048941 2024-04-26T21:52:47.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.439 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:47.439 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.439 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.439 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.439 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T21:52:47.439 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 798832 2024-04-26T21:52:47.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.439 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:52:47.440 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 3083, 2024-04-26T21:52:47.440 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 3084, 2024-04-26T21:52:47.440 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 290316191, 2024-04-26T21:52:47.440 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 980134, 2024-04-26T21:52:47.440 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 6, 2024-04-26T21:52:47.440 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 5, 2024-04-26T21:52:47.440 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.407692180, 2024-04-26T21:52:47.440 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.077788019, 2024-04-26T21:52:47.440 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.220606707, 2024-04-26T21:52:47.441 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.085714445, 2024-04-26T21:52:47.441 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:47.441 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 3082, 2024-04-26T21:52:47.441 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.117512079, 2024-04-26T21:52:47.441 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000038128 2024-04-26T21:52:47.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.441 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:47.441 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.441 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.441 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.442 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T21:52:47.442 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T21:52:47.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.442 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-26T21:52:47.442 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-26T21:52:47.442 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-26T21:52:47.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.442 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-26T21:52:47.443 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-26T21:52:47.443 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-26T21:52:47.443 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 10682, 2024-04-26T21:52:47.443 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.291806558, 2024-04-26T21:52:47.443 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000027317 2024-04-26T21:52:47.443 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:47.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.443 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-26T21:52:47.443 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-26T21:52:47.444 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-26T21:52:47.444 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.444 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.444 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.444 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:47.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.444 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-26T21:52:47.444 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 0, 2024-04-26T21:52:47.445 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 0, 2024-04-26T21:52:47.445 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-26T21:52:47.445 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.445 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.445 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.445 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-26T21:52:47.445 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 0, 2024-04-26T21:52:47.446 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 0, 2024-04-26T21:52:47.446 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-26T21:52:47.446 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 0, 2024-04-26T21:52:47.446 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 0, 2024-04-26T21:52:47.446 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 0, 2024-04-26T21:52:47.446 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 30507, 2024-04-26T21:52:47.446 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 9339, 2024-04-26T21:52:47.446 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 21168, 2024-04-26T21:52:47.447 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 0, 2024-04-26T21:52:47.447 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 17166, 2024-04-26T21:52:47.447 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 0, 2024-04-26T21:52:47.447 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 0, 2024-04-26T21:52:47.447 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 0, 2024-04-26T21:52:47.447 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 11, 2024-04-26T21:52:47.447 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 0, 2024-04-26T21:52:47.447 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 0, 2024-04-26T21:52:47.447 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 0, 2024-04-26T21:52:47.448 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 0, 2024-04-26T21:52:47.448 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 0, 2024-04-26T21:52:47.448 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-26T21:52:47.448 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 0, 2024-04-26T21:52:47.448 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 0, 2024-04-26T21:52:47.448 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-26T21:52:47.448 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 0, 2024-04-26T21:52:47.448 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 0, 2024-04-26T21:52:47.449 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 0, 2024-04-26T21:52:47.449 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 0, 2024-04-26T21:52:47.449 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-26T21:52:47.449 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 0, 2024-04-26T21:52:47.449 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-26T21:52:47.449 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 0, 2024-04-26T21:52:47.449 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 0, 2024-04-26T21:52:47.449 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 0, 2024-04-26T21:52:47.450 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T21:52:47.450 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-26T21:52:47.450 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T21:52:47.450 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-26T21:52:47.450 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 0, 2024-04-26T21:52:47.450 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T21:52:47.450 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T21:52:47.450 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 0, 2024-04-26T21:52:47.450 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 0, 2024-04-26T21:52:47.451 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T21:52:47.451 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 0, 2024-04-26T21:52:47.451 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T21:52:47.451 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T21:52:47.451 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T21:52:47.451 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 0, 2024-04-26T21:52:47.451 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 0, 2024-04-26T21:52:47.451 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 0, 2024-04-26T21:52:47.452 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 0 2024-04-26T21:52:47.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.452 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-26T21:52:47.452 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 0, 2024-04-26T21:52:47.452 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 0, 2024-04-26T21:52:47.452 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 0, 2024-04-26T21:52:47.452 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 0, 2024-04-26T21:52:47.452 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 0, 2024-04-26T21:52:47.452 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 0, 2024-04-26T21:52:47.452 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-26T21:52:47.453 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-26T21:52:47.453 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 0, 2024-04-26T21:52:47.453 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 0, 2024-04-26T21:52:47.453 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-26T21:52:47.453 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-26T21:52:47.453 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-26T21:52:47.453 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-26T21:52:47.453 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-26T21:52:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 0, 2024-04-26T21:52:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 0, 2024-04-26T21:52:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-26T21:52:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-26T21:52:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T21:52:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 0, 2024-04-26T21:52:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-26T21:52:47.454 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 0, 2024-04-26T21:52:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-26T21:52:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-26T21:52:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-26T21:52:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-26T21:52:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 0, 2024-04-26T21:52:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 0, 2024-04-26T21:52:47.455 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 0, 2024-04-26T21:52:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 161767, 2024-04-26T21:52:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-26T21:52:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 0, 2024-04-26T21:52:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 0, 2024-04-26T21:52:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 0, 2024-04-26T21:52:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 0, 2024-04-26T21:52:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 144, 2024-04-26T21:52:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 15, 2024-04-26T21:52:47.456 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-26T21:52:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 0, 2024-04-26T21:52:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 544230835, 2024-04-26T21:52:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 1117386011, 2024-04-26T21:52:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 1117369546, 2024-04-26T21:52:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-26T21:52:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.457 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 161767 2024-04-26T21:52:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-26T21:52:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 2264, 2024-04-26T21:52:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 31051, 2024-04-26T21:52:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 28787, 2024-04-26T21:52:47.458 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 105, 2024-04-26T21:52:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 233, 2024-04-26T21:52:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 128, 2024-04-26T21:52:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 30507, 2024-04-26T21:52:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 31789, 2024-04-26T21:52:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 1282, 2024-04-26T21:52:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 0, 2024-04-26T21:52:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 0, 2024-04-26T21:52:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 0, 2024-04-26T21:52:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 92240, 2024-04-26T21:52:47.459 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 215416 2024-04-26T21:52:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-26T21:52:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 0, 2024-04-26T21:52:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 0, 2024-04-26T21:52:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 0, 2024-04-26T21:52:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 0, 2024-04-26T21:52:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 0, 2024-04-26T21:52:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-26T21:52:47.460 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-26T21:52:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.461 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-26T21:52:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-26T21:52:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.463 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-26T21:52:47.463 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.463 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.463 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.463 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.463 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-26T21:52:47.463 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.463 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.463 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.464 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-26T21:52:47.464 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.464 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.464 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.464 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-26T21:52:47.464 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.464 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.465 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.465 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-26T21:52:47.465 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.465 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.465 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.465 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-26T21:52:47.466 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.466 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.466 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.466 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-26T21:52:47.466 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.466 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.466 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.467 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.467 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-26T21:52:47.467 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.467 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.467 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.468 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-26T21:52:47.468 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.468 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.468 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.468 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-26T21:52:47.468 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.469 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.469 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.469 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-26T21:52:47.469 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.469 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.469 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.469 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.469 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-26T21:52:47.470 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.470 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.470 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.470 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-26T21:52:47.470 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.470 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.470 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.471 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-26T21:52:47.471 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.471 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.471 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.471 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-26T21:52:47.471 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.472 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.472 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.472 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-26T21:52:47.472 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.472 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.472 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.473 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-26T21:52:47.473 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.473 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.473 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.473 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-26T21:52:47.473 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.473 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.474 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.474 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-26T21:52:47.474 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.474 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.474 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.475 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-26T21:52:47.475 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.475 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.475 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.475 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-26T21:52:47.475 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.475 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.475 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.476 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-26T21:52:47.476 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.476 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.476 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.476 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.476 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-26T21:52:47.477 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.477 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.477 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.477 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-26T21:52:47.477 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.477 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.477 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.478 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-26T21:52:47.478 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.478 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.478 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.478 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.478 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-26T21:52:47.478 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-26T21:52:47.478 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-26T21:52:47.478 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.479 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.479 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.479 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:47.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.479 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-26T21:52:47.479 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 2, 2024-04-26T21:52:47.479 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 4, 2024-04-26T21:52:47.479 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 2, 2024-04-26T21:52:47.479 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 4, 2024-04-26T21:52:47.480 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-26T21:52:47.480 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 0, 2024-04-26T21:52:47.480 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 0, 2024-04-26T21:52:47.480 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 4482, 2024-04-26T21:52:47.480 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-26T21:52:47.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.480 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-26T21:52:47.480 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:52:47.480 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-26T21:52:47.481 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:52:47.481 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:52:47.481 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:52:47.481 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:52:47.481 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:52:47.481 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:52:47.481 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:52:47.481 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:52:47.482 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:52:47.482 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:52:47.482 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:52:47.482 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:52:47.482 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:52:47.482 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-26T21:52:47.482 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:52:47.482 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-26T21:52:47.483 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:52:47.483 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:52:47.483 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:52:47.483 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:52:47.483 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:52:47.483 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:52:47.483 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:52:47.483 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-26T21:52:47.484 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:52:47.484 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:52:47.484 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:52:47.484 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-26T21:52:47.484 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-26T21:52:47.484 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-26T21:52:47.484 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:52:47.484 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:52:47.484 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:52:47.485 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:52:47.485 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 348023, 2024-04-26T21:52:47.485 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 2956, 2024-04-26T21:52:47.485 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-26T21:52:47.485 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-26T21:52:47.485 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-26T21:52:47.485 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-26T21:52:47.485 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:52:47.485 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-26T21:52:47.486 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:52:47.486 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-26T21:52:47.486 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 10040, 2024-04-26T21:52:47.486 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 211, 2024-04-26T21:52:47.486 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:52:47.486 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:52:47.486 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-26T21:52:47.486 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-26T21:52:47.487 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 27732443, 2024-04-26T21:52:47.487 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 188425, 2024-04-26T21:52:47.487 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-26T21:52:47.487 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-26T21:52:47.487 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-26T21:52:47.487 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-26T21:52:47.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.487 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-26T21:52:47.487 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-26T21:52:47.487 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-26T21:52:47.488 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 6652, 2024-04-26T21:52:47.488 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 21, 2024-04-26T21:52:47.488 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 3, 2024-04-26T21:52:47.488 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 6652, 2024-04-26T21:52:47.488 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-26T21:52:47.488 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 6652, 2024-04-26T21:52:47.488 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 25.633521236, 2024-04-26T21:52:47.488 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.003853505 2024-04-26T21:52:47.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.489 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-26T21:52:47.489 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-26T21:52:47.489 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 6652, 2024-04-26T21:52:47.489 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 6653 2024-04-26T21:52:47.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.489 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 6652, 2024-04-26T21:52:47.489 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 6652, 2024-04-26T21:52:47.489 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 0, 2024-04-26T21:52:47.489 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-26T21:52:47.490 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-26T21:52:47.490 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 3771, 2024-04-26T21:52:47.490 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 0, 2024-04-26T21:52:47.490 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 2879, 2024-04-26T21:52:47.490 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 0, 2024-04-26T21:52:47.490 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 0, 2024-04-26T21:52:47.490 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-26T21:52:47.490 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-26T21:52:47.491 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-26T21:52:47.491 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 0, 2024-04-26T21:52:47.491 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 0, 2024-04-26T21:52:47.491 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-26T21:52:47.491 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-26T21:52:47.491 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-26T21:52:47.491 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 1, 2024-04-26T21:52:47.491 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 0, 2024-04-26T21:52:47.491 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-26T21:52:47.491 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-26T21:52:47.492 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-26T21:52:47.492 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-26T21:52:47.492 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-26T21:52:47.492 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-26T21:52:47.492 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T21:52:47.492 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-26T21:52:47.492 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-26T21:52:47.493 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-26T21:52:47.493 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-26T21:52:47.493 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-26T21:52:47.493 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-26T21:52:47.493 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-26T21:52:47.493 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-26T21:52:47.493 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-26T21:52:47.493 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-26T21:52:47.493 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-26T21:52:47.493 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-26T21:52:47.494 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-26T21:52:47.494 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-26T21:52:47.494 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-26T21:52:47.494 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-26T21:52:47.494 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-26T21:52:47.494 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-26T21:52:47.494 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-26T21:52:47.495 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 134, 2024-04-26T21:52:47.495 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-26T21:52:47.495 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 43, 2024-04-26T21:52:47.495 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 11, 2024-04-26T21:52:47.495 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 11, 2024-04-26T21:52:47.495 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-26T21:52:47.495 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-26T21:52:47.495 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 0, 2024-04-26T21:52:47.495 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 2, 2024-04-26T21:52:47.496 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 0 2024-04-26T21:52:47.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.496 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-26T21:52:47.496 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 0, 2024-04-26T21:52:47.496 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-26T21:52:47.496 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-26T21:52:47.496 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-26T21:52:47.496 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.496 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-26T21:52:47.497 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 0 2024-04-26T21:52:47.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.497 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-26T21:52:47.497 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-26T21:52:47.497 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T21:52:47.497 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-26T21:52:47.497 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 0, 2024-04-26T21:52:47.497 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 0, 2024-04-26T21:52:47.497 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 0, 2024-04-26T21:52:47.498 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-26T21:52:47.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.498 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:52:47.498 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:47.498 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-26T21:52:47.498 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:47.498 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 7169, 2024-04-26T21:52:47.498 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1116235700, 2024-04-26T21:52:47.498 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:47.499 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 7169, 2024-04-26T21:52:47.499 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-26T21:52:47.499 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-26T21:52:47.499 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 7169, 2024-04-26T21:52:47.499 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1116235700, 2024-04-26T21:52:47.499 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:47.499 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.499 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.500 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:47.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.500 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-26T21:52:47.500 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:47.500 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-26T21:52:47.500 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:47.500 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-26T21:52:47.501 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-26T21:52:47.501 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:47.501 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:47.501 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 6652, 2024-04-26T21:52:47.501 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 1113226602, 2024-04-26T21:52:47.501 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 6652, 2024-04-26T21:52:47.501 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1113226602, 2024-04-26T21:52:47.501 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:47.501 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.502 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.502 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.502 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:47.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.502 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-26T21:52:47.502 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:47.502 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-26T21:52:47.502 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:47.503 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-26T21:52:47.503 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-26T21:52:47.503 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:47.503 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:47.503 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 6652, 2024-04-26T21:52:47.503 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 6652, 2024-04-26T21:52:47.503 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 6652, 2024-04-26T21:52:47.503 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 6652, 2024-04-26T21:52:47.503 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:47.503 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.504 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.504 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.504 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:47.504 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.504 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-26T21:52:47.504 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:47.504 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-26T21:52:47.504 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:47.505 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-26T21:52:47.505 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-26T21:52:47.505 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:47.505 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:47.505 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-26T21:52:47.505 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-26T21:52:47.505 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-26T21:52:47.505 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-26T21:52:47.506 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:47.506 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.506 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.506 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.506 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:47.506 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:47.506 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x55f0df9306e0": { 2024-04-26T21:52:47.506 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:47.506 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-26T21:52:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-26T21:52:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-26T21:52:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-26T21:52:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-26T21:52:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 0, 2024-04-26T21:52:47.507 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 0, 2024-04-26T21:52:47.508 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:47.508 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:47.508 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:47.508 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:47.508 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:47.508 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:47.508 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-26T21:52:48.396 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-26T21:52:48.397 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-26T21:52:48.397 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-04-26T21:52:48.397 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-04-26T21:52:48.397 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-04-26T21:52:48.397 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-04-26T21:52:48.397 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T21:52:48.397 DEBUG:tasks.check_counter:Getting stats from h 2024-04-26T21:52:48.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-26T21:52:48.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:48 smithi001 ceph-mon[28054]: pgmap v715: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 324 KiB/s wr, 40 op/s 2024-04-26T21:52:48.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:48 smithi038 ceph-mon[32724]: pgmap v715: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 324 KiB/s wr, 40 op/s 2024-04-26T21:52:48.682 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:52:48.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:48 smithi186 ceph-mon[32991]: pgmap v715: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 324 KiB/s wr, 40 op/s 2024-04-26T21:52:50.378 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:50 smithi038 ceph-mon[32724]: pgmap v716: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 100 KiB/s wr, 22 op/s 2024-04-26T21:52:50.513 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-26T21:52:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:52:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 640015, 2024-04-26T21:52:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 812877, 2024-04-26T21:52:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 208505630, 2024-04-26T21:52:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 735366838, 2024-04-26T21:52:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 14, 2024-04-26T21:52:50.513 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 4, 2024-04-26T21:52:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 45.053966959, 2024-04-26T21:52:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 19.156249071, 2024-04-26T21:52:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 30.777586115, 2024-04-26T21:52:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 1.828599850, 2024-04-26T21:52:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 812877, 2024-04-26T21:52:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 68.975517485, 2024-04-26T21:52:50.514 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000084853 2024-04-26T21:52:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1211137, 2024-04-26T21:52:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.250883508, 2024-04-26T21:52:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000207 2024-04-26T21:52:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 138516272, 2024-04-26T21:52:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 424245024 2024-04-26T21:52:50.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:52:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 55334, 2024-04-26T21:52:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 55458, 2024-04-26T21:52:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 13982931, 2024-04-26T21:52:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 613315085, 2024-04-26T21:52:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 14, 2024-04-26T21:52:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 6, 2024-04-26T21:52:50.516 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 5.656768073, 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 1.840210267, 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 1.054850913, 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 2.426509602, 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 55455, 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 5.869152531, 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000105836 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 127, 2024-04-26T21:52:50.517 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000027478, 2024-04-26T21:52:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000216 2024-04-26T21:52:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 9248, 2024-04-26T21:52:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 258192 2024-04-26T21:52:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:52:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 255144, 2024-04-26T21:52:50.518 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 302513, 2024-04-26T21:52:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 62966047, 2024-04-26T21:52:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 401240833, 2024-04-26T21:52:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 8, 2024-04-26T21:52:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 6, 2024-04-26T21:52:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 21.896991671, 2024-04-26T21:52:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 10.428756028, 2024-04-26T21:52:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 11.004515454, 2024-04-26T21:52:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 1.897893037, 2024-04-26T21:52:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:50.519 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 302511, 2024-04-26T21:52:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 18.784343936, 2024-04-26T21:52:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000062094 2024-04-26T21:52:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 483397, 2024-04-26T21:52:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.096340099, 2024-04-26T21:52:50.520 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000199 2024-04-26T21:52:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-26T21:52:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 573904 2024-04-26T21:52:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-26T21:52:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 1, 2024-04-26T21:52:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-26T21:52:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-MDSRank": { 2024-04-26T21:52:50.521 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-26T21:52:50.522 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-26T21:52:50.522 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 105959, 2024-04-26T21:52:50.522 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 158.467619900, 2024-04-26T21:52:50.522 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001495556 2024-04-26T21:52:50.522 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:50.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "finisher-PurgeQueue": { 2024-04-26T21:52:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "queue_len": 0, 2024-04-26T21:52:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "complete_latency": { 2024-04-26T21:52:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 50828, 2024-04-26T21:52:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 11.911776966, 2024-04-26T21:52:50.523 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000234354 2024-04-26T21:52:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "mds": { 2024-04-26T21:52:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "request": 259930, 2024-04-26T21:52:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "reply": 226174, 2024-04-26T21:52:50.524 INFO:teuthology.orchestra.run.smithi038.stdout: "reply_latency": { 2024-04-26T21:52:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 226174, 2024-04-26T21:52:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 1744.814740370, 2024-04-26T21:52:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.007714479 2024-04-26T21:52:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_reply": 0, 2024-04-26T21:52:50.525 INFO:teuthology.orchestra.run.smithi038.stdout: "forward": 33756, 2024-04-26T21:52:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_complete": 31, 2024-04-26T21:52:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_fetch_keys": 0, 2024-04-26T21:52:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_commit": 168, 2024-04-26T21:52:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_split": 12, 2024-04-26T21:52:50.526 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_merge": 7, 2024-04-26T21:52:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes": 3462, 2024-04-26T21:52:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_top": 1470, 2024-04-26T21:52:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_bottom": 1924, 2024-04-26T21:52:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pin_tail": 68, 2024-04-26T21:52:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_pinned": 1359, 2024-04-26T21:52:50.527 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_expired": 2099, 2024-04-26T21:52:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "inodes_with_caps": 270, 2024-04-26T21:52:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "caps": 270, 2024-04-26T21:52:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "subtrees": 12, 2024-04-26T21:52:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse": 394129, 2024-04-26T21:52:50.528 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_hit": 233827, 2024-04-26T21:52:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_forward": 33736, 2024-04-26T21:52:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_discover": 126, 2024-04-26T21:52:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_dir_fetch": 6, 2024-04-26T21:52:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_remote_ino": 0, 2024-04-26T21:52:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "traverse_lock": 3169, 2024-04-26T21:52:50.529 INFO:teuthology.orchestra.run.smithi038.stdout: "load_cent": 20149, 2024-04-26T21:52:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "q": 0, 2024-04-26T21:52:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "exported": 20, 2024-04-26T21:52:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "exported_inodes": 1369, 2024-04-26T21:52:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "imported": 14, 2024-04-26T21:52:50.530 INFO:teuthology.orchestra.run.smithi038.stdout: "imported_inodes": 597, 2024-04-26T21:52:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_dir_fetch": 0, 2024-04-26T21:52:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_backtrace_fetch": 0, 2024-04-26T21:52:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "openino_peer_discover": 0, 2024-04-26T21:52:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rfiles": 4, 2024-04-26T21:52:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rbytes": 234, 2024-04-26T21:52:50.531 INFO:teuthology.orchestra.run.smithi038.stdout: "root_rsnaps": 0, 2024-04-26T21:52:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T21:52:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_set_tag": 0, 2024-04-26T21:52:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T21:52:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_inotable_repaired": 0, 2024-04-26T21:52:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_inodes": 0, 2024-04-26T21:52:50.532 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T21:52:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T21:52:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_file_inodes": 0, 2024-04-26T21:52:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_inode_file_caps": 27, 2024-04-26T21:52:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_revoke": 35673, 2024-04-26T21:52:50.533 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_grant": 106314, 2024-04-26T21:52:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T21:52:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T21:52:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T21:52:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps": 82271, 2024-04-26T21:52:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_caps_dirty": 26613, 2024-04-26T21:52:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_cap_release": 10837, 2024-04-26T21:52:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: "process_request_cap_release": 148543 2024-04-26T21:52:50.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache": { 2024-04-26T21:52:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update": 275, 2024-04-26T21:52:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_update_receipt": 121, 2024-04-26T21:52:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_try_discover": 78, 2024-04-26T21:52:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_send_discover": 170, 2024-04-26T21:52:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "dir_handle_discover": 1529, 2024-04-26T21:52:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays": 0, 2024-04-26T21:52:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_delayed": 0, 2024-04-26T21:52:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "num_strays_enqueuing": 0, 2024-04-26T21:52:50.535 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_created": 27957, 2024-04-26T21:52:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_enqueued": 27957, 2024-04-26T21:52:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_reintegrated": 0, 2024-04-26T21:52:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "strays_migrated": 0, 2024-04-26T21:52:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_processing": 0, 2024-04-26T21:52:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_enqueued": 0, 2024-04-26T21:52:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "num_recovering_prioritized": 0, 2024-04-26T21:52:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_started": 0, 2024-04-26T21:52:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_completed": 0, 2024-04-26T21:52:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_path": 0, 2024-04-26T21:52:50.536 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_quiesce_inode": 0, 2024-04-26T21:52:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T21:52:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_exportdir": 22, 2024-04-26T21:52:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_flush": 0, 2024-04-26T21:52:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragmentdir": 19, 2024-04-26T21:52:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_fragstats": 0, 2024-04-26T21:52:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "ireq_inodestats": 0 2024-04-26T21:52:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log": { 2024-04-26T21:52:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "evlrg": 0, 2024-04-26T21:52:50.537 INFO:teuthology.orchestra.run.smithi038.stdout: "evadd": 161782, 2024-04-26T21:52:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "evex": 96725, 2024-04-26T21:52:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "evtrm": 86031, 2024-04-26T21:52:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "ev": 75751, 2024-04-26T21:52:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "evexg": 0, 2024-04-26T21:52:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "evexd": 10694, 2024-04-26T21:52:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "segadd": 292, 2024-04-26T21:52:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "segex": 164, 2024-04-26T21:52:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "segtrm": 148, 2024-04-26T21:52:50.538 INFO:teuthology.orchestra.run.smithi038.stdout: "seg": 144, 2024-04-26T21:52:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "segmjr": 7, 2024-04-26T21:52:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "segexg": 0, 2024-04-26T21:52:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "segexd": 16, 2024-04-26T21:52:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "expos": 544230835, 2024-04-26T21:52:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "wrpos": 1117740443, 2024-04-26T21:52:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "rdpos": 0, 2024-04-26T21:52:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "jlat": { 2024-04-26T21:52:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 57629, 2024-04-26T21:52:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 554.010983855, 2024-04-26T21:52:50.539 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.009613406 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "replayed": 0 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mem": { 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "ino": 2244, 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "ino+": 31498, 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "ino-": 29254, 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "dir": 220, 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "dir+": 377, 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "dir-": 157, 2024-04-26T21:52:50.540 INFO:teuthology.orchestra.run.smithi038.stdout: "dn": 3470, 2024-04-26T21:52:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "dn+": 61392, 2024-04-26T21:52:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "dn-": 57922, 2024-04-26T21:52:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "cap": 262, 2024-04-26T21:52:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "cap+": 31369, 2024-04-26T21:52:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "cap-": 31107, 2024-04-26T21:52:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "rss": 86128, 2024-04-26T21:52:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "heap": 223608 2024-04-26T21:52:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.541 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_server": { 2024-04-26T21:52:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_client_request": 367367, 2024-04-26T21:52:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "dispatch_server_request": 36917, 2024-04-26T21:52:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_request": 259930, 2024-04-26T21:52:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_client_session": 3433, 2024-04-26T21:52:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "handle_peer_request": 73719, 2024-04-26T21:52:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "req_create_latency": { 2024-04-26T21:52:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 28780, 2024-04-26T21:52:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 262.409392114, 2024-04-26T21:52:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.009117769 2024-04-26T21:52:50.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getattr_latency": { 2024-04-26T21:52:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 29442, 2024-04-26T21:52:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 364.220566670, 2024-04-26T21:52:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.012370782 2024-04-26T21:52:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getfilelock_latency": { 2024-04-26T21:52:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "req_link_latency": { 2024-04-26T21:52:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookup_latency": { 2024-04-26T21:52:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 60024, 2024-04-26T21:52:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 316.657856500, 2024-04-26T21:52:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.005275520 2024-04-26T21:52:50.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookuphash_latency": { 2024-04-26T21:52:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupino_latency": { 2024-04-26T21:52:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupname_latency": { 2024-04-26T21:52:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupparent_latency": { 2024-04-26T21:52:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.546 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lookupsnap_latency": { 2024-04-26T21:52:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.547 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "req_lssnap_latency": { 2024-04-26T21:52:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mkdir_latency": { 2024-04-26T21:52:50.548 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 112, 2024-04-26T21:52:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.538103623, 2024-04-26T21:52:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.004804496 2024-04-26T21:52:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mknod_latency": { 2024-04-26T21:52:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "req_mksnap_latency": { 2024-04-26T21:52:50.549 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "req_open_latency": { 2024-04-26T21:52:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 9, 2024-04-26T21:52:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.010450036, 2024-04-26T21:52:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001161115 2024-04-26T21:52:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "req_readdir_latency": { 2024-04-26T21:52:50.550 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 73161, 2024-04-26T21:52:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 521.283049433, 2024-04-26T21:52:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.007125149 2024-04-26T21:52:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rename_latency": { 2024-04-26T21:52:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 5680, 2024-04-26T21:52:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 61.297466422, 2024-04-26T21:52:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.010791807 2024-04-26T21:52:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.551 INFO:teuthology.orchestra.run.smithi038.stdout: "req_renamesnap_latency": { 2024-04-26T21:52:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "req_snapdiff_latency": { 2024-04-26T21:52:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.552 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmdir_latency": { 2024-04-26T21:52:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 1, 2024-04-26T21:52:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.001316117, 2024-04-26T21:52:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.001316117 2024-04-26T21:52:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmsnap_latency": { 2024-04-26T21:52:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "req_rmxattr_latency": { 2024-04-26T21:52:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setattr_latency": { 2024-04-26T21:52:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 148, 2024-04-26T21:52:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 5.738357973, 2024-04-26T21:52:50.554 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.038772689 2024-04-26T21:52:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setdirlayout_latency": { 2024-04-26T21:52:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setfilelock_latency": { 2024-04-26T21:52:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 860, 2024-04-26T21:52:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 3.452133768, 2024-04-26T21:52:50.555 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.004014109 2024-04-26T21:52:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setlayout_latency": { 2024-04-26T21:52:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "req_setxattr_latency": { 2024-04-26T21:52:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.556 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "req_symlink_latency": { 2024-04-26T21:52:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "req_unlink_latency": { 2024-04-26T21:52:50.557 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 27957, 2024-04-26T21:52:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 209.206047714, 2024-04-26T21:52:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.007483136 2024-04-26T21:52:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_revoke_eviction": 0, 2024-04-26T21:52:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "cap_acquisition_throttle": 0, 2024-04-26T21:52:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "req_getvxattr_latency": { 2024-04-26T21:52:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.558 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessions": { 2024-04-26T21:52:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "session_count": 2, 2024-04-26T21:52:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "session_add": 4, 2024-04-26T21:52:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "session_remove": 2, 2024-04-26T21:52:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_open": 2, 2024-04-26T21:52:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "sessions_stale": 0, 2024-04-26T21:52:50.559 INFO:teuthology.orchestra.run.smithi038.stdout: "total_load": 26634, 2024-04-26T21:52:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "average_load": 13317, 2024-04-26T21:52:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "avg_session_uptime": 1328, 2024-04-26T21:52:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "mdthresh_evicted": 0 2024-04-26T21:52:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-26T21:52:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 360, 2024-04-26T21:52:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 360, 2024-04-26T21:52:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:52:50.560 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:52:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:52:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:52:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:52:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:52:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:52:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:52:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:52:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:52:50.561 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:52:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:52:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:52:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-26T21:52:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:52:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-26T21:52:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:52:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:52:50.562 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:52:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:52:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:52:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:52:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:52:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-26T21:52:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:52:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:52:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:52:50.563 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-26T21:52:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-26T21:52:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-26T21:52:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:52:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:52:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:52:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:52:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 341457, 2024-04-26T21:52:50.564 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 1259, 2024-04-26T21:52:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 352, 2024-04-26T21:52:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 4, 2024-04-26T21:52:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-26T21:52:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-26T21:52:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:52:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-26T21:52:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:52:50.565 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-26T21:52:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 10040, 2024-04-26T21:52:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 211, 2024-04-26T21:52:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:52:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:52:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-26T21:52:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-26T21:52:50.566 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 8906538, 2024-04-26T21:52:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 79449, 2024-04-26T21:52:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-26T21:52:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-26T21:52:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-26T21:52:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-26T21:52:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.567 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter": { 2024-04-26T21:52:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "op_active": 0, 2024-04-26T21:52:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "op_laggy": 0, 2024-04-26T21:52:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send": 118079, 2024-04-26T21:52:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "op_send_bytes": 1121866251, 2024-04-26T21:52:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "op_resend": 27, 2024-04-26T21:52:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "op_reply": 118079, 2024-04-26T21:52:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "op_latency": { 2024-04-26T21:52:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 118079, 2024-04-26T21:52:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 1785.961680882, 2024-04-26T21:52:50.568 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.015125142 2024-04-26T21:52:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "op_inflight": 0, 2024-04-26T21:52:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "oplen_avg": { 2024-04-26T21:52:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 118079, 2024-04-26T21:52:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 120565 2024-04-26T21:52:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "op": 118079, 2024-04-26T21:52:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "op_r": 971, 2024-04-26T21:52:50.569 INFO:teuthology.orchestra.run.smithi038.stdout: "op_w": 117108, 2024-04-26T21:52:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "op_rmw": 0, 2024-04-26T21:52:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "op_pg": 0, 2024-04-26T21:52:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_stat": 13, 2024-04-26T21:52:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_create": 505, 2024-04-26T21:52:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_read": 940, 2024-04-26T21:52:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_write": 58569, 2024-04-26T21:52:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writefull": 520, 2024-04-26T21:52:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_writesame": 0, 2024-04-26T21:52:50.570 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_append": 0, 2024-04-26T21:52:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_zero": 0, 2024-04-26T21:52:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_truncate": 12, 2024-04-26T21:52:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_delete": 56375, 2024-04-26T21:52:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_mapext": 0, 2024-04-26T21:52:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_sparse_read": 0, 2024-04-26T21:52:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_clonerange": 0, 2024-04-26T21:52:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_getxattr": 10, 2024-04-26T21:52:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_setxattr": 794, 2024-04-26T21:52:50.571 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_cmpxattr": 0, 2024-04-26T21:52:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_rmxattr": 0, 2024-04-26T21:52:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_resetxattrs": 0, 2024-04-26T21:52:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_call": 0, 2024-04-26T21:52:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_watch": 0, 2024-04-26T21:52:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_notify": 0, 2024-04-26T21:52:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T21:52:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls": 0, 2024-04-26T21:52:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_pgls_filter": 0, 2024-04-26T21:52:50.572 INFO:teuthology.orchestra.run.smithi038.stdout: "osdop_other": 0, 2024-04-26T21:52:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_active": 0, 2024-04-26T21:52:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_send": 0, 2024-04-26T21:52:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_resend": 0, 2024-04-26T21:52:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "linger_ping": 0, 2024-04-26T21:52:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_active": 0, 2024-04-26T21:52:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_send": 0, 2024-04-26T21:52:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "poolop_resend": 0, 2024-04-26T21:52:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_active": 0, 2024-04-26T21:52:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_send": 0, 2024-04-26T21:52:50.573 INFO:teuthology.orchestra.run.smithi038.stdout: "poolstat_resend": 0, 2024-04-26T21:52:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_active": 0, 2024-04-26T21:52:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_send": 0, 2024-04-26T21:52:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "statfs_resend": 0, 2024-04-26T21:52:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "command_active": 0, 2024-04-26T21:52:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "command_send": 0, 2024-04-26T21:52:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "command_resend": 0, 2024-04-26T21:52:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "map_epoch": 134, 2024-04-26T21:52:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "map_full": 0, 2024-04-26T21:52:50.574 INFO:teuthology.orchestra.run.smithi038.stdout: "map_inc": 43, 2024-04-26T21:52:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_sessions": 12, 2024-04-26T21:52:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_open": 12, 2024-04-26T21:52:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_session_close": 0, 2024-04-26T21:52:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_laggy": 0, 2024-04-26T21:52:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_wr": 2152, 2024-04-26T21:52:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_rd": 62, 2024-04-26T21:52:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_del": 613 2024-04-26T21:52:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "oft": { 2024-04-26T21:52:50.575 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_objs": 1, 2024-04-26T21:52:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_kv_pairs": 0, 2024-04-26T21:52:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_updates": 0, 2024-04-26T21:52:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "omap_total_removes": 0 2024-04-26T21:52:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "osd-slow-ops": { 2024-04-26T21:52:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "slow_ops_count": 0 2024-04-26T21:52:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.576 INFO:teuthology.orchestra.run.smithi038.stdout: "purge_queue": { 2024-04-26T21:52:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops": 0, 2024-04-26T21:52:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_ops_high_water": 16, 2024-04-26T21:52:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing": 0, 2024-04-26T21:52:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executing_high_water": 8, 2024-04-26T21:52:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed_ops": 55914, 2024-04-26T21:52:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_executed": 27957, 2024-04-26T21:52:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "pq_item_in_journal": 0 2024-04-26T21:52:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:52:50.577 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-26T21:52:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 950489, 2024-04-26T21:52:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 183130259, 2024-04-26T21:52:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 950489, 2024-04-26T21:52:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-26T21:52:50.578 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-26T21:52:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 950489, 2024-04-26T21:52:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 183130259, 2024-04-26T21:52:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.579 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_bytes": { 2024-04-26T21:52:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-26T21:52:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-26T21:52:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-26T21:52:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:50.580 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 118079, 2024-04-26T21:52:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 1124918748, 2024-04-26T21:52:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 118079, 2024-04-26T21:52:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1124918748, 2024-04-26T21:52:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.581 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-objecter_ops": { 2024-04-26T21:52:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 1024, 2024-04-26T21:52:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 0, 2024-04-26T21:52:50.582 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 0, 2024-04-26T21:52:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 118079, 2024-04-26T21:52:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 118079, 2024-04-26T21:52:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 118079, 2024-04-26T21:52:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 118079, 2024-04-26T21:52:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle": { 2024-04-26T21:52:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-26T21:52:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:50.584 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 27957, 2024-04-26T21:52:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 3052993, 2024-04-26T21:52:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 27957, 2024-04-26T21:52:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-26T21:52:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-26T21:52:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 940, 2024-04-26T21:52:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 3052993, 2024-04-26T21:52:50.585 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-write_buf_throttle-0x5564d6e7a460": { 2024-04-26T21:52:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 3758096384, 2024-04-26T21:52:50.586 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 161782, 2024-04-26T21:52:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1113546139, 2024-04-26T21:52:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 161782, 2024-04-26T21:52:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-26T21:52:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-26T21:52:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 57629, 2024-04-26T21:52:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1113546139, 2024-04-26T21:52:50.587 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:50.588 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:50.588 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-26T21:52:50.589 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:50 smithi001 ceph-mon[28054]: pgmap v716: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 100 KiB/s wr, 22 op/s 2024-04-26T21:52:50.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:50 smithi186 ceph-mon[32991]: pgmap v716: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 100 KiB/s wr, 22 op/s 2024-04-26T21:52:51.326 INFO:tasks.check_counter:Daemon mds.h mds.exported=20 2024-04-26T21:52:51.326 INFO:tasks.check_counter:Daemon mds.h mds.imported=14 2024-04-26T21:52:51.326 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=275 2024-04-26T21:52:51.326 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=121 2024-04-26T21:52:51.326 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-04-26T21:52:51.326 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-26T21:52:51.326 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T21:52:51.326 DEBUG:tasks.check_counter:Getting stats from k 2024-04-26T21:52:51.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-26T21:52:51.575 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T21:52:52.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:52 smithi038 ceph-mon[32724]: pgmap v717: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 130 KiB/s wr, 21 op/s 2024-04-26T21:52:52.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:52 smithi186 ceph-mon[32991]: pgmap v717: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 130 KiB/s wr, 21 op/s 2024-04-26T21:52:52.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:52 smithi001 ceph-mon[28054]: pgmap v717: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 130 KiB/s wr, 21 op/s 2024-04-26T21:52:53.418 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-04-26T21:52:53.418 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:52:53.418 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 432, 2024-04-26T21:52:53.419 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 421, 2024-04-26T21:52:53.419 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1649856, 2024-04-26T21:52:53.419 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 260896, 2024-04-26T21:52:53.419 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-04-26T21:52:53.419 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-04-26T21:52:53.419 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.063566045, 2024-04-26T21:52:53.419 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.020238609, 2024-04-26T21:52:53.419 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.039646518, 2024-04-26T21:52:53.420 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.009464221, 2024-04-26T21:52:53.420 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:53.420 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 420, 2024-04-26T21:52:53.420 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.019245833, 2024-04-26T21:52:53.420 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000045823 2024-04-26T21:52:53.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.420 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:53.420 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:53.421 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:53.421 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:53.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.421 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1649856, 2024-04-26T21:52:53.421 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 260896 2024-04-26T21:52:53.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.421 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:52:53.421 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 7, 2024-04-26T21:52:53.421 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 325, 2024-04-26T21:52:53.421 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 3472, 2024-04-26T21:52:53.422 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 293216, 2024-04-26T21:52:53.422 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 6, 2024-04-26T21:52:53.422 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-04-26T21:52:53.422 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.016265227, 2024-04-26T21:52:53.422 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.011004648, 2024-04-26T21:52:53.422 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.002059853, 2024-04-26T21:52:53.422 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:52:53.422 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:53.423 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 323, 2024-04-26T21:52:53.423 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.045188196, 2024-04-26T21:52:53.423 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000139901 2024-04-26T21:52:53.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.423 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:53.423 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:53.423 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:53.423 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:53.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.424 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-04-26T21:52:53.424 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 293216 2024-04-26T21:52:53.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.424 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:52:53.424 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-04-26T21:52:53.424 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-04-26T21:52:53.424 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-04-26T21:52:53.424 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-04-26T21:52:53.425 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-04-26T21:52:53.425 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-04-26T21:52:53.425 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.000183375, 2024-04-26T21:52:53.425 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T21:52:53.425 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000119444, 2024-04-26T21:52:53.425 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:52:53.425 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:53.425 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:53.425 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:53.426 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:53.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.426 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:53.426 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:53.426 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:53.426 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:53.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.426 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T21:52:53.427 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T21:52:53.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.427 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-04-26T21:52:53.427 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-04-26T21:52:53.427 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-04-26T21:52:53.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.427 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-04-26T21:52:53.427 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:52:53.427 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-04-26T21:52:53.427 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:52:53.428 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:52:53.428 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:52:53.428 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:52:53.428 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:52:53.428 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:52:53.428 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:52:53.428 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:52:53.428 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:52:53.428 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:52:53.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:52:53.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:52:53.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:52:53.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-04-26T21:52:53.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:52:53.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-04-26T21:52:53.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:52:53.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:52:53.429 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:52:53.430 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:52:53.430 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:52:53.430 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:52:53.430 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:52:53.430 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-04-26T21:52:53.430 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:52:53.430 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:52:53.430 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:52:53.430 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-04-26T21:52:53.431 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-04-26T21:52:53.431 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-04-26T21:52:53.431 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:52:53.431 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:52:53.431 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:52:53.431 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:52:53.431 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148732, 2024-04-26T21:52:53.431 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 31, 2024-04-26T21:52:53.431 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-04-26T21:52:53.432 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-04-26T21:52:53.432 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-04-26T21:52:53.432 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-04-26T21:52:53.432 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:52:53.432 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-04-26T21:52:53.432 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:52:53.432 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-04-26T21:52:53.432 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-04-26T21:52:53.433 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-04-26T21:52:53.433 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:52:53.433 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:52:53.433 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-04-26T21:52:53.433 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-04-26T21:52:53.433 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-04-26T21:52:53.433 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-04-26T21:52:53.433 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-04-26T21:52:53.433 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-04-26T21:52:53.433 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-04-26T21:52:53.434 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-04-26T21:52:53.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-04-26T21:52:53.434 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:52:53.434 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-04-26T21:52:53.434 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-04-26T21:52:53.434 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-04-26T21:52:53.434 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 435, 2024-04-26T21:52:53.434 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1588837, 2024-04-26T21:52:53.434 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:53.435 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 435, 2024-04-26T21:52:53.435 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-04-26T21:52:53.435 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-04-26T21:52:53.435 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 435, 2024-04-26T21:52:53.435 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1588837, 2024-04-26T21:52:53.435 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-04-26T21:52:53.435 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-04-26T21:52:53.435 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-04-26T21:52:53.435 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-04-26T21:52:53.436 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:53.436 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-04-26T21:52:53.436 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-04-26T21:52:54.153 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-26T21:52:54.153 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-26T21:52:54.153 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-04-26T21:52:54.154 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-04-26T21:52:54.154 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-26T21:52:54.154 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-26T21:52:54.154 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-26T21:52:54.154 DEBUG:tasks.check_counter:Getting stats from c 2024-04-26T21:52:54.154 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-26T21:52:54.408 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:52:54.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:54 smithi186 ceph-mon[32991]: pgmap v718: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T21:52:54.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:54 smithi038 ceph-mon[32724]: pgmap v718: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T21:52:54.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:54 smithi001 ceph-mon[28054]: pgmap v718: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 57 KiB/s wr, 9 op/s 2024-04-26T21:52:56.277 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T21:52:56.278 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:52:56.278 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 3696, 2024-04-26T21:52:56.278 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 3696, 2024-04-26T21:52:56.278 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 290489408, 2024-04-26T21:52:56.278 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 1174511, 2024-04-26T21:52:56.279 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 8, 2024-04-26T21:52:56.279 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-04-26T21:52:56.279 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.467531604, 2024-04-26T21:52:56.279 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.094823622, 2024-04-26T21:52:56.279 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.241325261, 2024-04-26T21:52:56.279 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.104548162, 2024-04-26T21:52:56.279 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:56.279 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3695, 2024-04-26T21:52:56.279 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.128222098, 2024-04-26T21:52:56.280 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000034701 2024-04-26T21:52:56.280 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.280 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:56.280 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.280 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.280 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.280 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.280 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T21:52:56.281 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T21:52:56.281 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.281 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:52:56.281 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 1428, 2024-04-26T21:52:56.281 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 1745, 2024-04-26T21:52:56.281 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 243653168, 2024-04-26T21:52:56.281 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 1176602, 2024-04-26T21:52:56.282 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 9, 2024-04-26T21:52:56.282 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 7, 2024-04-26T21:52:56.282 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.254840817, 2024-04-26T21:52:56.282 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.040146573, 2024-04-26T21:52:56.282 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.167616886, 2024-04-26T21:52:56.282 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.034616776, 2024-04-26T21:52:56.282 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:56.282 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1742, 2024-04-26T21:52:56.283 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.078327234, 2024-04-26T21:52:56.283 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000044963 2024-04-26T21:52:56.283 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.283 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:56.283 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.283 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.283 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.283 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.283 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-26T21:52:56.284 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 723072 2024-04-26T21:52:56.284 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.284 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:52:56.284 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 1483, 2024-04-26T21:52:56.284 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 1435, 2024-04-26T21:52:56.284 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 160196832, 2024-04-26T21:52:56.284 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 575518, 2024-04-26T21:52:56.284 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 6, 2024-04-26T21:52:56.284 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-04-26T21:52:56.285 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.212495866, 2024-04-26T21:52:56.285 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.045748944, 2024-04-26T21:52:56.285 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.138773475, 2024-04-26T21:52:56.285 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.031474519, 2024-04-26T21:52:56.285 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:56.285 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1434, 2024-04-26T21:52:56.285 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.047816214, 2024-04-26T21:52:56.285 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000033344 2024-04-26T21:52:56.285 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.286 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:56.286 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.286 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.286 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.286 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.286 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1857312, 2024-04-26T21:52:56.286 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 267376 2024-04-26T21:52:56.286 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.287 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-26T21:52:56.287 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-04-26T21:52:56.287 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-26T21:52:56.287 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.287 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-04-26T21:52:56.287 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-26T21:52:56.287 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-26T21:52:56.287 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 10136, 2024-04-26T21:52:56.288 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.253082266, 2024-04-26T21:52:56.288 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000024968 2024-04-26T21:52:56.288 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:56.288 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.288 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-04-26T21:52:56.288 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-26T21:52:56.288 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-26T21:52:56.289 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.289 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.289 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.289 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:56.289 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.289 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-04-26T21:52:56.289 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 0, 2024-04-26T21:52:56.289 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 0, 2024-04-26T21:52:56.290 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-04-26T21:52:56.290 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.290 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.290 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.290 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.290 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-04-26T21:52:56.290 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 0, 2024-04-26T21:52:56.290 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-04-26T21:52:56.291 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-04-26T21:52:56.291 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 0, 2024-04-26T21:52:56.291 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 0, 2024-04-26T21:52:56.291 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-04-26T21:52:56.291 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 31777, 2024-04-26T21:52:56.291 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 8484, 2024-04-26T21:52:56.291 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 23293, 2024-04-26T21:52:56.291 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-04-26T21:52:56.291 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 19658, 2024-04-26T21:52:56.291 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 0, 2024-04-26T21:52:56.292 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 0, 2024-04-26T21:52:56.292 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 0, 2024-04-26T21:52:56.292 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 7, 2024-04-26T21:52:56.292 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 0, 2024-04-26T21:52:56.292 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 0, 2024-04-26T21:52:56.292 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 0, 2024-04-26T21:52:56.292 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 0, 2024-04-26T21:52:56.293 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-04-26T21:52:56.293 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 0, 2024-04-26T21:52:56.293 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 0, 2024-04-26T21:52:56.293 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 0, 2024-04-26T21:52:56.293 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 0, 2024-04-26T21:52:56.293 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-04-26T21:52:56.293 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-04-26T21:52:56.293 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 0, 2024-04-26T21:52:56.294 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 0, 2024-04-26T21:52:56.294 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-04-26T21:52:56.294 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 0, 2024-04-26T21:52:56.294 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 0, 2024-04-26T21:52:56.294 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 0, 2024-04-26T21:52:56.294 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 0, 2024-04-26T21:52:56.294 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 0, 2024-04-26T21:52:56.294 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T21:52:56.295 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-04-26T21:52:56.295 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T21:52:56.295 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-04-26T21:52:56.295 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-04-26T21:52:56.295 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T21:52:56.295 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T21:52:56.295 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 0, 2024-04-26T21:52:56.295 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 0, 2024-04-26T21:52:56.295 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T21:52:56.296 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 0, 2024-04-26T21:52:56.296 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T21:52:56.296 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T21:52:56.296 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T21:52:56.296 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 0, 2024-04-26T21:52:56.296 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 0, 2024-04-26T21:52:56.296 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 0, 2024-04-26T21:52:56.296 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 0 2024-04-26T21:52:56.296 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.297 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-04-26T21:52:56.297 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-04-26T21:52:56.297 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-04-26T21:52:56.297 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-04-26T21:52:56.297 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 0, 2024-04-26T21:52:56.297 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 0, 2024-04-26T21:52:56.297 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 0, 2024-04-26T21:52:56.297 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-04-26T21:52:56.298 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-04-26T21:52:56.298 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 0, 2024-04-26T21:52:56.298 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 0, 2024-04-26T21:52:56.298 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 0, 2024-04-26T21:52:56.298 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 0, 2024-04-26T21:52:56.298 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-04-26T21:52:56.298 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-04-26T21:52:56.298 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-04-26T21:52:56.299 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-04-26T21:52:56.299 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-04-26T21:52:56.299 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 0, 2024-04-26T21:52:56.299 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 0, 2024-04-26T21:52:56.299 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T21:52:56.299 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-04-26T21:52:56.299 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-04-26T21:52:56.299 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 0, 2024-04-26T21:52:56.299 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-04-26T21:52:56.300 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-04-26T21:52:56.300 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.300 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-04-26T21:52:56.300 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-04-26T21:52:56.300 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 0, 2024-04-26T21:52:56.300 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-04-26T21:52:56.300 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-04-26T21:52:56.300 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 149387, 2024-04-26T21:52:56.301 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-04-26T21:52:56.301 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-04-26T21:52:56.301 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 0, 2024-04-26T21:52:56.301 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-04-26T21:52:56.301 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-04-26T21:52:56.301 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 140, 2024-04-26T21:52:56.301 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 14, 2024-04-26T21:52:56.301 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-04-26T21:52:56.301 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-04-26T21:52:56.302 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 152553741, 2024-04-26T21:52:56.302 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 694956825, 2024-04-26T21:52:56.302 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 694949380, 2024-04-26T21:52:56.302 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-04-26T21:52:56.302 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.302 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.302 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.302 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.303 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 149387 2024-04-26T21:52:56.303 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.303 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-04-26T21:52:56.303 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 1761, 2024-04-26T21:52:56.303 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 31727, 2024-04-26T21:52:56.303 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 29966, 2024-04-26T21:52:56.303 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 84, 2024-04-26T21:52:56.303 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 114, 2024-04-26T21:52:56.303 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 30, 2024-04-26T21:52:56.303 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 31777, 2024-04-26T21:52:56.304 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 32136, 2024-04-26T21:52:56.304 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 359, 2024-04-26T21:52:56.304 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 0, 2024-04-26T21:52:56.304 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 0, 2024-04-26T21:52:56.304 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 0, 2024-04-26T21:52:56.304 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 89984, 2024-04-26T21:52:56.304 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 215416 2024-04-26T21:52:56.304 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.304 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-04-26T21:52:56.305 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 0, 2024-04-26T21:52:56.305 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 0, 2024-04-26T21:52:56.305 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 0, 2024-04-26T21:52:56.305 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 0, 2024-04-26T21:52:56.305 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 0, 2024-04-26T21:52:56.305 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-04-26T21:52:56.305 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.305 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.305 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.306 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.306 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-04-26T21:52:56.306 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.306 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.306 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.306 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.306 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-04-26T21:52:56.306 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.306 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.306 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.307 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.307 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-04-26T21:52:56.307 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.307 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.307 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.307 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.307 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-04-26T21:52:56.307 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.307 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.307 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.308 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.308 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-04-26T21:52:56.308 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.308 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.308 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.308 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.308 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-04-26T21:52:56.308 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.308 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-04-26T21:52:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-04-26T21:52:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.310 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.310 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.310 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-04-26T21:52:56.310 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.310 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.310 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.310 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.310 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-04-26T21:52:56.310 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.311 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.311 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.311 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.311 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-04-26T21:52:56.312 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.312 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.312 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.312 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.312 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-04-26T21:52:56.312 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.312 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.312 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.312 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.313 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-04-26T21:52:56.313 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.313 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.313 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.313 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.313 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-04-26T21:52:56.313 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.313 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.313 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.313 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.314 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-04-26T21:52:56.314 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.314 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.314 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.314 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.314 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-04-26T21:52:56.314 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-04-26T21:52:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-04-26T21:52:56.315 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.316 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.316 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.316 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.316 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-04-26T21:52:56.316 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.316 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.316 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.316 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.317 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-04-26T21:52:56.317 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.317 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.317 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.317 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.317 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-04-26T21:52:56.317 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.317 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.317 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.317 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.318 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-04-26T21:52:56.318 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.318 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.318 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.318 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.318 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-04-26T21:52:56.318 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.318 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.318 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.319 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.319 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-04-26T21:52:56.319 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.319 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.319 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.319 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.319 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-04-26T21:52:56.319 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.319 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.319 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.320 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.320 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-04-26T21:52:56.320 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.320 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.320 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.320 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.320 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-04-26T21:52:56.320 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.320 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.321 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.321 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.321 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-04-26T21:52:56.321 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.321 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.321 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.321 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.321 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-04-26T21:52:56.321 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-04-26T21:52:56.322 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-04-26T21:52:56.322 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.322 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.322 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.322 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:56.322 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.322 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-04-26T21:52:56.322 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 2, 2024-04-26T21:52:56.322 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 2, 2024-04-26T21:52:56.323 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 0, 2024-04-26T21:52:56.323 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 2, 2024-04-26T21:52:56.323 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-04-26T21:52:56.323 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 0, 2024-04-26T21:52:56.323 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 0, 2024-04-26T21:52:56.323 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 2904, 2024-04-26T21:52:56.323 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-04-26T21:52:56.323 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.324 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-26T21:52:56.324 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:52:56.324 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-26T21:52:56.324 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:52:56.324 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:52:56.324 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:52:56.324 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:52:56.324 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:52:56.324 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:52:56.325 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:52:56.325 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:52:56.325 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:52:56.325 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:52:56.325 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:52:56.325 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:52:56.325 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:52:56.325 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-26T21:52:56.326 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:52:56.326 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-26T21:52:56.326 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:52:56.326 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:52:56.326 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:52:56.326 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:52:56.326 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:52:56.326 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:52:56.327 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:52:56.327 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-26T21:52:56.327 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:52:56.327 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:52:56.327 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:52:56.327 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-26T21:52:56.327 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-26T21:52:56.327 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-26T21:52:56.328 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:52:56.328 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:52:56.328 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:52:56.328 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:52:56.328 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 316495, 2024-04-26T21:52:56.328 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 2104, 2024-04-26T21:52:56.328 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-26T21:52:56.328 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-26T21:52:56.328 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-26T21:52:56.329 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-26T21:52:56.329 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:52:56.329 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-26T21:52:56.329 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:52:56.329 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-26T21:52:56.329 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 10040, 2024-04-26T21:52:56.329 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 211, 2024-04-26T21:52:56.329 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:52:56.330 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:52:56.330 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-26T21:52:56.330 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-26T21:52:56.330 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 27045697, 2024-04-26T21:52:56.330 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 174656, 2024-04-26T21:52:56.330 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-26T21:52:56.330 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-26T21:52:56.330 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-26T21:52:56.330 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-26T21:52:56.331 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.331 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-04-26T21:52:56.331 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-04-26T21:52:56.331 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-04-26T21:52:56.331 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 6087, 2024-04-26T21:52:56.331 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 21, 2024-04-26T21:52:56.331 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 3, 2024-04-26T21:52:56.331 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 6087, 2024-04-26T21:52:56.331 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-04-26T21:52:56.332 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 6087, 2024-04-26T21:52:56.332 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 15.580718822, 2024-04-26T21:52:56.332 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.002559671 2024-04-26T21:52:56.332 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.332 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-04-26T21:52:56.332 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-04-26T21:52:56.332 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 6087, 2024-04-26T21:52:56.332 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 6088 2024-04-26T21:52:56.332 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.333 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 6087, 2024-04-26T21:52:56.333 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 6087, 2024-04-26T21:52:56.333 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 0, 2024-04-26T21:52:56.333 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-04-26T21:52:56.333 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-04-26T21:52:56.333 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 3334, 2024-04-26T21:52:56.333 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 0, 2024-04-26T21:52:56.333 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 2751, 2024-04-26T21:52:56.333 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 0, 2024-04-26T21:52:56.334 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 0, 2024-04-26T21:52:56.334 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-04-26T21:52:56.334 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-04-26T21:52:56.334 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-04-26T21:52:56.334 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-04-26T21:52:56.334 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 0, 2024-04-26T21:52:56.334 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-04-26T21:52:56.334 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-04-26T21:52:56.335 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-04-26T21:52:56.335 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 1, 2024-04-26T21:52:56.335 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 0, 2024-04-26T21:52:56.335 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-04-26T21:52:56.335 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-04-26T21:52:56.335 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-04-26T21:52:56.335 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-04-26T21:52:56.335 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-04-26T21:52:56.335 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-04-26T21:52:56.336 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T21:52:56.336 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-04-26T21:52:56.336 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-04-26T21:52:56.336 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-04-26T21:52:56.336 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-04-26T21:52:56.336 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-04-26T21:52:56.336 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-04-26T21:52:56.336 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-04-26T21:52:56.337 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-04-26T21:52:56.337 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-04-26T21:52:56.337 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-04-26T21:52:56.337 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-04-26T21:52:56.337 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-04-26T21:52:56.337 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-04-26T21:52:56.337 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-04-26T21:52:56.337 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-04-26T21:52:56.337 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-04-26T21:52:56.338 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-04-26T21:52:56.338 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-04-26T21:52:56.338 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-04-26T21:52:56.338 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 134, 2024-04-26T21:52:56.338 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-04-26T21:52:56.338 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 43, 2024-04-26T21:52:56.338 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 11, 2024-04-26T21:52:56.338 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 12, 2024-04-26T21:52:56.338 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 1, 2024-04-26T21:52:56.339 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-04-26T21:52:56.339 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 0, 2024-04-26T21:52:56.339 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 2, 2024-04-26T21:52:56.339 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 0 2024-04-26T21:52:56.339 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.339 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-04-26T21:52:56.339 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 0, 2024-04-26T21:52:56.339 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-04-26T21:52:56.339 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-04-26T21:52:56.340 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-04-26T21:52:56.340 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.340 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-04-26T21:52:56.340 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 0 2024-04-26T21:52:56.340 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.340 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-04-26T21:52:56.340 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-04-26T21:52:56.340 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T21:52:56.340 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-04-26T21:52:56.341 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 0, 2024-04-26T21:52:56.341 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 0, 2024-04-26T21:52:56.341 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 0, 2024-04-26T21:52:56.341 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-04-26T21:52:56.341 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.341 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:52:56.341 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:52:56.341 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T21:52:56.341 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:52:56.342 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 6603, 2024-04-26T21:52:56.342 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 693716037, 2024-04-26T21:52:56.342 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:56.342 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 6603, 2024-04-26T21:52:56.342 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T21:52:56.342 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T21:52:56.342 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 6603, 2024-04-26T21:52:56.342 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 693716037, 2024-04-26T21:52:56.343 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:52:56.343 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.343 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.343 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.343 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:56.343 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.343 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-04-26T21:52:56.343 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:52:56.343 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T21:52:56.344 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:52:56.344 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T21:52:56.344 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T21:52:56.344 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:56.344 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:56.344 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 6087, 2024-04-26T21:52:56.344 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 690804220, 2024-04-26T21:52:56.344 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 6087, 2024-04-26T21:52:56.345 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 690804220, 2024-04-26T21:52:56.345 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:52:56.345 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.345 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.345 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.345 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:56.345 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.345 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-04-26T21:52:56.345 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-04-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 6087, 2024-04-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 6087, 2024-04-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 6087, 2024-04-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 6087, 2024-04-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-04-26T21:52:56.348 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:52:56.348 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-26T21:52:56.348 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:52:56.348 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T21:52:56.348 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T21:52:56.348 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:56.348 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:56.348 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T21:52:56.349 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T21:52:56.349 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-26T21:52:56.349 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-26T21:52:56.349 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:52:56.349 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.349 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.349 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.349 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:56.349 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:56.350 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x562e7fd1c6e0": { 2024-04-26T21:52:56.350 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:52:56.350 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-26T21:52:56.350 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:52:56.350 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T21:52:56.350 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T21:52:56.350 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:56.350 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T21:52:56.351 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T21:52:56.351 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T21:52:56.351 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-26T21:52:56.351 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-26T21:52:56.351 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:52:56.351 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:56.351 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:56.351 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:56.351 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:56.352 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:56.352 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T21:52:56.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:56 smithi186 ceph-mon[32991]: pgmap v719: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 12 op/s 2024-04-26T21:52:56.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:56 smithi001 ceph-mon[28054]: pgmap v719: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 12 op/s 2024-04-26T21:52:56.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:56 smithi038 ceph-mon[32724]: pgmap v719: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 12 op/s 2024-04-26T21:52:57.082 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-26T21:52:57.082 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-26T21:52:57.082 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-04-26T21:52:57.082 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-04-26T21:52:57.082 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-04-26T21:52:57.083 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-26T21:52:57.083 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T21:52:57.083 DEBUG:tasks.check_counter:Getting stats from f 2024-04-26T21:52:57.083 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-26T21:52:57.335 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:52:58.454 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:52:58 smithi186 ceph-mon[32991]: pgmap v720: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 8 op/s 2024-04-26T21:52:58.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:52:58 smithi001 ceph-mon[28054]: pgmap v720: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 8 op/s 2024-04-26T21:52:58.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:52:58 smithi038 ceph-mon[32724]: pgmap v720: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 70 KiB/s wr, 8 op/s 2024-04-26T21:52:59.020 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T21:52:59.020 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:52:59.020 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 4, 2024-04-26T21:52:59.020 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 4, 2024-04-26T21:52:59.021 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 2992, 2024-04-26T21:52:59.021 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 448, 2024-04-26T21:52:59.021 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 4, 2024-04-26T21:52:59.021 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 0, 2024-04-26T21:52:59.021 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.001419104, 2024-04-26T21:52:59.021 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.000042614, 2024-04-26T21:52:59.021 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.000566000, 2024-04-26T21:52:59.021 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:52:59.021 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:59.021 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3, 2024-04-26T21:52:59.022 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000679070, 2024-04-26T21:52:59.022 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000226356 2024-04-26T21:52:59.022 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.022 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:59.022 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:59.022 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:59.022 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:59.022 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.022 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-26T21:52:59.023 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 448 2024-04-26T21:52:59.023 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.023 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:52:59.023 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 3, 2024-04-26T21:52:59.023 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 321, 2024-04-26T21:52:59.023 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 480, 2024-04-26T21:52:59.023 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 292416, 2024-04-26T21:52:59.023 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 6, 2024-04-26T21:52:59.023 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 3, 2024-04-26T21:52:59.023 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.017574145, 2024-04-26T21:52:59.024 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.012759852, 2024-04-26T21:52:59.024 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.001473115, 2024-04-26T21:52:59.024 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:52:59.024 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:59.024 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 318, 2024-04-26T21:52:59.024 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.026606059, 2024-04-26T21:52:59.024 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000083666 2024-04-26T21:52:59.024 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.024 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:59.025 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:59.025 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:59.025 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:59.025 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.025 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T21:52:59.025 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 292416 2024-04-26T21:52:59.025 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.025 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:52:59.025 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 430, 2024-04-26T21:52:59.026 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 419, 2024-04-26T21:52:59.026 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1649312, 2024-04-26T21:52:59.026 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 259616, 2024-04-26T21:52:59.026 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 2, 2024-04-26T21:52:59.026 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 1, 2024-04-26T21:52:59.026 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.068523328, 2024-04-26T21:52:59.026 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.022782861, 2024-04-26T21:52:59.026 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.042644607, 2024-04-26T21:52:59.026 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.009880609, 2024-04-26T21:52:59.027 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:52:59.027 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 418, 2024-04-26T21:52:59.027 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.020362650, 2024-04-26T21:52:59.027 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000048714 2024-04-26T21:52:59.027 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.027 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:52:59.027 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:59.027 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:59.027 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:59.028 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.028 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1649312, 2024-04-26T21:52:59.028 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 259616 2024-04-26T21:52:59.028 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.028 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-26T21:52:59.028 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 0, 2024-04-26T21:52:59.028 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-26T21:52:59.028 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.028 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-26T21:52:59.028 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:52:59.029 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-26T21:52:59.029 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:52:59.029 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:52:59.029 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:52:59.029 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:52:59.029 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:52:59.029 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:52:59.029 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:52:59.029 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:52:59.030 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:52:59.030 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:52:59.030 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:52:59.030 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:52:59.030 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:52:59.030 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-26T21:52:59.030 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:52:59.030 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-26T21:52:59.030 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:52:59.031 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:52:59.031 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:52:59.031 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:52:59.031 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:52:59.031 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:52:59.031 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:52:59.031 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-26T21:52:59.031 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:52:59.031 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:52:59.031 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:52:59.032 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-26T21:52:59.032 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-26T21:52:59.032 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-26T21:52:59.032 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:52:59.032 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:52:59.032 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:52:59.032 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:52:59.032 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 148754, 2024-04-26T21:52:59.032 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 31, 2024-04-26T21:52:59.032 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-26T21:52:59.033 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-26T21:52:59.033 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-26T21:52:59.033 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-26T21:52:59.033 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:52:59.033 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-26T21:52:59.033 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:52:59.033 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-26T21:52:59.033 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 0, 2024-04-26T21:52:59.033 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 0, 2024-04-26T21:52:59.033 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:52:59.034 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:52:59.034 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-26T21:52:59.034 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-26T21:52:59.034 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 928, 2024-04-26T21:52:59.034 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 2, 2024-04-26T21:52:59.034 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-26T21:52:59.034 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-26T21:52:59.034 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-26T21:52:59.034 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-26T21:52:59.035 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:52:59.035 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:52:59.035 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:52:59.035 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T21:52:59.035 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:52:59.035 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 433, 2024-04-26T21:52:59.035 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 1588577, 2024-04-26T21:52:59.035 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:52:59.035 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 433, 2024-04-26T21:52:59.035 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T21:52:59.036 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T21:52:59.036 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 433, 2024-04-26T21:52:59.036 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1588577, 2024-04-26T21:52:59.036 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:52:59.036 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:52:59.036 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:52:59.036 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:52:59.036 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:59.036 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:52:59.036 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T21:52:59.775 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.f 2024-04-26T21:52:59.775 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.f 2024-04-26T21:52:59.775 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.f 2024-04-26T21:52:59.775 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.f 2024-04-26T21:52:59.775 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-04-26T21:52:59.775 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-04-26T21:52:59.776 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-04-26T21:52:59.776 DEBUG:tasks.check_counter:Getting stats from i 2024-04-26T21:52:59.776 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-26T21:53:00.017 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:53:00.417 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:00 smithi186 ceph-mon[32991]: pgmap v721: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 71 KiB/s wr, 10 op/s 2024-04-26T21:53:00.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:00 smithi001 ceph-mon[28054]: pgmap v721: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 71 KiB/s wr, 10 op/s 2024-04-26T21:53:00.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:00 smithi038 ceph-mon[32724]: pgmap v721: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 71 KiB/s wr, 10 op/s 2024-04-26T21:53:01.748 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T21:53:01.748 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:53:01.748 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 584, 2024-04-26T21:53:01.748 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 586, 2024-04-26T21:53:01.748 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 62872646, 2024-04-26T21:53:01.749 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 185722, 2024-04-26T21:53:01.749 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 8, 2024-04-26T21:53:01.749 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-04-26T21:53:01.749 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.080870243, 2024-04-26T21:53:01.749 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.014094081, 2024-04-26T21:53:01.749 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.044395823, 2024-04-26T21:53:01.749 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.017482115, 2024-04-26T21:53:01.749 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:53:01.749 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 584, 2024-04-26T21:53:01.750 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.028773439, 2024-04-26T21:53:01.750 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000049269 2024-04-26T21:53:01.750 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.750 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:53:01.750 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.750 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.750 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.750 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.751 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T21:53:01.751 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T21:53:01.751 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.751 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:53:01.751 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 320, 2024-04-26T21:53:01.751 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 315, 2024-04-26T21:53:01.751 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 50587233, 2024-04-26T21:53:01.751 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 119034, 2024-04-26T21:53:01.752 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 5, 2024-04-26T21:53:01.752 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-04-26T21:53:01.752 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.055032548, 2024-04-26T21:53:01.752 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.008825457, 2024-04-26T21:53:01.752 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.037128943, 2024-04-26T21:53:01.752 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.007958700, 2024-04-26T21:53:01.752 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:53:01.752 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 315, 2024-04-26T21:53:01.753 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.053907059, 2024-04-26T21:53:01.753 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000171133 2024-04-26T21:53:01.753 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.753 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:53:01.753 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.753 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.753 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.753 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.753 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 197952, 2024-04-26T21:53:01.754 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 49008 2024-04-26T21:53:01.754 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.754 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:53:01.754 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 351, 2024-04-26T21:53:01.754 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 407, 2024-04-26T21:53:01.754 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 78713079, 2024-04-26T21:53:01.754 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 294725, 2024-04-26T21:53:01.754 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 6, 2024-04-26T21:53:01.755 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 5, 2024-04-26T21:53:01.755 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.071296967, 2024-04-26T21:53:01.755 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.009097882, 2024-04-26T21:53:01.755 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.049691457, 2024-04-26T21:53:01.755 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.009755905, 2024-04-26T21:53:01.755 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:53:01.755 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 405, 2024-04-26T21:53:01.755 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.020000300, 2024-04-26T21:53:01.755 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000049383 2024-04-26T21:53:01.756 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.756 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:53:01.756 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.756 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.756 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.756 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.756 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-26T21:53:01.756 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 183424 2024-04-26T21:53:01.756 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.757 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-26T21:53:01.757 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-04-26T21:53:01.757 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-26T21:53:01.757 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.757 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-04-26T21:53:01.757 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-26T21:53:01.757 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-26T21:53:01.757 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1872, 2024-04-26T21:53:01.757 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.124015066, 2024-04-26T21:53:01.758 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000066247 2024-04-26T21:53:01.758 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:01.758 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.758 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-04-26T21:53:01.758 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-26T21:53:01.758 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-26T21:53:01.758 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.758 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.759 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.759 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:01.759 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.759 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-04-26T21:53:01.759 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 0, 2024-04-26T21:53:01.759 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 0, 2024-04-26T21:53:01.759 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-04-26T21:53:01.759 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.759 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.760 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.760 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.760 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-04-26T21:53:01.760 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 0, 2024-04-26T21:53:01.760 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-04-26T21:53:01.760 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-04-26T21:53:01.760 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 0, 2024-04-26T21:53:01.760 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 0, 2024-04-26T21:53:01.761 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-04-26T21:53:01.761 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 10234, 2024-04-26T21:53:01.761 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 6538, 2024-04-26T21:53:01.761 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 3696, 2024-04-26T21:53:01.761 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-04-26T21:53:01.761 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 894, 2024-04-26T21:53:01.761 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 0, 2024-04-26T21:53:01.761 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 0, 2024-04-26T21:53:01.761 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 0, 2024-04-26T21:53:01.762 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 6, 2024-04-26T21:53:01.762 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 0, 2024-04-26T21:53:01.762 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 0, 2024-04-26T21:53:01.762 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 0, 2024-04-26T21:53:01.762 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 0, 2024-04-26T21:53:01.762 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-04-26T21:53:01.762 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 0, 2024-04-26T21:53:01.762 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 0, 2024-04-26T21:53:01.762 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 0, 2024-04-26T21:53:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 0, 2024-04-26T21:53:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-04-26T21:53:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-04-26T21:53:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 0, 2024-04-26T21:53:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 0, 2024-04-26T21:53:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-04-26T21:53:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 0, 2024-04-26T21:53:01.763 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 0, 2024-04-26T21:53:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 645, 2024-04-26T21:53:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 213712935, 2024-04-26T21:53:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 6, 2024-04-26T21:53:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 0, 2024-04-26T21:53:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-04-26T21:53:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-04-26T21:53:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-04-26T21:53:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-04-26T21:53:01.764 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-04-26T21:53:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-04-26T21:53:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 0, 2024-04-26T21:53:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 0, 2024-04-26T21:53:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 0, 2024-04-26T21:53:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 0, 2024-04-26T21:53:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-04-26T21:53:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-26T21:53:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-26T21:53:01.765 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 0, 2024-04-26T21:53:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 0, 2024-04-26T21:53:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 0, 2024-04-26T21:53:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 0 2024-04-26T21:53:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-04-26T21:53:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-04-26T21:53:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-04-26T21:53:01.766 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-04-26T21:53:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 0, 2024-04-26T21:53:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 0, 2024-04-26T21:53:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 0, 2024-04-26T21:53:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-04-26T21:53:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-04-26T21:53:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 0, 2024-04-26T21:53:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 0, 2024-04-26T21:53:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 0, 2024-04-26T21:53:01.767 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 0, 2024-04-26T21:53:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-04-26T21:53:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-04-26T21:53:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-04-26T21:53:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-04-26T21:53:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-04-26T21:53:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 0, 2024-04-26T21:53:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 0, 2024-04-26T21:53:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-04-26T21:53:01.768 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-04-26T21:53:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-04-26T21:53:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 0, 2024-04-26T21:53:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-04-26T21:53:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-04-26T21:53:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-04-26T21:53:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-04-26T21:53:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 0, 2024-04-26T21:53:01.769 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-04-26T21:53:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-04-26T21:53:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 44780, 2024-04-26T21:53:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-04-26T21:53:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-04-26T21:53:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 0, 2024-04-26T21:53:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-04-26T21:53:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-04-26T21:53:01.770 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 2, 2024-04-26T21:53:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 16, 2024-04-26T21:53:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-04-26T21:53:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-04-26T21:53:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 424421466, 2024-04-26T21:53:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 430944266, 2024-04-26T21:53:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 430940103, 2024-04-26T21:53:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-04-26T21:53:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.771 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 44780 2024-04-26T21:53:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-04-26T21:53:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 908, 2024-04-26T21:53:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 9959, 2024-04-26T21:53:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 9051, 2024-04-26T21:53:01.772 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 80, 2024-04-26T21:53:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 81, 2024-04-26T21:53:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 1, 2024-04-26T21:53:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 10234, 2024-04-26T21:53:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 10234, 2024-04-26T21:53:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 0, 2024-04-26T21:53:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 0, 2024-04-26T21:53:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 0, 2024-04-26T21:53:01.773 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 0, 2024-04-26T21:53:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 62744, 2024-04-26T21:53:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 215392 2024-04-26T21:53:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-04-26T21:53:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 0, 2024-04-26T21:53:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 0, 2024-04-26T21:53:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 0, 2024-04-26T21:53:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 0, 2024-04-26T21:53:01.774 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 0, 2024-04-26T21:53:01.775 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-04-26T21:53:01.775 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.775 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.775 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.775 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.775 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-04-26T21:53:01.775 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.775 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.775 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.776 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.776 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-04-26T21:53:01.776 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.776 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.776 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.776 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.776 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-04-26T21:53:01.776 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.776 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.777 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.777 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.777 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-04-26T21:53:01.777 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.777 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.777 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.777 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.777 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-04-26T21:53:01.778 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.778 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.778 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.778 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.778 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-04-26T21:53:01.778 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.778 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.778 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.778 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.779 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-04-26T21:53:01.779 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.779 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.779 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.779 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.779 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-04-26T21:53:01.779 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.779 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.779 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.780 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.780 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-04-26T21:53:01.780 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.780 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.780 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.780 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.780 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-04-26T21:53:01.780 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.780 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.781 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.781 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.781 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-04-26T21:53:01.782 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.782 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.782 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.782 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.782 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-04-26T21:53:01.782 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.782 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.783 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.783 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.783 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-04-26T21:53:01.783 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.783 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.783 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.783 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.783 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-04-26T21:53:01.783 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.784 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.784 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.784 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.784 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-04-26T21:53:01.784 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.784 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.784 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.784 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.785 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-04-26T21:53:01.785 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.785 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.785 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.785 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.785 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-04-26T21:53:01.785 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.785 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.786 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.786 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.786 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-04-26T21:53:01.786 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.786 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.786 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.786 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.786 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-04-26T21:53:01.787 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.787 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.787 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.787 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.787 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-04-26T21:53:01.787 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.787 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.787 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.787 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.788 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-04-26T21:53:01.788 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.788 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.788 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.788 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.788 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-04-26T21:53:01.788 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.788 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.789 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.789 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.789 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-04-26T21:53:01.789 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.789 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.789 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.789 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.789 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-04-26T21:53:01.789 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.790 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.790 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.790 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.790 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-04-26T21:53:01.790 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.790 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.790 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.790 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.790 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-04-26T21:53:01.791 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.791 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.791 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.791 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.791 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-04-26T21:53:01.791 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.791 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.791 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.792 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.792 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-04-26T21:53:01.792 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.792 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.792 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.792 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.792 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-04-26T21:53:01.792 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-04-26T21:53:01.792 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-04-26T21:53:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-04-26T21:53:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 2, 2024-04-26T21:53:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 2, 2024-04-26T21:53:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 0, 2024-04-26T21:53:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 0, 2024-04-26T21:53:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-04-26T21:53:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 0, 2024-04-26T21:53:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 0, 2024-04-26T21:53:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 2908, 2024-04-26T21:53:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-04-26T21:53:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.794 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-26T21:53:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:53:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-26T21:53:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:53:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:53:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:53:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:53:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:53:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:53:01.795 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:53:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:53:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:53:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:53:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:53:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:53:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:53:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-26T21:53:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:53:01.796 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-26T21:53:01.797 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:53:01.797 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:53:01.797 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:53:01.797 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:53:01.797 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:53:01.797 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:53:01.797 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:53:01.797 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-26T21:53:01.797 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:53:01.798 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:53:01.798 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:53:01.798 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-26T21:53:01.798 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-26T21:53:01.798 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-26T21:53:01.798 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:53:01.798 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:53:01.798 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:53:01.798 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:53:01.799 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 269335, 2024-04-26T21:53:01.799 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 1142, 2024-04-26T21:53:01.799 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-26T21:53:01.799 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-26T21:53:01.799 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-26T21:53:01.799 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-26T21:53:01.799 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:53:01.799 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-26T21:53:01.800 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:53:01.800 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-26T21:53:01.800 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 10040, 2024-04-26T21:53:01.800 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 211, 2024-04-26T21:53:01.800 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:53:01.800 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:53:01.800 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-26T21:53:01.800 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-26T21:53:01.800 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 9626110, 2024-04-26T21:53:01.801 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 65913, 2024-04-26T21:53:01.801 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-26T21:53:01.801 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-26T21:53:01.801 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-26T21:53:01.801 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-26T21:53:01.801 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.801 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-04-26T21:53:01.801 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-04-26T21:53:01.802 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-04-26T21:53:01.802 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 1154, 2024-04-26T21:53:01.802 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 26, 2024-04-26T21:53:01.802 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 4, 2024-04-26T21:53:01.802 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 1154, 2024-04-26T21:53:01.802 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-04-26T21:53:01.802 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1154, 2024-04-26T21:53:01.802 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 7.436867467, 2024-04-26T21:53:01.802 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.006444425 2024-04-26T21:53:01.803 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.803 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-04-26T21:53:01.803 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-04-26T21:53:01.803 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1154, 2024-04-26T21:53:01.803 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 1155 2024-04-26T21:53:01.803 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.803 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 1154, 2024-04-26T21:53:01.803 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 1154, 2024-04-26T21:53:01.803 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 0, 2024-04-26T21:53:01.804 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-04-26T21:53:01.804 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-04-26T21:53:01.804 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 626, 2024-04-26T21:53:01.804 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 0, 2024-04-26T21:53:01.804 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 525, 2024-04-26T21:53:01.804 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 0, 2024-04-26T21:53:01.804 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 0, 2024-04-26T21:53:01.804 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-04-26T21:53:01.804 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-04-26T21:53:01.805 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-04-26T21:53:01.805 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-04-26T21:53:01.805 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 0, 2024-04-26T21:53:01.805 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-04-26T21:53:01.805 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-04-26T21:53:01.805 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-04-26T21:53:01.805 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 2, 2024-04-26T21:53:01.805 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 0, 2024-04-26T21:53:01.805 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-04-26T21:53:01.806 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-04-26T21:53:01.806 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-04-26T21:53:01.806 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-04-26T21:53:01.806 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-04-26T21:53:01.806 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-04-26T21:53:01.806 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-04-26T21:53:01.806 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-04-26T21:53:01.806 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-04-26T21:53:01.807 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-04-26T21:53:01.807 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-04-26T21:53:01.807 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-04-26T21:53:01.807 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-04-26T21:53:01.807 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-04-26T21:53:01.807 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-04-26T21:53:01.807 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-04-26T21:53:01.807 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-04-26T21:53:01.807 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-04-26T21:53:01.808 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-04-26T21:53:01.808 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-04-26T21:53:01.808 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-04-26T21:53:01.808 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-04-26T21:53:01.808 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-04-26T21:53:01.808 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-04-26T21:53:01.808 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-04-26T21:53:01.808 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-04-26T21:53:01.808 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 134, 2024-04-26T21:53:01.809 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-04-26T21:53:01.809 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 11, 2024-04-26T21:53:01.809 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 10, 2024-04-26T21:53:01.809 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 10, 2024-04-26T21:53:01.809 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 0, 2024-04-26T21:53:01.809 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-04-26T21:53:01.809 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 0, 2024-04-26T21:53:01.809 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 2, 2024-04-26T21:53:01.810 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 0 2024-04-26T21:53:01.810 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.810 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-04-26T21:53:01.810 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 0, 2024-04-26T21:53:01.810 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-04-26T21:53:01.810 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-04-26T21:53:01.810 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-04-26T21:53:01.810 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.810 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-04-26T21:53:01.811 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 0 2024-04-26T21:53:01.811 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.811 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-04-26T21:53:01.811 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-04-26T21:53:01.811 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 0, 2024-04-26T21:53:01.811 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-04-26T21:53:01.811 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 0, 2024-04-26T21:53:01.811 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 0, 2024-04-26T21:53:01.812 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 0, 2024-04-26T21:53:01.812 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-04-26T21:53:01.812 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.812 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:53:01.812 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:53:01.812 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T21:53:01.812 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:53:01.812 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 1251, 2024-04-26T21:53:01.813 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 192052503, 2024-04-26T21:53:01.813 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:53:01.813 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 1251, 2024-04-26T21:53:01.813 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T21:53:01.813 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T21:53:01.813 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1251, 2024-04-26T21:53:01.813 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 192052503, 2024-04-26T21:53:01.813 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:53:01.813 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.814 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.814 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.814 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:01.814 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.814 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-04-26T21:53:01.814 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:53:01.814 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T21:53:01.814 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:53:01.814 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T21:53:01.815 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T21:53:01.815 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:53:01.815 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T21:53:01.815 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 1154, 2024-04-26T21:53:01.815 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 191653429, 2024-04-26T21:53:01.815 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1154, 2024-04-26T21:53:01.815 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 191653429, 2024-04-26T21:53:01.815 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:53:01.816 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.816 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.816 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.816 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:01.816 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.816 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-04-26T21:53:01.816 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:53:01.816 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-04-26T21:53:01.816 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:53:01.817 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T21:53:01.817 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T21:53:01.817 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:53:01.817 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T21:53:01.817 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 1154, 2024-04-26T21:53:01.817 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 1154, 2024-04-26T21:53:01.817 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1154, 2024-04-26T21:53:01.817 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1154, 2024-04-26T21:53:01.818 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:53:01.818 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.818 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.818 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.818 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:01.818 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.818 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-04-26T21:53:01.818 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:53:01.819 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-26T21:53:01.819 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:53:01.819 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T21:53:01.819 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T21:53:01.819 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:53:01.819 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T21:53:01.819 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T21:53:01.819 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T21:53:01.820 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-26T21:53:01.820 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-26T21:53:01.820 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:53:01.820 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.820 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.820 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.820 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:01.820 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:01.820 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x555d5f55c6e0": { 2024-04-26T21:53:01.820 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:53:01.821 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-26T21:53:01.821 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:53:01.821 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-26T21:53:01.821 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-26T21:53:01.821 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:53:01.821 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-26T21:53:01.821 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T21:53:01.821 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T21:53:01.822 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-26T21:53:01.822 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-26T21:53:01.822 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:53:01.822 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:01.822 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:01.822 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:01.822 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:01.822 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:01.823 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T21:53:02.493 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:02 smithi186 ceph-mon[32991]: pgmap v722: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 143 KiB/s wr, 14 op/s 2024-04-26T21:53:02.519 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-26T21:53:02.520 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-04-26T21:53:02.520 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=0 2024-04-26T21:53:02.520 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=0 2024-04-26T21:53:02.520 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-26T21:53:02.520 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-26T21:53:02.520 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-26T21:53:02.520 DEBUG:tasks.check_counter:Getting stats from l 2024-04-26T21:53:02.521 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387 shell --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-26T21:53:02.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:02 smithi001 ceph-mon[28054]: pgmap v722: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 143 KiB/s wr, 14 op/s 2024-04-26T21:53:02.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:02 smithi038 ceph-mon[32724]: pgmap v722: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 143 KiB/s wr, 14 op/s 2024-04-26T21:53:02.760 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/mon.c/config 2024-04-26T21:53:04.484 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:04 smithi186 ceph-mon[32991]: pgmap v723: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 24 op/s 2024-04-26T21:53:04.518 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-26T21:53:04.518 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-26T21:53:04.518 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 0, 2024-04-26T21:53:04.518 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 2, 2024-04-26T21:53:04.518 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 0, 2024-04-26T21:53:04.518 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 0, 2024-04-26T21:53:04.518 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 4, 2024-04-26T21:53:04.519 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 0, 2024-04-26T21:53:04.519 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.000910099, 2024-04-26T21:53:04.519 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.000000000, 2024-04-26T21:53:04.519 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.000107238, 2024-04-26T21:53:04.519 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:53:04.519 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:53:04.519 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:04.519 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:04.519 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:04.520 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.520 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:53:04.520 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:04.520 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:04.520 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:04.520 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.520 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-26T21:53:04.520 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 0 2024-04-26T21:53:04.521 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.521 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-26T21:53:04.521 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 440, 2024-04-26T21:53:04.521 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 427, 2024-04-26T21:53:04.521 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1653936, 2024-04-26T21:53:04.521 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 263904, 2024-04-26T21:53:04.521 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 2, 2024-04-26T21:53:04.521 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 1, 2024-04-26T21:53:04.521 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.069217119, 2024-04-26T21:53:04.522 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.022224112, 2024-04-26T21:53:04.522 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.043853773, 2024-04-26T21:53:04.522 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.009839729, 2024-04-26T21:53:04.522 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:53:04.522 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 427, 2024-04-26T21:53:04.522 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.016667632, 2024-04-26T21:53:04.522 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000039034 2024-04-26T21:53:04.522 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.523 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:53:04.523 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:04.523 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:04.523 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:04.523 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.523 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1653936, 2024-04-26T21:53:04.523 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 263904 2024-04-26T21:53:04.523 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.524 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-26T21:53:04.524 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 3, 2024-04-26T21:53:04.524 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 326, 2024-04-26T21:53:04.524 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 480, 2024-04-26T21:53:04.524 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 372112, 2024-04-26T21:53:04.524 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 7, 2024-04-26T21:53:04.524 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 3, 2024-04-26T21:53:04.524 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.018526342, 2024-04-26T21:53:04.524 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.013020613, 2024-04-26T21:53:04.525 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.002031505, 2024-04-26T21:53:04.525 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-26T21:53:04.525 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-26T21:53:04.525 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 323, 2024-04-26T21:53:04.525 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.043383410, 2024-04-26T21:53:04.525 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000134313 2024-04-26T21:53:04.525 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-26T21:53:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 480, 2024-04-26T21:53:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 372112 2024-04-26T21:53:04.526 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.527 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-26T21:53:04.527 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 0, 2024-04-26T21:53:04.527 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-26T21:53:04.527 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.527 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-26T21:53:04.527 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-26T21:53:04.527 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-26T21:53:04.527 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-26T21:53:04.528 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-26T21:53:04.528 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-26T21:53:04.528 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-26T21:53:04.528 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-26T21:53:04.528 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-26T21:53:04.528 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-26T21:53:04.528 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-26T21:53:04.528 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-26T21:53:04.528 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-26T21:53:04.529 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-26T21:53:04.529 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-26T21:53:04.529 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-26T21:53:04.529 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-26T21:53:04.529 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-26T21:53:04.529 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-26T21:53:04.529 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-26T21:53:04.529 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-26T21:53:04.529 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-26T21:53:04.530 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-26T21:53:04.530 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-26T21:53:04.530 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-26T21:53:04.530 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-26T21:53:04.530 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-26T21:53:04.530 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-26T21:53:04.530 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-26T21:53:04.530 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-26T21:53:04.530 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-26T21:53:04.531 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-26T21:53:04.531 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-26T21:53:04.531 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-26T21:53:04.531 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-26T21:53:04.531 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-26T21:53:04.531 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-26T21:53:04.531 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 148692, 2024-04-26T21:53:04.531 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 30, 2024-04-26T21:53:04.532 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-26T21:53:04.532 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-26T21:53:04.532 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-26T21:53:04.532 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-26T21:53:04.532 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-26T21:53:04.532 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-26T21:53:04.532 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-26T21:53:04.532 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-26T21:53:04.532 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 0, 2024-04-26T21:53:04.533 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 0, 2024-04-26T21:53:04.533 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-26T21:53:04.533 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-26T21:53:04.533 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-26T21:53:04.533 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-26T21:53:04.533 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 928, 2024-04-26T21:53:04.533 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 2, 2024-04-26T21:53:04.533 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-26T21:53:04.534 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-26T21:53:04.534 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-26T21:53:04.534 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-26T21:53:04.534 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-26T21:53:04.534 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-26T21:53:04.534 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-26T21:53:04.534 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-26T21:53:04.535 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-26T21:53:04.535 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 439, 2024-04-26T21:53:04.535 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 1589357, 2024-04-26T21:53:04.535 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-26T21:53:04.535 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 439, 2024-04-26T21:53:04.535 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-26T21:53:04.535 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-26T21:53:04.535 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 439, 2024-04-26T21:53:04.535 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1589357, 2024-04-26T21:53:04.536 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-26T21:53:04.536 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-26T21:53:04.536 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-26T21:53:04.536 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-26T21:53:04.536 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:04.536 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-26T21:53:04.536 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-26T21:53:04.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:04 smithi001 ceph-mon[28054]: pgmap v723: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 24 op/s 2024-04-26T21:53:04.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:04 smithi038 ceph-mon[32724]: pgmap v723: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 24 op/s 2024-04-26T21:53:05.253 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-26T21:53:05.254 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-26T21:53:05.254 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-04-26T21:53:05.254 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-04-26T21:53:05.254 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-26T21:53:05.254 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-26T21:53:05.254 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-26T21:53:05.255 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-26T21:53:05.266 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-26T21:53:05.266 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:53:05.266 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T21:53:05.337 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-04-26T21:53:05.337 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-26T21:53:05.394 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:53:05.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.619+0000 7fb501000640 1 -- 172.21.15.1:0/3920874595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4fc1052e0 msgr2=0x7fb4fc1117f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:05.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.619+0000 7fb501000640 1 --2- 172.21.15.1:0/3920874595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4fc1052e0 0x7fb4fc1117f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4009c10 tx=0x7fb4f402a6e0 comp rx=0 tx=0).stop 2024-04-26T21:53:05.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.620+0000 7fb501000640 1 -- 172.21.15.1:0/3920874595 shutdown_connections 2024-04-26T21:53:05.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.620+0000 7fb501000640 1 --2- 172.21.15.1:0/3920874595 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4fc1052e0 0x7fb4fc1117f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:05.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.620+0000 7fb501000640 1 --2- 172.21.15.1:0/3920874595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4fc104920 0x7fb4fc104da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:05.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.620+0000 7fb501000640 1 --2- 172.21.15.1:0/3920874595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4fc10a930 0x7fb4fc10ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:05.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.620+0000 7fb501000640 1 -- 172.21.15.1:0/3920874595 >> 172.21.15.1:0/3920874595 conn(0x7fb4fc1003d0 msgr2=0x7fb4fc1027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:05.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.620+0000 7fb501000640 1 -- 172.21.15.1:0/3920874595 shutdown_connections 2024-04-26T21:53:05.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.620+0000 7fb501000640 1 -- 172.21.15.1:0/3920874595 wait complete. 2024-04-26T21:53:05.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb501000640 1 Processor -- start 2024-04-26T21:53:05.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb501000640 1 -- start start 2024-04-26T21:53:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb501000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4fc104920 0x7fb4fc19b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb501000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4fc1052e0 0x7fb4fc19b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb501000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4fc10a930 0x7fb4fc19f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb501000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc114bf0 con 0x7fb4fc104920 2024-04-26T21:53:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb501000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc114a70 con 0x7fb4fc10a930 2024-04-26T21:53:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb501000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb4fc114d70 con 0x7fb4fc1052e0 2024-04-26T21:53:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb4fb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4fc104920 0x7fb4fc19b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:05.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb4fb400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4fc104920 0x7fb4fc19b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34568/0 (socket says 172.21.15.1:34568) 2024-04-26T21:53:05.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.621+0000 7fb4fb400640 1 -- 172.21.15.1:0/2807224102 learned_addr learned my addr 172.21.15.1:0/2807224102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:05.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4faa00640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4fc1052e0 0x7fb4fc19b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:05.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4fbe00640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4fc10a930 0x7fb4fc19f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:05.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4fb400640 1 -- 172.21.15.1:0/2807224102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4fc1052e0 msgr2=0x7fb4fc19b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:05.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4fb400640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4fc1052e0 0x7fb4fc19b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:05.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4fb400640 1 -- 172.21.15.1:0/2807224102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4fc10a930 msgr2=0x7fb4fc19f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:05.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4fb400640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4fc10a930 0x7fb4fc19f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:05.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4fb400640 1 -- 172.21.15.1:0/2807224102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4fc1a0000 con 0x7fb4fc104920 2024-04-26T21:53:05.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4fb400640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4fc104920 0x7fb4fc19b010 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00e9c0 tx=0x7fb4ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:05.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4f3e00640 1 -- 172.21.15.1:0/2807224102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec00ccc0 con 0x7fb4fc104920 2024-04-26T21:53:05.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4f3e00640 1 -- 172.21.15.1:0/2807224102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb4ec0054d0 con 0x7fb4fc104920 2024-04-26T21:53:05.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.622+0000 7fb4f3e00640 1 -- 172.21.15.1:0/2807224102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ec010430 con 0x7fb4fc104920 2024-04-26T21:53:05.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.623+0000 7fb501000640 1 -- 172.21.15.1:0/2807224102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4fc1a02f0 con 0x7fb4fc104920 2024-04-26T21:53:05.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.623+0000 7fb501000640 1 -- 172.21.15.1:0/2807224102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4fc1a7b80 con 0x7fb4fc104920 2024-04-26T21:53:05.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.624+0000 7fb501000640 1 -- 172.21.15.1:0/2807224102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4fc106060 con 0x7fb4fc104920 2024-04-26T21:53:05.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.626+0000 7fb4f3e00640 1 -- 172.21.15.1:0/2807224102 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb4ec004f80 con 0x7fb4fc104920 2024-04-26T21:53:05.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.627+0000 7fb4f3e00640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb4e00a3e40 0x7fb4e00a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:05.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.627+0000 7fb4f3e00640 1 -- 172.21.15.1:0/2807224102 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9642+0+0 (secure 0 0 0) 0x7fb4ec0d0fb0 con 0x7fb4fc104920 2024-04-26T21:53:05.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.628+0000 7fb4faa00640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb4e00a3e40 0x7fb4e00a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:05.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.628+0000 7fb4faa00640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb4e00a3e40 0x7fb4e00a6300 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb4e80095f0 tx=0x7fb4e8005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:05.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.632+0000 7fb4f3e00640 1 -- 172.21.15.1:0/2807224102 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb4ec09d190 con 0x7fb4fc104920 2024-04-26T21:53:05.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.837+0000 7fb501000640 1 -- 172.21.15.1:0/2807224102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4fc19c360 con 0x7fb4fc104920 2024-04-26T21:53:05.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.840+0000 7fb4f3e00640 1 -- 172.21.15.1:0/2807224102 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) ==== 74+0+25387 (secure 0 0 0) 0x7fb4ec09ca90 con 0x7fb4fc104920 2024-04-26T21:53:05.842 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:53:05.842 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":134,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:52:24.234120+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":133,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"133","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":133,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7},{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2275413710":"2024-04-27T21:30:18.553822+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2563183900":"2024-04-27T21:30:18.553822+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/4024107554":"2024-04-27T21:30:18.553822+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/2080672016":"2024-04-27T21:30:18.553822+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/2786715792":"2024-04-27T21:30:18.553822+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/2786715792":"2024-04-27T21:30:18.553822+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/590615206":"2024-04-27T21:30:18.553822+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]},{"pool":4,"snaps":[{"begin":26,"length":1},{"begin":39,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:53:05.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.842+0000 7fb501000640 1 -- 172.21.15.1:0/2807224102 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb4e00a3e40 msgr2=0x7fb4e00a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:05.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb4e00a3e40 0x7fb4e00a6300 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb4e80095f0 tx=0x7fb4e8005f90 comp rx=0 tx=0).stop 2024-04-26T21:53:05.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 -- 172.21.15.1:0/2807224102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4fc104920 msgr2=0x7fb4fc19b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:05.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4fc104920 0x7fb4fc19b010 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb4ec00e9c0 tx=0x7fb4ec00ee80 comp rx=0 tx=0).stop 2024-04-26T21:53:05.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 -- 172.21.15.1:0/2807224102 shutdown_connections 2024-04-26T21:53:05.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb4e00a3e40 0x7fb4e00a6300 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:05.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4fc1052e0 0x7fb4fc19b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:05.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb4fc10a930 0x7fb4fc19f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 --2- 172.21.15.1:0/2807224102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4fc104920 0x7fb4fc19b010 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 -- 172.21.15.1:0/2807224102 >> 172.21.15.1:0/2807224102 conn(0x7fb4fc1003d0 msgr2=0x7fb4fc100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 -- 172.21.15.1:0/2807224102 shutdown_connections 2024-04-26T21:53:05.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:05.843+0000 7fb501000640 1 -- 172.21.15.1:0/2807224102 wait complete. 2024-04-26T21:53:05.866 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T21:21:43.328190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-26T21:27:00.381106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-26T21:27:01.743898+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 39, 'snap_epoch': 133, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-26T21:27:05.341216+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '133', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 39, 'snap_epoch': 133, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-26T21:53:05.867 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T21:53:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.086+0000 7fba75200640 1 -- 172.21.15.1:0/523674208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70108e70 msgr2=0x7fba701092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.086+0000 7fba75200640 1 --2- 172.21.15.1:0/523674208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70108e70 0x7fba701092d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009930 tx=0x7fba5c02a420 comp rx=0 tx=0).stop 2024-04-26T21:53:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.086+0000 7fba75200640 1 -- 172.21.15.1:0/523674208 shutdown_connections 2024-04-26T21:53:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.086+0000 7fba75200640 1 --2- 172.21.15.1:0/523674208 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba70109810 0x7fba70115be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.086+0000 7fba75200640 1 --2- 172.21.15.1:0/523674208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70108e70 0x7fba701092d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.086+0000 7fba75200640 1 --2- 172.21.15.1:0/523674208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba7010edb0 0x7fba7010f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.086+0000 7fba75200640 1 -- 172.21.15.1:0/523674208 >> 172.21.15.1:0/523674208 conn(0x7fba70078740 msgr2=0x7fba70107750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.087+0000 7fba75200640 1 -- 172.21.15.1:0/523674208 shutdown_connections 2024-04-26T21:53:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.087+0000 7fba75200640 1 -- 172.21.15.1:0/523674208 wait complete. 2024-04-26T21:53:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.087+0000 7fba75200640 1 Processor -- start 2024-04-26T21:53:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba75200640 1 -- start start 2024-04-26T21:53:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba75200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70108e70 0x7fba7019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba75200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba70109810 0x7fba7019faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba75200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba7010edb0 0x7fba701a3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba75200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fba70118f90 con 0x7fba70108e70 2024-04-26T21:53:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba75200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fba70118e10 con 0x7fba70109810 2024-04-26T21:53:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba75200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fba70119110 con 0x7fba7010edb0 2024-04-26T21:53:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba6f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70108e70 0x7fba7019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba6f400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70108e70 0x7fba7019f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34584/0 (socket says 172.21.15.1:34584) 2024-04-26T21:53:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba6f400640 1 -- 172.21.15.1:0/2469301700 learned_addr learned my addr 172.21.15.1:0/2469301700 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.088+0000 7fba6ea00640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba70109810 0x7fba7019faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.089+0000 7fba6fe00640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba7010edb0 0x7fba701a3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.089+0000 7fba6f400640 1 -- 172.21.15.1:0/2469301700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba7010edb0 msgr2=0x7fba701a3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.089+0000 7fba6f400640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba7010edb0 0x7fba701a3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.089+0000 7fba6f400640 1 -- 172.21.15.1:0/2469301700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba70109810 msgr2=0x7fba7019faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.089+0000 7fba6f400640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba70109810 0x7fba7019faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.089+0000 7fba6f400640 1 -- 172.21.15.1:0/2469301700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba701a45a0 con 0x7fba70108e70 2024-04-26T21:53:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.089+0000 7fba6f400640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70108e70 0x7fba7019f5b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e9f0 tx=0x7fba6000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.089+0000 7fba67e00640 1 -- 172.21.15.1:0/2469301700 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6000ccc0 con 0x7fba70108e70 2024-04-26T21:53:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.090+0000 7fba67e00640 1 -- 172.21.15.1:0/2469301700 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba6000ce60 con 0x7fba70108e70 2024-04-26T21:53:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.090+0000 7fba67e00640 1 -- 172.21.15.1:0/2469301700 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba60010430 con 0x7fba70108e70 2024-04-26T21:53:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.090+0000 7fba75200640 1 -- 172.21.15.1:0/2469301700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba701a4890 con 0x7fba70108e70 2024-04-26T21:53:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.090+0000 7fba75200640 1 -- 172.21.15.1:0/2469301700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba701ac120 con 0x7fba70108e70 2024-04-26T21:53:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.091+0000 7fba75200640 1 -- 172.21.15.1:0/2469301700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba700506c0 con 0x7fba70108e70 2024-04-26T21:53:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.093+0000 7fba67e00640 1 -- 172.21.15.1:0/2469301700 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fba600046c0 con 0x7fba70108e70 2024-04-26T21:53:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.094+0000 7fba67e00640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fba540a3d20 0x7fba540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.095+0000 7fba67e00640 1 -- 172.21.15.1:0/2469301700 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9642+0+0 (secure 0 0 0) 0x7fba600d0fe0 con 0x7fba70108e70 2024-04-26T21:53:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.095+0000 7fba6ea00640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fba540a3d20 0x7fba540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:06.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.096+0000 7fba6ea00640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fba540a3d20 0x7fba540a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009900 tx=0x7fba5c000bc0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:06.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.099+0000 7fba67e00640 1 -- 172.21.15.1:0/2469301700 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fba6009d200 con 0x7fba70108e70 2024-04-26T21:53:06.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.306+0000 7fba75200640 1 -- 172.21.15.1:0/2469301700 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fba7002dad0 con 0x7fba70108e70 2024-04-26T21:53:06.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.306+0000 7fba67e00640 1 -- 172.21.15.1:0/2469301700 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) ==== 93+0+10 (secure 0 0 0) 0x7fba6009cb00 con 0x7fba70108e70 2024-04-26T21:53:06.308 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-04-26T21:53:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 -- 172.21.15.1:0/2469301700 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fba540a3d20 msgr2=0x7fba540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fba540a3d20 0x7fba540a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009900 tx=0x7fba5c000bc0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 -- 172.21.15.1:0/2469301700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70108e70 msgr2=0x7fba7019f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70108e70 0x7fba7019f5b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fba6000e9f0 tx=0x7fba6000edf0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 -- 172.21.15.1:0/2469301700 shutdown_connections 2024-04-26T21:53:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fba540a3d20 0x7fba540a61e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba7010edb0 0x7fba701a3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba70109810 0x7fba7019faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 --2- 172.21.15.1:0/2469301700 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba70108e70 0x7fba7019f5b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 -- 172.21.15.1:0/2469301700 >> 172.21.15.1:0/2469301700 conn(0x7fba70078740 msgr2=0x7fba700799c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:06.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 -- 172.21.15.1:0/2469301700 shutdown_connections 2024-04-26T21:53:06.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.309+0000 7fba75200640 1 -- 172.21.15.1:0/2469301700 wait complete. 2024-04-26T21:53:06.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-26T21:53:06.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.548+0000 7f2e5aa00640 1 -- 172.21.15.1:0/4069715098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e54101e40 msgr2=0x7f2e5410eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:06.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.548+0000 7f2e5aa00640 1 --2- 172.21.15.1:0/4069715098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e54101e40 0x7f2e5410eb00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009900 tx=0x7f2e4002a400 comp rx=0 tx=0).stop 2024-04-26T21:53:06.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.549+0000 7f2e5aa00640 1 -- 172.21.15.1:0/4069715098 shutdown_connections 2024-04-26T21:53:06.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.549+0000 7f2e5aa00640 1 --2- 172.21.15.1:0/4069715098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5410f040 0x7f2e54111430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.549+0000 7f2e5aa00640 1 --2- 172.21.15.1:0/4069715098 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e54101e40 0x7f2e5410eb00 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.549+0000 7f2e5aa00640 1 --2- 172.21.15.1:0/4069715098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e54101490 0x7f2e54101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.549+0000 7f2e5aa00640 1 -- 172.21.15.1:0/4069715098 >> 172.21.15.1:0/4069715098 conn(0x7f2e540783e0 msgr2=0x7f2e540ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:06.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.549+0000 7f2e5aa00640 1 -- 172.21.15.1:0/4069715098 shutdown_connections 2024-04-26T21:53:06.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.549+0000 7f2e5aa00640 1 -- 172.21.15.1:0/4069715098 wait complete. 2024-04-26T21:53:06.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e5aa00640 1 Processor -- start 2024-04-26T21:53:06.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e5aa00640 1 -- start start 2024-04-26T21:53:06.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e5aa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e54101490 0x7f2e5419b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:06.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e5aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e54101e40 0x7f2e5419b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:06.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e5aa00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5410f040 0x7f2e5419fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:06.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e5aa00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54114c50 con 0x7f2e54101490 2024-04-26T21:53:06.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e5aa00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54114ad0 con 0x7f2e54101e40 2024-04-26T21:53:06.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e5aa00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e54114dd0 con 0x7f2e5410f040 2024-04-26T21:53:06.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e59600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e54101490 0x7f2e5419b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:06.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e59600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e54101490 0x7f2e5419b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34612/0 (socket says 172.21.15.1:34612) 2024-04-26T21:53:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.550+0000 7f2e59600640 1 -- 172.21.15.1:0/1330263268 learned_addr learned my addr 172.21.15.1:0/1330263268 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.551+0000 7f2e5a000640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5410f040 0x7f2e5419fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.551+0000 7f2e58c00640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e54101e40 0x7f2e5419b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.551+0000 7f2e59600640 1 -- 172.21.15.1:0/1330263268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5410f040 msgr2=0x7f2e5419fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.551+0000 7f2e59600640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5410f040 0x7f2e5419fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.551+0000 7f2e59600640 1 -- 172.21.15.1:0/1330263268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e54101e40 msgr2=0x7f2e5419b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.551+0000 7f2e59600640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e54101e40 0x7f2e5419b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.551+0000 7f2e59600640 1 -- 172.21.15.1:0/1330263268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e541a0190 con 0x7f2e54101490 2024-04-26T21:53:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.551+0000 7f2e59600640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e54101490 0x7f2e5419b150 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2e44009d50 tx=0x7f2e4400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.551+0000 7f2e52000640 1 -- 172.21.15.1:0/1330263268 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44003b70 con 0x7f2e54101490 2024-04-26T21:53:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.551+0000 7f2e52000640 1 -- 172.21.15.1:0/1330263268 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e44003d10 con 0x7f2e54101490 2024-04-26T21:53:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.552+0000 7f2e5aa00640 1 -- 172.21.15.1:0/1330263268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e541a0480 con 0x7f2e54101490 2024-04-26T21:53:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.552+0000 7f2e52000640 1 -- 172.21.15.1:0/1330263268 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e440108a0 con 0x7f2e54101490 2024-04-26T21:53:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.552+0000 7f2e5aa00640 1 -- 172.21.15.1:0/1330263268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e541a7cc0 con 0x7f2e54101490 2024-04-26T21:53:06.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.553+0000 7f2e5aa00640 1 -- 172.21.15.1:0/1330263268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e540506c0 con 0x7f2e54101490 2024-04-26T21:53:06.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.555+0000 7f2e52000640 1 -- 172.21.15.1:0/1330263268 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f2e44004000 con 0x7f2e54101490 2024-04-26T21:53:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.556+0000 7f2e52000640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2e380a3da0 0x7f2e380a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.557+0000 7f2e52000640 1 -- 172.21.15.1:0/1330263268 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9642+0+0 (secure 0 0 0) 0x7f2e440d12b0 con 0x7f2e54101490 2024-04-26T21:53:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.557+0000 7f2e58c00640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2e380a3da0 0x7f2e380a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:06.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.557+0000 7f2e58c00640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2e380a3da0 0x7f2e380a6260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2e40005ab0 tx=0x7f2e40005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:06.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.560+0000 7f2e52000640 1 -- 172.21.15.1:0/1330263268 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f2e4409d490 con 0x7f2e54101490 2024-04-26T21:53:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:06 smithi001 ceph-mon[28054]: pgmap v724: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 203 KiB/s wr, 31 op/s 2024-04-26T21:53:06.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:06 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2807224102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:53:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:06 smithi038 ceph-mon[32724]: pgmap v724: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 203 KiB/s wr, 31 op/s 2024-04-26T21:53:06.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:06 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2807224102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:53:06.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:06 smithi186 ceph-mon[32991]: pgmap v724: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 203 KiB/s wr, 31 op/s 2024-04-26T21:53:06.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:06 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2807224102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:53:06.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.768+0000 7f2e5aa00640 1 -- 172.21.15.1:0/1330263268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2e5402dad0 con 0x7f2e54101490 2024-04-26T21:53:06.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.769+0000 7f2e52000640 1 -- 172.21.15.1:0/1330263268 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v134) ==== 104+0+11 (secure 0 0 0) 0x7f2e4409cd90 con 0x7f2e54101490 2024-04-26T21:53:06.771 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:53:06.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.772+0000 7f2e5aa00640 1 -- 172.21.15.1:0/1330263268 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2e380a3da0 msgr2=0x7f2e380a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:06.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.772+0000 7f2e5aa00640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2e380a3da0 0x7f2e380a6260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2e40005ab0 tx=0x7f2e40005a20 comp rx=0 tx=0).stop 2024-04-26T21:53:06.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.772+0000 7f2e5aa00640 1 -- 172.21.15.1:0/1330263268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e54101490 msgr2=0x7f2e5419b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:06.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.772+0000 7f2e5aa00640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e54101490 0x7f2e5419b150 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2e44009d50 tx=0x7f2e4400c6a0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.772+0000 7f2e5aa00640 1 -- 172.21.15.1:0/1330263268 shutdown_connections 2024-04-26T21:53:06.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.772+0000 7f2e5aa00640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f2e380a3da0 0x7f2e380a6260 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.772+0000 7f2e5aa00640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5410f040 0x7f2e5419fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.772+0000 7f2e5aa00640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2e54101e40 0x7f2e5419b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.772+0000 7f2e5aa00640 1 --2- 172.21.15.1:0/1330263268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e54101490 0x7f2e5419b150 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:06.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.772+0000 7f2e5aa00640 1 -- 172.21.15.1:0/1330263268 >> 172.21.15.1:0/1330263268 conn(0x7f2e540783e0 msgr2=0x7f2e54100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:06.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.773+0000 7f2e5aa00640 1 -- 172.21.15.1:0/1330263268 shutdown_connections 2024-04-26T21:53:06.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:06.773+0000 7f2e5aa00640 1 -- 172.21.15.1:0/1330263268 wait complete. 2024-04-26T21:53:06.794 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-26T21:53:07.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.009+0000 7f9fc2200640 1 -- 172.21.15.1:0/2778063699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fbc076e60 msgr2=0x7f9fbc077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:07.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.009+0000 7f9fc2200640 1 --2- 172.21.15.1:0/2778063699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fbc076e60 0x7f9fbc077240 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009900 tx=0x7f9fac02a410 comp rx=0 tx=0).stop 2024-04-26T21:53:07.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.010+0000 7f9fc2200640 1 -- 172.21.15.1:0/2778063699 shutdown_connections 2024-04-26T21:53:07.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.010+0000 7f9fc2200640 1 --2- 172.21.15.1:0/2778063699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fbc077780 0x7f9fbc113050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.010+0000 7f9fc2200640 1 --2- 172.21.15.1:0/2778063699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fbc113590 0x7f9fbc115980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.010+0000 7f9fc2200640 1 --2- 172.21.15.1:0/2778063699 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fbc076e60 0x7f9fbc077240 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.010+0000 7f9fc2200640 1 -- 172.21.15.1:0/2778063699 >> 172.21.15.1:0/2778063699 conn(0x7f9fbc1003a0 msgr2=0x7f9fbc1027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:07.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.010+0000 7f9fc2200640 1 -- 172.21.15.1:0/2778063699 shutdown_connections 2024-04-26T21:53:07.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.010+0000 7f9fc2200640 1 -- 172.21.15.1:0/2778063699 wait complete. 2024-04-26T21:53:07.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.011+0000 7f9fc2200640 1 Processor -- start 2024-04-26T21:53:07.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.011+0000 7f9fc2200640 1 -- start start 2024-04-26T21:53:07.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.011+0000 7f9fc2200640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fbc076e60 0x7f9fbc19f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:07.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.011+0000 7f9fc2200640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fbc077780 0x7f9fbc19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:07.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.011+0000 7f9fc2200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fbc113590 0x7f9fbc1a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:07.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.011+0000 7f9fc2200640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc118f90 con 0x7f9fbc076e60 2024-04-26T21:53:07.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.011+0000 7f9fc2200640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc118e10 con 0x7f9fbc077780 2024-04-26T21:53:07.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.011+0000 7f9fc2200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc119110 con 0x7f9fbc113590 2024-04-26T21:53:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.012+0000 7f9fc0e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fbc076e60 0x7f9fbc19f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.012+0000 7f9fc0e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fbc076e60 0x7f9fbc19f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34618/0 (socket says 172.21.15.1:34618) 2024-04-26T21:53:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.012+0000 7f9fc0e00640 1 -- 172.21.15.1:0/3924717230 learned_addr learned my addr 172.21.15.1:0/3924717230 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.012+0000 7f9fc1800640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fbc113590 0x7f9fbc1a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.012+0000 7f9fb3e00640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fbc077780 0x7f9fbc19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.012+0000 7f9fc0e00640 1 -- 172.21.15.1:0/3924717230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fbc113590 msgr2=0x7f9fbc1a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.012+0000 7f9fc0e00640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fbc113590 0x7f9fbc1a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.012+0000 7f9fc0e00640 1 -- 172.21.15.1:0/3924717230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fbc077780 msgr2=0x7f9fbc19fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:07.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.012+0000 7f9fc0e00640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fbc077780 0x7f9fbc19fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.012+0000 7f9fc0e00640 1 -- 172.21.15.1:0/3924717230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fbc1a4500 con 0x7f9fbc076e60 2024-04-26T21:53:07.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.013+0000 7f9fc0e00640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fbc076e60 0x7f9fbc19f510 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9fac006010 tx=0x7f9fac0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:07.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.013+0000 7f9fba000640 1 -- 172.21.15.1:0/3924717230 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac03f030 con 0x7f9fbc076e60 2024-04-26T21:53:07.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.013+0000 7f9fba000640 1 -- 172.21.15.1:0/3924717230 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fac0044c0 con 0x7f9fbc076e60 2024-04-26T21:53:07.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.013+0000 7f9fc2200640 1 -- 172.21.15.1:0/3924717230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fbc1a4790 con 0x7f9fbc076e60 2024-04-26T21:53:07.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.013+0000 7f9fba000640 1 -- 172.21.15.1:0/3924717230 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fac035070 con 0x7f9fbc076e60 2024-04-26T21:53:07.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.013+0000 7f9fc2200640 1 -- 172.21.15.1:0/3924717230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fbc1ac080 con 0x7f9fbc076e60 2024-04-26T21:53:07.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.014+0000 7f9fc2200640 1 -- 172.21.15.1:0/3924717230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fbc0506c0 con 0x7f9fbc076e60 2024-04-26T21:53:07.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.017+0000 7f9fba000640 1 -- 172.21.15.1:0/3924717230 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f9fac042030 con 0x7f9fbc076e60 2024-04-26T21:53:07.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.019+0000 7f9fba000640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9fa00a3d90 0x7f9fa00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:07.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.019+0000 7f9fba000640 1 -- 172.21.15.1:0/3924717230 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(134..134 src has 1..134) ==== 9642+0+0 (secure 0 0 0) 0x7f9fac0f0fc0 con 0x7f9fbc076e60 2024-04-26T21:53:07.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.019+0000 7f9fb3e00640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9fa00a3d90 0x7f9fa00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:07.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.020+0000 7f9fb3e00640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9fa00a3d90 0x7f9fa00a6250 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc0799f0 tx=0x7f9fa4005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:07.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.023+0000 7f9fba000640 1 -- 172.21.15.1:0/3924717230 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9fac0bd1a0 con 0x7f9fbc076e60 2024-04-26T21:53:07.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.231+0000 7f9fc2200640 1 -- 172.21.15.1:0/3924717230 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9fbc10e720 con 0x7f9fbc076e60 2024-04-26T21:53:07.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.232+0000 7f9fba000640 1 -- 172.21.15.1:0/3924717230 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v134) ==== 100+0+11 (secure 0 0 0) 0x7f9fac0bcaa0 con 0x7f9fbc076e60 2024-04-26T21:53:07.233 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-04-26T21:53:07.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 -- 172.21.15.1:0/3924717230 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9fa00a3d90 msgr2=0x7f9fa00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:07.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9fa00a3d90 0x7f9fa00a6250 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc0799f0 tx=0x7f9fa4005f90 comp rx=0 tx=0).stop 2024-04-26T21:53:07.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 -- 172.21.15.1:0/3924717230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fbc076e60 msgr2=0x7f9fbc19f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:07.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fbc076e60 0x7f9fbc19f510 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9fac006010 tx=0x7f9fac0040c0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 -- 172.21.15.1:0/3924717230 shutdown_connections 2024-04-26T21:53:07.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9fa00a3d90 0x7f9fa00a6250 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9fbc113590 0x7f9fbc1a3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9fbc077780 0x7f9fbc19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 --2- 172.21.15.1:0/3924717230 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9fbc076e60 0x7f9fbc19f510 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 -- 172.21.15.1:0/3924717230 >> 172.21.15.1:0/3924717230 conn(0x7f9fbc1003a0 msgr2=0x7f9fbc102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:07.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 -- 172.21.15.1:0/3924717230 shutdown_connections 2024-04-26T21:53:07.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.235+0000 7f9fc2200640 1 -- 172.21.15.1:0/3924717230 wait complete. 2024-04-26T21:53:07.256 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-26T21:53:07.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.554+0000 7fcdc3400640 1 -- 172.21.15.1:0/3006972357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc103ba0 msgr2=0x7fcdbc103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:07.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.554+0000 7fcdc3400640 1 --2- 172.21.15.1:0/3006972357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc103ba0 0x7fcdbc103f80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4009ca0 tx=0x7fcdb402a960 comp rx=0 tx=0).stop 2024-04-26T21:53:07.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.555+0000 7fcdc3400640 1 -- 172.21.15.1:0/3006972357 shutdown_connections 2024-04-26T21:53:07.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.555+0000 7fcdc3400640 1 --2- 172.21.15.1:0/3006972357 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc10cee0 0x7fcdbc10f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.555+0000 7fcdc3400640 1 --2- 172.21.15.1:0/3006972357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdbc1044c0 0x7fcdbc10c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.555+0000 7fcdc3400640 1 --2- 172.21.15.1:0/3006972357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc103ba0 0x7fcdbc103f80 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.555+0000 7fcdc3400640 1 -- 172.21.15.1:0/3006972357 >> 172.21.15.1:0/3006972357 conn(0x7fcdbc0789e0 msgr2=0x7fcdbc078df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:07.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.555+0000 7fcdc3400640 1 -- 172.21.15.1:0/3006972357 shutdown_connections 2024-04-26T21:53:07.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.555+0000 7fcdc3400640 1 -- 172.21.15.1:0/3006972357 wait complete. 2024-04-26T21:53:07.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.556+0000 7fcdc3400640 1 Processor -- start 2024-04-26T21:53:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.557+0000 7fcdc3400640 1 -- start start 2024-04-26T21:53:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.558+0000 7fcdc3400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdbc103ba0 0x7fcdbc1017f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:07.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.558+0000 7fcdc3400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1044c0 0x7fcdbc101d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:07.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.558+0000 7fcdc3400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc10cee0 0x7fcdbc102270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:07.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.558+0000 7fcdc0a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdbc103ba0 0x7fcdbc1017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:07.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.558+0000 7fcdc0a00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdbc103ba0 0x7fcdbc1017f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34624/0 (socket says 172.21.15.1:34624) 2024-04-26T21:53:07.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.558+0000 7fcdbbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1044c0 0x7fcdbc101d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:07.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.558+0000 7fcdbbe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1044c0 0x7fcdbc101d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:41632/0 (socket says 172.21.15.1:41632) 2024-04-26T21:53:07.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.559+0000 7fcdc0a00640 1 -- 172.21.15.1:0/614130266 learned_addr learned my addr 172.21.15.1:0/614130266 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.559+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc112ca0 con 0x7fcdbc103ba0 2024-04-26T21:53:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.559+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc112b20 con 0x7fcdbc1044c0 2024-04-26T21:53:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.559+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc112e20 con 0x7fcdbc10cee0 2024-04-26T21:53:07.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.559+0000 7fcdc1400640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc10cee0 0x7fcdbc102270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.559+0000 7fcdc0a00640 1 -- 172.21.15.1:0/614130266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc10cee0 msgr2=0x7fcdbc102270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.559+0000 7fcdc0a00640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc10cee0 0x7fcdbc102270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.559+0000 7fcdc0a00640 1 -- 172.21.15.1:0/614130266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1044c0 msgr2=0x7fcdbc101d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.559+0000 7fcdc0a00640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1044c0 0x7fcdbc101d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.559+0000 7fcdc0a00640 1 -- 172.21.15.1:0/614130266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdbc1abc00 con 0x7fcdbc103ba0 2024-04-26T21:53:07.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.560+0000 7fcdc0a00640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdbc103ba0 0x7fcdbc1017f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcdb402af50 tx=0x7fcdb402fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.560+0000 7fcdb9600640 1 -- 172.21.15.1:0/614130266 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4034070 con 0x7fcdbc103ba0 2024-04-26T21:53:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.560+0000 7fcdb9600640 1 -- 172.21.15.1:0/614130266 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdb4030d10 con 0x7fcdbc103ba0 2024-04-26T21:53:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.560+0000 7fcdb9600640 1 -- 172.21.15.1:0/614130266 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4039420 con 0x7fcdbc103ba0 2024-04-26T21:53:07.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.560+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdbc1abdd0 con 0x7fcdbc103ba0 2024-04-26T21:53:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.561+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdbc1ac210 con 0x7fcdbc103ba0 2024-04-26T21:53:07.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.563+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdbc06a440 con 0x7fcdbc103ba0 2024-04-26T21:53:07.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.565+0000 7fcdb9600640 1 -- 172.21.15.1:0/614130266 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fcdb4037020 con 0x7fcdbc103ba0 2024-04-26T21:53:07.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:07 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2469301700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:53:07.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:07 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1330263268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:53:07.568 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:07 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3924717230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:53:07.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.568+0000 7fcdb9600640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcda00a3ef0 0x7fcda00a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:07.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.568+0000 7fcdb9600640 1 -- 172.21.15.1:0/614130266 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9730+0+0 (secure 0 0 0) 0x7fcdb40f1920 con 0x7fcdbc103ba0 2024-04-26T21:53:07.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.568+0000 7fcdbbe00640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcda00a3ef0 0x7fcda00a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:07.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.570+0000 7fcdbbe00640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcda00a3ef0 0x7fcda00a63b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc102fb0 tx=0x7fcda8009110 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:07.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:07 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2469301700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:53:07.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:07 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1330263268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:53:07.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:07 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3924717230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:53:07.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.576+0000 7fcdb9600640 1 -- 172.21.15.1:0/614130266 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcdb40098b0 con 0x7fcdbc103ba0 2024-04-26T21:53:07.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:07 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2469301700' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T21:53:07.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:07 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1330263268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-26T21:53:07.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:07 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3924717230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-26T21:53:07.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.967+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fcdbc02dad0 con 0x7fcdbc103ba0 2024-04-26T21:53:07.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.969+0000 7fcdb9600640 1 -- 172.21.15.1:0/614130266 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v135) ==== 103+0+11 (secure 0 0 0) 0x7fcdb40bdaa0 con 0x7fcdbc103ba0 2024-04-26T21:53:07.970 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-04-26T21:53:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.972+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcda00a3ef0 msgr2=0x7fcda00a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.972+0000 7fcdc3400640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcda00a3ef0 0x7fcda00a63b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc102fb0 tx=0x7fcda8009110 comp rx=0 tx=0).stop 2024-04-26T21:53:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.973+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdbc103ba0 msgr2=0x7fcdbc1017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:07.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.973+0000 7fcdc3400640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdbc103ba0 0x7fcdbc1017f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcdb402af50 tx=0x7fcdb402fb20 comp rx=0 tx=0).stop 2024-04-26T21:53:07.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.977+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 shutdown_connections 2024-04-26T21:53:07.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.977+0000 7fcdc3400640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcda00a3ef0 0x7fcda00a63b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.977+0000 7fcdc3400640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc10cee0 0x7fcdbc102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.977+0000 7fcdc3400640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1044c0 0x7fcdbc101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.977+0000 7fcdc3400640 1 --2- 172.21.15.1:0/614130266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcdbc103ba0 0x7fcdbc1017f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:07.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.977+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 >> 172.21.15.1:0/614130266 conn(0x7fcdbc0789e0 msgr2=0x7fcdbc0fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:07.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.977+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 shutdown_connections 2024-04-26T21:53:07.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:07.977+0000 7fcdc3400640 1 -- 172.21.15.1:0/614130266 wait complete. 2024-04-26T21:53:08.007 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-26T21:53:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.306+0000 7f15bc800640 1 -- 172.21.15.1:0/3437262757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b4109930 msgr2=0x7f15b4109d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.306+0000 7f15bc800640 1 --2- 172.21.15.1:0/3437262757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b4109930 0x7f15b4109d10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f15a8009900 tx=0x7f15a802a4d0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.306+0000 7f15bc800640 1 -- 172.21.15.1:0/3437262757 shutdown_connections 2024-04-26T21:53:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.306+0000 7f15bc800640 1 --2- 172.21.15.1:0/3437262757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15b410a250 0x7f15b4115540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.306+0000 7f15bc800640 1 --2- 172.21.15.1:0/3437262757 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15b4115a80 0x7f15b4117e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.306+0000 7f15bc800640 1 --2- 172.21.15.1:0/3437262757 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b4109930 0x7f15b4109d10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.306+0000 7f15bc800640 1 -- 172.21.15.1:0/3437262757 >> 172.21.15.1:0/3437262757 conn(0x7f15b406ed70 msgr2=0x7f15b406f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.307+0000 7f15bc800640 1 -- 172.21.15.1:0/3437262757 shutdown_connections 2024-04-26T21:53:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.307+0000 7f15bc800640 1 -- 172.21.15.1:0/3437262757 wait complete. 2024-04-26T21:53:08.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.308+0000 7f15bc800640 1 Processor -- start 2024-04-26T21:53:08.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.309+0000 7f15bc800640 1 -- start start 2024-04-26T21:53:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.309+0000 7f15bc800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b410a250 0x7f15b41a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.309+0000 7f15bc800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15b4115a80 0x7f15b41a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.309+0000 7f15bc800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15b41a8670 0x7f15b41a8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:08.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.310+0000 7f15b9400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15b4115a80 0x7f15b41a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.310+0000 7f15b9400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15b4115a80 0x7f15b41a4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54940/0 (socket says 172.21.15.1:54940) 2024-04-26T21:53:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.310+0000 7f15b9400640 1 -- 172.21.15.1:0/4186630857 learned_addr learned my addr 172.21.15.1:0/4186630857 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.310+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f15b411b440 con 0x7f15b410a250 2024-04-26T21:53:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.310+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f15b411b2c0 con 0x7f15b41a8670 2024-04-26T21:53:08.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.310+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15b411b5c0 con 0x7f15b4115a80 2024-04-26T21:53:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.310+0000 7f15ba800640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15b41a8670 0x7f15b41a8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.311+0000 7f15b9400640 1 -- 172.21.15.1:0/4186630857 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15b41a8670 msgr2=0x7f15b41a8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.311+0000 7f15b9400640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15b41a8670 0x7f15b41a8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.311+0000 7f15b9400640 1 -- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b410a250 msgr2=0x7f15b41a3e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T21:53:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.311+0000 7f15b9e00640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b410a250 0x7f15b41a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.311+0000 7f15b9400640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b410a250 0x7f15b41a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.311+0000 7f15b9400640 1 -- 172.21.15.1:0/4186630857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b41a9160 con 0x7f15b4115a80 2024-04-26T21:53:08.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.312+0000 7f15b9400640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15b4115a80 0x7f15b41a4340 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f15a400e9f0 tx=0x7f15a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:08.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.314+0000 7f15b9e00640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b410a250 0x7f15b41a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T21:53:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.314+0000 7f15aea00640 1 -- 172.21.15.1:0/4186630857 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a400cd80 con 0x7f15b4115a80 2024-04-26T21:53:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.314+0000 7f15aea00640 1 -- 172.21.15.1:0/4186630857 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15a40108e0 con 0x7f15b4115a80 2024-04-26T21:53:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.314+0000 7f15aea00640 1 -- 172.21.15.1:0/4186630857 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4018880 con 0x7f15b4115a80 2024-04-26T21:53:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.314+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b41a9450 con 0x7f15b4115a80 2024-04-26T21:53:08.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.314+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b41b0c40 con 0x7f15b4115a80 2024-04-26T21:53:08.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.316+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b40697e0 con 0x7f15b4115a80 2024-04-26T21:53:08.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.321+0000 7f15aea00640 1 -- 172.21.15.1:0/4186630857 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f15a4010430 con 0x7f15b4115a80 2024-04-26T21:53:08.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.323+0000 7f15aea00640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f159c0a3f20 0x7f159c0a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.323+0000 7f15aea00640 1 -- 172.21.15.1:0/4186630857 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9730+0+0 (secure 0 0 0) 0x7f15a40d1170 con 0x7f15b4115a80 2024-04-26T21:53:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.324+0000 7f15b9e00640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f159c0a3f20 0x7f159c0a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:08.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.324+0000 7f15aea00640 1 -- 172.21.15.1:0/4186630857 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f15a409d370 con 0x7f15b4115a80 2024-04-26T21:53:08.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.327+0000 7f15b9e00640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f159c0a3f20 0x7f159c0a63e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f15a8005ab0 tx=0x7f15a8005fd0 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:08 smithi001 ceph-mon[28054]: pgmap v725: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 165 KiB/s wr, 27 op/s 2024-04-26T21:53:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:08 smithi001 ceph-mon[28054]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T21:53:08.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:08 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/614130266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:53:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:08 smithi038 ceph-mon[32724]: pgmap v725: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 165 KiB/s wr, 27 op/s 2024-04-26T21:53:08.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:08 smithi038 ceph-mon[32724]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T21:53:08.577 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:08 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/614130266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:53:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:08 smithi186 ceph-mon[32991]: pgmap v725: 129 pgs: 129 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 165 KiB/s wr, 27 op/s 2024-04-26T21:53:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:08 smithi186 ceph-mon[32991]: osdmap e135: 12 total, 12 up, 12 in 2024-04-26T21:53:08.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:08 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/614130266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-26T21:53:08.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.738+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f15b41b0f30 con 0x7f15b4115a80 2024-04-26T21:53:08.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.739+0000 7f15aea00640 1 -- 172.21.15.1:0/4186630857 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v135) ==== 81+0+982 (secure 0 0 0) 0x7f15a409cc70 con 0x7f15b4115a80 2024-04-26T21:53:08.746 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2275413710 2024-04-27T21:30:18.553822+0000 2024-04-26T21:53:08.746 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3004159799 2024-04-27T21:18:55.151699+0000 2024-04-26T21:53:08.746 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/1051804891 2024-04-27T21:18:20.022040+0000 2024-04-26T21:53:08.746 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/278183393 2024-04-27T21:18:20.022040+0000 2024-04-26T21:53:08.746 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2563183900 2024-04-27T21:30:18.553822+0000 2024-04-26T21:53:08.746 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/678258621 2024-04-27T21:18:20.022040+0000 2024-04-26T21:53:08.747 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/4024107554 2024-04-27T21:30:18.553822+0000 2024-04-26T21:53:08.747 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/3159681858 2024-04-27T21:18:55.151699+0000 2024-04-26T21:53:08.747 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/3822624839 2024-04-27T21:18:55.151699+0000 2024-04-26T21:53:08.747 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2080672016 2024-04-27T21:30:18.553822+0000 2024-04-26T21:53:08.747 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/3159681858 2024-04-27T21:18:55.151699+0000 2024-04-26T21:53:08.747 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/2786715792 2024-04-27T21:30:18.553822+0000 2024-04-26T21:53:08.747 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/304032214 2024-04-27T21:18:20.022040+0000 2024-04-26T21:53:08.747 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6801/2786715792 2024-04-27T21:30:18.553822+0000 2024-04-26T21:53:08.747 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/2647613577 2024-04-27T21:18:55.151699+0000 2024-04-26T21:53:08.748 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:6800/304032214 2024-04-27T21:18:20.022040+0000 2024-04-26T21:53:08.748 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.15.1:0/590615206 2024-04-27T21:30:18.553822+0000 2024-04-26T21:53:08.749 INFO:teuthology.orchestra.run.smithi001.stderr:listed 17 entries 2024-04-26T21:53:08.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f159c0a3f20 msgr2=0x7f159c0a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:08.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f159c0a3f20 0x7f159c0a63e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f15a8005ab0 tx=0x7f15a8005fd0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15b4115a80 msgr2=0x7f15b41a4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:08.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15b4115a80 0x7f15b41a4340 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f15a400e9f0 tx=0x7f15a400eeb0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 shutdown_connections 2024-04-26T21:53:08.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f159c0a3f20 0x7f159c0a63e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15b4115a80 0x7f15b41a4340 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15b41a8670 0x7f15b41a8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 --2- 172.21.15.1:0/4186630857 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f15b410a250 0x7f15b41a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:08.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 >> 172.21.15.1:0/4186630857 conn(0x7f15b406ed70 msgr2=0x7f15b410b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:08.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.745+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 shutdown_connections 2024-04-26T21:53:08.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:08.746+0000 7f15bc800640 1 -- 172.21.15.1:0/4186630857 wait complete. 2024-04-26T21:53:08.780 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-26T21:53:08.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:53:08.780 DEBUG:teuthology.orchestra.run.smithi001:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-26T21:53:09.209 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T21:53:09.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:53:09.209 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-26T21:53:09.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:53:09.297 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-04-26T21:53:09.401 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-26T21:53:09.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:53:09.402 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-26T21:53:09.501 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:09 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/4186630857' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:53:09.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-26T21:53:09.521 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-04-26T21:53:09.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:09 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/4186630857' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:53:09.666 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T21:53:09.679 INFO:tasks.cephadm:Teardown begin 2024-04-26T21:53:09.679 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-26T21:53:09.679 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:53:09.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:09 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/4186630857' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T21:53:09.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.967+0000 7f9ac1000640 1 -- 172.21.15.1:0/4096137115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9abc10b330 msgr2=0x7f9abc10b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:09.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.967+0000 7f9ac1000640 1 --2- 172.21.15.1:0/4096137115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9abc10b330 0x7f9abc10b710 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009860 tx=0x7f9aac02a360 comp rx=0 tx=0).stop 2024-04-26T21:53:09.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.968+0000 7f9ac1000640 1 -- 172.21.15.1:0/4096137115 shutdown_connections 2024-04-26T21:53:09.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.968+0000 7f9ac1000640 1 --2- 172.21.15.1:0/4096137115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9abc110400 0x7f9abc117fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:09.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.968+0000 7f9ac1000640 1 --2- 172.21.15.1:0/4096137115 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc10bce0 0x7f9abc10fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:09.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.968+0000 7f9ac1000640 1 --2- 172.21.15.1:0/4096137115 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9abc10b330 0x7f9abc10b710 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:09.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.968+0000 7f9ac1000640 1 -- 172.21.15.1:0/4096137115 >> 172.21.15.1:0/4096137115 conn(0x7f9abc070a60 msgr2=0x7f9abc072e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:09.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.969+0000 7f9ac1000640 1 -- 172.21.15.1:0/4096137115 shutdown_connections 2024-04-26T21:53:09.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.969+0000 7f9ac1000640 1 -- 172.21.15.1:0/4096137115 wait complete. 2024-04-26T21:53:09.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.970+0000 7f9ac1000640 1 Processor -- start 2024-04-26T21:53:09.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.970+0000 7f9ac1000640 1 -- start start 2024-04-26T21:53:09.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9ac1000640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9abc10b330 0x7f9abc1a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:09.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9ac1000640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9abc110400 0x7f9abc1a9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:09.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9ac1000640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc1a3910 0x7f9abc1a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:09.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9ac1000640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc11b370 con 0x7f9abc10b330 2024-04-26T21:53:09.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9ac1000640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc11b1f0 con 0x7f9abc1a3910 2024-04-26T21:53:09.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9ac1000640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9abc11b4f0 con 0x7f9abc110400 2024-04-26T21:53:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9abc110400 0x7f9abc1a9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9ab9400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9abc110400 0x7f9abc1a9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54956/0 (socket says 172.21.15.1:54956) 2024-04-26T21:53:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9ab9400640 1 -- 172.21.15.1:0/3102950064 learned_addr learned my addr 172.21.15.1:0/3102950064 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9aba800640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc1a3910 0x7f9abc1a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:09.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.971+0000 7f9ab9e00640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9abc10b330 0x7f9abc1a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.972+0000 7f9ab9400640 1 -- 172.21.15.1:0/3102950064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc1a3910 msgr2=0x7f9abc1a3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.972+0000 7f9ab9400640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc1a3910 0x7f9abc1a3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.972+0000 7f9ab9400640 1 -- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9abc10b330 msgr2=0x7f9abc1a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.972+0000 7f9ab9400640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9abc10b330 0x7f9abc1a9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:09.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.972+0000 7f9ab9400640 1 -- 172.21.15.1:0/3102950064 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9abc1a4430 con 0x7f9abc110400 2024-04-26T21:53:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.972+0000 7f9ab9e00640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9abc10b330 0x7f9abc1a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:53:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.972+0000 7f9ab9400640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9abc110400 0x7f9abc1a9ca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8005c80 tx=0x7f9aa800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.973+0000 7f9ab2a00640 1 -- 172.21.15.1:0/3102950064 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8017030 con 0x7f9abc110400 2024-04-26T21:53:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.973+0000 7f9ab2a00640 1 -- 172.21.15.1:0/3102950064 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aa8004020 con 0x7f9abc110400 2024-04-26T21:53:09.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.973+0000 7f9ab2a00640 1 -- 172.21.15.1:0/3102950064 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8003be0 con 0x7f9abc110400 2024-04-26T21:53:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.974+0000 7f9ac1000640 1 -- 172.21.15.1:0/3102950064 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9abc1a4720 con 0x7f9abc110400 2024-04-26T21:53:09.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.974+0000 7f9ac1000640 1 -- 172.21.15.1:0/3102950064 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9abc1122a0 con 0x7f9abc110400 2024-04-26T21:53:09.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.980+0000 7f9ac1000640 1 -- 172.21.15.1:0/3102950064 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9abc0697e0 con 0x7f9abc110400 2024-04-26T21:53:09.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.982+0000 7f9ab2a00640 1 -- 172.21.15.1:0/3102950064 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f9aa801a070 con 0x7f9abc110400 2024-04-26T21:53:09.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.984+0000 7f9ab2a00640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9aa00a3ef0 0x7f9aa00a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:09.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.984+0000 7f9ab2a00640 1 -- 172.21.15.1:0/3102950064 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9730+0+0 (secure 0 0 0) 0x7f9aa80d0120 con 0x7f9abc110400 2024-04-26T21:53:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.987+0000 7f9ab2a00640 1 -- 172.21.15.1:0/3102950064 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9aa8021020 con 0x7f9abc110400 2024-04-26T21:53:09.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.988+0000 7f9ab9e00640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9aa00a3ef0 0x7f9aa00a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:09.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:09.989+0000 7f9ab9e00640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9aa00a3ef0 0x7f9aa00a63b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009830 tx=0x7f9aac030340 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:10.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.470+0000 7f9ac1000640 1 -- 172.21.15.1:0/3102950064 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9abc1a4e40 con 0x7f9abc110400 2024-04-26T21:53:10.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.473+0000 7f9ab2a00640 1 -- 172.21.15.1:0/3102950064 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+11857 (secure 0 0 0) 0x7f9aa809c2a0 con 0x7f9abc110400 2024-04-26T21:53:10.474 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:53:10.474 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":59,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:27:02.795246+0000","modified":"2024-04-26T21:50:42.156119+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24572,"mds_2":24548},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24548":{"gid":24548,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26905":{"gid":26905,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/3084942661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3084942661},{"type":"v1","addr":"172.21.15.186:6839","nonce":3084942661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24548,24572,24602]},"id":1}]} 2024-04-26T21:53:10.474 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 59 2024-04-26T21:53:10.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.476+0000 7f9ac1000640 1 -- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9aa00a3ef0 msgr2=0x7f9aa00a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:10.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.476+0000 7f9ac1000640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9aa00a3ef0 0x7f9aa00a63b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009830 tx=0x7f9aac030340 comp rx=0 tx=0).stop 2024-04-26T21:53:10.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.476+0000 7f9ac1000640 1 -- 172.21.15.1:0/3102950064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9abc110400 msgr2=0x7f9abc1a9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:10.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.476+0000 7f9ac1000640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9abc110400 0x7f9abc1a9ca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8005c80 tx=0x7f9aa800dde0 comp rx=0 tx=0).stop 2024-04-26T21:53:10.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.477+0000 7f9ac1000640 1 -- 172.21.15.1:0/3102950064 shutdown_connections 2024-04-26T21:53:10.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.477+0000 7f9ac1000640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9aa00a3ef0 0x7f9aa00a63b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:10.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.477+0000 7f9ac1000640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9abc110400 0x7f9abc1a9ca0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:10.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.477+0000 7f9ac1000640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9abc1a3910 0x7f9abc1a3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:10.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.477+0000 7f9ac1000640 1 --2- 172.21.15.1:0/3102950064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9abc10b330 0x7f9abc1a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:10.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.477+0000 7f9ac1000640 1 -- 172.21.15.1:0/3102950064 >> 172.21.15.1:0/3102950064 conn(0x7f9abc070a60 msgr2=0x7f9abc070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:10.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.477+0000 7f9ac1000640 1 -- 172.21.15.1:0/3102950064 shutdown_connections 2024-04-26T21:53:10.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.477+0000 7f9ac1000640 1 -- 172.21.15.1:0/3102950064 wait complete. 2024-04-26T21:53:10.508 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-26T21:53:10.567 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:10 smithi001 ceph-mon[28054]: pgmap v727: 129 pgs: 6 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 106 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 196 KiB/s wr, 31 op/s 2024-04-26T21:53:10.576 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:10 smithi038 ceph-mon[32724]: pgmap v727: 129 pgs: 6 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 106 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 196 KiB/s wr, 31 op/s 2024-04-26T21:53:10.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:10 smithi186 ceph-mon[32991]: pgmap v727: 129 pgs: 6 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 106 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 196 KiB/s wr, 31 op/s 2024-04-26T21:53:10.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.813+0000 7f7b79e00640 1 -- 172.21.15.1:0/4224934164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b7410ab90 msgr2=0x7f7b7410af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:10.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.813+0000 7f7b79e00640 1 --2- 172.21.15.1:0/4224934164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b7410ab90 0x7f7b7410af70 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7b64009900 tx=0x7f7b6402a410 comp rx=0 tx=0).stop 2024-04-26T21:53:10.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.814+0000 7f7b79e00640 1 -- 172.21.15.1:0/4224934164 shutdown_connections 2024-04-26T21:53:10.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.814+0000 7f7b79e00640 1 --2- 172.21.15.1:0/4224934164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b741054b0 0x7f7b741119c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:10.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.814+0000 7f7b79e00640 1 --2- 172.21.15.1:0/4224934164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b74104af0 0x7f7b74104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:10.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.814+0000 7f7b79e00640 1 --2- 172.21.15.1:0/4224934164 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b7410ab90 0x7f7b7410af70 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:10.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.814+0000 7f7b79e00640 1 -- 172.21.15.1:0/4224934164 >> 172.21.15.1:0/4224934164 conn(0x7f7b741005e0 msgr2=0x7f7b74102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:10.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.814+0000 7f7b79e00640 1 -- 172.21.15.1:0/4224934164 shutdown_connections 2024-04-26T21:53:10.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.815+0000 7f7b79e00640 1 -- 172.21.15.1:0/4224934164 wait complete. 2024-04-26T21:53:10.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.816+0000 7f7b79e00640 1 Processor -- start 2024-04-26T21:53:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.816+0000 7f7b79e00640 1 -- start start 2024-04-26T21:53:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b79e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b74104af0 0x7f7b74073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b79e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b741054b0 0x7f7b74074510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b79e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b7410ab90 0x7f7b7406e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b79e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7406ca70 con 0x7f7b7410ab90 2024-04-26T21:53:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b79e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7406c8f0 con 0x7f7b74104af0 2024-04-26T21:53:10.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b79e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7406cbf0 con 0x7f7b741054b0 2024-04-26T21:53:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b72a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b74104af0 0x7f7b74073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b72a00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b74104af0 0x7f7b74073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:58886/0 (socket says 172.21.15.1:58886) 2024-04-26T21:53:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b72a00640 1 -- 172.21.15.1:0/2046092573 learned_addr learned my addr 172.21.15.1:0/2046092573 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b72000640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b741054b0 0x7f7b74074510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.817+0000 7f7b73400640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b7410ab90 0x7f7b7406e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.818+0000 7f7b72000640 1 -- 172.21.15.1:0/2046092573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b74104af0 msgr2=0x7f7b74073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.818+0000 7f7b72000640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b74104af0 0x7f7b74073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:10.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.818+0000 7f7b72000640 1 -- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b7410ab90 msgr2=0x7f7b7406e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.818+0000 7f7b72000640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b7410ab90 0x7f7b7406e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.818+0000 7f7b72000640 1 -- 172.21.15.1:0/2046092573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7406ea50 con 0x7f7b741054b0 2024-04-26T21:53:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.818+0000 7f7b73400640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b7410ab90 0x7f7b7406e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T21:53:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.818+0000 7f7b72000640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b741054b0 0x7f7b74074510 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000c970 tx=0x7f7b6000ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.819+0000 7f7b78a00640 1 -- 172.21.15.1:0/2046092573 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60007bb0 con 0x7f7b741054b0 2024-04-26T21:53:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.819+0000 7f7b78a00640 1 -- 172.21.15.1:0/2046092573 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b60007d50 con 0x7f7b741054b0 2024-04-26T21:53:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.819+0000 7f7b79e00640 1 -- 172.21.15.1:0/2046092573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7406ed40 con 0x7f7b741054b0 2024-04-26T21:53:10.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.819+0000 7f7b78a00640 1 -- 172.21.15.1:0/2046092573 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60004df0 con 0x7f7b741054b0 2024-04-26T21:53:10.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.822+0000 7f7b79e00640 1 -- 172.21.15.1:0/2046092573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7411a900 con 0x7f7b741054b0 2024-04-26T21:53:10.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.827+0000 7f7b78a00640 1 -- 172.21.15.1:0/2046092573 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f7b6000f530 con 0x7f7b741054b0 2024-04-26T21:53:10.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.831+0000 7f7b79e00640 1 -- 172.21.15.1:0/2046092573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b741061f0 con 0x7f7b741054b0 2024-04-26T21:53:10.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.832+0000 7f7b78a00640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7b580a3ea0 0x7f7b580a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:10.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.832+0000 7f7b78a00640 1 -- 172.21.15.1:0/2046092573 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9730+0+0 (secure 0 0 0) 0x7f7b600d3170 con 0x7f7b741054b0 2024-04-26T21:53:10.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.832+0000 7f7b72a00640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7b580a3ea0 0x7f7b580a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:10.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.834+0000 7f7b72a00640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7b580a3ea0 0x7f7b580a6360 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7b640096c0 tx=0x7f7b6402a910 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:10.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:10.837+0000 7f7b78a00640 1 -- 172.21.15.1:0/2046092573 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f7b60014030 con 0x7f7b741054b0 2024-04-26T21:53:11.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.310+0000 7f7b79e00640 1 -- 172.21.15.1:0/2046092573 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b7410d060 con 0x7f7b741054b0 2024-04-26T21:53:11.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.311+0000 7f7b78a00640 1 -- 172.21.15.1:0/2046092573 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+11857 (secure 0 0 0) 0x7f7b6009bf10 con 0x7f7b741054b0 2024-04-26T21:53:11.332 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:53:11.332 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2332941873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6834","nonce":2332941873},{"type":"v1","addr":"172.21.15.1:6835","nonce":2332941873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6841/2045981022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6840","nonce":2045981022},{"type":"v1","addr":"172.21.15.1:6841","nonce":2045981022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24451,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1838831066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1838831066},{"type":"v1","addr":"172.21.15.186:6833","nonce":1838831066}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24467,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6833/1237951568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":1237951568},{"type":"v1","addr":"172.21.15.38:6833","nonce":1237951568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24503,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/532288494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":532288494},{"type":"v1","addr":"172.21.15.1:6837","nonce":532288494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24511,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1778623983","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1778623983},{"type":"v1","addr":"172.21.15.186:6835","nonce":1778623983}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[{"mdsmap":{"epoch":59,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-26T21:27:02.795246+0000","modified":"2024-04-26T21:50:42.156119+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24602,"mds_1":24572,"mds_2":24548},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.38:6835/3801556553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":3801556553},{"type":"v1","addr":"172.21.15.38:6835","nonce":3801556553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6837/3582845013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3582845013},{"type":"v1","addr":"172.21.15.186:6837","nonce":3582845013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24548":{"gid":24548,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.38:6837/1033703980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":1033703980},{"type":"v1","addr":"172.21.15.38:6837","nonce":1033703980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":11,"addr":"172.21.15.1:6839/338659174","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6838","nonce":338659174},{"type":"v1","addr":"172.21.15.1:6839","nonce":338659174}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24602":{"gid":24602,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.38:6839/1675711617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":1675711617},{"type":"v1","addr":"172.21.15.38:6839","nonce":1675711617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26905":{"gid":26905,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6839/3084942661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":3084942661},{"type":"v1","addr":"172.21.15.186:6839","nonce":3084942661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24602,"qdb_cluster":[24548,24572,24602]},"id":1}]} 2024-04-26T21:53:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 59 2024-04-26T21:53:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.314+0000 7f7b79e00640 1 -- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7b580a3ea0 msgr2=0x7f7b580a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.314+0000 7f7b79e00640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7b580a3ea0 0x7f7b580a6360 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7b640096c0 tx=0x7f7b6402a910 comp rx=0 tx=0).stop 2024-04-26T21:53:11.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.314+0000 7f7b79e00640 1 -- 172.21.15.1:0/2046092573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b741054b0 msgr2=0x7f7b74074510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.314+0000 7f7b79e00640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b741054b0 0x7f7b74074510 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000c970 tx=0x7f7b6000ce30 comp rx=0 tx=0).stop 2024-04-26T21:53:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.315+0000 7f7b79e00640 1 -- 172.21.15.1:0/2046092573 shutdown_connections 2024-04-26T21:53:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.315+0000 7f7b79e00640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f7b580a3ea0 0x7f7b580a6360 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.315+0000 7f7b79e00640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b741054b0 0x7f7b74074510 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.315+0000 7f7b79e00640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b7410ab90 0x7f7b7406e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.315+0000 7f7b79e00640 1 --2- 172.21.15.1:0/2046092573 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b74104af0 0x7f7b74073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.315+0000 7f7b79e00640 1 -- 172.21.15.1:0/2046092573 >> 172.21.15.1:0/2046092573 conn(0x7f7b741005e0 msgr2=0x7f7b74100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:11.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.315+0000 7f7b79e00640 1 -- 172.21.15.1:0/2046092573 shutdown_connections 2024-04-26T21:53:11.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.315+0000 7f7b79e00640 1 -- 172.21.15.1:0/2046092573 wait complete. 2024-04-26T21:53:11.349 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T21:53:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.649+0000 7fcde1e00640 1 -- 172.21.15.1:0/578762791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc06b0c0 msgr2=0x7fcddc06b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:11.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.649+0000 7fcde1e00640 1 --2- 172.21.15.1:0/578762791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc06b0c0 0x7fcddc06b520 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009900 tx=0x7fcdc802a410 comp rx=0 tx=0).stop 2024-04-26T21:53:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.650+0000 7fcde1e00640 1 -- 172.21.15.1:0/578762791 shutdown_connections 2024-04-26T21:53:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.650+0000 7fcde1e00640 1 --2- 172.21.15.1:0/578762791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc06b0c0 0x7fcddc06b520 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.650+0000 7fcde1e00640 1 --2- 172.21.15.1:0/578762791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcddc06ba60 0x7fcddc10f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.650+0000 7fcde1e00640 1 --2- 172.21.15.1:0/578762791 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc1089a0 0x7fcddc108d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.650+0000 7fcde1e00640 1 -- 172.21.15.1:0/578762791 >> 172.21.15.1:0/578762791 conn(0x7fcddc0785f0 msgr2=0x7fcddc078a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:11.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.650+0000 7fcde1e00640 1 -- 172.21.15.1:0/578762791 shutdown_connections 2024-04-26T21:53:11.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.650+0000 7fcde1e00640 1 -- 172.21.15.1:0/578762791 wait complete. 2024-04-26T21:53:11.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.652+0000 7fcde1e00640 1 Processor -- start 2024-04-26T21:53:11.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.652+0000 7fcde1e00640 1 -- start start 2024-04-26T21:53:11.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.653+0000 7fcde1e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcddc06b0c0 0x7fcddc10c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:11.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.653+0000 7fcde1e00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc06ba60 0x7fcddc10ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:11.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.653+0000 7fcde1e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1089a0 0x7fcddc1a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:11.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.653+0000 7fcde1e00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc112c20 con 0x7fcddc06b0c0 2024-04-26T21:53:11.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.653+0000 7fcde1e00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc112aa0 con 0x7fcddc06ba60 2024-04-26T21:53:11.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.653+0000 7fcde1e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc112da0 con 0x7fcddc1089a0 2024-04-26T21:53:11.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.654+0000 7fcddb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1089a0 0x7fcddc1a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:11.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.654+0000 7fcddaa00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcddc06b0c0 0x7fcddc10c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:11.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.654+0000 7fcddb400640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1089a0 0x7fcddc1a3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:54996/0 (socket says 172.21.15.1:54996) 2024-04-26T21:53:11.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.654+0000 7fcddb400640 1 -- 172.21.15.1:0/1410555378 learned_addr learned my addr 172.21.15.1:0/1410555378 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:11.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.654+0000 7fcdda000640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc06ba60 0x7fcddc10ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:11.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.654+0000 7fcddb400640 1 -- 172.21.15.1:0/1410555378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc06ba60 msgr2=0x7fcddc10ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:11.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.654+0000 7fcddb400640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc06ba60 0x7fcddc10ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.654+0000 7fcddb400640 1 -- 172.21.15.1:0/1410555378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcddc06b0c0 msgr2=0x7fcddc10c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:11.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.654+0000 7fcddb400640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcddc06b0c0 0x7fcddc10c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.654+0000 7fcddb400640 1 -- 172.21.15.1:0/1410555378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcddc1a4700 con 0x7fcddc1089a0 2024-04-26T21:53:11.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.655+0000 7fcddb400640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1089a0 0x7fcddc1a3fd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400b970 tx=0x7fcdd400be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:11.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.655+0000 7fcde0a00640 1 -- 172.21.15.1:0/1410555378 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4019070 con 0x7fcddc1089a0 2024-04-26T21:53:11.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.655+0000 7fcde0a00640 1 -- 172.21.15.1:0/1410555378 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd4014440 con 0x7fcddc1089a0 2024-04-26T21:53:11.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.655+0000 7fcde1e00640 1 -- 172.21.15.1:0/1410555378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcddc1a49f0 con 0x7fcddc1089a0 2024-04-26T21:53:11.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.655+0000 7fcde0a00640 1 -- 172.21.15.1:0/1410555378 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4007630 con 0x7fcddc1089a0 2024-04-26T21:53:11.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.658+0000 7fcde1e00640 1 -- 172.21.15.1:0/1410555378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcddc1ac280 con 0x7fcddc1089a0 2024-04-26T21:53:11.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.659+0000 7fcde1e00640 1 -- 172.21.15.1:0/1410555378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcddc069c30 con 0x7fcddc1089a0 2024-04-26T21:53:11.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.660+0000 7fcde0a00640 1 -- 172.21.15.1:0/1410555378 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fcdd401c070 con 0x7fcddc1089a0 2024-04-26T21:53:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.662+0000 7fcde0a00640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcdc00a3d20 0x7fcdc00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:11.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.663+0000 7fcddaa00640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcdc00a3d20 0x7fcdc00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:11.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.663+0000 7fcde0a00640 1 -- 172.21.15.1:0/1410555378 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9730+0+0 (secure 0 0 0) 0x7fcdd4099c80 con 0x7fcddc1089a0 2024-04-26T21:53:11.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.666+0000 7fcddaa00640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcdc00a3d20 0x7fcdc00a61e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009b60 tx=0x7fcdcc005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:11.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.667+0000 7fcde0a00640 1 -- 172.21.15.1:0/1410555378 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcdd409c2e0 con 0x7fcddc1089a0 2024-04-26T21:53:11.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:11 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/3102950064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:53:11.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:11 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2046092573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:53:11.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:11 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/3102950064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:53:11.817 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:11 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2046092573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:53:11.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:11 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/3102950064' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:53:11.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:11 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2046092573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-26T21:53:11.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.991+0000 7fcde1e00640 1 -- 172.21.15.1:0/1410555378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcddc1ac510 con 0x7fcddc1089a0 2024-04-26T21:53:11.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.994+0000 7fcde0a00640 1 -- 172.21.15.1:0/1410555378 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) ==== 74+0+25524 (secure 0 0 0) 0x7fcdd409bbe0 con 0x7fcddc1089a0 2024-04-26T21:53:11.995 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-04-26T21:53:11.995 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":135,"fsid":"44ef3d6e-0412-11ef-bc93-c7b262605968","created":"2024-04-26T21:17:46.700596+0000","modified":"2024-04-26T21:53:07.284461+0000","last_up_change":"2024-04-26T21:26:03.319646+0000","last_in_change":"2024-04-26T21:25:43.630938+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T21:21:43.328190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-26T21:27:00.381106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-26T21:27:01.743898+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":41,"snap_epoch":135,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-26T21:27:05.341216+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"135","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":41,"snap_epoch":135,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"17d90e9e-d845-4bcc-b7a4-aad2e0a71b7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6803","nonce":3716675555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6805","nonce":3716675555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6809","nonce":3716675555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":3716675555},{"type":"v1","addr":"172.21.15.1:6807","nonce":3716675555}]},"public_addr":"172.21.15.1:6803/3716675555","cluster_addr":"172.21.15.1:6805/3716675555","heartbeat_back_addr":"172.21.15.1:6809/3716675555","heartbeat_front_addr":"172.21.15.1:6807/3716675555","state":["exists","up"]},{"osd":1,"uuid":"bf91a985-daf1-49bb-9ad9-d5eaabbe6896","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6801","nonce":2314678612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6803","nonce":2314678612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6807","nonce":2314678612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":2314678612},{"type":"v1","addr":"172.21.15.38:6805","nonce":2314678612}]},"public_addr":"172.21.15.38:6801/2314678612","cluster_addr":"172.21.15.38:6803/2314678612","heartbeat_back_addr":"172.21.15.38:6807/2314678612","heartbeat_front_addr":"172.21.15.38:6805/2314678612","state":["exists","up"]},{"osd":2,"uuid":"02453d9d-0297-4356-a632-9b786e930f63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6801","nonce":2555417177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6803","nonce":2555417177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6807","nonce":2555417177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2555417177},{"type":"v1","addr":"172.21.15.186:6805","nonce":2555417177}]},"public_addr":"172.21.15.186:6801/2555417177","cluster_addr":"172.21.15.186:6803/2555417177","heartbeat_back_addr":"172.21.15.186:6807/2555417177","heartbeat_front_addr":"172.21.15.186:6805/2555417177","state":["exists","up"]},{"osd":3,"uuid":"418b3cb6-8d70-41b8-a9b3-56fc10a2b1cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6811","nonce":851547730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6813","nonce":851547730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6817","nonce":851547730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":851547730},{"type":"v1","addr":"172.21.15.1:6815","nonce":851547730}]},"public_addr":"172.21.15.1:6811/851547730","cluster_addr":"172.21.15.1:6813/851547730","heartbeat_back_addr":"172.21.15.1:6817/851547730","heartbeat_front_addr":"172.21.15.1:6815/851547730","state":["exists","up"]},{"osd":4,"uuid":"e5b9f1af-09e2-4cbb-87fa-38cb698dfa2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6809","nonce":2096224}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6811","nonce":2096224}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6815","nonce":2096224}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2096224},{"type":"v1","addr":"172.21.15.38:6813","nonce":2096224}]},"public_addr":"172.21.15.38:6809/2096224","cluster_addr":"172.21.15.38:6811/2096224","heartbeat_back_addr":"172.21.15.38:6815/2096224","heartbeat_front_addr":"172.21.15.38:6813/2096224","state":["exists","up"]},{"osd":5,"uuid":"fb2ab549-8bab-42e1-8a6a-3a41b28301c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6809","nonce":577720361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6811","nonce":577720361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6815","nonce":577720361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":577720361},{"type":"v1","addr":"172.21.15.186:6813","nonce":577720361}]},"public_addr":"172.21.15.186:6809/577720361","cluster_addr":"172.21.15.186:6811/577720361","heartbeat_back_addr":"172.21.15.186:6815/577720361","heartbeat_front_addr":"172.21.15.186:6813/577720361","state":["exists","up"]},{"osd":6,"uuid":"55da783d-0f5f-479b-b13f-80ba6bd8eb72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6819","nonce":1579861250}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6821","nonce":1579861250}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6825","nonce":1579861250}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":1579861250},{"type":"v1","addr":"172.21.15.1:6823","nonce":1579861250}]},"public_addr":"172.21.15.1:6819/1579861250","cluster_addr":"172.21.15.1:6821/1579861250","heartbeat_back_addr":"172.21.15.1:6825/1579861250","heartbeat_front_addr":"172.21.15.1:6823/1579861250","state":["exists","up"]},{"osd":7,"uuid":"dfdf3d21-894f-4b0d-9055-152108098b66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6817","nonce":826797783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6819","nonce":826797783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6823","nonce":826797783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":826797783},{"type":"v1","addr":"172.21.15.38:6821","nonce":826797783}]},"public_addr":"172.21.15.38:6817/826797783","cluster_addr":"172.21.15.38:6819/826797783","heartbeat_back_addr":"172.21.15.38:6823/826797783","heartbeat_front_addr":"172.21.15.38:6821/826797783","state":["exists","up"]},{"osd":8,"uuid":"74279a09-821c-4a4f-94b9-1ac7a56d6cf7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6817","nonce":758273463}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6819","nonce":758273463}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6823","nonce":758273463}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":758273463},{"type":"v1","addr":"172.21.15.186:6821","nonce":758273463}]},"public_addr":"172.21.15.186:6817/758273463","cluster_addr":"172.21.15.186:6819/758273463","heartbeat_back_addr":"172.21.15.186:6823/758273463","heartbeat_front_addr":"172.21.15.186:6821/758273463","state":["exists","up"]},{"osd":9,"uuid":"f7bd72f3-4a1d-4aca-b57f-9939e36d9786","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6827","nonce":4290308342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6829","nonce":4290308342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6833","nonce":4290308342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":4290308342},{"type":"v1","addr":"172.21.15.1:6831","nonce":4290308342}]},"public_addr":"172.21.15.1:6827/4290308342","cluster_addr":"172.21.15.1:6829/4290308342","heartbeat_back_addr":"172.21.15.1:6833/4290308342","heartbeat_front_addr":"172.21.15.1:6831/4290308342","state":["exists","up"]},{"osd":10,"uuid":"03733579-57ef-431d-a55f-d01a168480a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6825","nonce":4252488524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6827","nonce":4252488524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6831","nonce":4252488524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":4252488524},{"type":"v1","addr":"172.21.15.38:6829","nonce":4252488524}]},"public_addr":"172.21.15.38:6825/4252488524","cluster_addr":"172.21.15.38:6827/4252488524","heartbeat_back_addr":"172.21.15.38:6831/4252488524","heartbeat_front_addr":"172.21.15.38:6829/4252488524","state":["exists","up"]},{"osd":11,"uuid":"bb81b189-ee25-44fd-bd06-ba73245f065c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6825","nonce":2966559392}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6827","nonce":2966559392}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6831","nonce":2966559392}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2966559392},{"type":"v1","addr":"172.21.15.186:6829","nonce":2966559392}]},"public_addr":"172.21.15.186:6825/2966559392","cluster_addr":"172.21.15.186:6827/2966559392","heartbeat_back_addr":"172.21.15.186:6831/2966559392","heartbeat_front_addr":"172.21.15.186:6829/2966559392","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:20:49.001985+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:14.781296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:21:39.768153+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:13.097481+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:22:39.584111+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:05.008176+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:23:40.735646+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:07.781004+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:24:33.658408+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:08.555882+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:25:35.764845+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T21:26:02.043228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":7},{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.9","mappings":[{"from":5,"to":10}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/2275413710":"2024-04-27T21:30:18.553822+0000","172.21.15.1:0/3004159799":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/1051804891":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/278183393":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/2563183900":"2024-04-27T21:30:18.553822+0000","172.21.15.1:0/678258621":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/4024107554":"2024-04-27T21:30:18.553822+0000","172.21.15.1:6800/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/3822624839":"2024-04-27T21:18:55.151699+0000","172.21.15.1:0/2080672016":"2024-04-27T21:30:18.553822+0000","172.21.15.1:6801/3159681858":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/2786715792":"2024-04-27T21:30:18.553822+0000","172.21.15.1:6801/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:6801/2786715792":"2024-04-27T21:30:18.553822+0000","172.21.15.1:0/2647613577":"2024-04-27T21:18:55.151699+0000","172.21.15.1:6800/304032214":"2024-04-27T21:18:20.022040+0000","172.21.15.1:0/590615206":"2024-04-27T21:30:18.553822+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":28,"length":1},{"begin":41,"length":1}]},{"pool":4,"snaps":[{"begin":28,"length":1},{"begin":41,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":28,"length":1},{"begin":41,"length":1}]},{"pool":4,"snaps":[{"begin":28,"length":1},{"begin":41,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T21:53:11.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.997+0000 7fcde1e00640 1 -- 172.21.15.1:0/1410555378 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcdc00a3d20 msgr2=0x7fcdc00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:11.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.997+0000 7fcde1e00640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcdc00a3d20 0x7fcdc00a61e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009b60 tx=0x7fcdcc005f90 comp rx=0 tx=0).stop 2024-04-26T21:53:11.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.997+0000 7fcde1e00640 1 -- 172.21.15.1:0/1410555378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1089a0 msgr2=0x7fcddc1a3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:11.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.997+0000 7fcde1e00640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1089a0 0x7fcddc1a3fd0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400b970 tx=0x7fcdd400be30 comp rx=0 tx=0).stop 2024-04-26T21:53:11.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.998+0000 7fcde1e00640 1 -- 172.21.15.1:0/1410555378 shutdown_connections 2024-04-26T21:53:11.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.998+0000 7fcde1e00640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcdc00a3d20 0x7fcdc00a61e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.998+0000 7fcde1e00640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1089a0 0x7fcddc1a3fd0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.998+0000 7fcde1e00640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcddc06ba60 0x7fcddc10ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.998+0000 7fcde1e00640 1 --2- 172.21.15.1:0/1410555378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcddc06b0c0 0x7fcddc10c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:11.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.998+0000 7fcde1e00640 1 -- 172.21.15.1:0/1410555378 >> 172.21.15.1:0/1410555378 conn(0x7fcddc0785f0 msgr2=0x7fcddc100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:11.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.998+0000 7fcde1e00640 1 -- 172.21.15.1:0/1410555378 shutdown_connections 2024-04-26T21:53:11.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:11.998+0000 7fcde1e00640 1 -- 172.21.15.1:0/1410555378 wait complete. 2024-04-26T21:53:12.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-26T21:53:12.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.262+0000 7f13d8c00640 1 -- 172.21.15.1:0/1512327190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13d4076e60 msgr2=0x7f13d4077240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:12.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.262+0000 7f13d8c00640 1 --2- 172.21.15.1:0/1512327190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13d4076e60 0x7f13d4077240 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f13c4009930 tx=0x7f13c402a440 comp rx=0 tx=0).stop 2024-04-26T21:53:12.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.263+0000 7f13d8c00640 1 -- 172.21.15.1:0/1512327190 shutdown_connections 2024-04-26T21:53:12.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.263+0000 7f13d8c00640 1 --2- 172.21.15.1:0/1512327190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13d4113850 0x7f13d4115c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.263+0000 7f13d8c00640 1 --2- 172.21.15.1:0/1512327190 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13d4077780 0x7f13d4113310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.263+0000 7f13d8c00640 1 --2- 172.21.15.1:0/1512327190 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13d4076e60 0x7f13d4077240 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.263+0000 7f13d8c00640 1 -- 172.21.15.1:0/1512327190 >> 172.21.15.1:0/1512327190 conn(0x7f13d41006c0 msgr2=0x7f13d4102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:12.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.263+0000 7f13d8c00640 1 -- 172.21.15.1:0/1512327190 shutdown_connections 2024-04-26T21:53:12.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.263+0000 7f13d8c00640 1 -- 172.21.15.1:0/1512327190 wait complete. 2024-04-26T21:53:12.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.264+0000 7f13d8c00640 1 Processor -- start 2024-04-26T21:53:12.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.264+0000 7f13d8c00640 1 -- start start 2024-04-26T21:53:12.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.265+0000 7f13d8c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13d4076e60 0x7f13d41a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:12.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.265+0000 7f13d8c00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13d4077780 0x7f13d41a59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:12.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.265+0000 7f13d8c00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13d4113850 0x7f13d419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:12.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.265+0000 7f13d8c00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f13d41192c0 con 0x7f13d4077780 2024-04-26T21:53:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.265+0000 7f13d8c00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f13d4119140 con 0x7f13d4113850 2024-04-26T21:53:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.265+0000 7f13d8c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f13d4119440 con 0x7f13d4076e60 2024-04-26T21:53:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.265+0000 7f13d2800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13d4113850 0x7f13d419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.265+0000 7f13d2800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13d4113850 0x7f13d419f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.1:58934/0 (socket says 172.21.15.1:58934) 2024-04-26T21:53:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.265+0000 7f13d2800640 1 -- 172.21.15.1:0/2379854287 learned_addr learned my addr 172.21.15.1:0/2379854287 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.265+0000 7f13d1e00640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13d4076e60 0x7f13d41a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.266+0000 7f13d1400640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13d4077780 0x7f13d41a59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.266+0000 7f13d2800640 1 -- 172.21.15.1:0/2379854287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13d4076e60 msgr2=0x7f13d41a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:12.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.266+0000 7f13d2800640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13d4076e60 0x7f13d41a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.266+0000 7f13d2800640 1 -- 172.21.15.1:0/2379854287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13d4077780 msgr2=0x7f13d41a59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:12.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.266+0000 7f13d2800640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13d4077780 0x7f13d41a59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.266+0000 7f13d2800640 1 -- 172.21.15.1:0/2379854287 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d419fda0 con 0x7f13d4113850 2024-04-26T21:53:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.266+0000 7f13d2800640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13d4113850 0x7f13d419f620 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00eb40 tx=0x7f13cc00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.267+0000 7f13caa00640 1 -- 172.21.15.1:0/2379854287 <== mon.2 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc004b20 con 0x7f13d4113850 2024-04-26T21:53:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.267+0000 7f13d8c00640 1 -- 172.21.15.1:0/2379854287 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d41a0090 con 0x7f13d4113850 2024-04-26T21:53:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.267+0000 7f13caa00640 1 -- 172.21.15.1:0/2379854287 <== mon.2 v2:172.21.15.38:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13cc004cc0 con 0x7f13d4113850 2024-04-26T21:53:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.267+0000 7f13caa00640 1 -- 172.21.15.1:0/2379854287 <== mon.2 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc010720 con 0x7f13d4113850 2024-04-26T21:53:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.267+0000 7f13d8c00640 1 -- 172.21.15.1:0/2379854287 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d41ac330 con 0x7f13d4113850 2024-04-26T21:53:12.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.268+0000 7f13d8c00640 1 -- 172.21.15.1:0/2379854287 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d4069c30 con 0x7f13d4113850 2024-04-26T21:53:12.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.271+0000 7f13caa00640 1 -- 172.21.15.1:0/2379854287 <== mon.2 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f13cc005560 con 0x7f13d4113850 2024-04-26T21:53:12.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.272+0000 7f13caa00640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f13b80a3da0 0x7f13b80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:12.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.272+0000 7f13caa00640 1 -- 172.21.15.1:0/2379854287 <== mon.2 v2:172.21.15.38:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 9730+0+0 (secure 0 0 0) 0x7f13cc0d1310 con 0x7f13d4113850 2024-04-26T21:53:12.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.276+0000 7f13d1e00640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f13b80a3da0 0x7f13b80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:12.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.277+0000 7f13d1e00640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f13b80a3da0 0x7f13b80a6260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f13c4031440 tx=0x7f13c4005a20 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:12.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.281+0000 7f13caa00640 1 -- 172.21.15.1:0/2379854287 <== mon.2 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f13cc09d490 con 0x7f13d4113850 2024-04-26T21:53:12.605 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:12 smithi186 ceph-mon[32991]: pgmap v728: 129 pgs: 19 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 64 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 230 KiB/s wr, 32 op/s 2024-04-26T21:53:12.606 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:12 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/1410555378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:53:12.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.635+0000 7f13d8c00640 1 -- 172.21.15.1:0/2379854287 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f13d402dad0 con 0x7f13d4113850 2024-04-26T21:53:12.639 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:12 smithi001 ceph-mon[28054]: pgmap v728: 129 pgs: 19 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 64 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 230 KiB/s wr, 32 op/s 2024-04-26T21:53:12.639 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:12 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/1410555378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:53:12.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.645+0000 7f13caa00640 1 -- 172.21.15.1:0/2379854287 <== mon.2 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v60) ==== 161+0+0 (secure 0 0 0) 0x7f13cc09cd90 con 0x7f13d4113850 2024-04-26T21:53:12.645 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-26T21:53:12.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.651+0000 7f13d8c00640 1 -- 172.21.15.1:0/2379854287 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f13b80a3da0 msgr2=0x7f13b80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:12.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.651+0000 7f13d8c00640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f13b80a3da0 0x7f13b80a6260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f13c4031440 tx=0x7f13c4005a20 comp rx=0 tx=0).stop 2024-04-26T21:53:12.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.651+0000 7f13d8c00640 1 -- 172.21.15.1:0/2379854287 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13d4113850 msgr2=0x7f13d419f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.651+0000 7f13d8c00640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13d4113850 0x7f13d419f620 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00eb40 tx=0x7f13cc00c2d0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.651+0000 7f13d8c00640 1 -- 172.21.15.1:0/2379854287 shutdown_connections 2024-04-26T21:53:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.651+0000 7f13d8c00640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f13b80a3da0 0x7f13b80a6260 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f13c4031440 tx=0x7f13c4005a20 comp rx=0 tx=0).stop 2024-04-26T21:53:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.651+0000 7f13d8c00640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f13d4113850 0x7f13d419f620 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.651+0000 7f13d8c00640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13d4077780 0x7f13d41a59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.651+0000 7f13d8c00640 1 --2- 172.21.15.1:0/2379854287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13d4076e60 0x7f13d41a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.651+0000 7f13d8c00640 1 -- 172.21.15.1:0/2379854287 >> 172.21.15.1:0/2379854287 conn(0x7f13d41006c0 msgr2=0x7f13d4114f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.652+0000 7f13d8c00640 1 -- 172.21.15.1:0/2379854287 shutdown_connections 2024-04-26T21:53:12.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.652+0000 7f13d8c00640 1 -- 172.21.15.1:0/2379854287 wait complete. 2024-04-26T21:53:12.678 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-26T21:53:12.738 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-mon[32724]: pgmap v728: 129 pgs: 19 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 64 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 230 KiB/s wr, 32 op/s 2024-04-26T21:53:12.738 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/1410555378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T21:53:12.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.894+0000 7f79f1600640 1 -- 172.21.15.1:0/4257234240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79ec104230 msgr2=0x7f79ec10c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.894+0000 7f79f1600640 1 --2- 172.21.15.1:0/4257234240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79ec104230 0x7f79ec10c710 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f79d8009900 tx=0x7f79d802a480 comp rx=0 tx=0).stop 2024-04-26T21:53:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.895+0000 7f79f1600640 1 -- 172.21.15.1:0/4257234240 shutdown_connections 2024-04-26T21:53:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.895+0000 7f79f1600640 1 --2- 172.21.15.1:0/4257234240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79ec10cd30 0x7f79ec10f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.895+0000 7f79f1600640 1 --2- 172.21.15.1:0/4257234240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79ec104230 0x7f79ec10c710 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.895+0000 7f79f1600640 1 --2- 172.21.15.1:0/4257234240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79ec103910 0x7f79ec103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.895+0000 7f79f1600640 1 -- 172.21.15.1:0/4257234240 >> 172.21.15.1:0/4257234240 conn(0x7f79ec0783e0 msgr2=0x7f79ec0ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.895+0000 7f79f1600640 1 -- 172.21.15.1:0/4257234240 shutdown_connections 2024-04-26T21:53:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.895+0000 7f79f1600640 1 -- 172.21.15.1:0/4257234240 wait complete. 2024-04-26T21:53:12.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.896+0000 7f79f1600640 1 Processor -- start 2024-04-26T21:53:12.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.896+0000 7f79f1600640 1 -- start start 2024-04-26T21:53:12.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.896+0000 7f79f1600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79ec103910 0x7f79ec196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:12.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.896+0000 7f79f1600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79ec104230 0x7f79ec1972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f1600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79ec10cd30 0x7f79ec19b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f1600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f79ec1107f0 con 0x7f79ec103910 2024-04-26T21:53:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f1600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f79ec110670 con 0x7f79ec104230 2024-04-26T21:53:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f1600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79ec110970 con 0x7f79ec10cd30 2024-04-26T21:53:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79ec10cd30 0x7f79ec19b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79eb400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79ec104230 0x7f79ec1972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79ebe00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79ec103910 0x7f79ec196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f0c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79ec10cd30 0x7f79ec19b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:55062/0 (socket says 172.21.15.1:55062) 2024-04-26T21:53:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f0c00640 1 -- 172.21.15.1:0/661664301 learned_addr learned my addr 172.21.15.1:0/661664301 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f0c00640 1 -- 172.21.15.1:0/661664301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79ec104230 msgr2=0x7f79ec1972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f0c00640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79ec104230 0x7f79ec1972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f0c00640 1 -- 172.21.15.1:0/661664301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79ec103910 msgr2=0x7f79ec196d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f0c00640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79ec103910 0x7f79ec196d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.897+0000 7f79f0c00640 1 -- 172.21.15.1:0/661664301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ec19bd80 con 0x7f79ec10cd30 2024-04-26T21:53:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.898+0000 7f79f0c00640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79ec10cd30 0x7f79ec19b650 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f79e400eb10 tx=0x7f79e400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:12.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.899+0000 7f79e8c00640 1 -- 172.21.15.1:0/661664301 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4004b20 con 0x7f79ec10cd30 2024-04-26T21:53:12.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.899+0000 7f79e8c00640 1 -- 172.21.15.1:0/661664301 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79e4004cc0 con 0x7f79ec10cd30 2024-04-26T21:53:12.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.899+0000 7f79e8c00640 1 -- 172.21.15.1:0/661664301 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4010670 con 0x7f79ec10cd30 2024-04-26T21:53:12.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.899+0000 7f79f1600640 1 -- 172.21.15.1:0/661664301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79ec19c070 con 0x7f79ec10cd30 2024-04-26T21:53:12.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.899+0000 7f79f1600640 1 -- 172.21.15.1:0/661664301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79ec1a3900 con 0x7f79ec10cd30 2024-04-26T21:53:12.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.901+0000 7f79f1600640 1 -- 172.21.15.1:0/661664301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79ec06a4f0 con 0x7f79ec10cd30 2024-04-26T21:53:12.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.903+0000 7f79e8c00640 1 -- 172.21.15.1:0/661664301 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f79e4005020 con 0x7f79ec10cd30 2024-04-26T21:53:12.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.905+0000 7f79e8c00640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f79d00a8280 0x7f79d00aa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:12.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.906+0000 7f79e8c00640 1 -- 172.21.15.1:0/661664301 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 9900+0+0 (secure 0 0 0) 0x7f79e40d1140 con 0x7f79ec10cd30 2024-04-26T21:53:12.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.906+0000 7f79ebe00640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f79d00a8280 0x7f79d00aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.907+0000 7f79ebe00640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f79d00a8280 0x7f79d00aa740 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f79dc005d00 tx=0x7f79dc00b410 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:12.909+0000 7f79e8c00640 1 -- 172.21.15.1:0/661664301 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f79e409d280 con 0x7f79ec10cd30 2024-04-26T21:53:12.954 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:53:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c[63550]: ignoring --setuser ceph since I am not root 2024-04-26T21:53:12.955 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:53:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c[63550]: ignoring --setgroup ceph since I am not root 2024-04-26T21:53:12.955 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:53:12 smithi186 ceph-mds[63554]: -- 172.21.15.186:0/1862549412 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fc17027a40 con 0x55fc16ffb000 2024-04-26T21:53:12.955 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:53:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c[63550]: starting mds.c at 2024-04-26T21:53:12.955 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:53:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: ignoring --setuser ceph since I am not root 2024-04-26T21:53:12.955 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:53:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: ignoring --setgroup ceph since I am not root 2024-04-26T21:53:12.956 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:53:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: starting mds.i at 2024-04-26T21:53:13.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a[28049]: 2024-04-26T21:53:12.637+0000 7faf41000700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T21:53:13.068 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:53:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g[66875]: ignoring --setuser ceph since I am not root 2024-04-26T21:53:13.068 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:53:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g[66875]: ignoring --setgroup ceph since I am not root 2024-04-26T21:53:13.068 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:53:12 smithi001 ceph-mds[66879]: -- 172.21.15.1:0/295525931 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5654d4a83a40 con 0x5654d4a57000 2024-04-26T21:53:13.068 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:53:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g[66875]: starting mds.g at 2024-04-26T21:53:13.076 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b[62819]: ignoring --setuser ceph since I am not root 2024-04-26T21:53:13.077 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b[62819]: ignoring --setgroup ceph since I am not root 2024-04-26T21:53:13.077 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-mds[62823]: -- 172.21.15.38:0/1292731529 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55597bcc1a40 con 0x55597bc94c00 2024-04-26T21:53:13.077 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b[62819]: starting mds.b at 2024-04-26T21:53:13.077 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e[62180]: ignoring --setuser ceph since I am not root 2024-04-26T21:53:13.078 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e[62180]: ignoring --setgroup ceph since I am not root 2024-04-26T21:53:13.078 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-mds[62184]: -- 172.21.15.38:0/2047153493 <== mon.2 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f09ae5a40 con 0x556f09ab8800 2024-04-26T21:53:13.078 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e[62180]: starting mds.e at 2024-04-26T21:53:13.078 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h[62492]: ignoring --setuser ceph since I am not root 2024-04-26T21:53:13.079 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h[62492]: ignoring --setgroup ceph since I am not root 2024-04-26T21:53:13.079 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-mds[62496]: -- 172.21.15.38:0/1595394204 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56240036a4e0 con 0x562400341000 2024-04-26T21:53:13.079 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-mds[62496]: -- 172.21.15.38:0/1595394204 <== mon.0 v2:172.21.15.1:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56240036da40 con 0x562400341000 2024-04-26T21:53:13.079 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:53:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h[62492]: starting mds.h at 2024-04-26T21:53:13.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.221+0000 7f79f1600640 1 -- 172.21.15.1:0/661664301 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f79ec107dd0 con 0x7f79ec10cd30 2024-04-26T21:53:13.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:13 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/2379854287' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T21:53:13.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:13 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T21:53:13.623 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:13 smithi186 ceph-mon[32991]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T21:53:13.624 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:13 smithi186 ceph-mon[32991]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T21:53:13.624 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:13 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T21:53:13.624 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:13 smithi186 ceph-mon[32991]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T21:53:13.624 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:13 smithi186 ceph-mon[32991]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T21:53:13.624 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:13 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/661664301' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:53:13.624 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:13 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:53:13.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.654+0000 7f79e8c00640 1 -- 172.21.15.1:0/661664301 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v61) ==== 104+0+0 (secure 0 0 0) 0x7f79e409cb80 con 0x7f79ec10cd30 2024-04-26T21:53:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.658+0000 7f79f1600640 1 -- 172.21.15.1:0/661664301 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f79d00a8280 msgr2=0x7f79d00aa740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.658+0000 7f79f1600640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f79d00a8280 0x7f79d00aa740 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f79dc005d00 tx=0x7f79dc00b410 comp rx=0 tx=0).stop 2024-04-26T21:53:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.658+0000 7f79f1600640 1 -- 172.21.15.1:0/661664301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79ec10cd30 msgr2=0x7f79ec19b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.658+0000 7f79f1600640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79ec10cd30 0x7f79ec19b650 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f79e400eb10 tx=0x7f79e400c2d0 comp rx=0 tx=0).stop 2024-04-26T21:53:13.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.659+0000 7f79f1600640 1 -- 172.21.15.1:0/661664301 shutdown_connections 2024-04-26T21:53:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.659+0000 7f79f1600640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f79d00a8280 0x7f79d00aa740 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.659+0000 7f79f1600640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79ec10cd30 0x7f79ec19b650 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.659+0000 7f79f1600640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79ec104230 0x7f79ec1972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.659+0000 7f79f1600640 1 --2- 172.21.15.1:0/661664301 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79ec103910 0x7f79ec196d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:13.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.659+0000 7f79f1600640 1 -- 172.21.15.1:0/661664301 >> 172.21.15.1:0/661664301 conn(0x7f79ec0783e0 msgr2=0x7f79ec0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.660+0000 7f79f1600640 1 -- 172.21.15.1:0/661664301 shutdown_connections 2024-04-26T21:53:13.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.660+0000 7f79f1600640 1 -- 172.21.15.1:0/661664301 wait complete. 2024-04-26T21:53:13.679 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:13 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/2379854287' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T21:53:13.680 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:13 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T21:53:13.680 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:13 smithi001 ceph-mon[28054]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T21:53:13.680 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:13 smithi001 ceph-mon[28054]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T21:53:13.680 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:13 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T21:53:13.680 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:13 smithi001 ceph-mon[28054]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T21:53:13.680 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:13 smithi001 ceph-mon[28054]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T21:53:13.680 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:13 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/661664301' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:53:13.680 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:13 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:53:13.683 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-26T21:53:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:13 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/2379854287' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T21:53:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:13 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-26T21:53:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:13 smithi038 ceph-mon[32724]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-26T21:53:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:13 smithi038 ceph-mon[32724]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-26T21:53:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:13 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-26T21:53:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:13 smithi038 ceph-mon[32724]: osdmap e136: 12 total, 12 up, 12 in 2024-04-26T21:53:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:13 smithi038 ceph-mon[32724]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-26T21:53:13.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:13 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/661664301' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:53:13.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:13 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T21:53:13.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.894+0000 7fcfdd800640 1 -- 172.21.15.1:0/2874910737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfd81048f0 msgr2=0x7fcfd8104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:13.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.894+0000 7fcfdd800640 1 --2- 172.21.15.1:0/2874910737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfd81048f0 0x7fcfd8104d70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4009860 tx=0x7fcfc402a340 comp rx=0 tx=0).stop 2024-04-26T21:53:13.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.894+0000 7fcfdd800640 1 -- 172.21.15.1:0/2874910737 shutdown_connections 2024-04-26T21:53:13.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.894+0000 7fcfdd800640 1 --2- 172.21.15.1:0/2874910737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd81052b0 0x7fcfd81117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:13.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.894+0000 7fcfdd800640 1 --2- 172.21.15.1:0/2874910737 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfd81048f0 0x7fcfd8104d70 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:13.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.894+0000 7fcfdd800640 1 --2- 172.21.15.1:0/2874910737 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfd810a900 0x7fcfd810ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:13.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.894+0000 7fcfdd800640 1 -- 172.21.15.1:0/2874910737 >> 172.21.15.1:0/2874910737 conn(0x7fcfd81003a0 msgr2=0x7fcfd81027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:13.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.894+0000 7fcfdd800640 1 -- 172.21.15.1:0/2874910737 shutdown_connections 2024-04-26T21:53:13.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.894+0000 7fcfdd800640 1 -- 172.21.15.1:0/2874910737 wait complete. 2024-04-26T21:53:13.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.895+0000 7fcfdd800640 1 Processor -- start 2024-04-26T21:53:13.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.895+0000 7fcfdd800640 1 -- start start 2024-04-26T21:53:13.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfdd800640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfd81048f0 0x7fcfd819b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:13.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfdd800640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfd81052b0 0x7fcfd819b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:13.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfdd800640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd810a900 0x7fcfd819f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:13.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfdd800640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8114bc0 con 0x7fcfd81048f0 2024-04-26T21:53:13.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfdd800640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8114a40 con 0x7fcfd81052b0 2024-04-26T21:53:13.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfdd800640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8114d40 con 0x7fcfd810a900 2024-04-26T21:53:13.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfd7e00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfd81048f0 0x7fcfd819b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:13.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfdce00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd810a900 0x7fcfd819f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:13.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfdce00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd810a900 0x7fcfd819f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.1:55076/0 (socket says 172.21.15.1:55076) 2024-04-26T21:53:13.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfdce00640 1 -- 172.21.15.1:0/433007417 learned_addr learned my addr 172.21.15.1:0/433007417 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:13.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.896+0000 7fcfd7400640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfd81052b0 0x7fcfd819b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:13.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfdce00640 1 -- 172.21.15.1:0/433007417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfd81052b0 msgr2=0x7fcfd819b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:13.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfdce00640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfd81052b0 0x7fcfd819b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:13.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfdce00640 1 -- 172.21.15.1:0/433007417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfd81048f0 msgr2=0x7fcfd819b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:13.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfdce00640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfd81048f0 0x7fcfd819b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:13.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfdce00640 1 -- 172.21.15.1:0/433007417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd81a0110 con 0x7fcfd810a900 2024-04-26T21:53:13.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfd7400640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfd81052b0 0x7fcfd819b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:53:13.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfdce00640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd810a900 0x7fcfd819f9e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000eb10 tx=0x7fcfd000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:13.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfd4c00640 1 -- 172.21.15.1:0/433007417 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd000ce20 con 0x7fcfd810a900 2024-04-26T21:53:13.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfdd800640 1 -- 172.21.15.1:0/433007417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd81a0400 con 0x7fcfd810a900 2024-04-26T21:53:13.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfd4c00640 1 -- 172.21.15.1:0/433007417 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcfd0005020 con 0x7fcfd810a900 2024-04-26T21:53:13.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfd4c00640 1 -- 172.21.15.1:0/433007417 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0010690 con 0x7fcfd810a900 2024-04-26T21:53:13.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.897+0000 7fcfdd800640 1 -- 172.21.15.1:0/433007417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd81a7c90 con 0x7fcfd810a900 2024-04-26T21:53:13.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.899+0000 7fcfdd800640 1 -- 172.21.15.1:0/433007417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd80506c0 con 0x7fcfd810a900 2024-04-26T21:53:13.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.900+0000 7fcfd4c00640 1 -- 172.21.15.1:0/433007417 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fcfd0005510 con 0x7fcfd810a900 2024-04-26T21:53:13.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.902+0000 7fcfd4c00640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcfbc0a3d20 0x7fcfbc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:13.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.902+0000 7fcfd4c00640 1 -- 172.21.15.1:0/433007417 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 9900+0+0 (secure 0 0 0) 0x7fcfd00d1440 con 0x7fcfd810a900 2024-04-26T21:53:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.902+0000 7fcfd7e00640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcfbc0a3d20 0x7fcfbc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:13.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.903+0000 7fcfd7e00640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcfbc0a3d20 0x7fcfbc0a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8005fd0 tx=0x7fcfc8009260 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:13.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:13.906+0000 7fcfd4c00640 1 -- 172.21.15.1:0/433007417 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fcfd009d520 con 0x7fcfd810a900 2024-04-26T21:53:14.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.111+0000 7fcfdd800640 1 -- 172.21.15.1:0/433007417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcfd802dad0 con 0x7fcfd810a900 2024-04-26T21:53:14.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.657+0000 7fcfd4c00640 1 -- 172.21.15.1:0/433007417 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v137) ==== 181+0+0 (secure 0 0 0) 0x7fcfd009ce20 con 0x7fcfd810a900 2024-04-26T21:53:14.658 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-04-26T21:53:14.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.662+0000 7fcfdd800640 1 -- 172.21.15.1:0/433007417 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcfbc0a3d20 msgr2=0x7fcfbc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:14.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.662+0000 7fcfdd800640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcfbc0a3d20 0x7fcfbc0a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8005fd0 tx=0x7fcfc8009260 comp rx=0 tx=0).stop 2024-04-26T21:53:14.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.662+0000 7fcfdd800640 1 -- 172.21.15.1:0/433007417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd810a900 msgr2=0x7fcfd819f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:14.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.662+0000 7fcfdd800640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd810a900 0x7fcfd819f9e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcfd000eb10 tx=0x7fcfd000ef10 comp rx=0 tx=0).stop 2024-04-26T21:53:14.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.663+0000 7fcfdd800640 1 -- 172.21.15.1:0/433007417 shutdown_connections 2024-04-26T21:53:14.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.663+0000 7fcfdd800640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fcfbc0a3d20 0x7fcfbc0a61e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:14.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.663+0000 7fcfdd800640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcfd810a900 0x7fcfd819f9e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:14.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.663+0000 7fcfdd800640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfd81052b0 0x7fcfd819b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:14.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.663+0000 7fcfdd800640 1 --2- 172.21.15.1:0/433007417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfd81048f0 0x7fcfd819b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:14.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.663+0000 7fcfdd800640 1 -- 172.21.15.1:0/433007417 >> 172.21.15.1:0/433007417 conn(0x7fcfd81003a0 msgr2=0x7fcfd8100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:14.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.664+0000 7fcfdd800640 1 -- 172.21.15.1:0/433007417 shutdown_connections 2024-04-26T21:53:14.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.664+0000 7fcfdd800640 1 -- 172.21.15.1:0/433007417 wait complete. 2024-04-26T21:53:14.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: pgmap v730: 129 pgs: 13 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 87 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 149 KiB/s wr, 8 op/s 2024-04-26T21:53:14.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T21:53:14.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T21:53:14.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: Cluster is now healthy 2024-04-26T21:53:14.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:53:14.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.186:6836/529937003,v1:172.21.15.186:6837/529937003] up:boot 2024-04-26T21:53:14.692 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.186:6838/2053862338,v1:172.21.15.186:6839/2053862338] up:boot 2024-04-26T21:53:14.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.38:6834/445449823,v1:172.21.15.38:6837/445449823] up:boot 2024-04-26T21:53:14.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.38:6835/321568626,v1:172.21.15.38:6838/321568626] up:boot 2024-04-26T21:53:14.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.38:6836/2512694167,v1:172.21.15.38:6839/2512694167] up:boot 2024-04-26T21:53:14.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: mds.? [v2:172.21.15.1:6838/3815241421,v1:172.21.15.1:6839/3815241421] up:boot 2024-04-26T21:53:14.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: fsmap 12 up:standby 2024-04-26T21:53:14.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:53:14.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:53:14.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:53:14.693 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:53:14.694 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:53:14.694 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:53:14.694 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/433007417' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:14.694 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:14 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:14.695 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-26T21:53:14.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: pgmap v730: 129 pgs: 13 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 87 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 149 KiB/s wr, 8 op/s 2024-04-26T21:53:14.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T21:53:14.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T21:53:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: Cluster is now healthy 2024-04-26T21:53:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:53:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.186:6836/529937003,v1:172.21.15.186:6837/529937003] up:boot 2024-04-26T21:53:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.186:6838/2053862338,v1:172.21.15.186:6839/2053862338] up:boot 2024-04-26T21:53:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.38:6834/445449823,v1:172.21.15.38:6837/445449823] up:boot 2024-04-26T21:53:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.38:6835/321568626,v1:172.21.15.38:6838/321568626] up:boot 2024-04-26T21:53:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.38:6836/2512694167,v1:172.21.15.38:6839/2512694167] up:boot 2024-04-26T21:53:14.705 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: mds.? [v2:172.21.15.1:6838/3815241421,v1:172.21.15.1:6839/3815241421] up:boot 2024-04-26T21:53:14.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: fsmap 12 up:standby 2024-04-26T21:53:14.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:53:14.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:53:14.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:53:14.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:53:14.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:53:14.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:53:14.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/433007417' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:14.706 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:14 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:14.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: pgmap v730: 129 pgs: 13 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 87 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 149 KiB/s wr, 8 op/s 2024-04-26T21:53:14.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-26T21:53:14.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-26T21:53:14.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: Cluster is now healthy 2024-04-26T21:53:14.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-26T21:53:14.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.186:6836/529937003,v1:172.21.15.186:6837/529937003] up:boot 2024-04-26T21:53:14.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.186:6838/2053862338,v1:172.21.15.186:6839/2053862338] up:boot 2024-04-26T21:53:14.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.38:6834/445449823,v1:172.21.15.38:6837/445449823] up:boot 2024-04-26T21:53:14.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.38:6835/321568626,v1:172.21.15.38:6838/321568626] up:boot 2024-04-26T21:53:14.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.38:6836/2512694167,v1:172.21.15.38:6839/2512694167] up:boot 2024-04-26T21:53:14.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: mds.? [v2:172.21.15.1:6838/3815241421,v1:172.21.15.1:6839/3815241421] up:boot 2024-04-26T21:53:14.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: fsmap 12 up:standby 2024-04-26T21:53:14.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-26T21:53:14.828 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-26T21:53:14.829 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-26T21:53:14.829 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-26T21:53:14.829 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-26T21:53:14.829 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: from='mgr.15291 172.21.15.1:0/2247070491' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-26T21:53:14.829 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/433007417' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:14.829 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:14 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.937+0000 7fb9ade00640 1 -- 172.21.15.1:0/4060631514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a810a920 msgr2=0x7fb9a810ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.937+0000 7fb9ade00640 1 --2- 172.21.15.1:0/4060631514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a810a920 0x7fb9a810ad00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb998009cd0 tx=0x7fb99802a8e0 comp rx=0 tx=0).stop 2024-04-26T21:53:14.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.937+0000 7fb9ade00640 1 -- 172.21.15.1:0/4060631514 shutdown_connections 2024-04-26T21:53:14.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.937+0000 7fb9ade00640 1 --2- 172.21.15.1:0/4060631514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a81052d0 0x7fb9a81117e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:14.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.937+0000 7fb9ade00640 1 --2- 172.21.15.1:0/4060631514 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9a8104910 0x7fb9a8104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:14.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.937+0000 7fb9ade00640 1 --2- 172.21.15.1:0/4060631514 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a810a920 0x7fb9a810ad00 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:14.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.937+0000 7fb9ade00640 1 -- 172.21.15.1:0/4060631514 >> 172.21.15.1:0/4060631514 conn(0x7fb9a8100400 msgr2=0x7fb9a8102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:14.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.937+0000 7fb9ade00640 1 -- 172.21.15.1:0/4060631514 shutdown_connections 2024-04-26T21:53:14.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.937+0000 7fb9ade00640 1 -- 172.21.15.1:0/4060631514 wait complete. 2024-04-26T21:53:14.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.938+0000 7fb9ade00640 1 Processor -- start 2024-04-26T21:53:14.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.938+0000 7fb9ade00640 1 -- start start 2024-04-26T21:53:14.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb9ade00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a8104910 0x7fb9a819b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:14.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb9ade00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9a81052d0 0x7fb9a819b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:14.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb9ade00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a810a920 0x7fb9a819fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:14.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb9ade00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a8114c00 con 0x7fb9a8104910 2024-04-26T21:53:14.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb9ade00640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a8114a80 con 0x7fb9a81052d0 2024-04-26T21:53:14.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb9ade00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a8114d80 con 0x7fb9a810a920 2024-04-26T21:53:14.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb9aca00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a8104910 0x7fb9a819b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:14.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb9aca00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a8104910 0x7fb9a819b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41852/0 (socket says 172.21.15.1:41852) 2024-04-26T21:53:14.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb9aca00640 1 -- 172.21.15.1:0/251675948 learned_addr learned my addr 172.21.15.1:0/251675948 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:14.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb99fe00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9a81052d0 0x7fb9a819b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:14.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.939+0000 7fb9ad400640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a810a920 0x7fb9a819fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:14.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9aca00640 1 -- 172.21.15.1:0/251675948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a810a920 msgr2=0x7fb9a819fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:14.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9aca00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a810a920 0x7fb9a819fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:14.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9aca00640 1 -- 172.21.15.1:0/251675948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9a81052d0 msgr2=0x7fb9a819b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:14.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9aca00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9a81052d0 0x7fb9a819b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:14.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9aca00640 1 -- 172.21.15.1:0/251675948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a81a0180 con 0x7fb9a8104910 2024-04-26T21:53:14.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9aca00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a8104910 0x7fb9a819b190 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb998005e30 tx=0x7fb9980039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:14.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9a6000640 1 -- 172.21.15.1:0/251675948 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998004040 con 0x7fb9a8104910 2024-04-26T21:53:14.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9a6000640 1 -- 172.21.15.1:0/251675948 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb998004690 con 0x7fb9a8104910 2024-04-26T21:53:14.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9a6000640 1 -- 172.21.15.1:0/251675948 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998046480 con 0x7fb9a8104910 2024-04-26T21:53:14.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9ade00640 1 -- 172.21.15.1:0/251675948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a81a0410 con 0x7fb9a8104910 2024-04-26T21:53:14.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.940+0000 7fb9ade00640 1 -- 172.21.15.1:0/251675948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a81a7d00 con 0x7fb9a8104910 2024-04-26T21:53:14.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.942+0000 7fb9ade00640 1 -- 172.21.15.1:0/251675948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a80506c0 con 0x7fb9a8104910 2024-04-26T21:53:14.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.944+0000 7fb9a6000640 1 -- 172.21.15.1:0/251675948 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7fb998030630 con 0x7fb9a8104910 2024-04-26T21:53:14.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.945+0000 7fb9a6000640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb98c0a3da0 0x7fb98c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:14.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.945+0000 7fb9a6000640 1 -- 172.21.15.1:0/251675948 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 9317+0+0 (secure 0 0 0) 0x7fb99802e030 con 0x7fb9a8104910 2024-04-26T21:53:14.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.945+0000 7fb99fe00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb98c0a3da0 0x7fb98c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:14.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.946+0000 7fb99fe00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb98c0a3da0 0x7fb98c0a6260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8102b20 tx=0x7fb990005f90 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:14.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:14.949+0000 7fb9a6000640 1 -- 172.21.15.1:0/251675948 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7fb99804b060 con 0x7fb9a8104910 2024-04-26T21:53:15.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.150+0000 7fb9ade00640 1 -- 172.21.15.1:0/251675948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb9a810cee0 con 0x7fb9a8104910 2024-04-26T21:53:15.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.668+0000 7fb9a6000640 1 -- 172.21.15.1:0/251675948 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v138) ==== 169+0+0 (secure 0 0 0) 0x7fb9980be040 con 0x7fb9a8104910 2024-04-26T21:53:15.669 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-04-26T21:53:15.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.673+0000 7fb9ade00640 1 -- 172.21.15.1:0/251675948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb98c0a3da0 msgr2=0x7fb98c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:15.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.673+0000 7fb9ade00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb98c0a3da0 0x7fb98c0a6260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8102b20 tx=0x7fb990005f90 comp rx=0 tx=0).stop 2024-04-26T21:53:15.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.673+0000 7fb9ade00640 1 -- 172.21.15.1:0/251675948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a8104910 msgr2=0x7fb9a819b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:15.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.673+0000 7fb9ade00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a8104910 0x7fb9a819b190 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb998005e30 tx=0x7fb9980039f0 comp rx=0 tx=0).stop 2024-04-26T21:53:15.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.674+0000 7fb9ade00640 1 -- 172.21.15.1:0/251675948 shutdown_connections 2024-04-26T21:53:15.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.674+0000 7fb9ade00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7fb98c0a3da0 0x7fb98c0a6260 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:15.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.674+0000 7fb9ade00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a810a920 0x7fb9a819fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:15.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.674+0000 7fb9ade00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9a81052d0 0x7fb9a819b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:15.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.674+0000 7fb9ade00640 1 --2- 172.21.15.1:0/251675948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9a8104910 0x7fb9a819b190 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:15.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.674+0000 7fb9ade00640 1 -- 172.21.15.1:0/251675948 >> 172.21.15.1:0/251675948 conn(0x7fb9a8100400 msgr2=0x7fb9a8100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:15.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.674+0000 7fb9ade00640 1 -- 172.21.15.1:0/251675948 shutdown_connections 2024-04-26T21:53:15.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.675+0000 7fb9ade00640 1 -- 172.21.15.1:0/251675948 wait complete. 2024-04-26T21:53:15.708 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-26T21:53:15.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.927+0000 7f9a32600640 1 -- 172.21.15.1:0/1136951921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a2c10ab60 msgr2=0x7f9a2c10afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:15.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.927+0000 7f9a32600640 1 --2- 172.21.15.1:0/1136951921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a2c10ab60 0x7f9a2c10afc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f9a18009900 tx=0x7f9a1802a400 comp rx=0 tx=0).stop 2024-04-26T21:53:15.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.927+0000 7f9a32600640 1 -- 172.21.15.1:0/1136951921 shutdown_connections 2024-04-26T21:53:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.927+0000 7f9a32600640 1 --2- 172.21.15.1:0/1136951921 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c10b500 0x7f9a2c115f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.927+0000 7f9a32600640 1 --2- 172.21.15.1:0/1136951921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a2c10ab60 0x7f9a2c10afc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.927+0000 7f9a32600640 1 --2- 172.21.15.1:0/1136951921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a2c1092d0 0x7f9a2c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.927+0000 7f9a32600640 1 -- 172.21.15.1:0/1136951921 >> 172.21.15.1:0/1136951921 conn(0x7f9a2c078720 msgr2=0x7f9a2c107750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.927+0000 7f9a32600640 1 -- 172.21.15.1:0/1136951921 shutdown_connections 2024-04-26T21:53:15.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.927+0000 7f9a32600640 1 -- 172.21.15.1:0/1136951921 wait complete. 2024-04-26T21:53:15.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.928+0000 7f9a32600640 1 Processor -- start 2024-04-26T21:53:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.928+0000 7f9a32600640 1 -- start start 2024-04-26T21:53:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a32600640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a2c1092d0 0x7f9a2c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a32600640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a2c10ab60 0x7f9a2c19fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a32600640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c10b500 0x7f9a2c1a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a32600640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2c1192f0 con 0x7f9a2c1092d0 2024-04-26T21:53:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a32600640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2c119170 con 0x7f9a2c10ab60 2024-04-26T21:53:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a32600640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2c119470 con 0x7f9a2c10b500 2024-04-26T21:53:15.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a2b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a2c1092d0 0x7f9a2c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a2aa00640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a2c10ab60 0x7f9a2c19fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a2b400640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a2c1092d0 0x7f9a2c19f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41880/0 (socket says 172.21.15.1:41880) 2024-04-26T21:53:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a2b400640 1 -- 172.21.15.1:0/346151294 learned_addr learned my addr 172.21.15.1:0/346151294 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-04-26T21:53:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a2be00640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c10b500 0x7f9a2c1a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.929+0000 7f9a2b400640 1 -- 172.21.15.1:0/346151294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c10b500 msgr2=0x7f9a2c1a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a2b400640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c10b500 0x7f9a2c1a3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:15.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a2b400640 1 -- 172.21.15.1:0/346151294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a2c10ab60 msgr2=0x7f9a2c19fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:15.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a2b400640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a2c10ab60 0x7f9a2c19fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:15.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a2b400640 1 -- 172.21.15.1:0/346151294 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2c1a4710 con 0x7f9a2c1092d0 2024-04-26T21:53:15.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a2aa00640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a2c10ab60 0x7f9a2c19fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T21:53:15.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a2b400640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a2c1092d0 0x7f9a2c19f770 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00e9f0 tx=0x7f9a1c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:15.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a23e00640 1 -- 172.21.15.1:0/346151294 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c00cd80 con 0x7f9a2c1092d0 2024-04-26T21:53:15.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a23e00640 1 -- 172.21.15.1:0/346151294 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a1c0108e0 con 0x7f9a2c1092d0 2024-04-26T21:53:15.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a32600640 1 -- 172.21.15.1:0/346151294 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2c1a4a00 con 0x7f9a2c1092d0 2024-04-26T21:53:15.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a23e00640 1 -- 172.21.15.1:0/346151294 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c018980 con 0x7f9a2c1092d0 2024-04-26T21:53:15.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.930+0000 7f9a32600640 1 -- 172.21.15.1:0/346151294 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a2c1ac290 con 0x7f9a2c1092d0 2024-04-26T21:53:15.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.931+0000 7f9a32600640 1 -- 172.21.15.1:0/346151294 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a2c069c30 con 0x7f9a2c1092d0 2024-04-26T21:53:15.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.934+0000 7f9a23e00640 1 -- 172.21.15.1:0/346151294 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 26) ==== 138776+0+0 (secure 0 0 0) 0x7f9a1c010430 con 0x7f9a2c1092d0 2024-04-26T21:53:15.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.935+0000 7f9a23e00640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a100a3d20 0x7f9a100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T21:53:15.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.935+0000 7f9a23e00640 1 -- 172.21.15.1:0/346151294 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 8713+0+0 (secure 0 0 0) 0x7f9a1c0d10b0 con 0x7f9a2c1092d0 2024-04-26T21:53:15.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.935+0000 7f9a2aa00640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a100a3d20 0x7f9a100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T21:53:15.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.936+0000 7f9a2aa00640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a100a3d20 0x7f9a100a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9a18000c00 tx=0x7f9a18005e60 comp rx=0 tx=0).ready entity=mgr.15291 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T21:53:15.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:15.937+0000 7f9a23e00640 1 -- 172.21.15.1:0/346151294 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188494 (secure 0 0 0) 0x7f9a1c09cfe0 con 0x7f9a2c1092d0 2024-04-26T21:53:15.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:15 smithi186 ceph-mon[32991]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:53:15.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:15 smithi186 ceph-mon[32991]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T21:53:15.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:15 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/251675948' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:15.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:15 smithi186 ceph-mon[32991]: pgmap v732: 97 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 88 active+clean; 319 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:53:16.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:15 smithi001 ceph-mon[28054]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:53:16.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:15 smithi001 ceph-mon[28054]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T21:53:16.067 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:15 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/251675948' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:16.068 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:15 smithi001 ceph-mon[28054]: pgmap v732: 97 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 88 active+clean; 319 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:53:16.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:15 smithi038 ceph-mon[32724]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:53:16.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:15 smithi038 ceph-mon[32724]: osdmap e137: 12 total, 12 up, 12 in 2024-04-26T21:53:16.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:15 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/251675948' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:16.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:15 smithi038 ceph-mon[32724]: pgmap v732: 97 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 88 active+clean; 319 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:53:16.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.169+0000 7f9a32600640 1 -- 172.21.15.1:0/346151294 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9a2c111550 con 0x7f9a2c1092d0 2024-04-26T21:53:16.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.671+0000 7f9a23e00640 1 -- 172.21.15.1:0/346151294 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v139) ==== 178+0+0 (secure 0 0 0) 0x7f9a1c0a12e0 con 0x7f9a2c1092d0 2024-04-26T21:53:16.672 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data_ec' removed 2024-04-26T21:53:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.675+0000 7f9a32600640 1 -- 172.21.15.1:0/346151294 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a100a3d20 msgr2=0x7f9a100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.675+0000 7f9a32600640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a100a3d20 0x7f9a100a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9a18000c00 tx=0x7f9a18005e60 comp rx=0 tx=0).stop 2024-04-26T21:53:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.675+0000 7f9a32600640 1 -- 172.21.15.1:0/346151294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a2c1092d0 msgr2=0x7f9a2c19f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T21:53:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.675+0000 7f9a32600640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a2c1092d0 0x7f9a2c19f770 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00e9f0 tx=0x7f9a1c00eeb0 comp rx=0 tx=0).stop 2024-04-26T21:53:16.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.676+0000 7f9a32600640 1 -- 172.21.15.1:0/346151294 shutdown_connections 2024-04-26T21:53:16.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.676+0000 7f9a32600640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.1:6800/3236768718,v1:172.21.15.1:6801/3236768718] conn(0x7f9a100a3d20 0x7f9a100a61e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:16.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.676+0000 7f9a32600640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c10b500 0x7f9a2c1a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:16.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.676+0000 7f9a32600640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a2c10ab60 0x7f9a2c19fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:16.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.676+0000 7f9a32600640 1 --2- 172.21.15.1:0/346151294 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a2c1092d0 0x7f9a2c19f770 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T21:53:16.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.676+0000 7f9a32600640 1 -- 172.21.15.1:0/346151294 >> 172.21.15.1:0/346151294 conn(0x7f9a2c078720 msgr2=0x7f9a2c078ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T21:53:16.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.679+0000 7f9a32600640 1 -- 172.21.15.1:0/346151294 shutdown_connections 2024-04-26T21:53:16.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-04-26T21:53:16.679+0000 7f9a32600640 1 -- 172.21.15.1:0/346151294 wait complete. 2024-04-26T21:53:16.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:53:16.802 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:53:16.836 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:53:16.864 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T21:53:16.864 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T21:53:16.910 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T21:53:16.927 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T21:53:16.942 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T21:53:16.942 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T21:53:16.942 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.a 2024-04-26T21:53:16.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:16 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/251675948' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:53:16.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:16 smithi186 ceph-mon[32991]: osdmap e138: 12 total, 12 up, 12 in 2024-04-26T21:53:16.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:16 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/346151294' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:17.015 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:16 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/251675948' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:53:17.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:16 smithi001 ceph-mon[28054]: osdmap e138: 12 total, 12 up, 12 in 2024-04-26T21:53:17.016 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:16 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/346151294' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:16 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/251675948' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:53:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:16 smithi038 ceph-mon[32724]: osdmap e138: 12 total, 12 up, 12 in 2024-04-26T21:53:17.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:16 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/346151294' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T21:53:17.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:17 smithi001 systemd[1]: Stopping Ceph mon.a for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:53:17.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:17 smithi186 ceph-mon[32991]: from='client.? 172.21.15.1:0/346151294' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:53:17.954 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:17 smithi186 ceph-mon[32991]: osdmap e139: 12 total, 12 up, 12 in 2024-04-26T21:53:17.955 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:17 smithi186 ceph-mon[32991]: pgmap v735: 1 pgs: 1 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:53:17.971 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:17 smithi001 ceph-mon[28054]: from='client.? 172.21.15.1:0/346151294' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:53:17.971 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:17 smithi001 ceph-mon[28054]: osdmap e139: 12 total, 12 up, 12 in 2024-04-26T21:53:17.971 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:17 smithi001 ceph-mon[28054]: pgmap v735: 1 pgs: 1 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:53:17.971 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a[28049]: 2024-04-26T21:53:17.838+0000 7faf47e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:53:17.972 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a[28049]: 2024-04-26T21:53:17.838+0000 7faf47e00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-26T21:53:17.972 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:17 smithi001 podman[92493]: 2024-04-26 21:53:17.966009225 +0000 UTC m=+0.553978862 container died acb0745845a29cd3647612d262c5a4fcb6636ebbd587c642c0996a67e66de0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-26T21:53:18.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:17 smithi038 ceph-mon[32724]: from='client.? 172.21.15.1:0/346151294' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T21:53:18.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:17 smithi038 ceph-mon[32724]: osdmap e139: 12 total, 12 up, 12 in 2024-04-26T21:53:18.077 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:17 smithi038 ceph-mon[32724]: pgmap v735: 1 pgs: 1 active+clean; 577 KiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-26T21:53:19.012 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:18 smithi001 podman[92493]: 2024-04-26 21:53:18.792863943 +0000 UTC m=+1.380833571 container cleanup acb0745845a29cd3647612d262c5a4fcb6636ebbd587c642c0996a67e66de0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-26T21:53:19.012 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:18 smithi001 bash[92493]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a 2024-04-26T21:53:19.317 INFO:journalctl@ceph.mon.a.smithi001.stdout:Apr 26 21:53:19 smithi001 podman[92505]: 2024-04-26 21:53:19.011443555 +0000 UTC m=+1.025446928 container remove acb0745845a29cd3647612d262c5a4fcb6636ebbd587c642c0996a67e66de0e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T21:53:19.561 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T21:53:19.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:53:19.708 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T21:53:19.709 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-26T21:53:19.709 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.b 2024-04-26T21:53:20.076 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:19 smithi038 systemd[1]: Stopping Ceph mon.b for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:53:20.545 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:20 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-b[32720]: 2024-04-26T21:53:20.110+0000 7fdeb6432700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:53:20.545 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:20 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-b[32720]: 2024-04-26T21:53:20.110+0000 7fdeb6432700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-26T21:53:20.545 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:20 smithi038 podman[71786]: 2024-04-26 21:53:20.243500032 +0000 UTC m=+0.256398035 container died c3b8a2fd1e24a8558a20f53fa749c21791a6c2076aecdf927cfede53ebda5592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-26T21:53:20.826 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:20 smithi038 podman[71786]: 2024-04-26 21:53:20.644546979 +0000 UTC m=+0.657444985 container cleanup c3b8a2fd1e24a8558a20f53fa749c21791a6c2076aecdf927cfede53ebda5592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-26T21:53:20.827 INFO:journalctl@ceph.mon.b.smithi038.stdout:Apr 26 21:53:20 smithi038 bash[71786]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-b 2024-04-26T21:53:21.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.b.service' 2024-04-26T21:53:21.288 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:53:21.288 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-26T21:53:21.288 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-26T21:53:21.289 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.c 2024-04-26T21:53:21.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:21 smithi186 systemd[1]: Stopping Ceph mon.c for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:53:22.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-c[32974]: 2024-04-26T21:53:21.981+0000 7f79a779f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:53:22.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-c[32974]: 2024-04-26T21:53:21.981+0000 7f79a779f700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-26T21:53:22.441 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:22 smithi186 podman[73906]: 2024-04-26 21:53:22.12467441 +0000 UTC m=+0.638856328 container died 2aa078f643b1ee5e91856a05810ce0e9817f9738a231231e2792e858ba4b30d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:53:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:22 smithi186 podman[73906]: 2024-04-26 21:53:22.540693076 +0000 UTC m=+1.054874994 container cleanup 2aa078f643b1ee5e91856a05810ce0e9817f9738a231231e2792e858ba4b30d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0) 2024-04-26T21:53:22.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 26 21:53:22 smithi186 bash[73906]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mon-c 2024-04-26T21:53:23.032 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mon.c.service' 2024-04-26T21:53:23.129 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:53:23.129 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-26T21:53:23.129 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-26T21:53:23.129 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.x 2024-04-26T21:53:23.567 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:53:23 smithi001 systemd[1]: Stopping Ceph mgr.x for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:53:24.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:53:23 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:53:23.804+0000 7fdb75a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:53:24.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:53:23 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x[28291]: 2024-04-26T21:53:23.804+0000 7fdb75a00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T21:53:24.068 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:53:23 smithi001 podman[92609]: 2024-04-26 21:53:23.91907991 +0000 UTC m=+0.297523742 container died 7e9c0a8e9d3d3b56a801776c3e57b4686c3d283dc3d59d8eb07ff308578d49f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:53:24.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:53:24 smithi001 podman[92609]: 2024-04-26 21:53:24.549436408 +0000 UTC m=+0.927880265 container cleanup 7e9c0a8e9d3d3b56a801776c3e57b4686c3d283dc3d59d8eb07ff308578d49f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True) 2024-04-26T21:53:24.818 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:53:24 smithi001 bash[92609]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x 2024-04-26T21:53:25.314 INFO:journalctl@ceph.mgr.x.smithi001.stdout:Apr 26 21:53:24 smithi001 podman[92623]: 2024-04-26 21:53:24.825710197 +0000 UTC m=+0.889104375 container remove 7e9c0a8e9d3d3b56a801776c3e57b4686c3d283dc3d59d8eb07ff308578d49f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bc612ffb9c1d5bc27536bc917a705262a36d4387, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:53:25.350 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-26T21:53:25.470 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:53:25.470 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-26T21:53:25.470 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-26T21:53:25.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.y 2024-04-26T21:53:25.576 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:53:25 smithi038 systemd[1]: Stopping Ceph mgr.y for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:53:26.134 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:53:25 smithi038 podman[71916]: 2024-04-26 21:53:25.831382613 +0000 UTC m=+0.155138630 container died 3a51d5f39ccf3a4d3c165b56e283539be6d1644ff13bc2156d0cb71bff48803d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-26T21:53:26.548 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:53:26 smithi038 podman[71916]: 2024-04-26 21:53:26.225434733 +0000 UTC m=+0.549190745 container cleanup 3a51d5f39ccf3a4d3c165b56e283539be6d1644ff13bc2156d0cb71bff48803d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T21:53:26.549 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Apr 26 21:53:26 smithi038 bash[71916]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-y 2024-04-26T21:53:26.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-26T21:53:26.763 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:53:26.763 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-26T21:53:26.763 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-26T21:53:26.763 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.z 2024-04-26T21:53:27.116 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:53:26 smithi186 systemd[1]: Stopping Ceph mgr.z for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:53:27.430 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:53:27 smithi186 podman[74027]: 2024-04-26 21:53:27.116368471 +0000 UTC m=+0.132146174 container died f6020480e4a4231b260effb0cc74e5681acbc7e2b9aca89c25aeec6cbf868a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-26T21:53:27.704 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:53:27 smithi186 podman[74027]: 2024-04-26 21:53:27.546254196 +0000 UTC m=+0.562031894 container cleanup f6020480e4a4231b260effb0cc74e5681acbc7e2b9aca89c25aeec6cbf868a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:53:27.704 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:53:27 smithi186 bash[74027]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z 2024-04-26T21:53:28.109 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-26T21:53:28.129 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:53:27 smithi186 podman[74041]: 2024-04-26 21:53:27.873103515 +0000 UTC m=+0.749731770 container remove f6020480e4a4231b260effb0cc74e5681acbc7e2b9aca89c25aeec6cbf868a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mgr-z, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T21:53:28.130 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:53:27 smithi186 systemd[1]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-04-26T21:53:28.130 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:53:28 smithi186 systemd[1]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.z.service: Failed with result 'exit-code'. 2024-04-26T21:53:28.130 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:53:28 smithi186 systemd[1]: Stopped Ceph mgr.z for 44ef3d6e-0412-11ef-bc93-c7b262605968. 2024-04-26T21:53:28.131 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 26 21:53:28 smithi186 systemd[1]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mgr.z.service: Consumed 23.413s CPU time. 2024-04-26T21:53:28.143 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:53:28.143 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-26T21:53:28.143 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T21:53:28.144 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.0 2024-04-26T21:53:28.567 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:28 smithi001 systemd[1]: Stopping Ceph osd.0 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:53:29.318 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:28 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0[40550]: 2024-04-26T21:53:28.871+0000 7f092e400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:53:29.318 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:28 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0[40550]: 2024-04-26T21:53:28.871+0000 7f092e400700 -1 osd.0 139 *** Got signal Terminated *** 2024-04-26T21:53:29.318 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:28 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0[40550]: 2024-04-26T21:53:28.871+0000 7f092e400700 -1 osd.0 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:53:34.818 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:34 smithi001 podman[92728]: 2024-04-26 21:53:34.465699037 +0000 UTC m=+5.823489289 container died 5e5a962be261a872ea06550e86a731a8f23f8aebab77781fa8048e29ec63f011 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:53:35.206 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:34 smithi001 podman[92728]: 2024-04-26 21:53:34.971314663 +0000 UTC m=+6.329104952 container cleanup 5e5a962be261a872ea06550e86a731a8f23f8aebab77781fa8048e29ec63f011 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:53:35.207 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:34 smithi001 bash[92728]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0 2024-04-26T21:53:35.567 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:35 smithi001 podman[92749]: 2024-04-26 21:53:35.205913649 +0000 UTC m=+0.722795246 container remove 5e5a962be261a872ea06550e86a731a8f23f8aebab77781fa8048e29ec63f011 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux ) 2024-04-26T21:53:36.473 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:36 smithi001 podman[92834]: 2024-04-26 21:53:36.004429145 +0000 UTC m=+0.033688081 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:53:36.473 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:36 smithi001 podman[92834]: 2024-04-26 21:53:36.148864783 +0000 UTC m=+0.178123756 container create c88fe7d367ceaba2087e10dc155b22d2a2ec7b4754033138172396005d30ab0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T21:53:36.817 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:36 smithi001 podman[92834]: 2024-04-26 21:53:36.472733138 +0000 UTC m=+0.501992118 container init c88fe7d367ceaba2087e10dc155b22d2a2ec7b4754033138172396005d30ab0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-26T21:53:36.818 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:36 smithi001 podman[92834]: 2024-04-26 21:53:36.484726854 +0000 UTC m=+0.513985817 container start c88fe7d367ceaba2087e10dc155b22d2a2ec7b4754033138172396005d30ab0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-04-26T21:53:36.818 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:36 smithi001 podman[92834]: 2024-04-26 21:53:36.585658721 +0000 UTC m=+0.614917748 container attach c88fe7d367ceaba2087e10dc155b22d2a2ec7b4754033138172396005d30ab0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-26T21:53:37.567 INFO:journalctl@ceph.osd.0.smithi001.stdout:Apr 26 21:53:37 smithi001 podman[92834]: 2024-04-26 21:53:37.287380361 +0000 UTC m=+1.316639374 container died c88fe7d367ceaba2087e10dc155b22d2a2ec7b4754033138172396005d30ab0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T21:53:38.070 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T21:53:38.194 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:53:38.194 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T21:53:38.194 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T21:53:38.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.1 2024-04-26T21:53:38.522 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:38 smithi038 systemd[1]: Stopping Ceph osd.1 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:53:38.827 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:38 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1[37337]: 2024-04-26T21:53:38.521+0000 7f6e1b5ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:53:38.827 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:38 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1[37337]: 2024-04-26T21:53:38.521+0000 7f6e1b5ab700 -1 osd.1 139 *** Got signal Terminated *** 2024-04-26T21:53:38.827 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:38 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1[37337]: 2024-04-26T21:53:38.521+0000 7f6e1b5ab700 -1 osd.1 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:53:44.327 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:44 smithi038 podman[72042]: 2024-04-26 21:53:44.067603942 +0000 UTC m=+5.666218760 container died cd0f4da29464ba67b8b162325ce6f2e27bbdbb5831168d0b402272937d633b67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-26T21:53:44.826 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:44 smithi038 podman[72042]: 2024-04-26 21:53:44.472420367 +0000 UTC m=+6.071035194 container cleanup cd0f4da29464ba67b8b162325ce6f2e27bbdbb5831168d0b402272937d633b67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS) 2024-04-26T21:53:44.827 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:44 smithi038 bash[72042]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1 2024-04-26T21:53:45.263 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:44 smithi038 podman[72060]: 2024-04-26 21:53:44.882702043 +0000 UTC m=+0.789446750 container remove cd0f4da29464ba67b8b162325ce6f2e27bbdbb5831168d0b402272937d633b67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-26T21:53:45.536 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:45 smithi038 podman[72146]: 2024-04-26 21:53:45.166470609 +0000 UTC m=+0.018385052 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:53:45.536 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:45 smithi038 podman[72146]: 2024-04-26 21:53:45.284064699 +0000 UTC m=+0.135979139 container create a53d1d82803b512909db8e2e87478e76230b06a8aba70145290132e806b84dc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-26T21:53:45.537 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:45 smithi038 podman[72146]: 2024-04-26 21:53:45.536891279 +0000 UTC m=+0.388805721 container init a53d1d82803b512909db8e2e87478e76230b06a8aba70145290132e806b84dc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:53:45.827 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:45 smithi038 podman[72146]: 2024-04-26 21:53:45.540366288 +0000 UTC m=+0.392280730 container start a53d1d82803b512909db8e2e87478e76230b06a8aba70145290132e806b84dc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-26T21:53:45.827 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:45 smithi038 podman[72146]: 2024-04-26 21:53:45.670345684 +0000 UTC m=+0.522260129 container attach a53d1d82803b512909db8e2e87478e76230b06a8aba70145290132e806b84dc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:53:46.326 INFO:journalctl@ceph.osd.1.smithi038.stdout:Apr 26 21:53:46 smithi038 podman[72146]: 2024-04-26 21:53:46.05305273 +0000 UTC m=+0.904967174 container died a53d1d82803b512909db8e2e87478e76230b06a8aba70145290132e806b84dc7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:53:46.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T21:53:46.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:53:46.826 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T21:53:46.826 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T21:53:46.826 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.2 2024-04-26T21:53:47.134 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:46 smithi186 systemd[1]: Stopping Ceph osd.2 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:53:47.454 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:47 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2[38145]: 2024-04-26T21:53:47.133+0000 7f0ee0ffb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:53:47.454 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:47 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2[38145]: 2024-04-26T21:53:47.133+0000 7f0ee0ffb700 -1 osd.2 139 *** Got signal Terminated *** 2024-04-26T21:53:47.455 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:47 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2[38145]: 2024-04-26T21:53:47.133+0000 7f0ee0ffb700 -1 osd.2 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:53:52.954 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:52 smithi186 podman[74150]: 2024-04-26 21:53:52.661861341 +0000 UTC m=+5.648869357 container died 0abb392a08c60735661b35e2989aaf65464bdf7b6078f95f595560ca0b77ab5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T21:53:53.346 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:53 smithi186 podman[74150]: 2024-04-26 21:53:53.101340225 +0000 UTC m=+6.088348239 container cleanup 0abb392a08c60735661b35e2989aaf65464bdf7b6078f95f595560ca0b77ab5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:53:53.346 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:53 smithi186 bash[74150]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2 2024-04-26T21:53:53.704 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:53 smithi186 podman[74179]: 2024-04-26 21:53:53.346045414 +0000 UTC m=+0.685875122 container remove 0abb392a08c60735661b35e2989aaf65464bdf7b6078f95f595560ca0b77ab5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-26T21:53:54.031 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:53 smithi186 podman[74272]: 2024-04-26 21:53:53.658795016 +0000 UTC m=+0.018026575 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:53:54.032 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:53 smithi186 podman[74272]: 2024-04-26 21:53:53.78879499 +0000 UTC m=+0.148026542 container create 13717087b3033e67719559d53b556227825ee583b77764bf579aa699026281e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:53:54.032 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:54 smithi186 podman[74272]: 2024-04-26 21:53:54.032131927 +0000 UTC m=+0.391363482 container init 13717087b3033e67719559d53b556227825ee583b77764bf579aa699026281e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T21:53:54.338 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:54 smithi186 podman[74272]: 2024-04-26 21:53:54.035601578 +0000 UTC m=+0.394833134 container start 13717087b3033e67719559d53b556227825ee583b77764bf579aa699026281e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T21:53:54.338 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:54 smithi186 podman[74272]: 2024-04-26 21:53:54.13213637 +0000 UTC m=+0.491367938 container attach 13717087b3033e67719559d53b556227825ee583b77764bf579aa699026281e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-26T21:53:54.694 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 26 21:53:54 smithi186 podman[74272]: 2024-04-26 21:53:54.338080522 +0000 UTC m=+0.697312085 container died 13717087b3033e67719559d53b556227825ee583b77764bf579aa699026281e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-26T21:53:55.014 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T21:53:55.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:53:55.052 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T21:53:55.053 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-26T21:53:55.053 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.3 2024-04-26T21:53:55.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:53:55 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:53:55.038+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:53:55.567 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:53:55 smithi001 systemd[1]: Stopping Ceph osd.3 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:53:55.923 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:53:55 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:53:55.699+0000 7f2529c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:53:55.924 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:53:55 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:53:55.699+0000 7f2529c00700 -1 osd.3 139 *** Got signal Terminated *** 2024-04-26T21:53:55.924 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:53:55 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:53:55.699+0000 7f2529c00700 -1 osd.3 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:53:55.924 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:53:55 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:53:55.702+0000 7f2520400700 -1 osd.3 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.667273+0000 front 2024-04-26T21:53:31.667845+0000 (oldest deadline 2024-04-26T21:53:55.167627+0000) 2024-04-26T21:53:56.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:53:56 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:53:56.051+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:53:57.068 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:53:56 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:53:56.741+0000 7f2520400700 -1 osd.3 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.667273+0000 front 2024-04-26T21:53:31.667845+0000 (oldest deadline 2024-04-26T21:53:55.167627+0000) 2024-04-26T21:53:57.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:53:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:53:57.031+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:53:57.791 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:53:57 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:53:57.395+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:53:57.792 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:53:57 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:53:57.432+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:53:58.067 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:53:57 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:53:57.790+0000 7f2520400700 -1 osd.3 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.667273+0000 front 2024-04-26T21:53:31.667845+0000 (oldest deadline 2024-04-26T21:53:55.167627+0000) 2024-04-26T21:53:58.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:53:58 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:53:58.004+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:53:58.766 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:53:58 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:53:58.429+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:53:58.767 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:53:58 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:53:58.434+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:53:59.067 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:53:58 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:53:58.765+0000 7f2520400700 -1 osd.3 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.667273+0000 front 2024-04-26T21:53:31.667845+0000 (oldest deadline 2024-04-26T21:53:55.167627+0000) 2024-04-26T21:53:59.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:53:58 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:53:58.988+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:53:59.718 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:53:59 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:53:59.474+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:53:59.719 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:53:59 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:53:59.432+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:00.067 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:53:59 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:53:59.718+0000 7f2520400700 -1 osd.3 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.667273+0000 front 2024-04-26T21:53:31.667845+0000 (oldest deadline 2024-04-26T21:53:55.167627+0000) 2024-04-26T21:54:00.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:00 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:00.002+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:00.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:00 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:00.133+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:00.817 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:00.492+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:00.818 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3[47039]: 2024-04-26T21:54:00.670+0000 7f2520400700 -1 osd.3 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.667273+0000 front 2024-04-26T21:53:31.667845+0000 (oldest deadline 2024-04-26T21:53:55.167627+0000) 2024-04-26T21:54:00.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:00 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:00.480+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:01.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:01 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:01.037+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:01.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:01 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:01.132+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:01.693 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:01 smithi001 podman[92938]: 2024-04-26 21:54:01.340825968 +0000 UTC m=+5.814558271 container died d01e8d00b130b33014b08a4bbb3f5c398475fa01017cd63719468ab3475ed725 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T21:54:01.693 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:01 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:01.512+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:01.694 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:01 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:01.433+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:02.045 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:01 smithi001 podman[92938]: 2024-04-26 21:54:01.834508739 +0000 UTC m=+6.308241039 container cleanup d01e8d00b130b33014b08a4bbb3f5c398475fa01017cd63719468ab3475ed725 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T21:54:02.045 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:01 smithi001 bash[92938]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3 2024-04-26T21:54:02.317 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:02 smithi001 podman[92967]: 2024-04-26 21:54:02.044224448 +0000 UTC m=+0.678519996 container remove d01e8d00b130b33014b08a4bbb3f5c398475fa01017cd63719468ab3475ed725 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-26T21:54:02.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:02 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:02.007+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:02.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:02 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:02.153+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:02.817 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:02 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:02.521+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:02.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:02 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:02.383+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:03.137 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:02 smithi001 podman[93054]: 2024-04-26 21:54:02.792702183 +0000 UTC m=+0.037003171 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:54:03.138 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:02 smithi001 podman[93054]: 2024-04-26 21:54:02.928645064 +0000 UTC m=+0.172946078 container create bc3d5ad04cf99beeba88c65e417e7946ace25bd0faaf9dba44667d7680899b5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:54:03.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:03 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:03.006+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:03.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:03 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:03.203+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:03.567 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:03 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:03.527+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:03.568 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:03 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:03.357+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:03.568 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:03 smithi001 podman[93054]: 2024-04-26 21:54:03.282915523 +0000 UTC m=+0.527216543 container init bc3d5ad04cf99beeba88c65e417e7946ace25bd0faaf9dba44667d7680899b5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-26T21:54:03.568 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:03 smithi001 podman[93054]: 2024-04-26 21:54:03.295000501 +0000 UTC m=+0.539301518 container start bc3d5ad04cf99beeba88c65e417e7946ace25bd0faaf9dba44667d7680899b5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T21:54:03.568 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:03 smithi001 podman[93054]: 2024-04-26 21:54:03.390197258 +0000 UTC m=+0.634498329 container attach bc3d5ad04cf99beeba88c65e417e7946ace25bd0faaf9dba44667d7680899b5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-26T21:54:04.068 INFO:journalctl@ceph.osd.3.smithi001.stdout:Apr 26 21:54:03 smithi001 podman[93054]: 2024-04-26 21:54:03.762093544 +0000 UTC m=+1.006394602 container died bc3d5ad04cf99beeba88c65e417e7946ace25bd0faaf9dba44667d7680899b5a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-04-26T21:54:04.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:03 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:03.959+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:04.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:04 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:04.168+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:04.567 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:04 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:04.391+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:04.725 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.3.service' 2024-04-26T21:54:04.841 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:04 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:04.574+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:04.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:54:04.845 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-26T21:54:04.846 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-26T21:54:04.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.4 2024-04-26T21:54:05.153 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:04 smithi038 systemd[1]: Stopping Ceph osd.4 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:54:05.154 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:04 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:04.919+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:05.154 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:05 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:05.037+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:05.404 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:05 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4[43989]: 2024-04-26T21:54:05.153+0000 7f67e4ef4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:54:05.405 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:05 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4[43989]: 2024-04-26T21:54:05.153+0000 7f67e4ef4700 -1 osd.4 139 *** Got signal Terminated *** 2024-04-26T21:54:05.405 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:05 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4[43989]: 2024-04-26T21:54:05.153+0000 7f67e4ef4700 -1 osd.4 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:54:05.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:05.178+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:05.817 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:05 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:05.541+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:05.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:05 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:05.412+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:06.076 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:05 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4[43989]: 2024-04-26T21:54:05.812+0000 7f67e14e8700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:39.520705+0000 front 2024-04-26T21:53:39.520736+0000 (oldest deadline 2024-04-26T21:54:04.820806+0000) 2024-04-26T21:54:06.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:06 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:06.003+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:06.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:05 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:05.937+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:06.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:05 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:05.937+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:06.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:06 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:06.213+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:06.817 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:06 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:06.577+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:06.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:06 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:06.424+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:07.076 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:06 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:06.997+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:07.077 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:06 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4[43989]: 2024-04-26T21:54:06.784+0000 7f67e14e8700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:39.520705+0000 front 2024-04-26T21:53:39.520736+0000 (oldest deadline 2024-04-26T21:54:04.820806+0000) 2024-04-26T21:54:07.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:06 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:06.906+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:07.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:06 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:06.906+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:07.204 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:06 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:06.800+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:07.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:07.231+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:07.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:07.231+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:07.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:07 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:07.464+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:07.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:07 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:07.568+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:07.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:07 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:07.568+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:08.076 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:07 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4[43989]: 2024-04-26T21:54:07.746+0000 7f67e14e8700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:39.520705+0000 front 2024-04-26T21:53:39.520736+0000 (oldest deadline 2024-04-26T21:54:04.820806+0000) 2024-04-26T21:54:08.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:07 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:07.987+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:08.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:07 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:07.956+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:08.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:07 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:07.956+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:08.199 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:07.829+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:08.199 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:07.772+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:08.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:08.198+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:08.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:08.198+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:08.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:08 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:08.424+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:08.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:08 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:08.616+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:08.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:08 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:08.616+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:09.077 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:08 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4[43989]: 2024-04-26T21:54:08.770+0000 7f67e14e8700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:39.520705+0000 front 2024-04-26T21:53:39.520736+0000 (oldest deadline 2024-04-26T21:54:04.820806+0000) 2024-04-26T21:54:09.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:08 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:08.942+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:09.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:08 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:08.984+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:09.078 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:08 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:08.984+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:09.204 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:08.813+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:09.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:08.729+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:09.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:09.237+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:09.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:09.237+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:09.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:09.427+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:09.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:09.427+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:09.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:09.589+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:09.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:09 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:09.589+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:10.077 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4[43989]: 2024-04-26T21:54:09.735+0000 7f67e14e8700 -1 osd.4 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:39.520705+0000 front 2024-04-26T21:53:39.520736+0000 (oldest deadline 2024-04-26T21:54:04.820806+0000) 2024-04-26T21:54:10.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:09.931+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:10.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:09.955+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:10.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:09 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:09.955+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:10.204 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:09.851+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:10.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:09.777+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:10.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:10.239+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:10.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:10.239+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:10.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:10 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:10.421+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:10.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:10 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:10.421+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:10.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:10 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:10.633+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:10.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:10 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:10.633+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:10.918 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:10 smithi038 podman[72263]: 2024-04-26 21:54:10.606286849 +0000 UTC m=+5.561436091 container died 5019cd953c5cd198593a6f06bf2e616fb239b5676b15a9d05bb124bfca47cb2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:54:11.204 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:10.852+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:11.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:10.795+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:11.263 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:10 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:10.958+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:11.263 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:10 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:10.918+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:11.263 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:10 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:10.918+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:11.263 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:11 smithi038 podman[72263]: 2024-04-26 21:54:11.028548259 +0000 UTC m=+5.983697500 container cleanup 5019cd953c5cd198593a6f06bf2e616fb239b5676b15a9d05bb124bfca47cb2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-26T21:54:11.264 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:11 smithi038 bash[72263]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4 2024-04-26T21:54:11.576 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:11 smithi038 podman[72280]: 2024-04-26 21:54:11.263287737 +0000 UTC m=+0.652171967 container remove 5019cd953c5cd198593a6f06bf2e616fb239b5676b15a9d05bb124bfca47cb2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD) 2024-04-26T21:54:11.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:11.212+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:11.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:11.212+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:11.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:11 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:11.457+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:11.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:11 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:11.457+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:11.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:11 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:11.652+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:11.819 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:11 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:11.652+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:11.947 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:11 smithi038 podman[72368]: 2024-04-26 21:54:11.598462003 +0000 UTC m=+0.018304344 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:54:11.948 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:11 smithi038 podman[72368]: 2024-04-26 21:54:11.73111038 +0000 UTC m=+0.150952763 container create 0ee0f65bc23e7fedea0a36851f336fece53c0f1ea9f62498b45a2c2a0ebb8c87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-26T21:54:12.163 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:11.843+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:12.163 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:11 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:11.878+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:12.301 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:11 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:11.991+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:12.302 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:11 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:11.947+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:12.302 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:11 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:11.947+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:12.302 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:11 smithi038 podman[72368]: 2024-04-26 21:54:11.992253659 +0000 UTC m=+0.412096013 container init 0ee0f65bc23e7fedea0a36851f336fece53c0f1ea9f62498b45a2c2a0ebb8c87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:54:12.302 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:11 smithi038 podman[72368]: 2024-04-26 21:54:11.995503201 +0000 UTC m=+0.415345549 container start 0ee0f65bc23e7fedea0a36851f336fece53c0f1ea9f62498b45a2c2a0ebb8c87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:54:12.302 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:12 smithi038 podman[72368]: 2024-04-26 21:54:12.092049087 +0000 UTC m=+0.511891427 container attach 0ee0f65bc23e7fedea0a36851f336fece53c0f1ea9f62498b45a2c2a0ebb8c87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:54:12.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:12.162+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:12.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:12.162+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:12.576 INFO:journalctl@ceph.osd.4.smithi038.stdout:Apr 26 21:54:12 smithi038 podman[72368]: 2024-04-26 21:54:12.301689175 +0000 UTC m=+0.721531519 container died 0ee0f65bc23e7fedea0a36851f336fece53c0f1ea9f62498b45a2c2a0ebb8c87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:54:12.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:12.495+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:12.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:12.495+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:12.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:12.682+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:12.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:12 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:12.682+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:13.068 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.4.service' 2024-04-26T21:54:13.114 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:54:13.114 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-26T21:54:13.115 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-26T21:54:13.115 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.5 2024-04-26T21:54:13.142 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:12.925+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:13.143 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:12 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:12.856+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:13.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:12.956+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:13.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:12.956+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:13.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:12.946+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:13.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:12 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:12.946+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:13.423 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:13 smithi186 systemd[1]: Stopping Ceph osd.5 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:54:13.423 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:13.197+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:13.423 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:13.197+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:13.704 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:13.422+0000 7f6e81f67700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:54:13.704 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:13.422+0000 7f6e81f67700 -1 osd.5 139 *** Got signal Terminated *** 2024-04-26T21:54:13.704 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:13.422+0000 7f6e81f67700 -1 osd.5 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:54:13.817 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:13.711+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:13.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:13.711+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:13.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:13.711+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:13.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:13.530+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:13.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:13.530+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:14.161 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:13.886+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:14.162 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:13 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:13.878+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:14.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:13 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:13.923+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:14.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:13 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:13.923+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:14.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:13 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:13.996+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:14.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:13 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:13.996+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:14.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:14.161+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:14.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:14.161+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:14.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:14.161+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:15.067 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:14.575+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:15.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:14.575+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:15.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:14.669+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:15.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:14.669+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:15.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:14 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:14.669+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:15.131 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:14.879+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:15.132 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:14 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:14.926+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:15.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:14 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:14.877+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:15.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:14 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:14.877+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:15.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:14 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:14.966+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:15.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:14 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:14.966+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:15.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:15 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:15.131+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:15.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:15 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:15.131+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:15.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:15 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:15.131+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:16.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:15 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:15.664+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:16.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:15 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:15.664+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:16.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:15 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:15.664+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:16.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:15 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:15.609+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:16.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:15 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:15.609+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:16.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:15 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:15.609+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:16.142 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:15 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:15.871+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:16.142 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:15 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:15.927+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:16.143 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:15 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:15.927+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:16.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:15 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:15.893+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:16.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:15 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:15.893+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:16.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:16 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:16.003+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:16.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:16 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:16.003+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:16.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:16.141+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:16.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:16.141+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:16.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:16.141+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:16.817 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:16.684+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:16.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:16.684+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:16.818 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:16.684+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:16.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:16.564+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:16.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:16.564+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:16.819 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:16 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:16.564+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:17.181 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:16.876+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:17.182 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:16.876+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.611841+0000 front 2024-04-26T21:53:51.611862+0000 (oldest deadline 2024-04-26T21:54:16.312036+0000) 2024-04-26T21:54:17.182 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:16.898+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:17.182 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:16 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:16.898+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:17.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:16 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:16.930+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:17.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:16 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:16.930+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:17.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:16 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:16.966+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:17.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:16 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:16.966+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:17.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:16 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:16.966+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:17.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:17 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:17.180+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:17.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:17 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:17.181+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:17.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:17 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:17.181+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:18.067 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:17.685+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:18.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:17.685+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:18.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:17.685+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:18.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:17.606+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:18.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:17.606+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:18.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:17 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:17.606+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:18.133 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:17 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:17.918+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.709755+0000 front 2024-04-26T21:53:41.709727+0000 (oldest deadline 2024-04-26T21:54:06.409822+0000) 2024-04-26T21:54:18.133 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:17 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5[44960]: 2024-04-26T21:54:17.918+0000 7f6e7a553700 -1 osd.5 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.611841+0000 front 2024-04-26T21:53:51.611862+0000 (oldest deadline 2024-04-26T21:54:16.312036+0000) 2024-04-26T21:54:18.133 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:17 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:17.860+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:18.134 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:17 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:17.860+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:18.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:17 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:17.920+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:18.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:17 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:17.920+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:18.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:17 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:17.950+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:18.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:17 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:17.950+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:18.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:17 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:17.950+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:18.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:18 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:18.133+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:18.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:18 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:18.133+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:18.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:18 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:18.133+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:19.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:18.664+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:19.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:18.664+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:19.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:18.664+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:19.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:18.632+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:19.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:18.632+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:19.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:18 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:18.632+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:19.204 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:19 smithi186 podman[74376]: 2024-04-26 21:54:19.020102294 +0000 UTC m=+5.707645626 container died ffe90ca75bf4ff335bd98e06286ae8fa0804014da7a754fe2a9fbed614427450 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:54:19.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:18 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:18.897+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:19.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:18 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:18.897+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:19.205 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:19 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:19.104+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:19.205 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:19 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:19.104+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:19.205 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:19 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:19.104+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:19.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:18 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:18.896+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:19.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:18 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:18.896+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:19.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:18 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:18.994+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:19.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:18 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:18.994+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:19.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:18 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:18.994+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:19.704 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:19 smithi186 podman[74376]: 2024-04-26 21:54:19.506801916 +0000 UTC m=+6.194345245 container cleanup ffe90ca75bf4ff335bd98e06286ae8fa0804014da7a754fe2a9fbed614427450 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-26T21:54:19.704 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:19 smithi186 bash[74376]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5 2024-04-26T21:54:20.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:19 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:19.702+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:20.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:19 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:19.702+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:20.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:19 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:19.702+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:20.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:19 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:19.625+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:20.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:19 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:19.625+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:20.069 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:19 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:19.625+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:20.073 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:19 smithi186 podman[74394]: 2024-04-26 21:54:19.70812486 +0000 UTC m=+0.686413712 container remove ffe90ca75bf4ff335bd98e06286ae8fa0804014da7a754fe2a9fbed614427450 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:54:20.073 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:19 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:19.860+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:20.074 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:19 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:19.860+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:20.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:19 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:19.851+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:20.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:19 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:19.851+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:20.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:19 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:19.995+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:20.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:19 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:19.995+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:20.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:19 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:19.995+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:20.353 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:20 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:20.073+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:20.353 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:20 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:20.073+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:20.353 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:20 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:20.073+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:20.354 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:20 smithi186 podman[74480]: 2024-04-26 21:54:20.015526495 +0000 UTC m=+0.017954722 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:54:20.354 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:20 smithi186 podman[74480]: 2024-04-26 21:54:20.142623208 +0000 UTC m=+0.145051434 container create 92caf4f5e43a88b8839f5021ccbc6af25691790233673c3e72e414473b564eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T21:54:20.354 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:20 smithi186 podman[74480]: 2024-04-26 21:54:20.353592333 +0000 UTC m=+0.356020559 container init 92caf4f5e43a88b8839f5021ccbc6af25691790233673c3e72e414473b564eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-26T21:54:20.648 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:20 smithi186 podman[74480]: 2024-04-26 21:54:20.356920178 +0000 UTC m=+0.359348406 container start 92caf4f5e43a88b8839f5021ccbc6af25691790233673c3e72e414473b564eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-26T21:54:20.649 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:20 smithi186 podman[74480]: 2024-04-26 21:54:20.470364356 +0000 UTC m=+0.472792601 container attach 92caf4f5e43a88b8839f5021ccbc6af25691790233673c3e72e414473b564eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-26T21:54:20.954 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:20 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:20.831+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:20.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:20 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:20.831+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:20.955 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 26 21:54:20 smithi186 podman[74480]: 2024-04-26 21:54:20.649127809 +0000 UTC m=+0.651556045 container died 92caf4f5e43a88b8839f5021ccbc6af25691790233673c3e72e414473b564eb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T21:54:21.067 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:20 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:20.679+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:21.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:20 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:20.679+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:21.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:20 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:20.679+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:21.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:20 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:20.624+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:21.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:20 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:20.624+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:21.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:20 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:20.624+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:21.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:20 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:20.870+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:21.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:20 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:20.870+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:21.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:20 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:20.963+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:21.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:20 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:20.963+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:21.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:20 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:20.963+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:21.332 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:21.091+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:21.332 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:21.091+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:21.332 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:21.091+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:21.356 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.5.service' 2024-04-26T21:54:21.434 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:54:21.434 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-26T21:54:21.435 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T21:54:21.435 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.6 2024-04-26T21:54:21.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:21 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:21.593+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:21.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:21 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:21.593+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:21.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:21 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:21.593+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:21.819 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:21 smithi001 systemd[1]: Stopping Ceph osd.6 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:54:21.819 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:21 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:21.700+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:21.819 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:21 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:21.700+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:21.819 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:21 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:21.700+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:22.136 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:21.805+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:22.137 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:21 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:21.805+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:22.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:21.838+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:22.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:21.838+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:22.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:21.943+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:22.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:21.943+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:22.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:21 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:21.943+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:22.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:22 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:22.136+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:22.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:22 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:22.136+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:22.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:22 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:22.136+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:22.567 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:22 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:22.077+0000 7f2d88800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:54:22.568 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:22 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:22.077+0000 7f2d88800700 -1 osd.6 139 *** Got signal Terminated *** 2024-04-26T21:54:22.568 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:22 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:22.077+0000 7f2d88800700 -1 osd.6 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:54:23.067 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:22 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:22.622+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:23.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:22 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:22.622+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:23.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:22 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:22.622+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:23.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:22 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:22.690+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:23.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:22 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:22.690+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:23.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:22 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:22.690+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:23.158 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:22 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:22.840+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:23.158 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:22 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:22.840+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:23.158 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:22 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:22.840+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:23.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:22 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:22.861+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:23.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:22 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:22.861+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:23.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:22 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:22.922+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:23.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:22 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:22.922+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:23.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:22 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:22.922+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:23.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:22 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:22.922+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:23.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:23 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:23.158+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:23.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:23 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:23.158+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:23.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:23 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:23.158+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:24.067 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:23 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:23.736+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:24.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:23 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:23.736+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:24.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:23 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:23.736+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:24.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:23 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:23.606+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:24.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:23 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:23.606+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:24.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:23 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:23.606+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:24.195 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:23 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:23.813+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:24.195 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:23 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:23.813+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:24.195 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:23 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:23.813+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:24.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:23 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:23.861+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:24.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:23 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:23.861+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:24.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:23 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:23.947+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:24.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:23 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:23.947+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:24.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:23 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:23.947+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:24.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:23 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:23.947+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:24.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:24 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:24.194+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:24.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:24 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:24.194+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:24.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:24 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:24.194+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:25.067 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:24 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:24.765+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:25.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:24 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:24.765+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:25.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:24 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:24.765+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:25.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:24 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:24.591+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:25.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:24 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:24.591+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:25.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:24 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:24.591+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:25.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:24 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:24.847+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:25.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:24 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:24.847+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:25.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:24 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:24.847+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:25.228 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:24.852+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:25.228 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:24.852+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:25.228 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:24.915+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:25.229 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:24.915+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:25.229 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:24.915+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:25.229 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:24 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:24.915+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:25.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:25 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:25.207+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:25.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:25 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:25.207+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:25.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:25 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:25.207+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:25.970 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:25.734+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:25.970 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:25.734+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:25.970 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:25.734+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:25.970 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:25.734+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.026933+0000 front 2024-04-26T21:54:00.027036+0000 (oldest deadline 2024-04-26T21:54:25.327351+0000) 2024-04-26T21:54:25.971 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:25.616+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:25.971 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:25.616+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:25.971 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:25.616+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:25.971 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:25 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:25.616+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:26.076 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:25 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:25.824+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:26.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:25 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:25.824+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:26.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:25 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:25.882+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:26.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:25 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:25.882+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:26.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:25 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:25.882+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:26.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:25 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:25.882+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:26.191 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:25 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:25.821+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:26.192 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:25 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:25.821+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:26.192 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:25 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:25.821+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:26.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:26 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:26.191+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:26.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:26 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:26.191+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:26.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:26 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:26.191+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:27.000 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:26 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:26.599+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:27.000 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:26 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:26.599+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:27.001 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:26 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:26.599+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:27.001 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:26 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:26.599+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:27.001 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:26 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:26.712+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:33.605466+0000 front 2024-04-26T21:53:33.605324+0000 (oldest deadline 2024-04-26T21:53:57.105405+0000) 2024-04-26T21:54:27.001 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:26 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:26.712+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:43.216456+0000 front 2024-04-26T21:53:43.216562+0000 (oldest deadline 2024-04-26T21:54:07.218491+0000) 2024-04-26T21:54:27.001 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:26 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:26.712+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.521275+0000 front 2024-04-26T21:53:49.521355+0000 (oldest deadline 2024-04-26T21:54:13.021552+0000) 2024-04-26T21:54:27.001 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:26 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6[53562]: 2024-04-26T21:54:26.712+0000 7f2d84000700 -1 osd.6 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.026933+0000 front 2024-04-26T21:54:00.027036+0000 (oldest deadline 2024-04-26T21:54:25.327351+0000) 2024-04-26T21:54:27.079 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:26 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:26.792+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:27.079 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:26 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:26.792+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:27.080 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:26 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:26.792+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.438795+0000 front 2024-04-26T21:54:00.438749+0000 (oldest deadline 2024-04-26T21:54:26.338799+0000) 2024-04-26T21:54:27.080 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:26 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:26.911+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:27.080 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:26 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:26.911+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:27.081 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:26 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:26.911+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:27.081 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:26 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:26.911+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:27.202 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:26 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:26.849+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:27.202 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:26 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:26.849+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:27.202 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:26 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:26.849+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:27.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:27 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:27.201+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:27.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:27 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:27.201+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:27.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:27 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:27.201+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:28.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:27 smithi001 podman[93158]: 2024-04-26 21:54:27.791284875 +0000 UTC m=+5.878469408 container died 01567e331cc23b2b6a99c9773b1e4e871328d794d690183ece0865edd0a2e00d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:54:28.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:27 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:27.636+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:28.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:27 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:27.636+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:28.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:27 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:27.636+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:28.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:27 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:27.636+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:28.076 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:27 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:27.768+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:28.076 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:27 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:27.768+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:28.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:27 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:27.768+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.438795+0000 front 2024-04-26T21:54:00.438749+0000 (oldest deadline 2024-04-26T21:54:26.338799+0000) 2024-04-26T21:54:28.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:27 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:27.958+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:28.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:27 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:27.958+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:28.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:27 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:27.958+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:28.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:27 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:27.958+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:28.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:27 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:27.851+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:28.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:27 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:27.851+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:28.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:27 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:27.851+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:28.568 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:28 smithi001 podman[93158]: 2024-04-26 21:54:28.230671262 +0000 UTC m=+6.317855851 container cleanup 01567e331cc23b2b6a99c9773b1e4e871328d794d690183ece0865edd0a2e00d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-26T21:54:28.568 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:28 smithi001 bash[93158]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6 2024-04-26T21:54:28.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:28 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:28.244+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:28.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:28 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:28.244+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:28.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:28 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:28.244+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:29.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:28 smithi001 podman[93188]: 2024-04-26 21:54:28.593569553 +0000 UTC m=+0.780421429 container remove 01567e331cc23b2b6a99c9773b1e4e871328d794d690183ece0865edd0a2e00d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:54:29.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:28 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:28.625+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:29.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:28 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:28.625+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:29.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:28 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:28.625+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:29.069 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:28 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:28.625+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:29.076 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:28 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:28.911+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:29.076 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:28 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:28.911+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:29.076 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:28 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:28.911+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:29.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:28 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:28.911+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:29.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:28 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:28.802+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:29.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:28 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:28.802+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:29.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:28 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:28.802+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.438795+0000 front 2024-04-26T21:54:00.438749+0000 (oldest deadline 2024-04-26T21:54:26.338799+0000) 2024-04-26T21:54:29.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:28 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:28.879+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:29.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:28 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:28.879+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:29.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:28 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:28.879+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:29.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:29 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:29.230+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:29.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:29 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:29.230+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:29.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:29 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:29.230+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:29.754 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:29 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:29.580+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:29.754 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:29 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:29.580+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:29.754 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:29 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:29.580+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:29.754 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:29 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:29.580+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:29.754 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:29 smithi001 podman[93277]: 2024-04-26 21:54:29.340806082 +0000 UTC m=+0.037346715 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:54:29.755 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:29 smithi001 podman[93277]: 2024-04-26 21:54:29.491116624 +0000 UTC m=+0.187657297 container create bb78677bc92141e7473e666194a2508b2c033783adbf5566d822ca4affe3a7ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T21:54:29.755 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:29 smithi001 podman[93277]: 2024-04-26 21:54:29.752870868 +0000 UTC m=+0.449411578 container init bb78677bc92141e7473e666194a2508b2c033783adbf5566d822ca4affe3a7ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T21:54:30.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:29 smithi001 podman[93277]: 2024-04-26 21:54:29.762685546 +0000 UTC m=+0.459226220 container start bb78677bc92141e7473e666194a2508b2c033783adbf5566d822ca4affe3a7ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-26T21:54:30.068 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:29 smithi001 podman[93277]: 2024-04-26 21:54:29.860238503 +0000 UTC m=+0.556779240 container attach bb78677bc92141e7473e666194a2508b2c033783adbf5566d822ca4affe3a7ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T21:54:30.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:29 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:29.905+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:30.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:29 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:29.906+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:30.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:29 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:29.906+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:30.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:29 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:29.834+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:30.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:29 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:29.834+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:30.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:29 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:29.834+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.438795+0000 front 2024-04-26T21:54:00.438749+0000 (oldest deadline 2024-04-26T21:54:26.338799+0000) 2024-04-26T21:54:30.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:29 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:29.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:30.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:29 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:29.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:30.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:29 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:29.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:30.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:29 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:29.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:30.559 INFO:journalctl@ceph.osd.6.smithi001.stdout:Apr 26 21:54:30 smithi001 podman[93277]: 2024-04-26 21:54:30.23808814 +0000 UTC m=+0.934628777 container died bb78677bc92141e7473e666194a2508b2c033783adbf5566d822ca4affe3a7ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-26T21:54:30.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:30 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:30.247+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:30.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:30 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:30.247+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:30.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:30 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:30.247+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:30.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:30.557+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:30.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:30.557+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:30.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:30.557+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:30.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:30.557+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:31.020 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T21:54:31.142 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:54:31.142 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T21:54:31.142 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-26T21:54:31.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.7 2024-04-26T21:54:31.170 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:30 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:30.829+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:31.170 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:30 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:30.829+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:31.170 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:30 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:30.829+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.438795+0000 front 2024-04-26T21:54:00.438749+0000 (oldest deadline 2024-04-26T21:54:26.338799+0000) 2024-04-26T21:54:31.170 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:30 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:30.882+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:31.171 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:30 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:30.882+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:31.171 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:30 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:30.882+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:31.171 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:30 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:30.882+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:31.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:30 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:30.864+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:31.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:30 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:30.864+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:31.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:30 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:30.864+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:31.460 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:31 smithi038 systemd[1]: Stopping Ceph osd.7 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:54:31.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:31 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:31.246+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:31.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:31 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:31.246+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:31.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:31 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:31.246+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:31.826 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:31.458+0000 7f066c5cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:54:31.827 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:31.458+0000 7f066c5cb700 -1 osd.7 139 *** Got signal Terminated *** 2024-04-26T21:54:31.827 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:31.458+0000 7f066c5cb700 -1 osd.7 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:54:32.067 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:31 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:31.601+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:32.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:31 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:31.601+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:32.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:31 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:31.601+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:32.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:31 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:31.601+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:32.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:31 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:31.843+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:32.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:31 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:31.843+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:32.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:31 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:31.843+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:32.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:31.841+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:32.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:31.841+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:32.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:31.841+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.438795+0000 front 2024-04-26T21:54:00.438749+0000 (oldest deadline 2024-04-26T21:54:26.338799+0000) 2024-04-26T21:54:32.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:31.841+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:06.339272+0000 front 2024-04-26T21:54:06.339301+0000 (oldest deadline 2024-04-26T21:54:31.039441+0000) 2024-04-26T21:54:32.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:31.898+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:32.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:31.898+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:32.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:31.898+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:32.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:31.898+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:32.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:31 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:31.898+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:32.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:32 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:32.249+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:32.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:32 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:32.249+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:32.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:32 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:32.249+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:32.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:32 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:32.249+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:32.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:32 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:32.553+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:32.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:32 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:32.553+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:32.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:32 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:32.553+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:32.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:32 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:32.553+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:32.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:32 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:32.553+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:33.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:32 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:32.857+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:33.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:32 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:32.857+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:33.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:32 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:32.857+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:33.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:32 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:32.874+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:33.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:32 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:32.874+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:33.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:32 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:32.874+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.438795+0000 front 2024-04-26T21:54:00.438749+0000 (oldest deadline 2024-04-26T21:54:26.338799+0000) 2024-04-26T21:54:33.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:32 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:32.874+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:06.339272+0000 front 2024-04-26T21:54:06.339301+0000 (oldest deadline 2024-04-26T21:54:31.039441+0000) 2024-04-26T21:54:33.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:32 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:32.902+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:33.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:32 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:32.902+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:33.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:32 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:32.902+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:33.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:32 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:32.902+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:33.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:32 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:32.902+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:33.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:33 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:33.219+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:33.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:33 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:33.219+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:33.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:33 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:33.219+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:33.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:33 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:33.219+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:33.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:33 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:33.550+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:33.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:33 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:33.550+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:33.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:33 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:33.550+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:33.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:33 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:33.550+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:33.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:33 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:33.550+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:34.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:33 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:33.832+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:34.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:33 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:33.832+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:34.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:33 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:33.832+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:34.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:33 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:33.853+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:34.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:33 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:33.853+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:34.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:33 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:33.853+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:34.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:33 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:33.853+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:34.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:33 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:33.853+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:34.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:33 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:33.850+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:34.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:33 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:33.850+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:34.328 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:33 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:33.850+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.438795+0000 front 2024-04-26T21:54:00.438749+0000 (oldest deadline 2024-04-26T21:54:26.338799+0000) 2024-04-26T21:54:34.328 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:33 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:33.850+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:06.339272+0000 front 2024-04-26T21:54:06.339301+0000 (oldest deadline 2024-04-26T21:54:31.039441+0000) 2024-04-26T21:54:34.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:34 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:34.259+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:34.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:34 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:34.259+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:34.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:34 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:34.259+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:34.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:34 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:34.259+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:34.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:34 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:34.523+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:34.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:34 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:34.523+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:34.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:34 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:34.523+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:34.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:34 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:34.523+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:34.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:34 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:34.523+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:35.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:34 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:34.834+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:35.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:34 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:34.834+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:35.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:34 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:34.834+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:35.078 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:34 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:34.834+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:35.078 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:34 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:34.834+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:35.078 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:34 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:34.821+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:35.079 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:34 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:34.821+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:35.079 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:34 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:34.821+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.438795+0000 front 2024-04-26T21:54:00.438749+0000 (oldest deadline 2024-04-26T21:54:26.338799+0000) 2024-04-26T21:54:35.079 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:34 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:34.821+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:06.339272+0000 front 2024-04-26T21:54:06.339301+0000 (oldest deadline 2024-04-26T21:54:31.039441+0000) 2024-04-26T21:54:35.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:34 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:34.872+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:35.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:34 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:34.872+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:35.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:34 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:34.872+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:35.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:35 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:35.288+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:35.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:35 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:35.288+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:35.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:35 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:35.288+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:35.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:35 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:35.288+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:35.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:35 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:35.506+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:35.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:35 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:35.506+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:35.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:35 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:35.506+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:35.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:35 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:35.506+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:35.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:35 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:35.506+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:36.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:35 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:35.792+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.534876+0000 front 2024-04-26T21:53:40.534906+0000 (oldest deadline 2024-04-26T21:54:04.635090+0000) 2024-04-26T21:54:36.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:35 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:35.792+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:48.736634+0000 front 2024-04-26T21:53:48.736561+0000 (oldest deadline 2024-04-26T21:54:12.836741+0000) 2024-04-26T21:54:36.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:35 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:35.792+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:54:00.438795+0000 front 2024-04-26T21:54:00.438749+0000 (oldest deadline 2024-04-26T21:54:26.338799+0000) 2024-04-26T21:54:36.077 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:35 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7[50859]: 2024-04-26T21:54:35.792+0000 7f0664bb7700 -1 osd.7 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:06.339272+0000 front 2024-04-26T21:54:06.339301+0000 (oldest deadline 2024-04-26T21:54:31.039441+0000) 2024-04-26T21:54:36.077 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:35 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:35.860+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:36.078 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:35 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:35.860+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:36.078 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:35 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:35.860+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:36.078 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:35 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:35.860+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:36.078 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:35 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:35.860+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:36.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:35 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:35.845+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:36.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:35 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:35.845+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:36.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:35 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:35.845+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:36.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:36 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:36.249+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:36.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:36 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:36.249+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:36.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:36 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:36.249+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:36.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:36 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:36.249+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:36.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:36 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:36.520+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:36.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:36 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:36.520+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:36.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:36 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:36.520+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:36.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:36 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:36.520+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:36.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:36 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:36.520+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:37.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:36 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:36.850+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:37.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:36 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:36.850+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:37.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:36 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:36.850+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:37.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:36 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:36.850+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:10.099324+0000 front 2024-04-26T21:54:10.099521+0000 (oldest deadline 2024-04-26T21:54:35.999292+0000) 2024-04-26T21:54:37.303 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:37 smithi038 podman[72485]: 2024-04-26 21:54:37.044547495 +0000 UTC m=+5.704082973 container died 1c4f4afb1f5b925acba63e9e2b353665bf3857c90d8f04b0b522742b70b7e163 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-04-26T21:54:37.303 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:36 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:36.897+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:37.304 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:36 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:36.897+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:37.304 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:36 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:36.897+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:37.304 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:36 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:36.897+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:37.304 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:36 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:36.897+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:37.576 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:37 smithi038 podman[72485]: 2024-04-26 21:54:37.386059168 +0000 UTC m=+6.045594643 container cleanup 1c4f4afb1f5b925acba63e9e2b353665bf3857c90d8f04b0b522742b70b7e163 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T21:54:37.577 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:37 smithi038 bash[72485]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7 2024-04-26T21:54:37.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:37 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:37.240+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:37.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:37 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:37.240+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:37.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:37 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:37.240+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:37.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:37 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:37.240+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:37.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:37 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:37.540+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:37.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:37 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:37.540+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:37.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:37 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:37.540+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:37.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:37 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:37.540+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:37.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:37 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:37.540+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:38.074 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:37 smithi038 podman[72502]: 2024-04-26 21:54:37.612522357 +0000 UTC m=+0.566701549 container remove 1c4f4afb1f5b925acba63e9e2b353665bf3857c90d8f04b0b522742b70b7e163 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-26T21:54:38.074 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:37 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:37.856+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:38.075 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:37 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:37.856+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:38.075 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:37 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:37.856+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:38.075 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:37 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:37.856+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:38.075 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:37 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:37.856+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:38.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:37 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:37.854+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:38.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:37 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:37.854+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:38.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:37 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:37.854+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:38.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:37 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:37.854+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:10.099324+0000 front 2024-04-26T21:54:10.099521+0000 (oldest deadline 2024-04-26T21:54:35.999292+0000) 2024-04-26T21:54:38.326 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:38 smithi038 podman[72590]: 2024-04-26 21:54:37.978101059 +0000 UTC m=+0.018172991 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:54:38.327 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:38 smithi038 podman[72590]: 2024-04-26 21:54:38.092868956 +0000 UTC m=+0.132940875 container create 5ec60b53d3ef9b1b83841f2dac640b9b3436e586084d50d875845fd593215797 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:54:38.647 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:38 smithi038 podman[72590]: 2024-04-26 21:54:38.333152263 +0000 UTC m=+0.373224182 container init 5ec60b53d3ef9b1b83841f2dac640b9b3436e586084d50d875845fd593215797 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0) 2024-04-26T21:54:38.648 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:38 smithi038 podman[72590]: 2024-04-26 21:54:38.336334424 +0000 UTC m=+0.376406533 container start 5ec60b53d3ef9b1b83841f2dac640b9b3436e586084d50d875845fd593215797 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-26T21:54:38.648 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:38 smithi038 podman[72590]: 2024-04-26 21:54:38.441568808 +0000 UTC m=+0.481640739 container attach 5ec60b53d3ef9b1b83841f2dac640b9b3436e586084d50d875845fd593215797 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T21:54:38.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:38 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:38.274+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:38.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:38 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:38.274+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:38.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:38 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:38.274+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:38.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:38 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:38.274+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:38.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:38 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:38.530+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:38.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:38 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:38.530+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:38.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:38 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:38.530+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:38.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:38 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:38.530+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:38.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:38 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:38.530+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:38.978 INFO:journalctl@ceph.osd.7.smithi038.stdout:Apr 26 21:54:38 smithi038 podman[72590]: 2024-04-26 21:54:38.648103002 +0000 UTC m=+0.688174914 container died 5ec60b53d3ef9b1b83841f2dac640b9b3436e586084d50d875845fd593215797 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-26T21:54:38.978 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:38 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:38.844+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:38.979 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:38 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:38.845+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:38.979 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:38 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:38.845+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:38.979 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:38 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:38.845+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:38.979 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:38 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:38.845+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:39.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:38 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:38.828+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:39.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:38 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:38.828+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:39.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:38 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:38.828+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:39.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:38 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:38.828+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:10.099324+0000 front 2024-04-26T21:54:10.099521+0000 (oldest deadline 2024-04-26T21:54:35.999292+0000) 2024-04-26T21:54:39.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.7.service' 2024-04-26T21:54:39.377 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:54:39.377 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-26T21:54:39.377 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-26T21:54:39.378 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.8 2024-04-26T21:54:39.685 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:39 smithi186 systemd[1]: Stopping Ceph osd.8 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:54:39.685 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:39.307+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:39.685 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:39.307+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:39.686 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:39.307+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:39.686 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:39.307+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:39.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:39 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:39.523+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:39.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:39 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:39.523+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:39.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:39 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:39.523+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:39.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:39 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:39.523+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:39.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:39 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:39.523+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:39.954 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:39.684+0000 7f182c9aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:54:39.954 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:39.684+0000 7f182c9aa700 -1 osd.8 139 *** Got signal Terminated *** 2024-04-26T21:54:39.954 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:39.684+0000 7f182c9aa700 -1 osd.8 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:54:39.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:39.836+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:39.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:39.836+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:39.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:39.836+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:39.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:39 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:39.836+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:10.099324+0000 front 2024-04-26T21:54:10.099521+0000 (oldest deadline 2024-04-26T21:54:35.999292+0000) 2024-04-26T21:54:40.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:39 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:39.842+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:40.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:39 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:39.842+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:40.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:39 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:39.842+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:40.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:39 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:39.842+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:40.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:39 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:39.842+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:40.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:39 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:39.842+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:40.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:40.281+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:40.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:40.281+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:40.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:40.281+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:40.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:40.281+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:40.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:40.281+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:40.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:40 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:40.566+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:40.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:40 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:40.566+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:40.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:40 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:40.566+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:40.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:40 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:40.566+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:40.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:40 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:40.566+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:40.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:40 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:40.566+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:41.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:40.789+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:41.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:40.789+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:41.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:40.789+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:41.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:40.789+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:10.099324+0000 front 2024-04-26T21:54:10.099521+0000 (oldest deadline 2024-04-26T21:54:35.999292+0000) 2024-04-26T21:54:41.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:40 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:40.789+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.999659+0000 front 2024-04-26T21:54:15.999705+0000 (oldest deadline 2024-04-26T21:54:40.099740+0000) 2024-04-26T21:54:41.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:40 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:40.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:41.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:40 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:40.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:41.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:40 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:40.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:41.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:40 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:40.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:41.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:40 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:40.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:41.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:40 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:40.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:41.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:41 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:41.258+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:41.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:41 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:41.258+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:41.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:41 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:41.258+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:41.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:41 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:41.258+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:41.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:41 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:41.258+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:41.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:41 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:41.539+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:41.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:41 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:41.539+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:41.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:41 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:41.539+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:41.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:41 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:41.539+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:41.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:41 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:41.539+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:41.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:41 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:41.539+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:42.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:41 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:41.752+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:42.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:41 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:41.752+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:42.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:41 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:41.752+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:42.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:41 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:41.752+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:10.099324+0000 front 2024-04-26T21:54:10.099521+0000 (oldest deadline 2024-04-26T21:54:35.999292+0000) 2024-04-26T21:54:42.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:41 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:41.752+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.999659+0000 front 2024-04-26T21:54:15.999705+0000 (oldest deadline 2024-04-26T21:54:40.099740+0000) 2024-04-26T21:54:42.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:41 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:41.852+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:42.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:41 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:41.852+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:42.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:41 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:41.852+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:42.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:41 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:41.852+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:42.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:41 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:41.852+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:42.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:41 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:41.852+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:42.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:42 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:42.236+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:42.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:42 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:42.236+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:42.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:42 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:42.236+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:42.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:42 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:42.236+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:42.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:42 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:42.236+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:42.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:42 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:42.548+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:42.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:42 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:42.548+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:42.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:42 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:42.548+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:42.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:42 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:42.548+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:42.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:42 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:42.548+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:42.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:42 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:42.548+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:43.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:42 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:42.714+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:43.204 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:42 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:42.714+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:43.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:42 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:42.714+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:43.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:42 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:42.714+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:10.099324+0000 front 2024-04-26T21:54:10.099521+0000 (oldest deadline 2024-04-26T21:54:35.999292+0000) 2024-04-26T21:54:43.205 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:42 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:42.714+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.999659+0000 front 2024-04-26T21:54:15.999705+0000 (oldest deadline 2024-04-26T21:54:40.099740+0000) 2024-04-26T21:54:43.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:42 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:42.846+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:43.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:42 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:42.846+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:43.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:42 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:42.846+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:43.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:42 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:42.846+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:43.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:42 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:42.846+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:43.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:42 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:42.846+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:43.671 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:43 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:43.282+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:43.671 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:43 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:43.282+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:43.672 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:43 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:43.282+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:43.672 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:43 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:43.282+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:43.672 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:43 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:43.282+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:43.954 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:43 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:43.670+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:43.954 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:43 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:43.670+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:43.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:43 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:43.670+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:43.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:43 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:43.670+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:10.099324+0000 front 2024-04-26T21:54:10.099521+0000 (oldest deadline 2024-04-26T21:54:35.999292+0000) 2024-04-26T21:54:43.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:43 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:43.670+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.999659+0000 front 2024-04-26T21:54:15.999705+0000 (oldest deadline 2024-04-26T21:54:40.099740+0000) 2024-04-26T21:54:44.067 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:43 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:43.592+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:44.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:43 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:43.592+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:44.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:43 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:43.592+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:44.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:43 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:43.592+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:44.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:43 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:43.592+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:44.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:43 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:43.592+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:44.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:43 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:43.848+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:44.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:43 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:43.848+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:44.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:43 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:43.848+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:44.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:43 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:43.848+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:44.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:43 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:43.848+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:44.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:43 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:43.848+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:44.629 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:44 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:44.275+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:44.630 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:44 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:44.275+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:44.630 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:44 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:44.275+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:44.630 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:44 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:44.275+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:44.630 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:44 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:44.275+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:44.954 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:44 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:44.628+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.690441+0000 front 2024-04-26T21:53:42.690493+0000 (oldest deadline 2024-04-26T21:54:07.291229+0000) 2024-04-26T21:54:44.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:44 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:44.628+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:50.192312+0000 front 2024-04-26T21:53:50.192341+0000 (oldest deadline 2024-04-26T21:54:15.393403+0000) 2024-04-26T21:54:44.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:44 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:44.628+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:58.695940+0000 front 2024-04-26T21:53:58.695787+0000 (oldest deadline 2024-04-26T21:54:22.696344+0000) 2024-04-26T21:54:44.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:44 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:44.628+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:10.099324+0000 front 2024-04-26T21:54:10.099521+0000 (oldest deadline 2024-04-26T21:54:35.999292+0000) 2024-04-26T21:54:44.955 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:44 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8[51827]: 2024-04-26T21:54:44.628+0000 7f1824f96700 -1 osd.8 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.999659+0000 front 2024-04-26T21:54:15.999705+0000 (oldest deadline 2024-04-26T21:54:40.099740+0000) 2024-04-26T21:54:45.067 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:44 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:44.597+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:45.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:44 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:44.597+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:45.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:44 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:44.597+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:45.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:44 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:44.597+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:45.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:44 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:44.597+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:45.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:44 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:44.597+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:45.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:44 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:44.870+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:45.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:44 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:44.870+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:45.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:44 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:44.870+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:45.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:44 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:44.870+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:45.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:44 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:44.870+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:45.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:44 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:44.870+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:45.454 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:45 smithi186 podman[74585]: 2024-04-26 21:54:45.163778911 +0000 UTC m=+5.593524024 container died a34c1233047affa0170cbd928371978849b005305f3af489a5bde9ba9ac46b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-26T21:54:45.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:45 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:45.289+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:45.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:45 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:45.289+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:45.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:45 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:45.289+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:45.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:45 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:45.289+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:45.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:45 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:45.289+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:45.811 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:45 smithi186 podman[74585]: 2024-04-26 21:54:45.552068942 +0000 UTC m=+5.981814053 container cleanup a34c1233047affa0170cbd928371978849b005305f3af489a5bde9ba9ac46b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T21:54:45.811 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:45 smithi186 bash[74585]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8 2024-04-26T21:54:46.067 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:45 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:45.583+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:46.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:45 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:45.583+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:46.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:45 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:45.583+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:46.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:45 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:45.583+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:46.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:45 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:45.583+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:46.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:45 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:45.583+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:46.163 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:45 smithi186 podman[74602]: 2024-04-26 21:54:45.811630013 +0000 UTC m=+0.645295367 container remove a34c1233047affa0170cbd928371978849b005305f3af489a5bde9ba9ac46b85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-26T21:54:46.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:45 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:45.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:46.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:45 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:45.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:46.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:45 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:45.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:46.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:45 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:45.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:46.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:45 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:45.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:46.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:45 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:45.881+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:46.454 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:46 smithi186 podman[74690]: 2024-04-26 21:54:46.066902147 +0000 UTC m=+0.018137814 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:54:46.455 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:46 smithi186 podman[74690]: 2024-04-26 21:54:46.170977642 +0000 UTC m=+0.122213336 container create 856fbc17479af4e50e16a459dec1ac4c308f02ee21c431c60f28ea02f51c5e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-26T21:54:46.455 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:46 smithi186 podman[74690]: 2024-04-26 21:54:46.406588929 +0000 UTC m=+0.357824581 container init 856fbc17479af4e50e16a459dec1ac4c308f02ee21c431c60f28ea02f51c5e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:54:46.455 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:46 smithi186 podman[74690]: 2024-04-26 21:54:46.409668665 +0000 UTC m=+0.360904321 container start 856fbc17479af4e50e16a459dec1ac4c308f02ee21c431c60f28ea02f51c5e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-26T21:54:46.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:46 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:46.263+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:46.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:46 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:46.263+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:46.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:46 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:46.263+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:46.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:46 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:46.263+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:46.456 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:46 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:46.263+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:46.708 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:46 smithi186 podman[74690]: 2024-04-26 21:54:46.515199084 +0000 UTC m=+0.466434745 container attach 856fbc17479af4e50e16a459dec1ac4c308f02ee21c431c60f28ea02f51c5e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-26T21:54:47.010 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 26 21:54:46 smithi186 podman[74717]: 2024-04-26 21:54:46.738235151 +0000 UTC m=+0.018599434 container died 856fbc17479af4e50e16a459dec1ac4c308f02ee21c431c60f28ea02f51c5e00 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-26T21:54:47.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:46.596+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:47.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:46.596+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:47.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:46.596+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:47.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:46.596+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:47.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:46.596+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:47.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:46 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:46.596+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:47.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:46 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:46.929+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:47.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:46 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:46.929+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:47.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:46 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:46.929+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:47.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:46 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:46.929+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:47.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:46 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:46.929+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:47.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:46 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:46.929+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:47.368 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:47 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:47.215+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:47.369 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:47 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:47.215+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:47.369 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:47 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:47.215+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:47.369 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:47 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:47.215+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:47.369 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:47 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:47.215+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:47.388 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.8.service' 2024-04-26T21:54:47.430 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:54:47.430 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-26T21:54:47.430 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-26T21:54:47.431 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.9 2024-04-26T21:54:47.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:47 smithi001 systemd[1]: Stopping Ceph osd.9 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:54:47.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:47 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:47.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:47.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:47 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:47.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:47.818 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:47 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:47.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:47.819 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:47 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:47.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:47.819 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:47 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:47.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:47.819 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:47 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:47.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:48.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:47 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:47.975+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:48.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:47 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:47.975+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:48.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:47 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:47.975+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:48.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:47 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:47.975+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:48.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:47 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:47.975+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:48.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:47 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:47.975+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:48.567 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:48 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:48.073+0000 7f88c6800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:54:48.568 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:48 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:48.073+0000 7f88c6800700 -1 osd.9 139 *** Got signal Terminated *** 2024-04-26T21:54:48.568 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:48 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:48.073+0000 7f88c6800700 -1 osd.9 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:54:48.587 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:48 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:48.265+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:48.588 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:48 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:48.265+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:48.588 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:48 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:48.265+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:48.588 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:48 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:48.265+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:48.588 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:48 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:48.265+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:49.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:48 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:48.639+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:49.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:48 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:48.639+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:49.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:48 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:48.639+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:49.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:48 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:48.639+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:49.069 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:48 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:48.639+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:49.069 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:48 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:48.639+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:49.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:48 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:48.933+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:49.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:48 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:48.933+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:49.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:48 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:48.933+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:49.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:48 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:48.933+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:49.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:48 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:48.933+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:49.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:48 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:48.933+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:49.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:49 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:49.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:49.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:49 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:49.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:49.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:49 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:49.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:49.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:49 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:49.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:49.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:49 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:49.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:50.067 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:49.683+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:50.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:49.683+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:50.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:49.683+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:50.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:49.683+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:50.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:49.683+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:50.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:49 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:49.683+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:50.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:49 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:49.982+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:50.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:49 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:49.982+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:50.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:49 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:49.982+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:50.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:49 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:49.982+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:50.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:49 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:49.982+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:50.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:49 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:49.982+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:50.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:50 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:50.218+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:50.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:50 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:50.218+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:50.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:50 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:50.218+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:50.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:50 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:50.218+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:50.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:50 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:50.218+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:50.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:50 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:50.218+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:54:51.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:50 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:50.674+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:51.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:50 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:50.674+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:51.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:50 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:50.674+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:51.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:50 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:50.674+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:51.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:50 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:50.674+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:51.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:50 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:50.674+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:51.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:50 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:50.977+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:51.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:50 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:50.977+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:51.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:50 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:50.977+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:51.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:50 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:50.977+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:51.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:50 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:50.977+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:51.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:50 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:50.977+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:51.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:51 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:51.261+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:51.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:51 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:51.261+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:51.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:51 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:51.261+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:51.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:51 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:51.261+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:51.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:51 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:51.261+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:51.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:51 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:51.261+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:54:52.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:51.629+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:52.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:51.629+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:52.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:51.629+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:52.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:51.629+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:52.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:51.629+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:52.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:51 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:51.629+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:52.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:52 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:52.003+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:52.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:52 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:52.003+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:52.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:52 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:52.003+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:52.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:52 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:52.003+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:52.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:52 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:52.003+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:52.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:52 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:52.003+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:52.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:52 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:52.003+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:54:52.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:52 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:52.252+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:52.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:52 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:52.252+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:52.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:52 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:52.252+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:52.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:52 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:52.252+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:52.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:52 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:52.252+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:52.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:52 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:52.252+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:54:53.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:52 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:52.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.297273+0000 front 2024-04-26T21:53:31.297352+0000 (oldest deadline 2024-04-26T21:53:56.597406+0000) 2024-04-26T21:54:53.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:52 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:52.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:41.704387+0000 front 2024-04-26T21:53:41.704488+0000 (oldest deadline 2024-04-26T21:54:09.306336+0000) 2024-04-26T21:54:53.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:52 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:52.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:49.307861+0000 front 2024-04-26T21:53:49.307795+0000 (oldest deadline 2024-04-26T21:54:15.208131+0000) 2024-04-26T21:54:53.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:52 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:52.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:55.209128+0000 front 2024-04-26T21:53:55.209177+0000 (oldest deadline 2024-04-26T21:54:24.611013+0000) 2024-04-26T21:54:53.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:52 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:52.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:04.612400+0000 front 2024-04-26T21:54:04.612657+0000 (oldest deadline 2024-04-26T21:54:31.614133+0000) 2024-04-26T21:54:53.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:52 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9[60307]: 2024-04-26T21:54:52.589+0000 7f88bd000700 -1 osd.9 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.316834+0000 front 2024-04-26T21:54:16.316891+0000 (oldest deadline 2024-04-26T21:54:39.817035+0000) 2024-04-26T21:54:53.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:53 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:53.037+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:53.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:53 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:53.037+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:53.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:53 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:53.037+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:53.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:53 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:53.037+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:53.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:53 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:53.037+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:53.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:53 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:53.037+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:53.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:53 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:53.037+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:54:53.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:53 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:53.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:53.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:53 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:53.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:53.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:53 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:53.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:53.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:53 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:53.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:53.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:53 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:53.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:53.705 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:53 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:53.227+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:54:54.068 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:53 smithi001 podman[93382]: 2024-04-26 21:54:53.777228487 +0000 UTC m=+5.895997470 container died b247478b9757c0f7fd3f6f72aed2f71ae0a18362b7bfd76200301f1011157611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T21:54:54.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:54 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:54.044+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:54.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:54 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:54.044+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:54.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:54 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:54.044+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:54.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:54 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:54.044+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:54.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:54 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:54.044+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:54.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:54 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:54.044+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:54.328 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:54 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:54.044+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:54:54.452 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:54 smithi001 podman[93382]: 2024-04-26 21:54:54.199989543 +0000 UTC m=+6.318758566 container cleanup b247478b9757c0f7fd3f6f72aed2f71ae0a18362b7bfd76200301f1011157611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-26T21:54:54.452 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:54 smithi001 bash[93382]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9 2024-04-26T21:54:54.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:54 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:54.190+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:54.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:54 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:54.190+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:54.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:54 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:54.190+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:54.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:54 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:54.190+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:54.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:54 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:54.190+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:54.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:54 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:54.190+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:54:54.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:54 smithi001 podman[93400]: 2024-04-26 21:54:54.450996849 +0000 UTC m=+0.654563547 container remove b247478b9757c0f7fd3f6f72aed2f71ae0a18362b7bfd76200301f1011157611 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-26T21:54:55.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:55 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:55.185+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:55.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:55 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:55.185+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:55.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:55 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:55.185+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:55.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:55 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:55.185+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:55.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:55 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:55.185+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:55.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:55 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:55.185+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:54:55.556 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:55 smithi001 podman[93487]: 2024-04-26 21:54:55.172868206 +0000 UTC m=+0.042374893 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:54:55.556 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:55 smithi001 podman[93487]: 2024-04-26 21:54:55.293491381 +0000 UTC m=+0.162998105 container create 8a956b8e4e2402d809af07bba5722aa14ab8b24cecf78446b604096e1bc708be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-26T21:54:55.557 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:55 smithi001 podman[93487]: 2024-04-26 21:54:55.555378552 +0000 UTC m=+0.424885283 container init 8a956b8e4e2402d809af07bba5722aa14ab8b24cecf78446b604096e1bc708be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:54:55.576 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:55 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:55.080+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:55.576 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:55 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:55.080+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:55.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:55 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:55.080+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:55.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:55 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:55.080+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:55.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:55 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:55.080+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:55.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:55 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:55.080+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:55.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:55 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:55.080+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:54:55.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:55 smithi001 podman[93487]: 2024-04-26 21:54:55.567587901 +0000 UTC m=+0.437094594 container start 8a956b8e4e2402d809af07bba5722aa14ab8b24cecf78446b604096e1bc708be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T21:54:55.817 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:55 smithi001 podman[93487]: 2024-04-26 21:54:55.64560252 +0000 UTC m=+0.515109277 container attach 8a956b8e4e2402d809af07bba5722aa14ab8b24cecf78446b604096e1bc708be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:54:56.146 INFO:journalctl@ceph.osd.9.smithi001.stdout:Apr 26 21:54:56 smithi001 podman[93487]: 2024-04-26 21:54:56.055611049 +0000 UTC m=+0.925117769 container died 8a956b8e4e2402d809af07bba5722aa14ab8b24cecf78446b604096e1bc708be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T21:54:56.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:56 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:56.054+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:56.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:56 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:56.054+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:56.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:56 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:56.054+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:56.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:56 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:56.054+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:56.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:56 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:56.054+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:56.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:56 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:56.054+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:56.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:56 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:56.054+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:54:56.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:56 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:56.144+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:56.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:56 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:56.144+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:56.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:56 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:56.144+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:56.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:56 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:56.144+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:56.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:56 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:56.144+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:56.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:56 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:56.144+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:54:56.860 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.9.service' 2024-04-26T21:54:56.958 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:54:56.958 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-26T21:54:56.959 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-26T21:54:56.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.10 2024-04-26T21:54:57.290 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 systemd[1]: Stopping Ceph osd.10 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:54:57.291 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:57.013+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:57.291 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:57.013+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:57.291 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:57.013+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:57.291 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:57.013+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:57.291 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:57.013+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:57.291 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:57.013+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:57.292 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:57.013+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:54:57.292 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:57.290+0000 7fa1d618a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:54:57.292 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:57.290+0000 7fa1d618a700 -1 osd.10 139 *** Got signal Terminated *** 2024-04-26T21:54:57.292 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:57 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:57.290+0000 7fa1d618a700 -1 osd.10 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:54:57.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:57 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:57.111+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:57.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:57 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:57.111+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:57.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:57 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:57.111+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:57.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:57 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:57.111+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:57.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:57 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:57.111+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:57.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:57 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:57.111+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:54:58.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:58 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:58.038+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:58.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:58 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:58.038+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:58.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:58 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:58.038+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:58.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:58 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:58.038+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:58.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:58 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:58.038+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:58.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:58 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:58.038+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:58.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:58 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:58.038+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:54:58.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:58 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:58.110+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:58.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:58 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:58.110+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:58.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:58 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:58.111+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:58.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:58 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:58.111+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:58.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:58 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:58.111+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:58.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:58 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:58.111+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:54:59.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:59 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:59.043+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:54:59.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:59 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:59.043+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:54:59.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:59 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:59.043+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:54:59.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:59 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:59.043+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:54:59.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:59 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:59.043+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:54:59.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:59 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:59.043+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:54:59.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:54:59 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:54:59.043+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:54:59.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:59 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:59.103+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:54:59.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:59 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:59.103+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:54:59.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:59 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:59.103+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:54:59.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:59 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:59.103+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:54:59.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:59 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:59.103+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:54:59.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:54:59 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:54:59.103+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:00.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:00 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:00.079+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:00.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:00 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:00.079+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:00.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:00 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:00.079+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:00.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:00 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:00.079+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:00.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:00 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:00.079+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:00.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:00 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:00.079+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:00.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:00 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:00.079+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:00.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:00 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:00.079+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:55:00.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:00 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:00.079+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:55:00.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:00 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:00.079+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:55:00.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:00 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:00.079+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:55:00.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:00 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:00.079+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:55:00.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:00 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:00.079+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:55:00.577 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:00 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:00.079+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:55:01.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:01 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:01.046+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:55:01.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:01 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:01.046+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:55:01.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:01 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:01.046+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:55:01.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:01 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:01.046+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:55:01.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:01 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:01.046+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:55:01.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:01 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:01.046+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:55:01.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:01 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:01.046+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:55:01.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:01 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:01.076+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:01.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:01 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:01.076+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:01.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:01 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:01.076+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:01.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:01 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:01.076+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:01.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:01 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:01.076+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:01.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:01 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:01.076+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:01.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:01 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:01.076+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:02.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:02 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:02.033+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:28.915000+0000 front 2024-04-26T21:53:28.915079+0000 (oldest deadline 2024-04-26T21:53:54.814837+0000) 2024-04-26T21:55:02.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:02 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:02.033+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:42.416679+0000 front 2024-04-26T21:53:42.416720+0000 (oldest deadline 2024-04-26T21:54:05.916807+0000) 2024-04-26T21:55:02.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:02 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:02.033+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:51.519636+0000 front 2024-04-26T21:53:51.519728+0000 (oldest deadline 2024-04-26T21:54:16.720469+0000) 2024-04-26T21:55:02.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:02 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:02.033+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6814 osd.3 since back 2024-04-26T21:53:56.721093+0000 front 2024-04-26T21:53:56.721004+0000 (oldest deadline 2024-04-26T21:54:22.621151+0000) 2024-04-26T21:55:02.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:02 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:02.033+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.622959+0000 front 2024-04-26T21:54:09.622985+0000 (oldest deadline 2024-04-26T21:54:31.323110+0000) 2024-04-26T21:55:02.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:02 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:02.033+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:16.624301+0000 front 2024-04-26T21:54:16.624398+0000 (oldest deadline 2024-04-26T21:54:39.524356+0000) 2024-04-26T21:55:02.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:02 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10[57448]: 2024-04-26T21:55:02.033+0000 7fa1d277e700 -1 osd.10 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:25.826878+0000 front 2024-04-26T21:54:25.826983+0000 (oldest deadline 2024-04-26T21:54:51.126625+0000) 2024-04-26T21:55:02.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:02 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:02.119+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:02.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:02 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:02.119+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:02.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:02 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:02.119+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:02.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:02 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:02.119+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:02.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:02 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:02.119+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:02.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:02 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:02.119+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:02.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:02 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:02.119+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:03.032 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:02 smithi038 podman[72708]: 2024-04-26 21:55:02.744479445 +0000 UTC m=+5.585993826 container died 9c3c231c5509a1bb35b127db7d26bf137f5902d4b9b93d55c5f0388f90e8693c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T21:55:03.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:03 smithi038 podman[72708]: 2024-04-26 21:55:03.122899899 +0000 UTC m=+5.964414289 container cleanup 9c3c231c5509a1bb35b127db7d26bf137f5902d4b9b93d55c5f0388f90e8693c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5) 2024-04-26T21:55:03.327 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:03 smithi038 bash[72708]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10 2024-04-26T21:55:03.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:03 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:03.107+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:03.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:03 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:03.107+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:03.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:03 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:03.107+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:03.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:03 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:03.107+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:03.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:03 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:03.107+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:03.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:03 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:03.107+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:03.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:03 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:03.107+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:03.658 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:03 smithi038 podman[72727]: 2024-04-26 21:55:03.365782056 +0000 UTC m=+0.611763918 container remove 9c3c231c5509a1bb35b127db7d26bf137f5902d4b9b93d55c5f0388f90e8693c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-26T21:55:03.943 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:03 smithi038 podman[72818]: 2024-04-26 21:55:03.658399866 +0000 UTC m=+0.104437401 container create ec9817508e9606d04d878424cf00fb425597efd9596efb51d2533c86f2c7a885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T21:55:03.944 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:03 smithi038 podman[72818]: 2024-04-26 21:55:03.572128415 +0000 UTC m=+0.018165965 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:55:03.944 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:03 smithi038 podman[72818]: 2024-04-26 21:55:03.869706465 +0000 UTC m=+0.315744016 container init ec9817508e9606d04d878424cf00fb425597efd9596efb51d2533c86f2c7a885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:55:03.944 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:03 smithi038 podman[72818]: 2024-04-26 21:55:03.873418256 +0000 UTC m=+0.319455795 container start ec9817508e9606d04d878424cf00fb425597efd9596efb51d2533c86f2c7a885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-26T21:55:03.944 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:03 smithi038 podman[72818]: 2024-04-26 21:55:03.944107297 +0000 UTC m=+0.390144848 container attach ec9817508e9606d04d878424cf00fb425597efd9596efb51d2533c86f2c7a885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True) 2024-04-26T21:55:04.326 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:04 smithi038 podman[72818]: 2024-04-26 21:55:04.187426482 +0000 UTC m=+0.633464018 container died ec9817508e9606d04d878424cf00fb425597efd9596efb51d2533c86f2c7a885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:55:04.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:04 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:04.125+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:04.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:04 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:04.125+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:04.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:04 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:04.125+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:04.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:04 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:04.125+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:04.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:04 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:04.125+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:04.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:04 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:04.125+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:04.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:04 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:04.125+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:04.899 INFO:journalctl@ceph.osd.10.smithi038.stdout:Apr 26 21:55:04 smithi038 podman[72844]: 2024-04-26 21:55:04.880760972 +0000 UTC m=+0.685059519 container remove ec9817508e9606d04d878424cf00fb425597efd9596efb51d2533c86f2c7a885 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:55:04.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.10.service' 2024-04-26T21:55:04.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:04.951 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-26T21:55:04.951 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-26T21:55:04.952 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.11 2024-04-26T21:55:05.272 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:04 smithi186 systemd[1]: Stopping Ceph osd.11 for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:05.272 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:05.150+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:05.272 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:05.150+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:05.272 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:05.150+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:05.272 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:05.150+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:05.272 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:05.150+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:05.272 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:05.150+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:05.273 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:05.150+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:05.273 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:05.271+0000 7f24a4d04700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:05.273 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:05.271+0000 7f24a4d04700 -1 osd.11 139 *** Got signal Terminated *** 2024-04-26T21:55:05.273 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:05.271+0000 7f24a4d04700 -1 osd.11 139 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T21:55:06.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:06 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:06.165+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:06.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:06 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:06.165+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:06.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:06 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:06.165+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:06.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:06 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:06.165+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:06.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:06 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:06.165+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:06.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:06 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:06.165+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:06.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:06 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:06.165+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:07.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:07.117+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:07.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:07.117+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:07.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:07.117+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:07.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:07.117+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:07.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:07.117+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:07.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:07.117+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:07.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:07.117+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:07.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:07 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:07.117+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6820 osd.8 since back 2024-04-26T21:54:43.529276+0000 front 2024-04-26T21:54:43.529297+0000 (oldest deadline 2024-04-26T21:55:06.429298+0000) 2024-04-26T21:55:08.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:08.152+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:08.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:08.152+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:08.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:08.152+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:08.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:08.152+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:08.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:08.152+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:08.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:08.152+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:08.456 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:08.152+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:08.456 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:08 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:08.152+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6820 osd.8 since back 2024-04-26T21:54:43.529276+0000 front 2024-04-26T21:54:43.529297+0000 (oldest deadline 2024-04-26T21:55:06.429298+0000) 2024-04-26T21:55:09.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:09.124+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:09.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:09.124+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:09.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:09.124+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:09.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:09.124+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:09.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:09.124+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:09.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:09.124+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:09.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:09.124+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:09.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:09 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:09.124+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6820 osd.8 since back 2024-04-26T21:54:43.529276+0000 front 2024-04-26T21:54:43.529297+0000 (oldest deadline 2024-04-26T21:55:06.429298+0000) 2024-04-26T21:55:10.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:10.148+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-04-26T21:53:31.214097+0000 front 2024-04-26T21:53:31.214010+0000 (oldest deadline 2024-04-26T21:53:59.414412+0000) 2024-04-26T21:55:10.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:10.148+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6804 osd.1 since back 2024-04-26T21:53:40.515920+0000 front 2024-04-26T21:53:40.516071+0000 (oldest deadline 2024-04-26T21:54:06.415862+0000) 2024-04-26T21:55:10.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:10.148+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-26T21:53:52.118040+0000 front 2024-04-26T21:53:52.118070+0000 (oldest deadline 2024-04-26T21:54:13.218209+0000) 2024-04-26T21:55:10.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:10.148+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6812 osd.4 since back 2024-04-26T21:54:09.922751+0000 front 2024-04-26T21:54:09.922773+0000 (oldest deadline 2024-04-26T21:54:31.622859+0000) 2024-04-26T21:55:10.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:10.148+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-26T21:54:15.723694+0000 front 2024-04-26T21:54:15.723726+0000 (oldest deadline 2024-04-26T21:54:39.823744+0000) 2024-04-26T21:55:10.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:10.148+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.1:6822 osd.6 since back 2024-04-26T21:54:26.126361+0000 front 2024-04-26T21:54:26.126279+0000 (oldest deadline 2024-04-26T21:54:49.626126+0000) 2024-04-26T21:55:10.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:10.148+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.38:6820 osd.7 since back 2024-04-26T21:54:34.128433+0000 front 2024-04-26T21:54:34.128465+0000 (oldest deadline 2024-04-26T21:55:00.028336+0000) 2024-04-26T21:55:10.455 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:10 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11[58574]: 2024-04-26T21:55:10.148+0000 7f24a12f8700 -1 osd.11 139 heartbeat_check: no reply from 172.21.15.186:6820 osd.8 since back 2024-04-26T21:54:43.529276+0000 front 2024-04-26T21:54:43.529297+0000 (oldest deadline 2024-04-26T21:55:06.429298+0000) 2024-04-26T21:55:11.046 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:10 smithi186 podman[74808]: 2024-04-26 21:55:10.792963456 +0000 UTC m=+5.651536317 container died fdc4c1f150d4b0f9aacdbb8673910003944b766b27e8041f842b4f3b6c925075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-26T21:55:11.414 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:11 smithi186 podman[74808]: 2024-04-26 21:55:11.146355166 +0000 UTC m=+6.004928017 container cleanup fdc4c1f150d4b0f9aacdbb8673910003944b766b27e8041f842b4f3b6c925075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-26T21:55:11.414 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:11 smithi186 bash[74808]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11 2024-04-26T21:55:11.704 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:11 smithi186 podman[74825]: 2024-04-26 21:55:11.414036961 +0000 UTC m=+0.619339254 container remove fdc4c1f150d4b0f9aacdbb8673910003944b766b27e8041f842b4f3b6c925075 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, GIT_CLEAN=True) 2024-04-26T21:55:12.075 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:11 smithi186 podman[74917]: 2024-04-26 21:55:11.658215975 +0000 UTC m=+0.017973615 image pull 55dc54019ad82626637ea289d349113ba783f0c02a3b8251fba6bc77c1a60c07 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2 2024-04-26T21:55:12.076 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:11 smithi186 podman[74917]: 2024-04-26 21:55:11.756887237 +0000 UTC m=+0.116644874 container create 872bcc950ffb0dab55ce7221717b415c72f9a2627bc231065d486a5e1a39f48b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-26T21:55:12.076 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:11 smithi186 podman[74917]: 2024-04-26 21:55:11.993109446 +0000 UTC m=+0.352867085 container init 872bcc950ffb0dab55ce7221717b415c72f9a2627bc231065d486a5e1a39f48b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-04-26T21:55:12.076 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:11 smithi186 podman[74917]: 2024-04-26 21:55:11.99638637 +0000 UTC m=+0.356144015 container start 872bcc950ffb0dab55ce7221717b415c72f9a2627bc231065d486a5e1a39f48b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:55:12.076 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:12 smithi186 podman[74917]: 2024-04-26 21:55:12.075697084 +0000 UTC m=+0.435454735 container attach 872bcc950ffb0dab55ce7221717b415c72f9a2627bc231065d486a5e1a39f48b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-04-26T21:55:12.454 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 26 21:55:12 smithi186 podman[74917]: 2024-04-26 21:55:12.290461689 +0000 UTC m=+0.650219318 container died 872bcc950ffb0dab55ce7221717b415c72f9a2627bc231065d486a5e1a39f48b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T21:55:12.912 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@osd.11.service' 2024-04-26T21:55:12.949 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:12.949 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-26T21:55:12.949 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-26T21:55:12.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.a 2024-04-26T21:55:13.317 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:55:13 smithi001 systemd[1]: Stopping Ceph mds.a for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:13.818 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:55:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-a[67238]: 2024-04-26T21:55:13.499+0000 7fb8b6200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:13.818 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:55:13 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-a[67238]: 2024-04-26T21:55:13.499+0000 7fb8b6200700 -1 mds.a *** got signal Terminated *** 2024-04-26T21:55:17.734 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:55:17 smithi001 podman[93602]: 2024-04-26 21:55:17.43873417 +0000 UTC m=+4.063264752 container died 57763fb7fce3c4f51e016af19f083e4037ff0ca12d73f99f67171a91ee491c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:55:18.068 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:55:17 smithi001 podman[93602]: 2024-04-26 21:55:17.842645044 +0000 UTC m=+4.467175643 container cleanup 57763fb7fce3c4f51e016af19f083e4037ff0ca12d73f99f67171a91ee491c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:55:18.068 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:55:17 smithi001 bash[93602]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-a 2024-04-26T21:55:18.565 INFO:journalctl@ceph.mds.a.smithi001.stdout:Apr 26 21:55:18 smithi001 podman[93616]: 2024-04-26 21:55:18.235554828 +0000 UTC m=+0.776002272 container remove 57763fb7fce3c4f51e016af19f083e4037ff0ca12d73f99f67171a91ee491c7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-a, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-26T21:55:18.607 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.a.service' 2024-04-26T21:55:18.720 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:18.720 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-26T21:55:18.720 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-26T21:55:18.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.d 2024-04-26T21:55:19.317 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:55:18 smithi001 systemd[1]: Stopping Ceph mds.d for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:19.817 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:55:19 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-d[66250]: 2024-04-26T21:55:19.396+0000 7faa77a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:19.817 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:55:19 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-d[66250]: 2024-04-26T21:55:19.396+0000 7faa77a00700 -1 mds.d *** got signal Terminated *** 2024-04-26T21:55:23.036 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:55:22 smithi001 podman[93721]: 2024-04-26 21:55:22.598411652 +0000 UTC m=+3.374473890 container died bdead4d57d832ebcef6d0d6942ca488969dba77d50715bd77b16a8d74a35fc0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-26T21:55:23.318 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:55:23 smithi001 podman[93721]: 2024-04-26 21:55:23.15510141 +0000 UTC m=+3.931163647 container cleanup bdead4d57d832ebcef6d0d6942ca488969dba77d50715bd77b16a8d74a35fc0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-26T21:55:23.318 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:55:23 smithi001 bash[93721]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-d 2024-04-26T21:55:23.818 INFO:journalctl@ceph.mds.d.smithi001.stdout:Apr 26 21:55:23 smithi001 podman[93736]: 2024-04-26 21:55:23.438959082 +0000 UTC m=+0.817917814 container remove bdead4d57d832ebcef6d0d6942ca488969dba77d50715bd77b16a8d74a35fc0b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:55:23.943 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.d.service' 2024-04-26T21:55:24.058 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:24.058 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-26T21:55:24.058 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-26T21:55:24.059 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.g 2024-04-26T21:55:24.567 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:55:24 smithi001 systemd[1]: Stopping Ceph mds.g for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:25.067 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:55:24 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g[66875]: 2024-04-26T21:55:24.799+0000 7fb175800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:25.068 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:55:24 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g[66875]: 2024-04-26T21:55:24.799+0000 7fb175800700 -1 mds.g *** got signal Terminated *** 2024-04-26T21:55:29.083 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:55:28 smithi001 podman[93842]: 2024-04-26 21:55:28.817883233 +0000 UTC m=+4.280429885 container died fa524eb6c02ac82e8ec307a3068e6e4ff62b65de4c0f256699afd6ea4649f0ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-26T21:55:29.527 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:55:29 smithi001 podman[93842]: 2024-04-26 21:55:29.225446323 +0000 UTC m=+4.687993005 container cleanup fa524eb6c02ac82e8ec307a3068e6e4ff62b65de4c0f256699afd6ea4649f0ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-26T21:55:29.527 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:55:29 smithi001 bash[93842]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g 2024-04-26T21:55:29.817 INFO:journalctl@ceph.mds.g.smithi001.stdout:Apr 26 21:55:29 smithi001 podman[93868]: 2024-04-26 21:55:29.526375017 +0000 UTC m=+0.691757526 container remove fa524eb6c02ac82e8ec307a3068e6e4ff62b65de4c0f256699afd6ea4649f0ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-26T21:55:30.085 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.g.service' 2024-04-26T21:55:30.203 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:30.203 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-26T21:55:30.203 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-26T21:55:30.204 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.j 2024-04-26T21:55:30.811 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:55:30 smithi001 systemd[1]: Stopping Ceph mds.j for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:31.067 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:55:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-j[65890]: 2024-04-26T21:55:30.809+0000 7fb489200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:31.067 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:55:30 smithi001 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-j[65890]: 2024-04-26T21:55:30.809+0000 7fb489200700 -1 mds.j *** got signal Terminated *** 2024-04-26T21:55:34.951 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:55:34 smithi001 podman[93974]: 2024-04-26 21:55:34.659298985 +0000 UTC m=+3.975528719 container died 31aa4a5f93ef579d4d4fbf9a91be9683795e4e3f0ba6f31b8292c0137b3893a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424) 2024-04-26T21:55:35.247 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:55:35 smithi001 podman[93974]: 2024-04-26 21:55:35.078923113 +0000 UTC m=+4.395152858 container cleanup 31aa4a5f93ef579d4d4fbf9a91be9683795e4e3f0ba6f31b8292c0137b3893a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-26T21:55:35.247 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:55:35 smithi001 bash[93974]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-j 2024-04-26T21:55:35.567 INFO:journalctl@ceph.mds.j.smithi001.stdout:Apr 26 21:55:35 smithi001 podman[93988]: 2024-04-26 21:55:35.24637629 +0000 UTC m=+0.571698406 container remove 31aa4a5f93ef579d4d4fbf9a91be9683795e4e3f0ba6f31b8292c0137b3893a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:55:35.829 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.j.service' 2024-04-26T21:55:35.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:35.943 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-26T21:55:35.944 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-26T21:55:35.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.b 2024-04-26T21:55:36.239 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:55:35 smithi038 systemd[1]: Stopping Ceph mds.b for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:36.576 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:55:36 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b[62819]: 2024-04-26T21:55:36.239+0000 7f460a87d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:36.577 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:55:36 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b[62819]: 2024-04-26T21:55:36.239+0000 7f460a87d700 -1 mds.b *** got signal Terminated *** 2024-04-26T21:55:40.548 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:55:40 smithi038 podman[72922]: 2024-04-26 21:55:40.247413573 +0000 UTC m=+4.105723232 container died 1ac8426148a506e6d13dd2447cafbdbdf57b35d83a6b3b42ad9903aa8ec8a02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:55:40.826 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:55:40 smithi038 podman[72922]: 2024-04-26 21:55:40.647630038 +0000 UTC m=+4.505939696 container cleanup 1ac8426148a506e6d13dd2447cafbdbdf57b35d83a6b3b42ad9903aa8ec8a02f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-26T21:55:40.827 INFO:journalctl@ceph.mds.b.smithi038.stdout:Apr 26 21:55:40 smithi038 bash[72922]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-b 2024-04-26T21:55:40.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.b.service' 2024-04-26T21:55:41.029 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:41.029 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-26T21:55:41.030 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-26T21:55:41.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.e 2024-04-26T21:55:41.326 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:55:41 smithi038 systemd[1]: Stopping Ceph mds.e for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:41.829 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:55:41 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e[62180]: 2024-04-26T21:55:41.376+0000 7f88b03e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:41.829 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:55:41 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e[62180]: 2024-04-26T21:55:41.376+0000 7f88b03e6700 -1 mds.e *** got signal Terminated *** 2024-04-26T21:55:45.060 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:55:44 smithi038 podman[73056]: 2024-04-26 21:55:44.764203543 +0000 UTC m=+3.507121633 container died ef0064fcc81e70980eb15dddbc6e74677cd5d2d770212fb155bf3a4ea87173ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T21:55:45.326 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:55:45 smithi038 podman[73056]: 2024-04-26 21:55:45.151019516 +0000 UTC m=+3.893937618 container cleanup ef0064fcc81e70980eb15dddbc6e74677cd5d2d770212fb155bf3a4ea87173ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-26T21:55:45.327 INFO:journalctl@ceph.mds.e.smithi038.stdout:Apr 26 21:55:45 smithi038 bash[73056]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-e 2024-04-26T21:55:45.543 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.e.service' 2024-04-26T21:55:45.577 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:45.577 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-26T21:55:45.578 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-26T21:55:45.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.h 2024-04-26T21:55:46.076 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:55:45 smithi038 systemd[1]: Stopping Ceph mds.h for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:46.077 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:55:45 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h[62492]: 2024-04-26T21:55:45.912+0000 7f0ade825700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:46.077 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:55:45 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h[62492]: 2024-04-26T21:55:45.912+0000 7f0ade825700 -1 mds.h *** got signal Terminated *** 2024-04-26T21:55:49.039 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:55:48 smithi038 podman[73181]: 2024-04-26 21:55:48.764194968 +0000 UTC m=+2.955586374 container died eaecce66b2abff9392c0b3d1575c4b5724f6d4848d633a9547e191475723d70f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-26T21:55:49.326 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:55:49 smithi038 podman[73181]: 2024-04-26 21:55:49.138093945 +0000 UTC m=+3.329485353 container cleanup eaecce66b2abff9392c0b3d1575c4b5724f6d4848d633a9547e191475723d70f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-26T21:55:49.327 INFO:journalctl@ceph.mds.h.smithi038.stdout:Apr 26 21:55:49 smithi038 bash[73181]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-h 2024-04-26T21:55:49.498 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.h.service' 2024-04-26T21:55:49.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:49.531 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-26T21:55:49.531 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-26T21:55:49.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.k 2024-04-26T21:55:49.826 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:55:49 smithi038 systemd[1]: Stopping Ceph mds.k for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:50.326 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:55:49 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-k[61868]: 2024-04-26T21:55:49.868+0000 7f4fc6600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:50.327 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:55:49 smithi038 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-k[61868]: 2024-04-26T21:55:49.868+0000 7f4fc6600700 -1 mds.k *** got signal Terminated *** 2024-04-26T21:55:54.165 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:55:53 smithi038 podman[73303]: 2024-04-26 21:55:53.882830825 +0000 UTC m=+4.129753538 container died aa549981267a2c2abad07c1fbf292ecd3d29d918479dccb5a52ba5e4b043da3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-26T21:55:54.500 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:55:54 smithi038 podman[73303]: 2024-04-26 21:55:54.241491587 +0000 UTC m=+4.488414357 container cleanup aa549981267a2c2abad07c1fbf292ecd3d29d918479dccb5a52ba5e4b043da3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.license=GPLv2) 2024-04-26T21:55:54.500 INFO:journalctl@ceph.mds.k.smithi038.stdout:Apr 26 21:55:54 smithi038 bash[73303]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-k 2024-04-26T21:55:54.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.k.service' 2024-04-26T21:55:54.659 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:54.659 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-26T21:55:54.660 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-26T21:55:54.660 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.c 2024-04-26T21:55:54.954 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:55:54 smithi186 systemd[1]: Stopping Ceph mds.c for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:55.214 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:55:54 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c[63550]: 2024-04-26T21:55:54.960+0000 7fc6115fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:55.214 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:55:54 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c[63550]: 2024-04-26T21:55:54.960+0000 7fc6115fb700 -1 mds.c *** got signal Terminated *** 2024-04-26T21:55:56.978 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:55:56 smithi186 podman[75022]: 2024-04-26 21:55:56.723158865 +0000 UTC m=+1.873610131 container died f83a8c0b275a4eb9fb6025209f200047d6f35df5d4e0458141b694a8b0757e2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-26T21:55:57.353 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:55:57 smithi186 podman[75022]: 2024-04-26 21:55:57.069264753 +0000 UTC m=+2.219716016 container cleanup f83a8c0b275a4eb9fb6025209f200047d6f35df5d4e0458141b694a8b0757e2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-26T21:55:57.354 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 26 21:55:57 smithi186 bash[75022]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-c 2024-04-26T21:55:57.487 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.c.service' 2024-04-26T21:55:57.522 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:55:57.523 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-26T21:55:57.523 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-26T21:55:57.524 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.f 2024-04-26T21:55:57.704 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:55:57 smithi186 systemd[1]: Stopping Ceph mds.f for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:55:58.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:55:57 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-f[63239]: 2024-04-26T21:55:57.831+0000 7f6436886700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:55:58.204 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:55:57 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-f[63239]: 2024-04-26T21:55:57.831+0000 7f6436886700 -1 mds.f *** got signal Terminated *** 2024-04-26T21:56:00.624 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:56:00 smithi186 podman[75158]: 2024-04-26 21:56:00.352370535 +0000 UTC m=+2.614704152 container died 1fbd5fa336a03bf24c191e18ff7f8f61dee295c7fbe0291746603ceedb7950c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff) 2024-04-26T21:56:00.954 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:56:00 smithi186 podman[75158]: 2024-04-26 21:56:00.785330281 +0000 UTC m=+3.047663891 container cleanup 1fbd5fa336a03bf24c191e18ff7f8f61dee295c7fbe0291746603ceedb7950c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-26T21:56:00.954 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 26 21:56:00 smithi186 bash[75158]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-f 2024-04-26T21:56:01.213 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.f.service' 2024-04-26T21:56:01.246 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:56:01.246 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-26T21:56:01.246 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-26T21:56:01.246 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.i 2024-04-26T21:56:01.568 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:56:01 smithi186 systemd[1]: Stopping Ceph mds.i for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:56:01.954 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:56:01 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: 2024-04-26T21:56:01.568+0000 7fe68d6ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:56:01.954 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:56:01 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i[63872]: 2024-04-26T21:56:01.568+0000 7fe68d6ea700 -1 mds.i *** got signal Terminated *** 2024-04-26T21:56:05.002 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:56:04 smithi186 podman[75279]: 2024-04-26 21:56:04.726660881 +0000 UTC m=+3.261837754 container died b230804d290a2ae26a1640ad057d80c27f7f903232c960f00cd4e500282471b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-26T21:56:05.361 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:56:05 smithi186 podman[75279]: 2024-04-26 21:56:05.102165623 +0000 UTC m=+3.637342498 container cleanup b230804d290a2ae26a1640ad057d80c27f7f903232c960f00cd4e500282471b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-26T21:56:05.361 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 26 21:56:05 smithi186 bash[75279]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-i 2024-04-26T21:56:05.520 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.i.service' 2024-04-26T21:56:05.553 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:56:05.553 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-26T21:56:05.553 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-26T21:56:05.553 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.l 2024-04-26T21:56:05.954 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:56:05 smithi186 systemd[1]: Stopping Ceph mds.l for 44ef3d6e-0412-11ef-bc93-c7b262605968... 2024-04-26T21:56:06.215 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:56:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-l[62932]: 2024-04-26T21:56:05.955+0000 7f8608b71700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-26T21:56:06.216 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:56:05 smithi186 ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-l[62932]: 2024-04-26T21:56:05.955+0000 7f8608b71700 -1 mds.l *** got signal Terminated *** 2024-04-26T21:56:07.739 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:56:07 smithi186 podman[75399]: 2024-04-26 21:56:07.460927607 +0000 UTC m=+1.689743440 container died 15b4c1050def987490d134da5857b047574c00e786f3ec3810b3989b982b5ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-26T21:56:08.081 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:56:07 smithi186 podman[75399]: 2024-04-26 21:56:07.822804137 +0000 UTC m=+2.051619971 container cleanup 15b4c1050def987490d134da5857b047574c00e786f3ec3810b3989b982b5ce6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:39d00e115f9430aba1116fce52d7d386b73fc3b9ed873e7f9c2c3b6cdcd253c2, name=ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240426.111959-bc612ff, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-26T21:56:08.082 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 26 21:56:07 smithi186 bash[75399]: ceph-44ef3d6e-0412-11ef-bc93-c7b262605968-mds-l 2024-04-26T21:56:08.249 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44ef3d6e-0412-11ef-bc93-c7b262605968@mds.l.service' 2024-04-26T21:56:08.279 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T21:56:08.280 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-26T21:56:08.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T21:56:08.609 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting cluster with fsid: 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:56:13.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T21:56:13.903 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:56:16.533 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T21:56:16.721 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T21:56:19.172 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:56:19.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:56:19.304 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T21:56:19.330 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T21:56:19.332 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204/remote/smithi001/crash 2024-04-26T21:56:19.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/crash -- . 2024-04-26T21:56:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:tar: /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T21:56:19.436 INFO:teuthology.orchestra.run.smithi001.stderr:tar: Error is not recoverable: exiting now 2024-04-26T21:56:19.441 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204/remote/smithi038/crash 2024-04-26T21:56:19.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/crash -- . 2024-04-26T21:56:19.485 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T21:56:19.486 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2024-04-26T21:56:19.488 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204/remote/smithi186/crash 2024-04-26T21:56:19.489 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/crash -- . 2024-04-26T21:56:19.526 INFO:teuthology.orchestra.run.smithi186.stderr:tar: /var/lib/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T21:56:19.527 INFO:teuthology.orchestra.run.smithi186.stderr:tar: Error is not recoverable: exiting now 2024-04-26T21:56:19.527 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T21:56:19.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-26T21:56:19.669 INFO:tasks.cephadm:Compressing logs... 2024-04-26T21:56:19.669 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T21:56:19.673 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T21:56:19.676 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T21:56:19.700 INFO:teuthology.orchestra.run.smithi186.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.log 2024-04-26T21:56:19.700 INFO:teuthology.orchestra.run.smithi186.stderr:: No such file or directory 2024-04-26T21:56:19.700 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T21:56:19.700 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-26T21:56:19.700 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T21:56:19.704 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-26T21:56:19.704 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-26T21:56:19.704 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-26T21:56:19.704 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-26T21:56:19.704 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-26T21:56:19.704 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mon.c.log: /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T21:56:19.705 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-26T21:56:19.705 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-26T21:56:19.705 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.cephadm.log: /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T21:56:19.705 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-26T21:56:19.705 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.8.log: /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T21:56:19.705 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T21:56:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-26T21:56:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.log 2024-04-26T21:56:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T21:56:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-26T21:56:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-26T21:56:19.707 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-26T21:56:19.711 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T21:56:19.711 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-26T21:56:19.711 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-26T21:56:19.711 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-26T21:56:19.711 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T21:56:19.711 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-26T21:56:19.711 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T21:56:19.711 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-26T21:56:19.712 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-26T21:56:19.712 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T21:56:19.712 INFO:teuthology.orchestra.run.smithi186.stderr: 92.6% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-26T21:56:19.712 INFO:teuthology.orchestra.run.smithi186.stderr: 94.4% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-26T21:56:19.732 INFO:teuthology.orchestra.run.smithi186.stderr: 84.1% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T21:56:19.772 INFO:teuthology.orchestra.run.smithi001.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-26T21:56:19.772 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.log 2024-04-26T21:56:19.773 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T21:56:19.774 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-26T21:56:19.776 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-26T21:56:19.778 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-26T21:56:19.778 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T21:56:19.779 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T21:56:19.780 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-26T21:56:19.782 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T21:56:19.783 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-26T21:56:19.784 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T21:56:19.786 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-26T21:56:19.787 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T21:56:19.788 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-26T21:56:19.789 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-26T21:56:19.789 INFO:teuthology.orchestra.run.smithi001.stderr: 94.4% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-26T21:56:19.790 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T21:56:19.792 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-26T21:56:19.794 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.admin.log: 94.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-26T21:56:19.795 INFO:teuthology.orchestra.run.smithi001.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-26T21:56:19.796 INFO:teuthology.orchestra.run.smithi186.stderr: 90.4% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T21:56:19.796 INFO:teuthology.orchestra.run.smithi186.stderr: 87.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T21:56:19.796 INFO:teuthology.orchestra.run.smithi001.stderr: 94.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-26T21:56:19.805 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 84.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T21:56:19.816 INFO:teuthology.orchestra.run.smithi001.stderr: 84.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T21:56:19.825 INFO:teuthology.orchestra.run.smithi186.stderr: 94.5% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-26T21:56:19.868 INFO:teuthology.orchestra.run.smithi001.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T21:56:19.899 INFO:teuthology.orchestra.run.smithi186.stderr: 91.0% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T21:56:19.900 INFO:teuthology.orchestra.run.smithi001.stderr: 90.3% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T21:56:19.983 INFO:teuthology.orchestra.run.smithi038.stderr: 94.4% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-26T21:56:20.017 INFO:teuthology.orchestra.run.smithi038.stderr: 92.5% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-26T21:56:20.066 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T21:56:20.107 INFO:teuthology.orchestra.run.smithi038.stderr: 91.1% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T21:56:20.478 INFO:teuthology.orchestra.run.smithi186.stderr: 91.7% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-26T21:56:20.997 INFO:teuthology.orchestra.run.smithi038.stderr: 90.4% 92.0% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.audit.log.gz -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T21:56:20.998 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-04-26T21:56:21.085 INFO:teuthology.orchestra.run.smithi038.stderr: 87.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T21:56:21.960 INFO:teuthology.orchestra.run.smithi001.stderr: 90.1% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-26T21:56:26.426 INFO:teuthology.orchestra.run.smithi186.stderr: 95.0% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-26T21:56:26.580 INFO:teuthology.orchestra.run.smithi001.stderr: 90.1% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T21:56:28.678 INFO:teuthology.orchestra.run.smithi186.stderr: 94.9% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-26T21:57:06.862 INFO:teuthology.orchestra.run.smithi186.stderr: 91.1% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-26T21:57:08.659 INFO:teuthology.orchestra.run.smithi186.stderr: 91.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-26T21:57:18.467 INFO:teuthology.orchestra.run.smithi186.stderr: 91.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T21:57:19.367 INFO:teuthology.orchestra.run.smithi038.stderr: 92.9% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-26T21:57:31.646 INFO:teuthology.orchestra.run.smithi186.stderr: 91.0% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-26T21:57:31.648 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T21:57:31.648 INFO:teuthology.orchestra.run.smithi186.stderr:real 1m11.958s 2024-04-26T21:57:31.648 INFO:teuthology.orchestra.run.smithi186.stderr:user 3m37.399s 2024-04-26T21:57:31.648 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m6.952s 2024-04-26T21:59:07.023 INFO:teuthology.orchestra.run.smithi038.stderr: 93.0% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-26T22:00:53.230 INFO:teuthology.orchestra.run.smithi038.stderr: 94.8% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-26T22:01:09.012 INFO:teuthology.orchestra.run.smithi001.stderr: 91.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T22:01:44.128 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-26T22:01:45.385 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-26T22:02:00.646 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-26T22:03:58.475 INFO:teuthology.orchestra.run.smithi038.stderr: 91.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-26T22:04:19.711 INFO:teuthology.orchestra.run.smithi001.stderr: 93.1% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-26T22:04:19.715 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-04-26T22:04:19.716 INFO:teuthology.orchestra.run.smithi001.stderr:real 8m0.005s 2024-04-26T22:04:19.716 INFO:teuthology.orchestra.run.smithi001.stderr:user 10m34.929s 2024-04-26T22:04:19.716 INFO:teuthology.orchestra.run.smithi001.stderr:sys 1m10.961s 2024-04-26T22:05:13.815 INFO:teuthology.orchestra.run.smithi038.stderr: 91.3% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T22:06:06.560 INFO:teuthology.orchestra.run.smithi038.stderr: 91.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-26T22:08:17.132 INFO:teuthology.orchestra.run.smithi038.stderr: 91.2% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-26T22:11:02.701 INFO:teuthology.orchestra.run.smithi038.stderr: 93.0% -- replaced with /var/log/ceph/44ef3d6e-0412-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-26T22:11:02.703 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-04-26T22:11:02.703 INFO:teuthology.orchestra.run.smithi038.stderr:real 14m43.012s 2024-04-26T22:11:02.703 INFO:teuthology.orchestra.run.smithi038.stderr:user 9m29.243s 2024-04-26T22:11:02.703 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m23.182s 2024-04-26T22:11:02.704 INFO:tasks.cephadm:Archiving logs... 2024-04-26T22:11:02.705 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204/remote/smithi001/log 2024-04-26T22:11:02.706 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T22:12:04.194 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204/remote/smithi038/log 2024-04-26T22:12:04.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T22:13:33.778 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204/remote/smithi186/log 2024-04-26T22:13:33.780 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T22:14:13.047 INFO:tasks.cephadm:Removing cluster... 2024-04-26T22:14:13.048 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 --force 2024-04-26T22:14:13.828 INFO:teuthology.orchestra.run.smithi001.stdout:Deleting cluster with fsid: 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T22:14:15.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 --force 2024-04-26T22:14:16.078 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T22:14:16.985 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44ef3d6e-0412-11ef-bc93-c7b262605968 --force 2024-04-26T22:14:17.177 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 44ef3d6e-0412-11ef-bc93-c7b262605968 2024-04-26T22:14:17.756 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T22:14:17.757 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T22:14:17.818 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T22:14:17.837 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T22:14:17.853 INFO:tasks.cephadm:Teardown complete 2024-04-26T22:14:17.853 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T22:14:17.866 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T22:14:17.866 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T22:14:17.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T22:14:17.880 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T22:14:18.034 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T22:14:18.058 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T22:14:18.058 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T22:14:18.066 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T22:14:18.069 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T22:14:18.082 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-26T22:14:18.083 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-04-26T22:14:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T22:14:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-26T22:14:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv01.front.sepia.ceph.com 3 6 377 28 +1491us[+1473us] +/- 44ms 2024-04-26T22:14:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 +92us[ +74us] +/- 61ms 2024-04-26T22:14:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -2471us[-2489us] +/- 71ms 2024-04-26T22:14:18.087 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:14:18.113 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-04-26T22:14:18.121 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T22:14:18.122 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-04-26T22:14:18.122 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +1493us[+1486us] +/- 45ms 2024-04-26T22:14:18.122 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 +95us[ +88us] +/- 61ms 2024-04-26T22:14:18.122 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -2453us[-2460us] +/- 71ms 2024-04-26T22:14:18.122 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:14:18.237 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T22:14:18.237 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-04-26T22:14:18.237 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv01.front.sepia.ceph.com 3 6 377 31 +1488us[+1502us] +/- 44ms 2024-04-26T22:14:18.238 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 +109us[ +124us] +/- 61ms 2024-04-26T22:14:18.238 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 -2456us[-2456us] +/- 71ms 2024-04-26T22:14:18.238 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T22:14:18.241 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T22:14:18.252 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T22:14:18.253 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T22:14:18.279 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T22:14:18.516 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T22:14:18.631 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T22:14:18.715 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T22:14:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:14:18.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T22:14:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:14:18.845 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-26T22:14:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:14:18.874 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T22:14:18.887 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T22:14:18.921 INFO:teuthology.task.internal:Duration was 3908.762191 seconds 2024-04-26T22:14:18.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T22:14:18.955 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T22:14:18.955 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T22:14:18.963 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T22:14:18.966 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T22:14:19.010 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T22:14:19.021 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T22:14:19.127 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T22:14:19.493 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T22:14:19.493 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-04-26T22:14:19.493 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T22:14:19.586 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-04-26T22:14:19.586 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T22:14:19.618 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T22:14:19.618 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T22:14:19.652 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T22:14:19.652 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T22:14:19.660 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T22:14:19.664 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T22:14:19.750 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T22:14:19.751 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T22:14:19.757 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T22:14:19.760 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T22:14:20.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T22:14:20.333 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T22:14:20.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T22:14:20.408 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T22:14:20.436 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T22:14:20.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T22:14:20.472 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T22:14:20.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T22:14:20.482 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T22:14:20.518 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-04-26T22:14:20.523 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-04-26T22:14:20.597 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-04-26T22:14:20.647 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T22:14:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:14:20.697 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T22:14:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:14:20.712 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T22:14:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T22:14:20.727 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T22:14:20.740 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T22:14:20.741 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204/remote/smithi001 2024-04-26T22:14:20.741 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T22:14:20.852 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204/remote/smithi038 2024-04-26T22:14:20.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T22:14:20.889 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204/remote/smithi186 2024-04-26T22:14:20.889 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T22:14:20.930 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T22:14:20.930 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T22:14:20.938 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T22:14:20.941 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T22:14:20.991 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T22:14:21.005 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T22:14:21.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T22:14:21.038 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T22:14:21.039 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T22:14:21.046 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T22:14:21.049 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T22:14:21.062 INFO:teuthology.orchestra.run.smithi038.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 22:14 /home/ubuntu/cephtest 2024-04-26T22:14:21.063 INFO:teuthology.orchestra.run.smithi186.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 22:14 /home/ubuntu/cephtest 2024-04-26T22:14:21.098 INFO:teuthology.orchestra.run.smithi001.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 22:14 /home/ubuntu/cephtest 2024-04-26T22:14:21.102 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T22:14:21.114 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T22:14:21.173 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T22:14:21.200 DEBUG:teuthology.nuke:shortname: smithi001 2024-04-26T22:14:21.201 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T22:14:21.225 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:00:10.173396', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFf2aXPvj/0ccN2dtFI0yyjVcw+DmIuTnxhXs2ZD1Tf2z0KzCQYseJR3tRObtZllJ0PcB+eblcoJCDqohzYzK/I='} 2024-04-26T22:14:21.229 DEBUG:teuthology.nuke:shortname: smithi038 2024-04-26T22:14:21.229 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T22:14:21.253 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:00:10.175128', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPaky2vOsP4VUCUWYzQGzwp0GzpvIqL1JGL08QbC+PVx3E8lrj4EDdrqyEW8fN7OTtLBboZJowJLVhXzsT1nIl8='} 2024-04-26T22:14:21.259 DEBUG:teuthology.nuke:shortname: smithi186 2024-04-26T22:14:21.259 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T22:14:21.289 INFO:teuthology.orchestra.console.smithi001:Power off 2024-04-26T22:14:21.289 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T22:14:21.300 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-26_19:30:57-fs-wip-rishabh-testing-20240426.111959-testing-default-smithi/7675204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 21:00:10.171386', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLXrztibfFKNHYapVSY5yz823xjnIE3KD4MFl4HgKwRzAZh7cjMScP0gqZkBBcKL1GtTp73rhjcigBMjh8FdHew='} 2024-04-26T22:14:21.313 DEBUG:teuthology.orchestra.console.smithi001:power off output: Chassis Power Control: Down/Off 2024-04-26T22:14:21.313 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T22:14:21.335 INFO:teuthology.orchestra.console.smithi038:Power off 2024-04-26T22:14:21.336 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T22:14:21.354 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-04-26T22:14:21.363 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-04-26T22:14:21.363 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T22:14:21.380 INFO:teuthology.orchestra.console.smithi186:Power off 2024-04-26T22:14:21.380 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T22:14:21.389 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-04-26T22:14:21.403 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-04-26T22:14:21.403 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T22:14:21.426 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-26T22:14:25.355 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T22:14:25.391 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T22:14:25.427 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T22:14:25.468 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is on 2024-04-26T22:14:25.503 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-04-26T22:14:25.537 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-04-26T22:14:29.471 DEBUG:teuthology.orchestra.console.smithi001:pexpect command: ipmitool -H smithi001.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T22:14:29.504 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T22:14:29.539 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T22:14:29.583 DEBUG:teuthology.orchestra.console.smithi001:check power output: Chassis Power is off 2024-04-26T22:14:29.615 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-04-26T22:14:29.651 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-04-26T22:14:29.684 INFO:teuthology.orchestra.console.smithi001:Power off completed 2024-04-26T22:14:29.716 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-04-26T22:14:29.753 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-04-26T22:14:29.861 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 3908.7621908187866 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi001 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc612ffb9c1d5bc27536bc917a705262a36d4387 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=31de5266af954439839393a95344e27f status: fail success: false 2024-04-26T22:14:29.862 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T22:14:29.958 INFO:teuthology.run:FAIL